I followed the directions on http://www.allbuttonspressed.com/projects/djangoappengine, but afterwards realized that the resulting project has almost 5,000 files, namely because of the django directory.
Am I not supposed to include django 1.3 and just use django 1.2 builtin with Google App Engine? Or am I missing something? I've heard that zipimport is not a good idea with django.
There are not a lot of solutions:
You can try to remove every lib unnecessary in the directory of Django
Use zipimport if you don't want use django 1.2 provides with
GAE. Reduce the number of files you use in your project.
But note that: For yours instances, load a lot of files is slower because there are a lot of reads in the file system. django.zip is reads only one time and stocked in the memory to unzip it. there is just one read on the file system not 3000 and more...
Related
Our website is build with heavy node_modules.
Many heavy package cause heavy node_modules and eventually affects heavy cost on AWS.
So we are trying to reduce total bundle size to reduce some cost.
What I found is, by splitting code with webpack, we can have more but smaller bundles. Though, overall size doesn't change a lot.
What i'm wondering is, does splitting code with webpack can save some AWS cost?
Sorry for poor question, but this is all I can think of right now.
Any great idea would very helpful Thx.
This is a complex topic and equally open-ended question too! To answer this, I will make few assumptions:
By saving AWS cost, it means reducing bundle size so that outgoing bandwidth cost is saved.
Application being built is 100% SPA i.e. fully client-side. Server-side optimization gets very complex quickly.
Out of box, Webpack will bundle everything into one big file/bundle. It contains your own code as well as code from third-party libraries. The fundamental idea here is that third-party code rarely changes while our own code frequently changes.
So, we can use Webpack to split our code into two distinct chucks using SplitChunksPlugin. One for our own code and another for third-party code i.e. code from node_modules folder; lets call it vendor bundle. Now as long as your node_modules folder remains constant i.e. your lock file - package-lock.json file is constant, it will always produce the same bundle with exact same content for third-party code.
Then next part of the idea is that you can simply take this vendor bundle and upload it to CDN and then use via CDN. The CDN and browser will do their caching magic and users will hardly need to download this file each time. CDN will use ETag and/or cache-control header to achieve this and browser will use that.
However, the reality is different. When you have too many dependencies and or using dependabot to update dependencies, you will often update your lock file. This means on each build a new vendor bundle gets generated even if there is a difference of single byte. The hash id generated by Webpack will be different. And in other scenario, the way you import the dependencies may also change the generated bundle content resulting in a different bundle.
So, architecturally, we do better vendor bundling by making use of CDNs. The first step is to distinguish between stable third-party module and frequently updated third-party module. For example, consider react, react-dom and rxjs, etc. These are not updated often. For these libraries, make use of third-party CDN like cloudflare, cdnjs or unpkg. Add these libraries as CDN based UMD packages.
For this, you will add these dependencies to your index.html file which is typically generated using html-webpack-plugin.
<!-- index.html -->
<script crossorigin src="https://unpkg.com/react#18/umd/react.production.min.js"></script>
<script crossorigin src="https://unpkg.com/react-dom#18/umd/react-dom.production.min.js"></script>
Now, simply tell Webpack to not bundle these dependencies which you have already made available via CDN script. Use Webpack externals to do that:
// webpack.config.js
module.exports = {
externals: {
'react': 'React',
'react-dom': 'ReactDOM'
},
};
With this configuration, Webpack is not only going to exclude React from the bundle but also will speed up your bundling. Where ever you import stuff from react library, Webpack will substitute this for global object React.
You can then extend this model to all the stable libraries that you are using. Another important advantage of using this way is that there would be other websites that your users may have already visited which would have cached this particular file in their browser using the same CDN.
To automate your workflow, you can customize Webpack or any bundler script to inject these scripts with exact version by reading the packge.json file for your dependencies and then generating the <script> tags. It means you would still have single source of truth for your dependencies versions. You can read CDN documentation to understand how they allow you to construct CDN URLs for the dependencies.
I've made a desktop app in Python to process .xls files with openpyxl, tkinter and other stuff. Now i'm looking to run this App on www.pythonanywhere.com. I was expecting to make an app to upload the file to the server, process it and then retrieve it changed to the user. But after long months struggling with Django i`ve reached the problem of static media. As I understood Django doesn’t serve files in production mode. Does this means that I can’t upload-process-download as I was planning? Or I can run in the view function the process algorithm on the request.file and then retrieve it changed, regardless of not serving static files? Am I missing something? Does Flask has the same issue? What’s the optimal solution? Apologize me for the many doubts.
PD: I took the time to read many similar questions and seems to be possible to upload and download, but then im missing something on handling static files and what that means
We're currently trying to convert html files to PDF on AppEngine using Python. The HTML files are from a third-party vendor so we have no control over their format. Both the Flexible and Standard environments are options, but every path we go down we seem to hit a roadblock:
PDFkit requires a wkhtml2pdf install, no PIP package available, however converts perfectly offline
xhtml2pdf / PISA - works even on GAE Standard but doesn't support many features such as float and badly formatted HTML
WeasyPrint - C dependencies in theory would run on the Flexible environment but no pip packages available for dependencies including Cairo and Pango
Has anyone got a robust solution running on AppEngine with any of the above? Or with other libraries I am missing?
I ran into this same problem a year back and concluded that this is currently not possible in App Engine, at least with a good quality conversion. (Someone please point out if things have changed)
xhtml2pdf - I was able to successfully run it in standard App Engine but not at all happy with the conversion quality.
PDFkit - Ran into a similar problem and came up with a different solution. Hosted PDFkit on a Compute Engine Instance and exposed an endpoint wherein a POST request with the HTML file will return the converted PDF as a response. This gave me the best/expected results in terms of quality/speed of processing.
It did incur some extra charges but I was able to utilize the instance for something else too ;). I chose the least possible configuration initially since I was not storing anything on the Compute Engine Instance.
Hi I have an issue that is preventing some additional security on my website,
Bearing in mine, when run locally this all works perfectly.
Basically when it is being used my site will zip/ rename/ re-zip files (mostly image files)every three minuets after a certain request is made. But since deploying on heroku this does not work. it says it cant fine the file after zipping or re-naming it.
I am using Django 1.7
Heroku has a read-only filesystem. You can only write to ./tmp and ./log directories:
https://devcenter.heroku.com/articles/read-only-filesystem
There are two directories that are writeable: ./tmp and ./log (under your application root). If you wish to drop a file temporarily for the duration of the request, you can write to a filename like #{RAILS_ROOT}/tmp/myfile_#{Process.pid}. There is no guarantee that this file will be there on subsequent requests (although it might be), so this should not be used for any kind of permanent storage.
More specifically, when using a backend application framework I generally am afforded some level of asset management which allows me to work with multiple files in development which are uncompressed and unminified and then in production mode those files become automatically minified, compressed, and concatenated into a single file.
I am looking to create an Ember application that is a single page app that interfaces with a separate RESTful services layer. I simply do not need the weight of a framework behind the Ember app and am hoping to serve it as static html+css+js, so I am looking for any guidance on how to easily manage development and deployment of a client-side only app without adding much overhead.
Right now my biggest issue is with including JS (and to a lesser extent, CSS) files. My HTML is static and I have an Ember app comprised of many files, so I have many script tags to include them all. This is clearly not appropriate for production so I imagine some kind of build tool will be needed to assemble my Javascript files and overwrite the script tags in the HTML file. Are there tools out there right now that will do this? Is there another approach that I may be overlooking?
This is my first fully client-side application so it's very possible that I just need to make a paradigm shift, having done server-side applications for so long.
Agreed this can be tricky without a backend framework. For sure script tags are not the way to go and you will need some kind of build tool for production deployment.
Ember App Kit is a solution a few of us have been working on. It's still early stages but i've used it for a couple of projects so far and it's been much better than trying to roll-my-own with grunt. I would expect it to become the default starting point for ember apps in near future, to try it now just download it as a zip then read the Getting Started Guide
There are many other solid solutions out there, consider checking out:
ember-tools
brunch-with-ember-reloaded
brunch-with-hapmsters
charcoal
I use a combination of requirejs and Grunt, using these lovely functions and this one, which can compile your ember-handlebars templates into functions. (The git-contrib includes the ability to watch for changes in your files and perform various build steps which may differ if you are in development or production. You can have separate grunt functions which run various tasks for production or development. Of course for all of this you are going to need node!