I am in a scenario where i have to resize existing images before displaying them on webpage. For that purpose is it wise to use different server for storing my images, as django server may take bandwidth to fetch images from another server.
Yes it is wise you use seperate server for static files. It is also mentioned in the django official documents.Check this serving static files from dedicated server
Absolutely. Especially for images, it is very meaningful to use infrastructures which were built for this purpose.
It is not without reason that cloudinary is a dedicated cdn service for images that can be manipulated on the fly.
You might even consider Flickr as a great cdn solution for photos or slideshows.
By all means, outsourcing photo delivery will boost the performance along with the perceived quality of services that your webiste provides.
Related
I've seen many questions on stackoverflow about handling static files in django during deployment. I saw that many answers said something like this - "With debug turned off Django won't handle static files for you any more - your production web server (Apache or something) should take care of that."
Why can't we use the server hosting the django project to host the static files too?
Static files don't require any kind of logic or processing. It is more efficient to deliver them directly to the end-user directly from disk via a web server, rather than running them through the middle layer of Django. That middle layer (such as gunicorn, uwsgi, or mod_wsgi) is what allows things like views to be processed and for the ORM to connect with a database. Since static files require none of that, bypassing it is the most efficient. The same is true for media files that are uploaded by the end user. Good luck!
I am looking into possibly setting up a CDN to use with my Wagtail sites. I am thinking that this will be a more efficient way to manage media uploads during stage/production pushes, since right now the media folder has to be manually copied from server to server on deploy. If all of the images were being accessed from a CDN then this wouldn't be an issue.
This would be my first time using a CDN so I'm looking for advice. There is lots of info on using a CDN with WordPress, but not a lot of documentation on setting one up with Wagtail/Django. I have the following questions about it:
Does anyone have any suggestions on the best way to implement the CDN with Wagtail?
How does it handle the uploads that the user submits through the CMS? Most of the images will be uploaded as part of the static files, but how does it work when the user uploads a photo as part of a post?
Which CDN companies have you had the best/worst experiences with? The sites I am planning to use this for are professional/business, but not e-commerce.
Also, if there is a more efficient way to handle the transfer of media uploads from one environment to another than using a CDN, I'd love to hear your suggestions for that too. As of right now I've had to copy the media folder over after doing the deploy, and I will have to do this every time I make a change to the site.
Thanks in advance for your assistance.
The following resources can be helpful for your required setup in Wagtail (later on today I can provide you some more details):
Frontend cache invalidator for pages (so not only for static and media files)
Link: http://docs.wagtail.io/en/latest/reference/contrib/frontendcache.html#frontendcache-aws-cloudfront
Storing media files in Amazon Web Services S3 buckets
This should be a better solution instead of copying media files from server to server. In this case Amazon Web Services CloudFront (CDN) would be a perfect choice.
Link: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#amazon-s3
More info CloudFront: https://aws.amazon.com/cloudfront/
Static file cache invalidation with Django Whitenoise
Can be relevant to clear the cache for a new deployment (the static files will have a unique filename so CDN will have a new file cache from its origin after the deployment)
Link: http://whitenoise.evans.io/en/stable/django.html
CloudFront from AWS will have my personal choice for CDN. Besides the awesome resources/services AWS has to offer, CloudFront is simple to setup and has one of the best CDN's out there.
Finally a CDN for serving static- and media files has nothing to do with Wagtail specifically. There are some (see list above) nice apps available for Django itself, but you are free to choose another CDN solution (like Cloudflare).
So setting up a AWS S3 Bucket for each environment (tst/acc/stg/prd) and use it for uploading you media files (so the files aren't on the server anymore) and setup a CloudFront distribution for these buckets would be a proper solution for your problem.
Best regards,
Rob Moorman
i'm building a restful web service with Django as the backend and am currently concerned with scaling.
I want to have an architecure like this: http://www.djangobook.com/en/2.0/_images/scaling-4.png
where i can just start up more django instances on high load.
My Problem is that i don't know how to externalize the user uploaded content (in django refered to as Media).
My Basic ideas are: Amazon S3, FTP Upload, SMT or NFS Mount, or some kind of file db like GridFS.
Because the EU canceled the Save Harbour Agreement and there is a lot of legal uncertainty for EU companies to put user data on non EU servers, i cant use Amazons S3 or any non EU based storage solution.
I have looked at Minio, a S3 API compatible storage server.
I could only get the posts to minio working with the latest (non stable) release, but not the gets. Other S3 compatible servers like CEPH are "too big" for my usecase.
I thought of uploading the files to an ftp server and serve them from there with nginx. The Problem is, that django-storages ftp functionality does not appear to be stable and django-resto is deprecated and no longer maintained.
SMB/NFS mounts are unstable and django would loose data it whats to write, when the mount is unavailabe due to network failures.
GridFS i could not get to work because it uses mongodb, which is non relational and django is relational. The non-relational django fork is not well maintained. In terms of serving files from GridFS there is nginx-gridfs and mod_gridfs, both are not maintained anymore.
Do you have any ideas on how to externalize django's media?
Well, I solved my problem by using Minio library.
Created special storage class, which inherits django's FileStorageClass. Overrided url() and save() methods.
Moved it to external module.
As stated in: https://docs.djangoproject.com/en/dev/howto/static-files/
When DEBUG is set to True, the server automatically serves the static file, but it states:
This method is grossly inefficient and probably insecure, so it is unsuitable for production.
But what exactly is inefficient and insecure about it? I just have a small-ish project on Heroku that I haven't set to "production" mode yet and I'm wondering what are the exact downsides.
Performance related reasons:
web servers are orders of magnitude better at serving static files.
AFAIK the development server is mono-threaded and can respond only one request at time, concurrent requests will block (most browsers make 4 concurrent requests trying to download assets in parallel).
Security related reasons:
using the app to serve static content is overkill (simplification is good for security)
the developers like to be on the safe side, so it is kind of a disclaimer
debug mode exposes a lot of information about the server
Django started in the news publishing industry where in general there is enough traffic to justify serving static content from a dedicated web server, probably the original developers have a bias for this arrangement.
That said, there are projects that replace the default development server by a more robust implementation based on gunicorn or tornado.
Kenneth (the author of requests, employed by Heroku) has a different opinion (source):
In reality, serving static files through Python/Django is fine for
production — those requests are no different than dynamic ones.
Performance will be fantastic, but not as good as nginx.
If you're that heavily concerned about efficiency then you shouldn't
be hosting those files on your server anyway, you'd be pushing them to
an CDN like S3+Cloudfront and the like.
Another benefit to this approach is development:production parity.
And on heroku, you can't use Nginx to server static files, actually you can't do it on most other PaaS too, I got the same problem on cloud foundry last year. But there is a workaround:
On Heroku, your application directly responds to HTTP requests,
instead of going through an additional web server like Apache or
Nginx.
We recommend most applications serve their assets strait from Django
or a CDN.
Django doesn't recommend serving static files in production because of
the design of its static file handler.
Luckily, there is a library called DJ-Static which makes uses a
production-ready WSGI asset server.
I've written up a guide for Django and Static Assets here:
https://devcenter.heroku.com/articles/django-assets
Read the following discussions for more details:
Serving static files for a Django app
serving static files via gunicorn
I'm about two months in with django. I've been following tutorials on youtube and such and one of the tutorials says that I have to use two servers when deploying my site. Django will be served from heroku and static files from amazon s3. I have to pay for two seperate servers to deploy a django app? I did not expect this and this would not be within my budget if this is so. Is he wrong or is this just for special cases like his? Any help would be appreciated
No, sounds like the video is a bit confusing. There is a distinction between static assets, ie the CSS/JS etc that makes up your site, and dynamic media, ie any user-uploaded content.
Heroku can quite happily serve static assets from the filesystem, and their docs on deploying Django state exactly how to do this. However you cannot store dynamically uploaded content on Heroku, since the filesystem is ephemeral. If your app allows this, you need to save them somewhere permanent such as S3.
Note however that S3 is really cheap; hosting media files there should only cost you pennies.
You don't need 2 different servers for deploying your django project. You can just use a single Amazon EC2 instance then install Nginx/Apache+Supervisor+Gunicorn+Python. After that, you just need to configure the location of your static files from your virtualhost. Here's a tutorial from digital ocean.