we have a django app on nginx where users upload media files. the media are huge such as 30min tv and radio programs resulting 100-300mb, and our shared hosting limits the upload to 30mb.
how to embed a smart uploader which will put chunks of 20-30mb instead of trying to upload the large file? we would like not to destroy our highly edited forms, so if there is an easy way to insert such tool as a bulletproof widget, you're awesome.
links, snippets, examples - highly appreciated, and any ideas are welcome. tx in advance.
You should consider alternative hosting (perhaps a virtual private server), as for any serious downloads you will quickly run into the limits of your shared hosting.
Related
I need to increased file upload speed in Django. Any ways to do this? I guess about to upload files as a background, when user send POST request i just redirect them to some page and start uploading files. Any ways to do this? Or do you know any ways to increased upload speed? Thank you in advance
Low upload speed could be a result of several issues.
It is a normal situation and your client doesn't have a possibility to upload at a higher speed.
Your server instance using an old HDD and can't write quickly.
Your server works on another pool of requests and serves your clients as fast as it could but it is overloaded.
Your instance doesn't have free space on the hard drive
Your server redirects the file as a stream somewhere else.
You've written a not optimized code of the upload handler.
etc.
You don't use a proxy-server that works perfectly with slow clients and when a file is on the proxy server's side give it to Django in a moment.
You are trying to upload a very big file.
etc.
Maybe you have more details on how you handle the uploads and your environment.
There are cases in a project where I'd like to store images on a model.
For example:
Company Logos
Profile Pictures
Programming Languages
Etc.
Recently I've been using AWS S3 for file storage (primarily hosting on Heroku) via ImageField uploads.
I feel like there's a better way to store files than what I've been doing.
For some things (like for the examples above) I think it would make sense to actually just get an image url from a more publically available url than take up space in my own database.
For the experts in the Django community who have built and deployed really professional projects, do you typically store files directly into the Django media folder via ImageField?
or do you normally use a URLField and then pull a url from an API or an image link from the web (e.g., go on any Google image, right click and copy then paste image URL)?
Bonus: What does your image storing setup look like?
Hope this makes sense.
Thanks in advance!
The standard is what you've described, using something like AWS S3 to store the actual image and handle the URL in your database. Here's a few reasons why:
It's cheap. like really cheap
Instead of making your web server serve the files, you're offloading that onto the client (e.g. their browser grabbing the file from S3)
If you're using an ephemeral system (like Heroku), your only option is to use something like S3.
Control. Sure, you can pull an image link from somewhere else that isn't managed by you. But this does not scale. What happens if that server goes offline? What if they take that image down? This way, you control what happens to the objects.
An example of a decently large internet company but not large enough to run their own infrastructure (like Facebook/Instagram, Google, etc.) is VSCO. They're taking a decent amount of photo uploads every day and they're handling them with AWS.
I am making a website in Django, and I am trying my best to make sure it is horizontally scalable.
Due to the application being horizontally scalable, I am unable to save Images that users upload locally, in the media folder.
I was wondering what are some ways I could save the images that the users upload, in such a way that would allow my application to be horizontal scalable?
I do have a MariaDB Galera Cluster that I use to store other data, but it seems like saving images in a shared database might not be the best idea due to performance reasons (Storing Images in DB - Yea or Nay?).
If I attempt to use the media folder, are there any solutions that could sync storage (folder) between different instances of the application?
In general, what are some good practices for serving(download/upload) static content like images for horizontally scalable websites, and does Django provide anything to assist with this matter out of the box?
Our website has about 200K images stored in sitecore database now. It runs more slowly than before. Does this large numbers of images stored in database will slow down the whole site?
If yes, how can I improve our image storage?
Thanks very much, our sitecore version is 6.2.
Have you considered setting up a CDN for your static assets? That would reduce load on your site and should speed it up.
Otherwise you might look at optimising the databases. Have a look at the Sitecore Optimisation Guide http://sdn.sitecore.net/upload/sitecore6/64/cms_tuning_guide_sc60-64-a4.pdf
In general, it depends on whether front-end or back-end is slow.
If you experience issues even when the site is not loaded with huge number of requests - then you should probably upgrade hardware.
If it's caused by high website load - there are two rather simple options:
1) Use dedicated image server for Sitecore http://pentialized.dk/2011/01/02/dedicated-image-server-in-sitecore-part-2/
2) Integrate Media Library with CDN, CloudFront is really simple and powerfull, see the example here: http://herskind.co.uk/blog/2012/04/using-cloudfront-for-sitecore-media-content
I would like to prepare my website for a possible influx in traffic. This is my first time using Django as a framework, so I'm unsure of the modifications that should be made to assure that I'm ready and won't go down. What are some of the common things one can do to prepare a Django website for production-level traffic?
I'm also wondering what to expect in terms of traffic numbers. I'm currently hosted at Webfaction with 600GB/month of traffic. Will this quickly run out? Are there statistics on how big 'slashdotted' events are?
Use memcache and caching middleware.
Be sure to offload serving statics.
Use CDN for statics. This doesn't directly affect Django, but will reduce your network traffic.
Anything beyond that — read up what others are using:
Scaling Django Web Apps By Mike Malone
Instagram Architecture
DISQUS Architecture
Since you are at Webfaction you have an easy answer for handling your statics:
Create a Static-only application. (Not the Static CGI/PHP app)
Add it under you current website.
Put all of your statics under it (or symlink to them, which is what I do).
This will serve all statics through their nginx frontend -- blindingly fast.
Regarding your bandwidth allocation:
You don't say what type of content you are offering. If it is anything even slightly vanilla you are unlikely to approach 600GB/mo. I have one customer who offers adult-oriented videos teaching tantric sex techniques and their video bandwidth (for both free & member-only videos) is about 400-450GB/mo. The HTML portion of the site (with tons of images) runs about 50-60GB/mo.