I am working on a django app to store user pics and photos.
What is the optimal approach to store individual user media.
File Sizes are no more than 5MB.
The data is persistent.
The approach i have in mind is:
On form data submission, Upload it to an FTP server using django-storages.
Store the url and fetch it via http later for user.
How to save upload files to another server
I have seen the answers and I don't know what type of queue needs to be used.
you'd usually save the file locally and then latter upload it to some cloud service asynchronously, preferably using something like django-celery
see this answer
Related
I have a Django app running on a server. Currently user uploads are stored on the server filesystem.
I already know how to set up S3 storage. What is the best way to migrate existing uploads to S3 without breaking the API and having existing uploads still available?
These files are served to the front end in two ways:
Directly through a /media/path/to/upload endpoint:
/media/<path> django.views.static.serve
Through a viewset:
/api/v1/users/<user_pk>/items/<item_pk>/attachments/<pk>/ project.app.views.ItemAttachmentsViewSet
Does the following make sense:
Change the storage backend
Go through all model objects
Save each model object to get the files uploaded
Have /media/path go to a new view that will serve the files similar to how ItemAttachmentsViewSet does it.
? Or is there a better way?
The procedure outlined in the question was what I ended up doing, with the exception of step 4 which turned out to be unnecessary.
I want to upload images Facebook style: select the images before submitting the form and when they are all uploaded, submit the form instantly. I know how to do it in the frontend, but the problem is in the backend. I have found some ways of managing the images in the backend but I'm not satisfied. The great deal of all this is avoiding to store the photos that won't be used, like if the user closes the browser while some photos are already uploaded. I have in mind 3 ways of doing the upload and I don't know which would be the best:
Create a "tmp" directory and place all the uploaded photos there, and when the form is submitted move all the used photos to another directory. (With this method there can be some concurrency problems)
Create a TempPhoto table in my database and do the same as the previous solution, moving the used photos from TempPhoto to the permanent table.
Add the photos directly to the permanent table and erase the ones that are not used (that are not related to other entity) at a scheduled time. (I suppose this would be the slower solution)
I think your first way with some reformations is the best way. You can create a tmp directory and attach a unique data (e.g. IP address) to each image that takes control on concurrency and then write some script in $(window).unload(...) for send a signal to the backend that remove the image(s) from tmp directory when user close window before submitting the form.
I can't seem to wrap my head around it. How do websites create forms for image uploads, store, and process it for display on their sites? My current situation is with django storing images.
You place a <form> on your page. It can have different inputs in it, like text, numbers, etc. There is also file input. Then the user submits the form and all data is sent to the webserver. The webserver, basically, receives the file as an array of bytes. You can read more about it here.
In case of Django you'll receive this data in request.FILES. You'll get name, size and other data of the file selected by user. You can then save it on disk or in your database. Read about Django-specific things here.
In general (in a very resumed/simplistic way):
A web page renders a form with an widget in it.
When you tell the browser what's the file you want to upload, it prepare an HTTP request to the webserver in which it stores the binary data of the file.
The request is sent to the web server through a TCP connection.
The server takes the request, see the file an decides what to do with it (depending on configuration files)
Later, the program handling the request, ask the global variables of the server (or something like that) what the heck it did with it files
The server give it the address where it was stored
The app do the rest of the job with the file uploaded :)
In Django:
Three first step are the same
The server ask django what to do with the file
Depending on settings.py the file is uploaded to memory or to a temporary file in disk
The application handles the file accessing it to request.FILES dictionary in your view
When you have it, you can pass it to your models, forms or do whatever you want.
If you want to go deeper with Django file uploads, this section of the documentation is really good. You can take a look at it.
Hope this helps!
I have a website where I allow people to download files, such as Powerpoint templates that they can reuse.
I added a column to the model that stores information and location of these files called 'Downloads' and in that column I'd like to use it as a counter to track how many times the file has been downloaded.
How would I go about this? Would it be a GET request on the page? Basically I just provide a link to the file and the download starts automatically, so not sure how to relay the fact that the link was clicked back to the model.
Many thanks :-)
If you create a view that handles the GET request you could put the updating code in that view. If your Django-app is not handling the uploading itself, you can create a view that simply redirects to the download link, after updating the counter in the database.
You have several ways to do this:
create a custom view, that "proxies" all files, while keeping track of downloads
create a middleware that does pretty much the same as above, filtering which requests to record
..but none of the above will be applicable if you want to count downloads of collected static files, that will be served directly by your http server, without passing through django. In this case, I'd retrieve the downloads count from the webserver logs; have a look if your webserver allows storing logs to database, otherwise I'd create a cron-running scripts that parses the logfiles and stores the count to a db, accessible from your django application.
Like redShadow said you can create a proxie view. This view could serve the files via mod_xsendfile (if you are using apache as webserver) and set a counter for the downloads.
My setup is: Django 1.3/Python 2.7.2/Win Server 2008 R2/IIS 7.5/MS SQL Server 2008 R2. I am developing an application whose main function is to analyze uploaded files and produce a report.
Reading over the documentation for django-filetransfers, I believe this is a solution to a problem I've been trying to solve for a while (i.e. form-based file uploads completely block all Django responses until the file-transfer finishes...horror for even moderate-sized files).
The documentation talks about piping uploads to S3 or Blobstore, and that might be what I end up doing eventually, but during development I thought maybe I could just set up my own "poor-man's S3" on a server that I control. This would basically just be another Django instance (or possibly a simple ASP.NET app) whose sole purpose is to receive uploaded files. This sounds like it should be possible with django-filetransfers and would solve the problem of Django responsiveness (???).
But I am missing some bits of understanding how this works in general, as well as some specifics. Maybe an example will help: let's say I have MyMainDjangoServer and MyFileUploadServer. MyMainDjangoServer will serve the views, including the upload form. MyFileUploadServer will "catch" the uploaded files. My questions/confusion are as follows:
My upload form will contain additional fields beyond just the file(s)...do I understand correctly that MyMainDjangoServer will somehow still get that form data, minus the file data (basically: request.POST), and the file data gets shunted over to MyFileUploadServer? How does this work? Will MyMainDjangoServer still block during the upload to MyFileUploadServer?
I assume that what I would need to do on MyFileUploadServer is have a view/URL that handles the form request and sucks out the request.FILES data. What else needs to happen? What happens to the rest of the form data?
How would I set up my settings.py for this scenario? The django-filetransfers examples seem to assume either S3 or GAE/Blobstore but maybe I am missing some basics.
Any advice/answers appreciated...this is a confusing and frustrating area of Django for me.
"MyMainDjangoServer will somehow still get that form data, minus the file data (basically: request.POST), and the file data gets shunted over to MyFileUploadServer? How does this work? Will MyMainDjangoServer still block during the upload to MyFileUploadServer?"
I know the GAE Blobstore, presumably S3 as well, handles this by requiring you to give it a success_url. In your case that would be the url on MyMainDjangoServer where your file receiving view on MyFileUploadServer would re-post the non-files form data to once the upload is complete.
Have a look at the create_upload_url method here: https://developers.google.com/appengine/docs/python/blobstore/functions
You need to recreate this functionality in some form (see below).
"How would I set up my settings.py for this scenario?"
You'd need to create your own filetransfers backend which would be a file with a prepare_upload function in it.
You can see the App Engine one here:
https://github.com/django-nonrel/djangoappengine/blob/develop/storage.py
The prepare_upload method just wraps the GAE create_upload_url method mentioned above.
So in your settings.py you'd have something like:
PREPARE_UPLOAD_BACKEND = 'myapp.filetransfers_backend.prepare_upload'
(i.e. the import path to your prepare_upload function)
For the rest you can start with the ones provided by filetransfers already:
SERVE_FILE_BACKEND = 'filetransfers.backends.url.serve_file'
# if you need it:
PUBLIC_DOWNLOAD_URL_BACKEND = 'filetransfers.backends.url.public_download_url'
These rely on the file_field.url being set (see Django docs) and since your files will be on a separate server you probably need to look into writing a custom storage backend for Django too. (the S3 and GAE cases assume you're using the custom Django storage backends from here)