I've been trying to find a way to upload a large file to GAE's datastore using Django's admin interface, but haven't found an answer that specifically addresses this issue. I'm fairly new to Python/Django, so there might be an angle that I'm not looking at.
I've been looking at the django-filetransfers solution, but I'm not sure how to integrate that into the admin interface. Any suggestions would be great!
File uploads to the appengine are handled using the blobstore and not the datastore.
https://developers.google.com/appengine/docs/python/blobstore/overview#Complete_Sample_App
The datastore filesize limit is 1 megabyte. So, attempting to upload a large file into the appengine's non-relational database isn't going to work. This is by design.
The documentation I provided above shows you how to implement large file uploads.
Related
I am using Django 1.4 on Google App Engine.
I have a model called Media, where the admins can upload files to use in their website. It has a field:
file = models.FileField(upload_to='/uploads/%Y/%m/%d')
It works perfectly with images (although the URL provided is weird), but that is not a problem.
The problem is when they try to upload a PDF (or anything else). Everything seems to work, but when you go to edit it, it doesn't contain any file - there is no "Currently" or anything else.
If I go to Google App Engine dashboard, the file is in Blob Viewer, and the record is also saved and available through the Datastore Viewer, with the correct blob key.
Why Django is not recognizing it? And how I can fix it?
djangoappengine contains a storage provider for the blobstore.
To me it doesn't seem like a full-featured solution, just something to get file uploads running.
I had to add this to get some functionality I needed (urls to the files).
It might make the admin work too:
https://github.com/dragonx/djangoappengine/commit/6a472a80c25bc077816c9104b45d5a64f3063273
I'm developing a django 1.4 application, and I'm using django-filepicker to upload images. They use a special modelField for images called FPFileField.
I'm interested in storing those images in a MongoDB database, instead of uploading it to a "media" folder or something similar.
Any idea how this can be achieved?
I've tried mongoengine with no luck.
I would recommend storing the url into your mongo Document directly and then making use of the filepicker.io URL and built in S3 support rather than storing the file on your servers directly.
You should look into GridFS
I'd also consider uploading it to amazon's S3 service too - depending on what your requirements are.
My setup is: Django 1.3/Python 2.7.2/Win Server 2008 R2/IIS 7.5/MS SQL Server 2008 R2. I am developing an application whose main function is to analyze uploaded files and produce a report.
Reading over the documentation for django-filetransfers, I believe this is a solution to a problem I've been trying to solve for a while (i.e. form-based file uploads completely block all Django responses until the file-transfer finishes...horror for even moderate-sized files).
The documentation talks about piping uploads to S3 or Blobstore, and that might be what I end up doing eventually, but during development I thought maybe I could just set up my own "poor-man's S3" on a server that I control. This would basically just be another Django instance (or possibly a simple ASP.NET app) whose sole purpose is to receive uploaded files. This sounds like it should be possible with django-filetransfers and would solve the problem of Django responsiveness (???).
But I am missing some bits of understanding how this works in general, as well as some specifics. Maybe an example will help: let's say I have MyMainDjangoServer and MyFileUploadServer. MyMainDjangoServer will serve the views, including the upload form. MyFileUploadServer will "catch" the uploaded files. My questions/confusion are as follows:
My upload form will contain additional fields beyond just the file(s)...do I understand correctly that MyMainDjangoServer will somehow still get that form data, minus the file data (basically: request.POST), and the file data gets shunted over to MyFileUploadServer? How does this work? Will MyMainDjangoServer still block during the upload to MyFileUploadServer?
I assume that what I would need to do on MyFileUploadServer is have a view/URL that handles the form request and sucks out the request.FILES data. What else needs to happen? What happens to the rest of the form data?
How would I set up my settings.py for this scenario? The django-filetransfers examples seem to assume either S3 or GAE/Blobstore but maybe I am missing some basics.
Any advice/answers appreciated...this is a confusing and frustrating area of Django for me.
"MyMainDjangoServer will somehow still get that form data, minus the file data (basically: request.POST), and the file data gets shunted over to MyFileUploadServer? How does this work? Will MyMainDjangoServer still block during the upload to MyFileUploadServer?"
I know the GAE Blobstore, presumably S3 as well, handles this by requiring you to give it a success_url. In your case that would be the url on MyMainDjangoServer where your file receiving view on MyFileUploadServer would re-post the non-files form data to once the upload is complete.
Have a look at the create_upload_url method here: https://developers.google.com/appengine/docs/python/blobstore/functions
You need to recreate this functionality in some form (see below).
"How would I set up my settings.py for this scenario?"
You'd need to create your own filetransfers backend which would be a file with a prepare_upload function in it.
You can see the App Engine one here:
https://github.com/django-nonrel/djangoappengine/blob/develop/storage.py
The prepare_upload method just wraps the GAE create_upload_url method mentioned above.
So in your settings.py you'd have something like:
PREPARE_UPLOAD_BACKEND = 'myapp.filetransfers_backend.prepare_upload'
(i.e. the import path to your prepare_upload function)
For the rest you can start with the ones provided by filetransfers already:
SERVE_FILE_BACKEND = 'filetransfers.backends.url.serve_file'
# if you need it:
PUBLIC_DOWNLOAD_URL_BACKEND = 'filetransfers.backends.url.public_download_url'
These rely on the file_field.url being set (see Django docs) and since your files will be on a separate server you probably need to look into writing a custom storage backend for Django too. (the S3 and GAE cases assume you're using the custom Django storage backends from here)
I met the django-uploadify (multiple file uploading) application, but the only template-use is described in wiki of the project. I need to integrate it with django admin. Any ideas?
#rebus and #Mordi have good suggestions; one of those is probably your best bet. Still...
The current implementation of file uploads in web browsers is single file only. That's a limitation of the HTML standards and the browsers themselves. When you see multiple file upload capability (all at once / select multiple) in some web app, something like Flash or Java is in play. You could potentially use a Flash or Java applet (check out Uploadify - about the easiest to work with you'll find), if you wanted, but you'd have to do some work tying it into Django: namely mapping the output of the applet to the input the Django Admin expects on POST.
Personally I found this package to be painful when I attempted to do this locally. It is 7 years old and not maintained. django-filer is a more modern implementation that has this exact functionality built into the admin panel, it was quick to set up and it seems to be maintained a lot better. And the UI is excellent!
If your really set on using uploadify my attempt was quite painful. When I went to the website to download the uploadify jQuery plugin it was a paid download. After looking a bit I found a fork, and was able to get it going locally, but because the python package is old it required a lot of changes from python 2.7..
To integrate this with your project in the admin will definitely require you to write your own custom admin template. In your admin.py you would have to override the default template and methods, this faq has some insights. In your new template you would include the uploadify jquery script and put the upload file field inside the form where your model fields would be. When you upload files the signal sender in the package will fire, and you would just have to intercept the signal and handle it with your intended logic
Very easy to implement multiple image upload with this project:
https://github.com/tstone/django-uploadify
I am sorry if this question is already posted somewhere.
I am very new to Django. I have been googling around for quite a long time for sample codes to upload large files using Django, but goes in vain.
Can anyone help me with some sample code for uploading large files in Django or provide relavant reliable link(s)?
Thanks in advance.
Django itself handles large files uploads quite well (the whole file is not loaded to memory since 1.0, see http://code.djangoproject.com/ticket/2070).
But django usually sits behind web server and there is often a limit over request body. So Web server config should be probably adjusted (if you are using apache, look at http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestbody ).
There is this called django-video. I haven't tried it though. I have used django-basic-apps
which can handle video, music and image uploads.