I'm currently using django. And now I need to save a file uploaded by a user to another server which is not the one that serves the django application. The file will be saved to file system not the database. Could somebody tell me how to do this?
Default Django behavior is to save file on the filesystem, not the database itself).
You have several options how to do this, the simplest one is to have a filesystem exported from you "other" machine and mounted on the machine with the django application.
For the filesystem export you can use NFS, MogileFS or GlusterFS (which I'm using) or many more :). If you do not need real-time save&serve, simple rsync may be an option too.
Second option is to utilize existing django mechanisms StogareAPI. There are already available different storage backeds you can use and may be helpful for you (e.g. ftp).
This wont work out of the box, you need to have a mechanism(write some code) to queue the files that are uploaded through django application, then use a middleware(can be in python) to transfer files from queue to your file server. so flow is basically like this:
Accept the uploaded file via django app.
django app. writes the file info(its temporary path and name) to a queue
A middleware app reads the next file in queue.
The middleware app use some transfering protocol(sftp, scp, ftp, http etc...) to copy the file to file server.
Middleware app deletes the file from where django app is hosted so django server dont have a copy of the file.
Related
I am working on a django app to store user pics and photos.
What is the optimal approach to store individual user media.
File Sizes are no more than 5MB.
The data is persistent.
The approach i have in mind is:
On form data submission, Upload it to an FTP server using django-storages.
Store the url and fetch it via http later for user.
How to save upload files to another server
I have seen the answers and I don't know what type of queue needs to be used.
you'd usually save the file locally and then latter upload it to some cloud service asynchronously, preferably using something like django-celery
see this answer
I currently have Django set up to upload files to:
/path/to/project/uploads
This works great. This folder is in the root folder of the project so the files cannot be served directly from a web URL, which is what I want, the files are "CVs" uploaded by users.
I've had a look at a third-party django app called filetransfers which would do the job, but I'm wondering if there is a way with Django core to serve files from outside the media folder.
Any help would be great.
Andy
Depending on what web server you are using I would recommend using X-sendfile if you use Apache or X-accel-redirect if you use Nginx. But remember you will need to change setting in your web server. But this is far more efficient way of serving files than using Django to do it.
If what you want is to keep control on how your files are served / who can see them etc, then the simplest solution is to write a custom view serving theses files. You just have to provide the file's content as the response body and set the appropriate response headers (file type, content length etc). Reading the FineManual(tm) part about the Response object should be a good starting point.
Resolved using FileWrapper().
Thanks anyway.
Hy,
I am really new to Pimcore (I'm a joomla guy) and my friend asked me if I could help him transfer a page based on Pimcore to another server. I made a sql dump and copied all the files from the server to my hd and after that on the new server, imported the sql database, changed the username in the db and copied all the files back on the server. Also I made the correct changes for the db in the system.xml file in the config folder. But now all I get is a blank screen but the backend works partially (I can't see the files and the page). I would really appreciate some help!
Thanks in advance
Several things could go wrong, but start with checking:
MySQL user has access to the imported views. This is quite a common issue when importing views between servers,
Apache Linux user has read/write access to the whole website/var folder, access to /var/config is not enough. Pimcore writes cache files to website/var/cache for example.
You are missing mod_rewrite in apache, or your vhost does not allow override.
My setup is: Django 1.3/Python 2.7.2/Win Server 2008 R2/IIS 7.5/MS SQL Server 2008 R2. I am developing an application whose main function is to analyze uploaded files and produce a report.
Reading over the documentation for django-filetransfers, I believe this is a solution to a problem I've been trying to solve for a while (i.e. form-based file uploads completely block all Django responses until the file-transfer finishes...horror for even moderate-sized files).
The documentation talks about piping uploads to S3 or Blobstore, and that might be what I end up doing eventually, but during development I thought maybe I could just set up my own "poor-man's S3" on a server that I control. This would basically just be another Django instance (or possibly a simple ASP.NET app) whose sole purpose is to receive uploaded files. This sounds like it should be possible with django-filetransfers and would solve the problem of Django responsiveness (???).
But I am missing some bits of understanding how this works in general, as well as some specifics. Maybe an example will help: let's say I have MyMainDjangoServer and MyFileUploadServer. MyMainDjangoServer will serve the views, including the upload form. MyFileUploadServer will "catch" the uploaded files. My questions/confusion are as follows:
My upload form will contain additional fields beyond just the file(s)...do I understand correctly that MyMainDjangoServer will somehow still get that form data, minus the file data (basically: request.POST), and the file data gets shunted over to MyFileUploadServer? How does this work? Will MyMainDjangoServer still block during the upload to MyFileUploadServer?
I assume that what I would need to do on MyFileUploadServer is have a view/URL that handles the form request and sucks out the request.FILES data. What else needs to happen? What happens to the rest of the form data?
How would I set up my settings.py for this scenario? The django-filetransfers examples seem to assume either S3 or GAE/Blobstore but maybe I am missing some basics.
Any advice/answers appreciated...this is a confusing and frustrating area of Django for me.
"MyMainDjangoServer will somehow still get that form data, minus the file data (basically: request.POST), and the file data gets shunted over to MyFileUploadServer? How does this work? Will MyMainDjangoServer still block during the upload to MyFileUploadServer?"
I know the GAE Blobstore, presumably S3 as well, handles this by requiring you to give it a success_url. In your case that would be the url on MyMainDjangoServer where your file receiving view on MyFileUploadServer would re-post the non-files form data to once the upload is complete.
Have a look at the create_upload_url method here: https://developers.google.com/appengine/docs/python/blobstore/functions
You need to recreate this functionality in some form (see below).
"How would I set up my settings.py for this scenario?"
You'd need to create your own filetransfers backend which would be a file with a prepare_upload function in it.
You can see the App Engine one here:
https://github.com/django-nonrel/djangoappengine/blob/develop/storage.py
The prepare_upload method just wraps the GAE create_upload_url method mentioned above.
So in your settings.py you'd have something like:
PREPARE_UPLOAD_BACKEND = 'myapp.filetransfers_backend.prepare_upload'
(i.e. the import path to your prepare_upload function)
For the rest you can start with the ones provided by filetransfers already:
SERVE_FILE_BACKEND = 'filetransfers.backends.url.serve_file'
# if you need it:
PUBLIC_DOWNLOAD_URL_BACKEND = 'filetransfers.backends.url.public_download_url'
These rely on the file_field.url being set (see Django docs) and since your files will be on a separate server you probably need to look into writing a custom storage backend for Django too. (the S3 and GAE cases assume you're using the custom Django storage backends from here)
I have a site which enables the user to download certain files. However I want to keep a download count for each file so going the usual way by putting the static files on a different subdomain and then letting apache do the heavy lifting is not a way as well as HttpResponseRedirecting the user to a subdomain isn't good because then the user 'sees' the proper download url and can therefore download the file without incrementing the download count. I could just build a view which then serve()s the file however i am worried about that "big fat disclaimer". How would you/did you implement this? I am quite shure I am not the only one with that problem.
About the Platform: I am using apache and mod_wsgi.
Thank you
We've implemented a system where we needed to control download access to (largish) static files, naturally not wanting Django to serve them itself. We came up with a scheme whereby the Django app, after validating that the user was allowed to download the file (or increment a counter, in your case) we would create a randomly-named symlink to the file, which Apache had access to (be careful: make sure directory indexing is off etc), and then redirect the user to that symlink to be served by Apache.
We have a "cleanup" cronjob that cleans up symlink a minute after they're created, so if they want to download it again, they have to go through Django and have it counted again. Now, theoretically they could download it more than once in that time, but is that likely to happen? You could clean up more than every minute: Apache just needs the symlink to exist at the beginning of the download, not throughout the whole thing.
I'd be curious to know how others address this problem, as I agree with the OP that it is a common scenario.
psj's answer is definitely one viable option. Another option you should investigate is putting a reverse-proxy server in-front of apache like Perlbal which supports "X-REPROXY-URL" headers.
Once you have the reverse-proxy server in place, instead of sending the user a redirect response, you can send a response with the "X-REPROXY-URL" header set to a URL where the proxy server can access but the user can't. The proxy server will then read in the file from the location you sent in the header, and then serve it out to your client. They'll do so in an efficient way and since all your Django app server needs to send is a response with a header set, it is free to handle another request.
The easiest way to do this is to use Apache's X-Sendfile header. Just set the value of the header to the file path and Apache will send the file for you. This blog post has some more details: http://francoisgaudin.com/2011/03/13/serving-static-files-with-apache-while-controlling-access-with-django/ .
I did this with django-counter not to long ago. Lets you keep track of the counts in the admin.
http://github.com/svetlyak40wt/django-counter/