Is there a way to make some files unaccessible via direct url? For example, an image appears on a page but the image location doesn't work on it's own.
How Things Normally Work
By default, all static and media files are served up from the static root and media root folders. A good practice is to have NGINX or Apache route to these, or to use something like Django Whitenoise to serve up static files.
In production, you definitely don't want to have the runserver serving up these files because 1) That doesn't scale well, 2) It means you're running in DEBUG mode on production which is an absolute no-no.
Protecting Those Files
You can keep this configuration for most files that you don't need to protect. Instead, you can serve up files of your own within Django from a different filepath. Use the upload_to parameter of the filefield to specify where you want those files to live. For example,
protectedfile = models.FileField(upload_to="protected/")
publicfile = models.FileField(upload_to="public/")
Then in NGINX make your block direct to /var/www/myproject/MEDIAROOT/public instead. That will enable you to continue serving up public files.
Meanwhile, for the protected files, those can be served up by the view with:
def view_to_serve_up_docs(request, document_id):
my_doc = MyDocumentModel.objects.get(id=document_id)
# Do some check against this user
if request.user.is_authenticated():
response = FileResponse(my_doc.privatefile)
response["Content-Disposition"] = "attachment; filename=" + my_doc.privatefile.name
else:
raise Http404()
return response
And link to this view within your templates
<a href='/downloadFileView/12345'>Download File #12345 Here!</a>
Reference
More about the FileResponse object: https://docs.djangoproject.com/en/1.11/ref/request-response/#fileresponse-objects
Related
My Flask application is hosted by Heroku and served on Nginx and uses Cloudflare as a CDN. There are times when I change static assets (images, CSS, JS, etc.) on the backend that get changed through deployment on Heroku. These changes will not change on the client's browser unless they manually purge their cache. The cache does expire on the client's browser every month as recommended, but I want the backend to manually tell client browsers to purge their cache for my website every time I deploy to Heroku and they load/reload my website after the fact. Is there a way to automize this process?
If you're using the same filenames it'll use a cached-copy so why not provide versioning on your static files using a filter? You don't have to change the filename at all. Although do read about the caveats in the link provided.
import os
from some_app import app
#app.template_filter('autoversion')
def autoversion_filter(filename):
# determining fullpath might be project specific
fullpath = os.path.join('some_app/', filename[1:])
try:
timestamp = str(os.path.getmtime(fullpath))
except OSError:
return filename
newfilename = "{0}?v={1}".format(filename, timestamp)
return newfilename
Via https://ana-balica.github.io/2014/02/01/autoversioning-static-assets-in-flask/
“Don’t include a query string in the URL for static resources.” It
says that most proxies will not cache static files with query
parameters. Consequently that will increase the bandwidth, since all
the resources will be downloaded on each request.
“To enable proxy caching for these resources, remove query strings
from references to static resources, and instead encode the parameters
into the file names themselves.” But this implies a slightly different
implementation :)
Using Django2 and nginx, users can upload files (mainly pic,vids) and I want to serve those files by masking the full url path.
This is the example structure I expect to see but I don't want users to know about this structure or even the image filename.
domain.com/media/user/pictures/Y/M/D/image1.jpg
I want the user to see the above image through a url like this and the random UUID number changes for each file and that number can point to any type of file.
domain.com/media/23kj23l9ak3
When the file is uploaded the original name, permissions assigned (public, friends, private), file path and the UUID that is generated stored in the database) but the file is stored on the filesystem or remote location.
I've never got to this point before and I would like to know what the modern way of doing it would be or let me know what technology/features of django/nginx can help me solve it.
I'm not entirely sure why you want to do this, rather than simply using the UUID as the filename of the uploaded file, but you certainly can do it.
One good way would be to route the request via Django, and use the custom X-Accel-Redirect header to tell nginx to respond with a specific file. You would need to store both the UID and the actual path in a Django model. So the nginx config would be something like:
location /protected/ {
internal;
alias /media/user/; # note the trailing slash
}
and the Django view would be:
def user_picture(request, uuid):
image = MyModel.objects.get(uuid=uuid)
response = HttpResponse(status=200)
response['X-Accel-Redirect'] = '/protected/' + image.file.path
return response
How to configure Django and Cherokee to serve media (user uploaded) files from Cherokee but to logged in users only as with #login_required on production.
Create a Django view which servers the file
Use #login_required on this view to restrict the access
Read the file from the disk using standard Python io operations
Use StreamingHttpResponse so there is no latency or memory overhead writing the response
Set response mimetype correctly
I will answer my own question
As you are using Cherokee
Remove direct access to media folder with the media URL as localhost/media/.. for exemple by removing the virtuelhost serving it
Activate (check) Allow X-Sendfile under Handler tab in Common CGI Options in the virtuelserver page that handle Django request.
Let's say you have users pictures under media/pictures to protect that will be visible to all users only. (can be modified as you want just an exemple)
Every user picture is stored in media/pictures/pk.jpg (1.jpg, 2.jpg ..)
Create a view :
#login_required(redirect_field_name=None)
def media_pictures(request,pk):
response = HttpResponse()
path=os.path.join (settings.MEDIA_ROOT,'pictures',pk+'.jpg')
if os.path.isfile(path):
#response['Content-Type']="" # add it if it's not working without ^^
response['X-Accel-Redirect'] = path
#response['X-Sendfile'] = path # same as previous line,
# X-Accel-Redirect is for NGINX and X-Sendfile is for apache , in our case cherokee is compatible with two , use one of them.
return response
return HttpResponseForbidden()
Cherokee now take care of serving the file , it's why we checked the Allow X-Sendfile , this will not work without
path variable here is the full path to the file, can be anywhere , just read accsible by cherokee user or group
4. Url conf
As we disable direct access of Media folder, we need to provide an url to access with from Django using the previous view
for exemple , To make image of user with id 17 accessible
localhost/media/pictures/17.jpg
url(r"^media/pictures/(?P<pk>\d+)\.jpg$", views.media_pictures,name="media_pictures"),
This will also work for Apache, Nginx etc , just configure your server to use X-Sendfile (or X-Accel-Redirect for Nginx), this can be easily found on docs
Using this, every logged user can view all users' pictures , feel free to add additional verifications before serving the file , per user check etc
Hope it will help someone
Currently have a project deployed on Heroku with static files loaded from S3. I'm using boto/django-storage to manage my S3 content, but if I call the same view or load the same page repeatedly, all the images/static content load twice and is not cached.
I've placed
AWS_HEADERS = {
'Cache-Control': 'max-age=2592000',
}
in my settings.py, but the reason seems the same exact images (refreshed + loaded twice) have different signatures in their URL? I've tried multiples headers, but the browser doesn't seem want to cache it and instead loads them all everytime.
try setting AWS_QUERYSTRING_AUTH = False. Then the URL generated will always be the same (public) URL. The default-ACL in S3BotoStorageis public-read, which shouldn't be changed then.
Two things not to forget:
perhaps you want to add public, max-age=XXX, so public proxies also can cache your content?
When you want the browser to cache that long, you should keep in mind that the filenames have to change when you change the content. One solution would be to S3BotoStorage combined with the Django-CachedStaticFilesStorage (see here, but I use it without the seperate cache-backend)
I've made a simple file server that runs on my raspberry pi (1/2 gb RAM, 1 CPU). It's running under gunicorn (3 workers) behind nginx (1 worker).
I've got a weird issue where when I try to download too many files simultaneously (say 5) they all get part way through and then just abort. There's no output from the django server (I get this issue using the development server too which is why it's now running behind gunicorn & nginx, but still no joy).
My download view is:
#never_cache
def download_media(request, user_id, session_key, id, filepath):
"Download an individual media file"
context = RequestContext(request)
# validate the user_id & session_key pair
if not __validate_session_key(user_id, session_key):
return HttpResponseRedirect(reverse('handle_logout'))
filepath = unicode(urllib.unquote(filepath))
if '..' in filepath:
raise SuspiciousOperation('Invalid characters in subdir parameter.')
location = MediaCollectionLocation.objects.get(id=id)
path = os.path.join(location.path, filepath)
response = HttpResponse(FileWrapper(file(path)), content_type='application/octet-stream')
response['Content-Disposition'] = 'attachment; filename=%s' % os.path.basename(path)
response['Content-Length'] = os.path.getsize(path)
response["Cache-Control"] = "no-cache, no-store, must-revalidate"
return response
I'm serving files this way because I want clients to authenticate (so don't just want to redirect and serve static content with nginx).
Anyone any idea why it'd drop out if I make several requests in parallel?
I'm not entirely sure why the files would be failing, but I'd imagine it'd have something to do with downloading more files than you have workers, or that there is a timeout happening between nginx and gunicorn.
You can get nginx to serve the files only after django has authenticated the user by making django set a particular header that nginx then reads (internal only) and serves the file itself.
XSendFile is what nginx uses to do this. You may then either create some middleware or a function to set the appropriate headers from django, or use something like django-sendfile with the nginx backend to have it all done for you.
If the issue you're experiencing is caused by a timeout between django and nginx, this fix should solve it. If it does not, increase the number of nginx workers as well, since it will be responsible for the serving of files now.