So, I have created a django application and deployed it with Heroku. Static files (as well as media) are served by a GCS bucket.
I am currently having a problem loading all of my static files. Notably, the png files are not loading, but everything else is. The png file requests return a 403 error, but the rest of the files (css, js, etc.) load fine.
Furthermore, jpg images that were uploaded via media are returned without error.
I have to assume that this is a GCS problem, but I'm not entirely sure of that. I have tried everything in including assigning permissions to my service agent, creating another service agent, and recreating another bucket with the same data.
I have not had this problem when using GCS buckets with applications running on Google App Engine or GKE.
Any point in the right direction would be great.
This is the relevant part of my settings file. Note that I put the django_heroku.settings(locals()) call above this block of code intentionally, because it overrides these settings.
DEBUG = False
django_heroku.settings(locals())
DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
STATICFILES_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
GS_BUCKET_NAME = 'hello-world'
GCS_URL = 'https://storage.googleapis.com/'
STATIC_URL = '{}/{}/static/'.format(GCS_URL, GS_BUCKET_NAME)
MEDIA_URL = '{}/{}/media/'.format(GCS_URL, GS_BUCKET_NAME)
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
UPLOAD_ROOT = 'media/uploads/'
I figured it out. The files that were missing were being called from the CSS file (background images, etc). Since they weren't being called with the {% static %} tag, the appropriate signatures weren't being added onto the call, and the bucket was seeing an unauthorized attempt to access the files.
My solution was to move these calls to bucket files inline, so that I could use the static tag.
Related
Perhaps a misconception on my part, I don't understand what is the purpose of MEDIA_URL and MEDIA_ROOT when using S3. The files are uploaded to the bucket so the root is never used, and the url of the ImageField is stored inside the image object so the MEDIA_URL is never used either. Is this correct? If not, please explain it so I can better understand how everything fits together.
My setup has three buckets:
static for js, css, templates, set as STATICFILES_STORAGE
media for user uploaded images, set as DEFAULT_FILE_STORAGE
spider for images scraped from other websites.
Given that you are using django-storages, you can safely remove these two variables from your settings.py file.
Both variables are used for when you are handling media files on the server itself.
okay so am using whitenoise to store my static files, thats not an issue when deploying to heroku that works perfectly well.
So I need to use S3 to serve my media files. I can upload perfectly to the bucket with no issues, from the website interface that is, but when it tries to load the actual images am trying to serve I instantly get a 404, when I check this 404 in the dev console it shows me the debug message and the link, when I visit the link it actually exists.
So my amazon credentials work, so does CORS and bucket policy, even accessing the images through the link that it claims does exist works.
This is also my MEDIA_ROOT
MEDIA_ROOT = 'https://%s/' % AWS_S3_CUSTOM_DOMAIN
and prints out correctly, is it an issue with using whitenoise and boto storage together?
edit:
so now my static files serve perfectly from S3 but not my media. seems to be ignoring my media_root and media_url
I have a Django web application which allows users to upload their display picture. Some users can see this display picture but not everyone (not all users) should have access to this picture. With that said, in development, I used to save it locally on my machine. In my settings file, I had
MEDIA_ROOT = '/home/myProfile/Documents/thisFolder/uPhotosFolder'
MEDIA_URL = '/media/'
and Django would save all user uploaded display pictures to media rood (the uPhotosFolder) which was located on my machine.
With that said, I deployed (or at least am trying to deploy) the app and when I left my MEDIA_ROOT and MEDIA_URL in my settings file as
MEDIA_ROOT = '/home/myProfile/Documents/thisFolder/uPhotosFolder'
MEDIA_URL = '/media/'
and tried uploading an image, it said access was denied. Apparently I have to go into my EC2 instance (I use Amazon Web Services) and create the
'/home/myProfile/Documents/thisFolder/uPhotosFolder'
directory and then start saving the display images.
My question is, is this the correct / best way to save private user uploaded images on the server? Or should I use S3 for this as well? (From what I read, images on S3 are stored on the cloud and can be accessed from a URL by anyone) Is there a way to use S3 for my situation right now?
The question whether you store the image on EC2 instance in a folder or on S3 Storage is independent of the file privacy.
The MEDIA_ROOT defines the location of Media Folder. The MEDIA_URL defines the URL formed when serving media.
Now for storing if we use default file system storage we end up saving file in MEDIA_ROOT. We can change the storage to S3 by some additional configuration in settings file. ( Docs Here )
Now securing your files on user basis can be done by restricting access to URLs. For this Media URLs can be protected using X-Accell-Redirect headers use when using nginx to serve files.( eg: Here ).
In case you go ahead and use the S3 storage for serving media files ( more preferable in production environment ) - The S3 storage provides functions to get a protected URL for media which can be passed to client to retrieve media. These urls have accessibility for limited time period.
Do static images expire after soon point and automatically get deleted?
My situation is that I retrieve images from my PostgreSQL database through a model's ImageField by setting its upload_to equal to static/images. Then, I access the images through mysitesurl.com/static/images/model.url This works perfectly initially. However, after several hours, what I notice is that the images are no longer accessible. When I try to access them through the same url, they no longer exist. I do not do any manual image deletions so the operations I perform should not interfere with this.
Is this something that Heroku does that I do not understand?
Also, one odd occurrence that I notice is that the image is still accessible through the url, but it doesn't actually get saved to static/images. At least, when I run heroku run ls static/images, it is not there. Is this the correct way to check a directory's contents in heroku?
EDIT: My configs
STATIC_ROOT = ''
STATIC_URL = '/static/'
STATICFILES_DIRS = (
'static',
'static/uploaded_stuff/',
)
You can run heroku run bash --app YOUR_APP from your repository directory. This creates a shell to your Heroku instance allowing you to navigate around your files.
You can check on your static images that way.
I would suggest using S3 to serve your static images though. You can view this tutorial on how to integrate it:
http://blog.doismellburning.co.uk/2012/07/14/using-amazon-s3-to-host-your-django-static-files/
After further details, you are trying to allow user uploads and Heroku's file system is read-only. You will need to use S3 or a similar hosting solution for static/uploaded files.
https://devcenter.heroku.com/articles/read-only-filesystem
I'm trying to serve the media and static files of my django app with a static service on dotcloud (http://stackoverflow.com/questions/13681183/serve-static-files-with-nginx-and-custom-service-dotcloud).
I created a static service where I put my static files. In my django settings, I have:
STATIC_ROOT = STATIC_DIR
STATIC_URL = 'http://mystaticapp.dotcloud.com/'
It works perfectly.
Concerning my media files, I put a folder media in my static service, and I have:
MEDIA_ROOT = 'http://mystaticapp.dotcloud.com/media/'
MEDIA_URL = 'http://mystaticapp.dotcloud.com/media/'
MEDIA_URL works find: my django app get the media from the static service.
But the problem happens with MEDIA_ROOT. It looks like the app is not uploading the files to my static app. When I upload profile picture for my users, it does not upload it to mystaticapp/media/profilepic, as it should.
Any idea on how I could solve that problem? Thank you.
The important thing to remember is that the static service and your django service on dotCloud are separate services, that have no common file system. So if you upload a file to your django service, it won't be available to your static service.
For this use case I would recommend uploading your files to S3, and serving those files from there.
I would recommend Django-storages for this. Here is some documentation: http://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html