I'm trying to setup django and the cms package mezzanine to use amazon s3 storage using django-storages and django-s3-folder-storage. All works well in the sense that I can use collectstatic without issue and upload my files to amazon, and also images get served correctly as does css/js.
However if a user tries to upload an image to the media-library or as a featured-image for a blog post. I get simply HTTP Error in the browser and POST /admin/media-library/upload_file/ HTTP/1.1" 500 146580 in the console. This is with DEBUG=True set, so I'm surprised there is not more of a trace/feedback.
I'm not sure what to do to fix or even begin debugging why this is occurring, anyone help?
My relevant settings.py are:
AWS_STORAGE_BUCKET_NAME = 'my_bucket'
AWS_SECRET_ACCESS_KEY = 'my_key'
AWS_ACCESS_KEY_ID = 'my_id'
AWS_PRELOAD_METADATA = True
AWS_QUERYSTRING_AUTH = False
AWS_S3_SECURE_URLS = False
AWS_S3_ENCRYPTION = False
from boto.s3.connection import ProtocolIndependentOrdinaryCallingFormat
AWS_S3_CALLING_FORMAT = ProtocolIndependentOrdinaryCallingFormat()
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
MEDIA_ROOT = ''
MEDIA_URL = ''
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
I had the same problem. What the problem turned out to be for me was in my setup I needed to change the permissions on this directory (and all its sub-directories) to be owned by the user used by my nginx.conf:
/var/lib/nginx
To correct this I did:
sudo chown -R <myuser>:<myuser> /var/lib/nginx
Where myuser is the nginx user specified in nginx.conf.
Think images you are uploading are placed in a temporary folder in there before reaching Amazon S3 and this temporary folder doesn't have the correct permissions to allow this.
What got me on the right track to fix this was I changed debug to DEBUG = True and then when I got the 500 error in my error logs I saw:
[crit] 26489#0: *374 open() "/var/lib/nginx/tmp/client_body/0000000050" failed (13: Permission denied),
Also have a look at this:
nginx 500 error, permission denied for tmp folder
Related
I tried moving from local static files to S3 using django-storages. I followed the documentation carefully but still there is no access to the static files.
In the local environment I have:
STATIC_URL = '/static/'
in the settings.py and everything works fine.
when I add all the S3 params as the documentation shows:
STATIC_URL = 'https://django-main.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = 'https://django-main.s3.amazonaws.com/admin/' # tried with this and also without this
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
AWS_ACCESS_KEY_ID = '<AWS_ACCESS_KEY_ID>'
AWS_SECRET_ACCESS_KEY = '<AWS_SECRET_ACCESS_KEY>'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
I ran python manage.py collectstatic which seemed to work fine and uploaded the static files to the bucket.
but running the server and going to the admin page it looks like this:
which is because it doesn't have access to the static files. No error is thrown/shown
Any ideas?
EDIT:
So apperently I'm getting a forbbiden call:
but I changed my settings.py to:
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
and made sure that access is available (put a breakpoint and downloaded a css file using boto3 from that bucket using these exact environment variables and still no solution
The issue was that the bucket read permissions were not public. Changing the permissions worked
My Django website allows users to upload photos. When the s3 bucket is on public the media content loads fine on the site. However once i block all public access the content no longer loads and a 403 forbidden error is shown. In my code I have added the values needed to allow for authenticate requests. There are no bucket policy's or any CORS configurations. I have tried several different suggestions from blogs and tutorials but nothing seems to work.
I have a user which has programmatic access and set the secret variables in the heroku environment and still get 403 error.
I have tried hard setting the secret variables and still no success.
settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
MEDIA_LOCATION = 'MediaStorage'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIA_LOCATION)
MEDIA_FILE_STORAGE = 'MediaStorage'
STATIC_ROOT = os.path.join(BASE_DIR, "static")
STATIC_LOCATION = 'static'
STATICFILES_LOCATION = 'StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATIC_LOCATION)
django_heroku.settings(locals())
storage_backends.py
from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3Boto3Storage):
location = settings.MEDIA_FILE_STORAGE
file_overwrite = False
custom_domain = False
All static files load fine but media files do not load at all.
I expect the files to load when displaying them in the view web page. however i just get a 403 forbidden error.
In your settings.py your have set?
AWS_DEFAULT_ACL = 'public-read'
Look this example is an public project for my state in Brazil.
Project
So after many iterations it came down to a line of code missing in the storage_backends.py file missing the line
custom_domain = False
I have updated the original post to match the correct storage_backends.py file
I am using a Django website running on an ubuntu EC2 instance with S3 for hosting my static and media. When I try to upload an image file to a model I receive the following.
"An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records."
I have an iam user with AmazonS3FullAccess and I copy and pasted the correct Access key into my settings.py file. I am able to run collectstatic and access the buckets through the terminal, and I even have my static served correctly on my website, but no user can upload images or files. I'v run and rerun aws configure, but still no luck.
Here is the relevant code from my utils.py and settings.py:
utils.py
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = 'static'
class MediaStorage(S3Boto3Storage):
location = 'media'
settings.py
AWS_ACCESS_KEY_ID = 'access_key'
AWS_SECRET_ACCESS_KEY = 'secret_key'
S3_REGION_NAME ='aws_region'
DEFAULT_FILE_STORAGE = 'my-project.utils.MediaStorage'
AWS_STORAGE_BUCKET_NAME = 'my-project-bucket'
STATICFILES_STORAGE = 'my-project.utils.StaticStorage'
AWS_S3_CUSTOM_DOMAIN = '//%s.s3-aws_region.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = AWS_S3_CUSTOM_DOMAIN + 'media/'
STATIC_URL = AWS_S3_CUSTOM_DOMAIN + 'static/'
MEDIA_ROOT = MEDIA_URL
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
my /.aws/config and credentials file also have the same copy and pasted Id and Keys.
Any and all help would be greatly appreciated
I have set my server up with django-storages and django-avatar. When I go to view my site, none of my css or images loads. When I inspect the url i see it is a https and that is causing a certificate error. If I remove the s to make it a normal http then it works fine. What setting have I messed up that is causing the issue?
Django 1.4.5
django-storages
django-avatar
import os
PROJECT_ROOT = os.path.dirname(__file__) + '/'
MEDIA_ROOT = PROJECT_ROOT + 'media/'
MEDIA_URL = 'http://static.XXXX.com.s3.amazonaws.com/'
STATIC_ROOT = PROJECT_ROOT + 'static/'
STATIC_URL = 'http://static.XXXX.com.s3.amazonaws.com/'
STATIC_DOC_ROOT = PROJECT_ROOT + 'static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXXX'
AWS_SECRET_ACCESS_KEY = 'XXXX'
AWS_STORAGE_BUCKET_NAME = 'static.XXXX.com'
AWS_QUERYSTRING_AUTH = False
AWS_S3_SECURE_URLS = False
#AVATAR
AVATAR_DEFAULT_SIZE = 80
AVATAR_THUMB_FORMAT = "PNG"
AVATAR_THUMB_QUALITY = 90
AVATAR_HASH_FILENAMES = False
AVATAR_HASH_USERDIRNAMES = False
AVATAR_GRAVATAR_BACKUP = False
AVATAR_DEFAULT_URL = MEDIA_URL + 'avatars/default.png'
I have used storages (on the same shared server) before with no issue so it leads me to believe that it is avatar that is the issue. Any pointers would be greatly appreciated.
EDIT:
linked files that are using {{ MEDIA_URL }} on the front end are fine
Images that were uploaded using the avatar are not showing due to the https
admin styles and images are not showing due to https
If it helps the images that are not showing are in a section that requires users to be logged in?
...I am just checking other images now
EDIT 2:
Other images (log & bg image) are ok , but are loaded via a css file on s3
I have fixed the issue. The way I did it was by replacing:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
with:
DEFAULT_FILE_STORAGE = 'storages.backends.s3.S3Storage'
from S3 import CallingFormat
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN
I am using boto with django-storages to upload images directly to S3 from my django form. Everything is ok, except image url is https. I think having https is overkill and not required for media files. How do I change the setting to serve images as http? Went through documentation, couldn't find right settings.
My AWS setting is as follows.
AWS_ACCESS_KEY_ID = 'xxx'
AWS_SECRET_ACCESS_KEY = 'xxx'
AWS_STORAGE_BUCKET_NAME = 'xxx'
AWS_DEFAULT_ACL = 'private'
AWS_LOCATION = '/media/'
Thanks in advance.
Ok, I read the django-storages source code, and it's there.
django-storages on github
All I need to do is set SECURE_URLS to false, like this.
AWS_S3_SECURE_URLS = False
:)