I am having problems using an s3 bucket for my sites static files. Currently using django-storages and boto3.
In my setting.py I have:
AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY")
AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME")
AWS_S3_FILE_OVERWITE = False
AWS_DEFAULT_ACL = None
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_LOCATION = 'static'
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
Running collectstatic sends static files to the s3 bucket successfully, but the static urls in my templates do not update.
I am using apache2 and have not changed the .conf file, do I need to?
Any help would be gratefully recieved
When you say but the static urls in my templates do not update. what did you meant by that, can you try printing the static_url in settings and see if you are getting url what you are expecting, and can you share your template where you are accessing this.
Related
I tried moving from local static files to S3 using django-storages. I followed the documentation carefully but still there is no access to the static files.
In the local environment I have:
STATIC_URL = '/static/'
in the settings.py and everything works fine.
when I add all the S3 params as the documentation shows:
STATIC_URL = 'https://django-main.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = 'https://django-main.s3.amazonaws.com/admin/' # tried with this and also without this
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
AWS_ACCESS_KEY_ID = '<AWS_ACCESS_KEY_ID>'
AWS_SECRET_ACCESS_KEY = '<AWS_SECRET_ACCESS_KEY>'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
I ran python manage.py collectstatic which seemed to work fine and uploaded the static files to the bucket.
but running the server and going to the admin page it looks like this:
which is because it doesn't have access to the static files. No error is thrown/shown
Any ideas?
EDIT:
So apperently I'm getting a forbbiden call:
but I changed my settings.py to:
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
and made sure that access is available (put a breakpoint and downloaded a css file using boto3 from that bucket using these exact environment variables and still no solution
The issue was that the bucket read permissions were not public. Changing the permissions worked
I have an droplet on digitalocean which for the most part is working great, however i have enabled the cdn option in digitalocean spaces (similar to aws s3 storage) and in trying to load static files using the cdn url, i cannot seem to get the url working correctly so suspect i am missing the obvious?
For example when i change the STATIC_URL in settings to the cdn no change is seen in the web page source?
If i change the AWS_S3_ENDPOINT_URL and MEDIA_ENDPOINT_URL then the source does change but the files are not found and as one can guess a collectstatic no longer works, So i assume that the AWS_S3_ENDPOINT_URL & MEDIA_ENDPOINT_URL need to remain as is and i merely need to ensure the static_url is used?
I did read somewhere that it was not good practice to change the templates from {%static.... to {%static_url so have not done that, is this something i should update or not?
Settings:
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
MEDIA_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com/media/'
AWS_STORAGE_BUCKET_NAME = 'mysitestaging'
#STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# if False it will create unique file names for every uploaded file
AWS_S3_FILE_OVERWRITE = False
STATICFILES_STORAGE = 'mysite.settings.storage_backends.StaticStorage'
DEFAULT_FILE_STORAGE = 'mysite.settings.storage_backends.MediaStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
# the sub-directories of media and static files
STATIC_ROOT = 'static'
MEDIA_ROOT = 'media'
AWS_DEFAULT_ACL = 'public-read'
BUCKET_ROOT = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, STATIC_ROOT)
# the regular Django file settings but with the custom S3 URLs
STATIC_URL = '{}/{}/'.format('https://cdn.mysite.com', STATIC_ROOT)
MEDIA_URL = '{}/{}/'.format('https://cdn.mysite.com', MEDIA_ROOT)
Source view returns:
https://nyc3.digitaloceanspaces.com/mysitestaging/static/img/apple-touch-icon.png?AWSAccessKeyId=37FLLPUJLEUO5IG7R4GQ&Signature=eof5%2BZvHPo%2FRSzvKQsrobXkcOZ0%3D&Expires=1586789962
cdn.mysite.com
is an alias of
mysitestaging.nyc3.cdn.digitaloceanspaces.com.
My storage_backends.py:
import os
from storages.backends.s3boto3 import S3Boto3Storage
from my_site.settings import core_settings
class StaticStorage(S3Boto3Storage):
location = core_settings.STATIC_ROOT
class MediaStorage(S3Boto3Storage):
location = core_settings.MEDIA_ROOT
Ok figured this out by actually re-reading the docs:
https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
Adding the following worked immediately:
AWS_S3_CUSTOM_DOMAIN = 'cdn.mysite.com'
Hope it helps someone else.
My Django website allows users to upload photos. When the s3 bucket is on public the media content loads fine on the site. However once i block all public access the content no longer loads and a 403 forbidden error is shown. In my code I have added the values needed to allow for authenticate requests. There are no bucket policy's or any CORS configurations. I have tried several different suggestions from blogs and tutorials but nothing seems to work.
I have a user which has programmatic access and set the secret variables in the heroku environment and still get 403 error.
I have tried hard setting the secret variables and still no success.
settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
MEDIA_LOCATION = 'MediaStorage'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIA_LOCATION)
MEDIA_FILE_STORAGE = 'MediaStorage'
STATIC_ROOT = os.path.join(BASE_DIR, "static")
STATIC_LOCATION = 'static'
STATICFILES_LOCATION = 'StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATIC_LOCATION)
django_heroku.settings(locals())
storage_backends.py
from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3Boto3Storage):
location = settings.MEDIA_FILE_STORAGE
file_overwrite = False
custom_domain = False
All static files load fine but media files do not load at all.
I expect the files to load when displaying them in the view web page. however i just get a 403 forbidden error.
In your settings.py your have set?
AWS_DEFAULT_ACL = 'public-read'
Look this example is an public project for my state in Brazil.
Project
So after many iterations it came down to a line of code missing in the storage_backends.py file missing the line
custom_domain = False
I have updated the original post to match the correct storage_backends.py file
I am using a Django website running on an ubuntu EC2 instance with S3 for hosting my static and media. When I try to upload an image file to a model I receive the following.
"An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records."
I have an iam user with AmazonS3FullAccess and I copy and pasted the correct Access key into my settings.py file. I am able to run collectstatic and access the buckets through the terminal, and I even have my static served correctly on my website, but no user can upload images or files. I'v run and rerun aws configure, but still no luck.
Here is the relevant code from my utils.py and settings.py:
utils.py
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = 'static'
class MediaStorage(S3Boto3Storage):
location = 'media'
settings.py
AWS_ACCESS_KEY_ID = 'access_key'
AWS_SECRET_ACCESS_KEY = 'secret_key'
S3_REGION_NAME ='aws_region'
DEFAULT_FILE_STORAGE = 'my-project.utils.MediaStorage'
AWS_STORAGE_BUCKET_NAME = 'my-project-bucket'
STATICFILES_STORAGE = 'my-project.utils.StaticStorage'
AWS_S3_CUSTOM_DOMAIN = '//%s.s3-aws_region.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = AWS_S3_CUSTOM_DOMAIN + 'media/'
STATIC_URL = AWS_S3_CUSTOM_DOMAIN + 'static/'
MEDIA_ROOT = MEDIA_URL
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
my /.aws/config and credentials file also have the same copy and pasted Id and Keys.
Any and all help would be greatly appreciated
I am using django-s3-folder-storage for my static and media files storage. I followed all the instructions mentioned in the documentation, but still I am not able to serve my static files. I am able to serve my user uploaded files, but the static files are not displayed on the website even after running collectstatic command, which copied the files to the bucket. I haven't created any CNAME records in my domain (not sure whether this makes any difference as I am able to see user uploaded images).
My settings file:
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
AWS_ACCESS_KEY_ID = # omitted
AWS_SECRET_ACCESS_KEY = # omitted
AWS_STORAGE_BUCKET_NAME = # omitted
MEDIA_ROOT = '/%s/' % DEFAULT_S3_PATH
MEDIA_URL = '//s3.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
STATICFILES_DIRS = (
root('static'),
)
And my project directory:
website_name
static
website_name
settings.py
I have changed the STATIC_URL to like below and it is working fine now.
STATIC_URL = '//bucketname.s3.amazonaws.com/static/'
But MEDIA_URL is like mentioned in the documentation itself and MEDIA_URL works fine. It is only STATIC_URL which needs modifying.
Looks like the documentation is wrong.
Generally S3 can use either the subdomain or subfolder path format. The default for django-storages, which is used by this particular storage, is subdomain. So it's interesting that changing to subfolder solved the problem. The OP mentions using the European region in a different question, and I suspect the root problem here has to do with the endpoint being used, or one of S3/S3BotoStorage's region-affecting configuration settings (e.g. AWS_CALLING_FORMAT).