I'm using S3 to store my static files, but I'm having an issue where only the admin static files are uploading to the bucket.
I expected to see the css and other folders from inside static upload to the bucket, but instead, this is all I got:
static/
- admin/
- flags/
What can I do to get my app's static folder to upload also?
Looking at other answers to this question, it seems like the main difference is that my project uses django-sass-processor, but I'm not sure why that would mean my entire folder is not picked up.
Project folder structure:
backend/
- accounts/
- core/
-- settings.py
-- ...
- static/
- templates/
settings.py
USE_S3 = os.getenv('USE_S3') == 'TRUE'
if USE_S3:
# aws settings
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
AWS_DEFAULT_ACL = 'public-read'
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
# s3 static settings
AWS_LOCATION = 'static'
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
else:
STATIC_URL = '/staticfiles/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'sass_processor.finders.CssFinder',
]
SASS_PROCESSOR_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
Related
I use AWS S3 to store my Django static files (using django-storages). And alternate cloudfront domain name to serve these files.
I am unable to finish Django compressor via python manage.py compress.
This is the error I see:
CommandError: An error occurred during rendering /home/ubuntu/foldername/templates/home/xxx.html: 'https://xxx.domainname.com/static/plugins/xxx-xxx/xxx.js' isn't accessible via COMPRESS_URL ('https://s3bucket-static.s3.amazonaws.com/') and can't be compressed
So I tried using the default cloudfront address, alternate domain name and S3 bucket for COMPRESS_URL:
https://s3bucket-static.s3.amazonaws.com/
https://s3bucket-static.s3.amazonaws.com/static/
https://d1231232131241123.cloudfront.net/
https://d1231232131241123.cloudfront.net/static/
https://xxx.domainname.com/
https://xxx.domainname.com/static/
Only https://domainname.com/static/ don't have error. What happen is that after compress is done, the CACHE folder appear inside the local instance staticfiles folder. Which is wrong since I wanted the CACHE folder to be inside the S3 bucket.
Here are my settings for django compressor:
# django-compressor
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'compressor.finders.CompressorFinder',
)
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True
if not DEBUG:
COMPRESS_URL = env('COMPRESS_URL')
Here are my django-storages settings:
if not DEBUG:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = env('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = env('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME')
AWS_DEFAULT_ACL = None
AWS_CLOUDFRONT_DOMAIN_NAME = env('AWS_CLOUDFRONT_DOMAIN_NAME')
AWS_S3_CUSTOM_DOMAIN = f'{AWS_CLOUDFRONT_DOMAIN_NAME}.com'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'
settings for static files:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles/')
STATICFILES_DIRS = os.path.join(BASE_DIR, 'static/')
It worked after I changed my django-compressor settings to this:
# django-compressor
COMPRESS_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'compressor.finders.CompressorFinder',
)
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True
if not DEBUG:
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/static/'
COMPRESS_ROOT = STATIC_ROOT
COMPRESS_URL = STATIC_URL
I want to upload my staticfiles to amazon s3 storage, but I can't stop django from just putting them in a directory staticfiles in the project root. I have boto3 in my requirements.txt and have set
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
I have successfully set up s3 with media files, to the same bucket (under different directory).
Collectstatic also seems disregards the STATIC_ROOT variable in settings.py when I comment out the s3-settings. When running python3 manage.py collectstatic I expect it to gather static files into STATIC_ROOT as it says in the docs here https://docs.djangoproject.com/en/2.1/ref/settings/ (under static_root). But even if I give a different value to STATIC_ROOT, collectstatic always collects into a directory on the root called staticfiles.
STATIC_ROOT = os.path.join(BASE_DIR, 'this_is_not_used_by_collectstatic')
The rest of my s3 settings:
# Amazon S3
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_STORAGE_BUCKET_NAME = 'my_bucket_name'
AWS_S3_REGION_NAME = 'eu-north-1'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_DEFAULT_ACL = None
AWS_LOCATION = 'static'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# media
DEFAULT_FILE_STORAGE = 'korvir.storage_backends.MediaStorage'
If you're using django-heroku and and you have the following in your code, as per the instructions:
django_heroku.settings(locals())
... then it will overwrite your STATIC_ROOT setting. If you want to preserve your own STATIC_ROOT setting, then make sure you set STATIC_ROOT after calling django_heroku.settings.
Intro: I have made my application in Django I am trying to get my static and media files hosted in aws s3. My Django project is on AWS Lambda and AWS Api gateway using Zappa. below is my settings.py
AWS_DEFAULT_ACL = None
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
DEFAULT_FILE_STORAGE = 'aws_storage_classes.MediaStorage'
AWS_ACCESS_KEY_ID = os.getenv("ACCESS_KEY")
AWS_SECRET_ACCESS_KEY = os.getenv("ACCESS_SECRET_KEY")
AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME")
STATICFILES_STORAGE = 'aws_storage_classes.StaticStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_S3_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://%s.static/' % AWS_S3_DOMAIN
MEDIA_URL = 'https://%s.media/' % AWS_S3_DOMAIN
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
I then created a file in my project folder the same as my manage.py called aws_storage_classes.py
Below are the contents of my file aws_storage_classes.py
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = 'static'
class MediaStorage(S3Boto3Storage):
location = 'media'
Below are my urls
if settings.DEBUG:
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Also in my settings.py
DEBUG = False
When I do python manage.py collectstatic all the staticfiles are downloaded and I don't get any errors. But when go on admin page the static files are not uploaded. See images below
Below is the image of my S3 bucket
Below is what is inside the static folder
Static files not loading
<link rel="stylesheet" type="text/css" href="https://<bucketname>.s3.amazonaws.com/static/style.css?AWSAccessKeyId=AKIAJHJGHJGJGJGJJGJGJMHIFQ&Signature=yfS3%2BvA0q15aUxw7OBySuQWZfjg%3D&x-amz-security-token=FQoGZXIvYXdgljdaksfgdjsgfkjertert;hre;thtires=151232173">
Updated settings.py
AWS_DEFAULT_ACL = None
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
DEFAULT_FILE_STORAGE = 'aws_storage_classes.MediaStorage'
AWS_ACCESS_KEY_ID = os.getenv("ACCESS_KEY")
AWS_SECRET_ACCESS_KEY = os.getenv("ACCESS_SECRET_KEY")
AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME")
STATICFILES_STORAGE = 'aws_storage_classes.StaticStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_S3_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://%s.static/' % AWS_S3_DOMAIN
MEDIA_URL = 'https://%s.media/' % AWS_S3_DOMAIN
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
Below is the link after adding ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
<link rel="stylesheet" type="text/css" href="https://django-static-media.s3.amazonaws.com/static/style.css?AWSAccessKeyId=AKIAGGGGGGGGGMHIFQ&Signature=6gFQTsOSDFSDFA%3D&x-amz-security-token=FQoGZvwbJhd9amp;Expires=1549168642">
Still staticfiles not loading
I have given my user full s3 access so I don't think I should need the below. Also I was able to successfully add the static files in the S3 and it is not giving me any permission related errors. still trying it out as per a suggestions below
Below are my permissions
I tried to give permission to lambda function but it gives me the bellow error
After changing the static and media URLS
I can see STATIC_URL and MEDIA_URL not setup correctly.
change
STATIC_URL = 'https://%s.static/' % AWS_S3_DOMAIN
MEDIA_URL = 'https://%s.media/' % AWS_S3_DOMAIN
To
STATIC_URL = 'https://%s/static/' % AWS_S3_DOMAIN
MEDIA_URL = 'https://%s/media/' % AWS_S3_DOMAIN
It should work.
The following link has a very good explanation.
https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html
I had the same issue but with me it was a ACL issue. I could only get collectstatic to run if I set AWS_DEFAULT_ACL=None but then all the files would be set to private by default and couldn't access them from the app.
To fix I had to change the S3 bucket Object Ownership to Bucket owner preferred so that it accepted the public-read ACL.
I have a working Django site where I've been using whitenoise to serve static files.
I've then switched to S3 storage for both site static and media. And it works really well.
I have one problem with this setup though: collecting the admin static files (select2, tinyMCE, etc) takes lots of times and regularly fails in Heroku.
So here's my ideal scenario:
keep everything as is BUT
DO NOT upload the admin static files to S3 (during collectsatic) and serve them from the app server using whitenoise
So how do I achieve that?
Here's the relevant excerpt from my settings file
ENVIRONMENT = os.environ.get('DJANGO_ENVIRONMENT', 'dev')
IS_PROD = ENVIRONMENT == 'production'
IS_DEV = not IS_PROD
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEBUG = bool(os.environ.get('DJANGO_DEBUG', IS_DEV))
# Static files (CSS, JavaScript, Images) and media
STATIC_URL = '/static/'
STATIC_ROOT = '.static'
if IS_PROD:
AWS_STORAGE_BUCKET_NAME = os.environ.get('S3_BUCKET', '')
AWS_S3_REGION_NAME = os.environ.get('S3_REGION', '')
AWS_ACCESS_KEY_ID = os.environ.get('S3_KEY_ID', '')
AWS_SECRET_ACCESS_KEY = os.environ.get('S3_SECRET', '')
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {
'Expires': 'Thu, 31 Dec 2100 20:00:00 GMT',
'CacheControl': 'max-age=94608000',
}
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{MEDIAFILES_LOCATION}/'
TINYMCE_JS_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/static/tiny_mce/tiny_mce.js'
else:
MEDIA_URL = '/media/'
MEDIA_ROOT = '.media'
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]
I'm trying to configure my django setup to serve static files in debug and production mode. I want in production mode to be from S3 and in debug mode to be from the local installation. I have the settings as below:
COMPRESS_ENABLED = not DEBUG
COMPRESS_PARSER = 'compressor.parser.LxmlParser'
COMPRESS_CSS_FILTERS = ['compressor.filters.cssmin.CSSMinFilter']
COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter']
COMPRESS_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'libs.storages.S3Storage.S3Storage'
AWS_ACCESS_KEY_ID = # Key
AWS_SECRET_ACCESS_KEY = # Secret
AWS_STORAGE_BUCKET_NAME = # Bucket
AWS_QUERYSTRING_AUTH = False
MEDIA_URL = '/media/'
if COMPRESS_ENABLED:
from boto.s3.connection import SubdomainCallingFormat
AWS_S3_CALLING_FORMAT = SubdomainCallingFormat()
COMPRESS_OFFLINE = True
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
# Path for all static files
STATIC_URL = 'https://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = STATIC_URL
else:
STATIC_ROOT = path.join(PROJECT_ROOT, '..', 'assets')
STATIC_URL = '/static/'
I'm able to get everything to work except one tiny thing which I'm not able to figure out. Often times my CSS files will have a background url as /images/logo.png. The folder /images/ exists in S3 but for my local it needs to be /static/images/. I tried to set the S3 URL as /static/ at the end but it wasn't working. Is there anything else i'm missing? How do I get the CSS image urls to redirect to /static/?
I read through the source code of s3boto.py. You can set the following property:
AWS_LOCATION = "static"
That variable creates a static folder in your s3. The other properties should be:
STATIC_URL = 'https://%s.s3.amazonaws.com/static/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = STATIC_URL