Django compressor and Cloudfront - django

I use AWS S3 to store my Django static files (using django-storages). And alternate cloudfront domain name to serve these files.
I am unable to finish Django compressor via python manage.py compress.
This is the error I see:
CommandError: An error occurred during rendering /home/ubuntu/foldername/templates/home/xxx.html: 'https://xxx.domainname.com/static/plugins/xxx-xxx/xxx.js' isn't accessible via COMPRESS_URL ('https://s3bucket-static.s3.amazonaws.com/') and can't be compressed
So I tried using the default cloudfront address, alternate domain name and S3 bucket for COMPRESS_URL:
https://s3bucket-static.s3.amazonaws.com/
https://s3bucket-static.s3.amazonaws.com/static/
https://d1231232131241123.cloudfront.net/
https://d1231232131241123.cloudfront.net/static/
https://xxx.domainname.com/
https://xxx.domainname.com/static/
Only https://domainname.com/static/ don't have error. What happen is that after compress is done, the CACHE folder appear inside the local instance staticfiles folder. Which is wrong since I wanted the CACHE folder to be inside the S3 bucket.
Here are my settings for django compressor:
# django-compressor
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'compressor.finders.CompressorFinder',
)
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True
if not DEBUG:
COMPRESS_URL = env('COMPRESS_URL')
Here are my django-storages settings:
if not DEBUG:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = env('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = env('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME')
AWS_DEFAULT_ACL = None
AWS_CLOUDFRONT_DOMAIN_NAME = env('AWS_CLOUDFRONT_DOMAIN_NAME')
AWS_S3_CUSTOM_DOMAIN = f'{AWS_CLOUDFRONT_DOMAIN_NAME}.com'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'
settings for static files:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles/')
STATICFILES_DIRS = os.path.join(BASE_DIR, 'static/')

It worked after I changed my django-compressor settings to this:
# django-compressor
COMPRESS_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'compressor.finders.CompressorFinder',
)
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True
if not DEBUG:
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/static/'
COMPRESS_ROOT = STATIC_ROOT
COMPRESS_URL = STATIC_URL

Related

Why is collectstatic only detecting admin static files?

I'm using S3 to store my static files, but I'm having an issue where only the admin static files are uploading to the bucket.
I expected to see the css and other folders from inside static upload to the bucket, but instead, this is all I got:
static/
- admin/
- flags/
What can I do to get my app's static folder to upload also?
Looking at other answers to this question, it seems like the main difference is that my project uses django-sass-processor, but I'm not sure why that would mean my entire folder is not picked up.
Project folder structure:
backend/
- accounts/
- core/
-- settings.py
-- ...
- static/
- templates/
settings.py
USE_S3 = os.getenv('USE_S3') == 'TRUE'
if USE_S3:
# aws settings
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
AWS_DEFAULT_ACL = 'public-read'
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
# s3 static settings
AWS_LOCATION = 'static'
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
else:
STATIC_URL = '/staticfiles/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'sass_processor.finders.CssFinder',
]
SASS_PROCESSOR_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

Django storages using incorrect urls for s3 static files

I am using Django storages to host my static files (css, js, images) on s3. When I load my webpage django keeps pointing to the incorrect url of my s3 public bucket. For example, it keeps returning https://mysite.amazonaws.com/assets/images/cat.png despite the correct public url for the file being https://mysite.s3-ap-southeast-2.amazonaws.com/assets/images/cat.png
settings.py
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = os.environ.get('aws_access_key_id')
AWS_SECRET_ACCESS_KEY = os.environ.get('aws_secret_key')
AWS_STORAGE_BUCKET_NAME = 'mysite'
AWS_DEFAULT_ACL = "private"
AWS_S3_SIGNATURE_VERSION = "s3v4"
AWS_S3_REGION_NAME = 'ap-southeast-2'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
STATIC_URL = 'https://mysite.s3-ap-southeast-2.amazonaws.com/static/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
Try setting a s3 custom domain.
AWS_S3_CUSTOM_DOMAIN = 'mysite.s3-ap-southeast-2.amazonaws.com'

Django app on heroku doesn't load staticfiles from S3 bucket

When I run heroku run python manage.py collectstatic it does upload staticfiles to AWS S3, however when I run django in production, my static files do not load to the page.
To be honest i dont know why do i have bad request, since i already did collectstatic to S3.
My AWS settings # settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
STATIC_URL = 'https://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
AWS_URL = os.environ.get('AWS_URL')
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
STATIC_URL = "https://" + AWS_STORAGE_BUCKET_NAME + ".s3.amazonaws.com/"
MEDIA_URL = STATIC_URL + "media/"
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)
STATIC_ROOT = 'staticfiles'
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

Django: serve admin static files with whitenoise and site's files from S3

I have a working Django site where I've been using whitenoise to serve static files.
I've then switched to S3 storage for both site static and media. And it works really well.
I have one problem with this setup though: collecting the admin static files (select2, tinyMCE, etc) takes lots of times and regularly fails in Heroku.
So here's my ideal scenario:
keep everything as is BUT
DO NOT upload the admin static files to S3 (during collectsatic) and serve them from the app server using whitenoise
So how do I achieve that?
Here's the relevant excerpt from my settings file
ENVIRONMENT = os.environ.get('DJANGO_ENVIRONMENT', 'dev')
IS_PROD = ENVIRONMENT == 'production'
IS_DEV = not IS_PROD
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEBUG = bool(os.environ.get('DJANGO_DEBUG', IS_DEV))
# Static files (CSS, JavaScript, Images) and media
STATIC_URL = '/static/'
STATIC_ROOT = '.static'
if IS_PROD:
AWS_STORAGE_BUCKET_NAME = os.environ.get('S3_BUCKET', '')
AWS_S3_REGION_NAME = os.environ.get('S3_REGION', '')
AWS_ACCESS_KEY_ID = os.environ.get('S3_KEY_ID', '')
AWS_SECRET_ACCESS_KEY = os.environ.get('S3_SECRET', '')
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {
'Expires': 'Thu, 31 Dec 2100 20:00:00 GMT',
'CacheControl': 'max-age=94608000',
}
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{MEDIAFILES_LOCATION}/'
TINYMCE_JS_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/static/tiny_mce/tiny_mce.js'
else:
MEDIA_URL = '/media/'
MEDIA_ROOT = '.media'
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]

Django serving static files

I'm trying to configure my django setup to serve static files in debug and production mode. I want in production mode to be from S3 and in debug mode to be from the local installation. I have the settings as below:
COMPRESS_ENABLED = not DEBUG
COMPRESS_PARSER = 'compressor.parser.LxmlParser'
COMPRESS_CSS_FILTERS = ['compressor.filters.cssmin.CSSMinFilter']
COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter']
COMPRESS_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'libs.storages.S3Storage.S3Storage'
AWS_ACCESS_KEY_ID = # Key
AWS_SECRET_ACCESS_KEY = # Secret
AWS_STORAGE_BUCKET_NAME = # Bucket
AWS_QUERYSTRING_AUTH = False
MEDIA_URL = '/media/'
if COMPRESS_ENABLED:
from boto.s3.connection import SubdomainCallingFormat
AWS_S3_CALLING_FORMAT = SubdomainCallingFormat()
COMPRESS_OFFLINE = True
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
# Path for all static files
STATIC_URL = 'https://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = STATIC_URL
else:
STATIC_ROOT = path.join(PROJECT_ROOT, '..', 'assets')
STATIC_URL = '/static/'
I'm able to get everything to work except one tiny thing which I'm not able to figure out. Often times my CSS files will have a background url as /images/logo.png. The folder /images/ exists in S3 but for my local it needs to be /static/images/. I tried to set the S3 URL as /static/ at the end but it wasn't working. Is there anything else i'm missing? How do I get the CSS image urls to redirect to /static/?
I read through the source code of s3boto.py. You can set the following property:
AWS_LOCATION = "static"
That variable creates a static folder in your s3. The other properties should be:
STATIC_URL = 'https://%s.s3.amazonaws.com/static/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = STATIC_URL