Django. Alternate between local & remote staticfiles - django

After collecting my staticfiles and storing them in an Amazon Bucket (AWS S3), when I run the project locally it still uses the staticfiles stored online, this is a problem cause when I want to make a change on a css file for ex, I have to run collectstatic or manually upload the file to Amazon. I tried adding a new setting variable "LOCAL_STATICFILES" like this:
settings.py
LOCAL_STATICFILES = False
if not LOCAL_STATICFILES:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWSAccessKeyId']
AWS_SECRET_ACCESS_KEY = os.environ['AWSSecretKey']
AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
if LOCAL_STATICFILES:
STATIC_URL = '/static/'
STATIC_ROOT = '/'
But when I turn LOCAL_STATICFILES to True and runserver, django can't find them.
The project's folders look like this:
project
app
app
static
css
js
img
templates
What am I doing wrong?

First of all: Ensure you have a way to distinguish whether you are, or not, in an environment supporting the Amazon bucket configuration. This means, usually this will be your production environment, where you already configured the amazon bucket settings.
So you will:
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
# this is the base project path
if 'AWSAccessKeyId' in os.environ:
# assume the presence of this key will determine whether
# we are, or not, in the bucket-supporting environment
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWSAccessKeyId']
AWS_SECRET_ACCESS_KEY = os.environ['AWSSecretKey']
AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
STATIC_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
# static url will be the re
STATIC_ROOT = None
# STATIC_ROOT doesn't matter since you will not invoke
# `manage.py collectstatic` from this environment. You
# can safely let it to None, or anything like:
# os.path.join(BASE_DIR, 'static')
else:
STATIC_URL = '/static/'
# the static files url will point to your local,
# development, server
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
# here, STATIC_ROOT matters, since you will invoke
# `manage.py collectstatic` from your local environment.

Related

Django-Compressor won't work in offline mode with custom S3 domain

I get the following error whenever a custom domain for the S3 endpoint is used.
# WORKS
AWS_S3_CUSTOM_DOMAIN = 'example.fra1.digitaloceanspaces.com/{}'.format(AWS_STORAGE_BUCKET_NAME)
# DOES NOT WORK ⁉️
AWS_S3_CUSTOM_DOMAIN = 'cdn.example.com/{}'.format(AWS_STORAGE_BUCKET_NAME)
CommandError: An error occurred during rendering /home/<user>/<app>/templates/public/index.html: 'https://cdn.example.com/storage/static/node_modules/nouislider/distribute/nouislider.min.css' isn't accessible via COMPRESS_URL ('https://example.fra1.digitaloceanspaces.com/storage/static/') and can't be compressed
If I go to either url the file is accessible, hence its likely that the CDN is ok, URLs are correctly defined, CORS are fine too.
Also without django-compressor subdomain delivery had been working fine, leading me to believe the issue is not with django-storages
I've been trying for several hours and ultimately had to do a temporary fix by setting the AWS_S3_CUSTOM_DOMAIN to be the same as the AWS_S3_ENDPOINT_URL. However this is not ideal.
Please see the implementation below.
/requirements.txt
Django==3.1.4
...
boto3~=1.16.46
botocore~=1.19.46
s3transfer~=0.3.3
...
django-storages~=1.11.1
django-compressor~=2.4
/config/settings.py
...
# ==========================================================
# DJANGO-STORAGES
# ==========================================================
if LOCALHOST_MODE:
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
STATIC_URL = '/static/'
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static/'), ]
else:
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_STORAGE_BUCKET_NAME = 'storage'
AWS_S3_ENDPOINT_URL = 'https://example.fra1.digitaloceanspaces.com'
# WORKS ⚠️
AWS_S3_CUSTOM_DOMAIN = 'example.fra1.digitaloceanspaces.com/{}'.format(AWS_STORAGE_BUCKET_NAME)
# DOES NOT WORK ⁉️
AWS_S3_CUSTOM_DOMAIN = 'cdn.example.com/{}'.format(AWS_STORAGE_BUCKET_NAME)
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'
AWS_DEFAULT_ACL = 'public-read'
STATICFILES_DIRS = (os.path.join(BASE_DIR, "static"), )
STATIC_URL = '{}/{}/{}/'.format(AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME, AWS_LOCATION)
STATICFILES_STORAGE = 'storage_backends.CachedS3Boto3Storage'
MEDIA_ROOT = os.path.join(BASE_DIR, "media")
MEDIA_URL = '{}/{}/{}/'.format(AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME, 'media')
DEFAULT_FILE_STORAGE = 'storage_backends.MediaStorage'
# ==========================================================
# DJANGO-COMPRESSOR
# ==========================================================
COMPRESS_ENABLED = True
STATIC_DEPS = True
COMPRESS_ROOT = os.path.join(BASE_DIR, "static")
COMPRESS_FILTERS = {
'css': ['compressor.filters.css_default.CssAbsoluteFilter', 'compressor.filters.cssmin.rCSSMinFilter'],
'js': ['compressor.filters.jsmin.JSMinFilter']
}
if LOCALHOST_MODE:
COMPRESS_OFFLINE = False
else:
COMPRESS_OFFLINE = True
COMPRESS_STORAGE = STATICFILES_STORAGE
COMPRESS_URL = STATIC_URL
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'compressor.finders.CompressorFinder',
]
...
/storage_backends.py
from django.core.files.storage import get_storage_class
from storages.backends.s3boto3 import S3Boto3Storage
from config import settings
class MediaStorage(S3Boto3Storage):
bucket_name = settings.AWS_STORAGE_BUCKET_NAME
location = 'media'
class CachedS3Boto3Storage(S3Boto3Storage):
def __init__(self, *args, **kwargs):
super(CachedS3Boto3Storage, self).__init__(*args, **kwargs)
self.local_storage = get_storage_class(
"compressor.storage.CompressorFileStorage")()
def save(self, name, content):
self.local_storage._save(name, content)
super(CachedS3Boto3Storage, self).save(name, self.local_storage._open(name))
return name
How to debug this error:
Looking at where the error is raised (in django-compressor.compressor.base) we find the following:
def get_basename(self, url):
try:
base_url = self.storage.base_url
except AttributeError:
base_url = settings.COMPRESS_URL
base_url = str(base_url)
if not url.startswith(base_url):
raise UncompressableFileError(
"'%s' isn't accessible via "
"COMPRESS_URL ('%s') and can't be "
"compressed" % (url, base_url)
)
Looking at your settings:
your COMPRESS_URL is set equal to STATIC_URL,
which in turn is equal to '{}/{}/{}/'.format(AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME, AWS_LOCATION).
This is https://example.fra1.digitaloceanspaces.com/storage/static/
But the url being passed in is https://cdn.example.com/storage/static/...
This is the route of the problem.
So what's happening??
COMPRESS_URL controls the URL that linked files will be read from and compressed files will be written to.
When you have a CSS or JS file with a link, rel or src attribute somewhere, django-compressor calls get_basename.
get_basename takes full path to a static file (eg. "/static/css/style.css") and
returns path with storage's base url removed (eg. "css/style.css").
Since the base path of some link, rel or src attribute doesn't begin with COMPRESS_URL (which it is expecting to be the base url) it can't remove it and so raises an error.
How to solve this
It's difficult to know exactly what is going wrong without seeing the whole project, however the below might help:
If you updated your AWS_S3_ENDPOINT_URL to match your AWS_S3_CUSTOM_DOMAIN. This would in turn update your STATIC_URL (which I suspect is affecting the filenames in your files), and will also update your COMPRESS_URL. I think this might fix it.
Alternatively, just update COMPRESS_URL to use the correct url (although I think this needs to match STATIC_URL with your current setup).
If any (link, rel or src) urls in files that are being compressed are hard-coded they need to be updated to match your COMPRESS_URL.
Django Compressor has to be configured for remote storage. Section 2.5.1 of the Django Compressor Documentation covers S3 and explains that you need to set
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
COMPRESS_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/'
and
COMPRESS_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
in settings.py. Before I could get my app working, I still had to debug further such that I had to run collectstatic locally so that my_project/static matched my S3 storage. Then, I had to set
STATICFILES_LOCATION = ''
The latter two bugs may be unrelated issues, but I thought I should mention them.

How to load static files from separate S3 bucket when hosting site on Elastic Beanstalk Aws React Django

Setup: React+Django hosted on Elastic Beanstalk. Static files hosted on separate S3 Bucket.
I'm trying to load an image using src="/static/images/logo.png".
In development it works perfectly, but in production it sends a request to XXX.elasticbeanstalk.com/static/images/logo.png while it should be requesting Bucket.amazonaws.com/static/images/logo.png.
Meanwhile the user uploaded media is working perfectly for both POST and GET requests, images are stored and fetched from the Bucket /media/ path.
I want to avoid conditionally coding the absolute url path depending on the environment.
I have a django production_settings.py file:
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATIC_ROOT = os.path.join(BASE_DIR, "..", "www", "static")
STATIC_URL = '/static/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
AWS_STORAGE_BUCKET_NAME = '****************'
AWS_S3_REGION_NAME = '****************'
AWS_ACCESS_KEY_ID = '****************'
AWS_SECRET_ACCESS_KEY = '****************'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
custom_storages.py:
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3Boto3Storage):
location = settings.MEDIAFILES_LOCATION
Thanks
It seems like your STATIC_URL set incorrectly. It refers to your web app project path instead of S3 bucket URL. You can try using this below:
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
You have already extended S3Boto3Storage and you could see there is multiple parameters, you could set additional env key for static storage class:
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
custom_domain = setting('AWS_S3_CUSTOM_DOMAIN_STATIC')

django's collectstatic collects into unexpected directory

I want to upload my staticfiles to amazon s3 storage, but I can't stop django from just putting them in a directory staticfiles in the project root. I have boto3 in my requirements.txt and have set
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
I have successfully set up s3 with media files, to the same bucket (under different directory).
Collectstatic also seems disregards the STATIC_ROOT variable in settings.py when I comment out the s3-settings. When running python3 manage.py collectstatic I expect it to gather static files into STATIC_ROOT as it says in the docs here https://docs.djangoproject.com/en/2.1/ref/settings/ (under static_root). But even if I give a different value to STATIC_ROOT, collectstatic always collects into a directory on the root called staticfiles.
STATIC_ROOT = os.path.join(BASE_DIR, 'this_is_not_used_by_collectstatic')
The rest of my s3 settings:
# Amazon S3
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_STORAGE_BUCKET_NAME = 'my_bucket_name'
AWS_S3_REGION_NAME = 'eu-north-1'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_DEFAULT_ACL = None
AWS_LOCATION = 'static'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# media
DEFAULT_FILE_STORAGE = 'korvir.storage_backends.MediaStorage'
If you're using django-heroku and and you have the following in your code, as per the instructions:
django_heroku.settings(locals())
... then it will overwrite your STATIC_ROOT setting. If you want to preserve your own STATIC_ROOT setting, then make sure you set STATIC_ROOT after calling django_heroku.settings.

Django collectstatic not pushing to AWS S3

I can't push my local files to amazon S3 using django 'collectstatic'.
My settings.py, all the settings that may be relevant:
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
INSTALLED_APPS += ('storages',)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWSAccessKeyId']
AWS_SECRET_ACCESS_KEY = os.environ['AWSSecretKey']
AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL
STATICFILES_DIRS = (
os.path.join(BASE_DIR, "static"),)
STATIC_ROOT = 'staticfiles'
If I run collectsatic with these settings, I get the error:
0 static files copied, 139 unmodified.
If I change the STATIC_ROOT to "/" (this is how I'd like it to be, given the folder distribution in the Amazon Bucket), I get the error:
OSError: [Errno 13] Permission denied: '/css'
Please help, I've run out of ideas. I've double-checked the credentials and I've also tried to run it from heroku.
heroku run python manage.py collectstatic
Same errors.
According to the docs, you'll need to add
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
to your settings file.

Django serving static files

I'm trying to configure my django setup to serve static files in debug and production mode. I want in production mode to be from S3 and in debug mode to be from the local installation. I have the settings as below:
COMPRESS_ENABLED = not DEBUG
COMPRESS_PARSER = 'compressor.parser.LxmlParser'
COMPRESS_CSS_FILTERS = ['compressor.filters.cssmin.CSSMinFilter']
COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter']
COMPRESS_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'libs.storages.S3Storage.S3Storage'
AWS_ACCESS_KEY_ID = # Key
AWS_SECRET_ACCESS_KEY = # Secret
AWS_STORAGE_BUCKET_NAME = # Bucket
AWS_QUERYSTRING_AUTH = False
MEDIA_URL = '/media/'
if COMPRESS_ENABLED:
from boto.s3.connection import SubdomainCallingFormat
AWS_S3_CALLING_FORMAT = SubdomainCallingFormat()
COMPRESS_OFFLINE = True
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
# Path for all static files
STATIC_URL = 'https://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = STATIC_URL
else:
STATIC_ROOT = path.join(PROJECT_ROOT, '..', 'assets')
STATIC_URL = '/static/'
I'm able to get everything to work except one tiny thing which I'm not able to figure out. Often times my CSS files will have a background url as /images/logo.png. The folder /images/ exists in S3 but for my local it needs to be /static/images/. I tried to set the S3 URL as /static/ at the end but it wasn't working. Is there anything else i'm missing? How do I get the CSS image urls to redirect to /static/?
I read through the source code of s3boto.py. You can set the following property:
AWS_LOCATION = "static"
That variable creates a static folder in your s3. The other properties should be:
STATIC_URL = 'https://%s.s3.amazonaws.com/static/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = STATIC_URL