I have a website developed using django and hosted on AWS EC2.
I am using amazon cloud front as my content delivery network with origin server as amazon S3. I have created multiple cnames and added it to the cloud front distribution.
Using just one cname and using it as Cloudfront domain works as expected.
Now I want to extend it so that content can be downloaded in parallel. But AWS_S3_CUSTOM_DOMAIN setting takes a particular url which can be assigned to only one cname at a time.
How to assign multiple cnames for static contents. Or is there some other way to achieve the same?
Following is the relevant content from my django settings.py file
INSTALLED_APPS += ('s3_folder_storage',)
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
AWS_STORAGE_BUCKET_NAME = "mybucket"
CLOUDFRONT_DOMAIN = "*******.cloudfront.net"
CLOUDFRONT_CNAME = "data1.example.com"
MEDIA_ROOT = '/%s/' % DEFAULT_S3_PATH
MEDIA_URL = '//%s/%s/' % (CLOUDFRONT_DOMAIN, DEFAULT_S3_PATH)
STATIC_S3_PATH = "static"
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//%s/%s/' % (CLOUDFRONT_DOMAIN, STATIC_S3_PATH)
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
AWS_S3_CUSTOM_DOMAIN = CLOUDFRONT_CNAME #important: no "http://"
AWS_S3_SECURE_URLS = True #default, but must set to false if using an alias on cloudfront
AWS_ACCESS_KEY_ID = '******'
AWS_SECRET_ACCESS_KEY = '*****'
Related
I have created a backend django app using AWS Beanstalk, and a frontend reactjs app deployed using cloudfront (plus S3)
I have a model in backend that does
class EnhancedUser(AbstractUser):
# some other attributes
picture = models.ImageField(blank=True)
my settings.py has
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '<my_elastic_beanstalk_domain>/media/'
Since I'm using cloudfront, if i just set the MEDIA_URL to /media/, it would just append /media/ to my cloudfront url, so I have to hardcode it to my backend url
and then, following the django docs, I added the static part to my urls.py
urlpatterns = [
path('admin/', admin.site.urls),
# some other urls
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Note that django doc does mention we can't use absolute url for MEDIA_URL, but I have no alternative solution at the moment
When I upload my image, it doesn't get stored in the right place, but I cannot open it with the url. It returns a 404 saying the img's url is not part of urls list
My question is:
How do I set it up so I can display the image
Since the images will be updated through users/admins, these will be stored in the EC2 instance created in beanstalk, so every time I deploy, I think they will be wiped. How do I prevent this?
Take a look at using django-storages to save your uploads. I use S3 for storing uploads of a django/docker/EB deployment, and include django settings that look something like this (I keep them in settings/deployment.py):
if 'AWS_ACCESS_KEY_ID' in os.environ:
# Use Amazon S3 for storage for uploaded media files
# Keep them private by default
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# Amazon S3 settings.
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
AWS_STORAGE_BUCKET_NAME = os.environ["AWS_STORAGE_BUCKET_NAME"]
AWS_S3_REGION_NAME = os.environ.get("AWS_S3_REGION_NAME", None)
AWS_S3_SIGNATURE_VERSION = 's3v4'
AWS_AUTO_CREATE_BUCKET = False
AWS_HEADERS = {"Cache-Control": "public, max-age=86400"}
AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = 'private'
AWS_QUERYSTING_AUTH = True
AWS_QUERYSTRING_EXPIRE = 600
AWS_S3_SECURE_URLS = True
AWS_REDUCED_REDUNDANCY = False
AWS_IS_GZIPPED = False
MEDIA_ROOT = '/'
MEDIA_URL = 'https://s3.{}.amazonaws.com/{}/'.format(
AWS_S3_REGION_NAME, AWS_STORAGE_BUCKET_NAME)
USING_AWS = True
I have an droplet on digitalocean which for the most part is working great, however i have enabled the cdn option in digitalocean spaces (similar to aws s3 storage) and in trying to load static files using the cdn url, i cannot seem to get the url working correctly so suspect i am missing the obvious?
For example when i change the STATIC_URL in settings to the cdn no change is seen in the web page source?
If i change the AWS_S3_ENDPOINT_URL and MEDIA_ENDPOINT_URL then the source does change but the files are not found and as one can guess a collectstatic no longer works, So i assume that the AWS_S3_ENDPOINT_URL & MEDIA_ENDPOINT_URL need to remain as is and i merely need to ensure the static_url is used?
I did read somewhere that it was not good practice to change the templates from {%static.... to {%static_url so have not done that, is this something i should update or not?
Settings:
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
MEDIA_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com/media/'
AWS_STORAGE_BUCKET_NAME = 'mysitestaging'
#STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# if False it will create unique file names for every uploaded file
AWS_S3_FILE_OVERWRITE = False
STATICFILES_STORAGE = 'mysite.settings.storage_backends.StaticStorage'
DEFAULT_FILE_STORAGE = 'mysite.settings.storage_backends.MediaStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
# the sub-directories of media and static files
STATIC_ROOT = 'static'
MEDIA_ROOT = 'media'
AWS_DEFAULT_ACL = 'public-read'
BUCKET_ROOT = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, STATIC_ROOT)
# the regular Django file settings but with the custom S3 URLs
STATIC_URL = '{}/{}/'.format('https://cdn.mysite.com', STATIC_ROOT)
MEDIA_URL = '{}/{}/'.format('https://cdn.mysite.com', MEDIA_ROOT)
Source view returns:
https://nyc3.digitaloceanspaces.com/mysitestaging/static/img/apple-touch-icon.png?AWSAccessKeyId=37FLLPUJLEUO5IG7R4GQ&Signature=eof5%2BZvHPo%2FRSzvKQsrobXkcOZ0%3D&Expires=1586789962
cdn.mysite.com
is an alias of
mysitestaging.nyc3.cdn.digitaloceanspaces.com.
My storage_backends.py:
import os
from storages.backends.s3boto3 import S3Boto3Storage
from my_site.settings import core_settings
class StaticStorage(S3Boto3Storage):
location = core_settings.STATIC_ROOT
class MediaStorage(S3Boto3Storage):
location = core_settings.MEDIA_ROOT
Ok figured this out by actually re-reading the docs:
https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
Adding the following worked immediately:
AWS_S3_CUSTOM_DOMAIN = 'cdn.mysite.com'
Hope it helps someone else.
I'm trying to host my Django Static and Media files on Amazon S3 and I've been following every guide out there, but I still end up getting S3ResponseError: 301 Moved Permanently errors on deployment of my Elastic Beanstalk Application when it tries to run collectstatic.
My S3 is working and I can access other files on it. I also have it set to a custom domain so you can access the same file in the following ways:
http://s3.condopilot.com.s3-eu-west-1.amazonaws.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
https://s3-eu-west-1.amazonaws.com/s3.condopilot.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
http://s3.condopilot.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
It is the third option that I want to use, but I've tried the other ones aswell. Both with and without https:// in the settings below.
My settings file look like this
#settings.py file
AWS_ACCESS_KEY_ID = 'XXX'
AWS_SECRET_ACCESS_KEY = 'XXX'
AWS_HEADERS = {
'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
'Cache-Control': 'max-age=94608000',
}
AWS_STORAGE_BUCKET_NAME = 's3.condopilot.com'
# I have also tried setting AWS_S3_CUSTOM_DOMAIN to the following:
# - "s3-eu-west-1.amazonaws.com/%s/" % AWS_STORAGE_BUCKET_NAME
# - "s3-eu-west-1.amazonaws.com/%s" % AWS_STORAGE_BUCKET_NAME
# - "s3.condopilot.com"
AWS_S3_CUSTOM_DOMAIN = "%s.s3-eu-west-1.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat'
AWS_S3_SECURE_URLS = False # Tried both True and False
AWS_S3_URL_PROTOCOL = 'http' # Tried with and without
STATICFILES_LOCATION = 'static'
STATIC_URL = "http://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = "http://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
The reason I have AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat' is because without it I get the following error:
ssl.CertificateError: hostname 's3.condopilot.com.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'. All advice I find online regarding that error says that OrdinaryCallingFormat should be used when bucket name contains dots, example s3.condopilot.com.
My custom storages looks like this
#custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
And yes, my S3 bucket is set up in eu-west-1.
I think you do not need to set the region S3 in the URL and if you are using django-storage replaces this app to django-storages-redux. You don't need the custom_storages.py file.
Keep things simple. This is enough.
from django.utils import six
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XxXxXxXxXxXxXxXxXxXxXxXxXxXxxXxX'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
AWS_AUTO_CREATE_BUCKET = False
AWS_QUERYSTRING_AUTH = False
AWS_EXPIRY = 60 * 60 * 24 * 7
AWS_HEADERS = {
'Cache-Control': six.b('max-age=%d, s-maxage=%d, must-revalidate' % (
AWS_EXPIRY, AWS_EXPIRY))
}
MEDIA_URL = 'https://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATICFILES_STORAGE = DEFAULT_FILE_STORAGE
STATIC_URL = MEDIA_URL
I have set my server up with django-storages and django-avatar. When I go to view my site, none of my css or images loads. When I inspect the url i see it is a https and that is causing a certificate error. If I remove the s to make it a normal http then it works fine. What setting have I messed up that is causing the issue?
Django 1.4.5
django-storages
django-avatar
import os
PROJECT_ROOT = os.path.dirname(__file__) + '/'
MEDIA_ROOT = PROJECT_ROOT + 'media/'
MEDIA_URL = 'http://static.XXXX.com.s3.amazonaws.com/'
STATIC_ROOT = PROJECT_ROOT + 'static/'
STATIC_URL = 'http://static.XXXX.com.s3.amazonaws.com/'
STATIC_DOC_ROOT = PROJECT_ROOT + 'static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXXX'
AWS_SECRET_ACCESS_KEY = 'XXXX'
AWS_STORAGE_BUCKET_NAME = 'static.XXXX.com'
AWS_QUERYSTRING_AUTH = False
AWS_S3_SECURE_URLS = False
#AVATAR
AVATAR_DEFAULT_SIZE = 80
AVATAR_THUMB_FORMAT = "PNG"
AVATAR_THUMB_QUALITY = 90
AVATAR_HASH_FILENAMES = False
AVATAR_HASH_USERDIRNAMES = False
AVATAR_GRAVATAR_BACKUP = False
AVATAR_DEFAULT_URL = MEDIA_URL + 'avatars/default.png'
I have used storages (on the same shared server) before with no issue so it leads me to believe that it is avatar that is the issue. Any pointers would be greatly appreciated.
EDIT:
linked files that are using {{ MEDIA_URL }} on the front end are fine
Images that were uploaded using the avatar are not showing due to the https
admin styles and images are not showing due to https
If it helps the images that are not showing are in a section that requires users to be logged in?
...I am just checking other images now
EDIT 2:
Other images (log & bg image) are ok , but are loaded via a css file on s3
I have fixed the issue. The way I did it was by replacing:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
with:
DEFAULT_FILE_STORAGE = 'storages.backends.s3.S3Storage'
from S3 import CallingFormat
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN
I am using django-s3-folder-storage for my static and media files storage. I followed all the instructions mentioned in the documentation, but still I am not able to serve my static files. I am able to serve my user uploaded files, but the static files are not displayed on the website even after running collectstatic command, which copied the files to the bucket. I haven't created any CNAME records in my domain (not sure whether this makes any difference as I am able to see user uploaded images).
My settings file:
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
AWS_ACCESS_KEY_ID = # omitted
AWS_SECRET_ACCESS_KEY = # omitted
AWS_STORAGE_BUCKET_NAME = # omitted
MEDIA_ROOT = '/%s/' % DEFAULT_S3_PATH
MEDIA_URL = '//s3.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
STATICFILES_DIRS = (
root('static'),
)
And my project directory:
website_name
static
website_name
settings.py
I have changed the STATIC_URL to like below and it is working fine now.
STATIC_URL = '//bucketname.s3.amazonaws.com/static/'
But MEDIA_URL is like mentioned in the documentation itself and MEDIA_URL works fine. It is only STATIC_URL which needs modifying.
Looks like the documentation is wrong.
Generally S3 can use either the subdomain or subfolder path format. The default for django-storages, which is used by this particular storage, is subdomain. So it's interesting that changing to subfolder solved the problem. The OP mentions using the European region in a different question, and I suspect the root problem here has to do with the endpoint being used, or one of S3/S3BotoStorage's region-affecting configuration settings (e.g. AWS_CALLING_FORMAT).