Django Static Files on S3: S3ResponseError: 301 Moved Permanently - django

I'm trying to host my Django Static and Media files on Amazon S3 and I've been following every guide out there, but I still end up getting S3ResponseError: 301 Moved Permanently errors on deployment of my Elastic Beanstalk Application when it tries to run collectstatic.
My S3 is working and I can access other files on it. I also have it set to a custom domain so you can access the same file in the following ways:
http://s3.condopilot.com.s3-eu-west-1.amazonaws.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
https://s3-eu-west-1.amazonaws.com/s3.condopilot.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
http://s3.condopilot.com/thumbs/big/3fca62e2150e8abec3f693a6eae8d2f79bb227fb.jpg
It is the third option that I want to use, but I've tried the other ones aswell. Both with and without https:// in the settings below.
My settings file look like this
#settings.py file
AWS_ACCESS_KEY_ID = 'XXX'
AWS_SECRET_ACCESS_KEY = 'XXX'
AWS_HEADERS = {
'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
'Cache-Control': 'max-age=94608000',
}
AWS_STORAGE_BUCKET_NAME = 's3.condopilot.com'
# I have also tried setting AWS_S3_CUSTOM_DOMAIN to the following:
# - "s3-eu-west-1.amazonaws.com/%s/" % AWS_STORAGE_BUCKET_NAME
# - "s3-eu-west-1.amazonaws.com/%s" % AWS_STORAGE_BUCKET_NAME
# - "s3.condopilot.com"
AWS_S3_CUSTOM_DOMAIN = "%s.s3-eu-west-1.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat'
AWS_S3_SECURE_URLS = False # Tried both True and False
AWS_S3_URL_PROTOCOL = 'http' # Tried with and without
STATICFILES_LOCATION = 'static'
STATIC_URL = "http://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = "http://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
The reason I have AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat' is because without it I get the following error:
ssl.CertificateError: hostname 's3.condopilot.com.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'. All advice I find online regarding that error says that OrdinaryCallingFormat should be used when bucket name contains dots, example s3.condopilot.com.
My custom storages looks like this
#custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
And yes, my S3 bucket is set up in eu-west-1.

I think you do not need to set the region S3 in the URL and if you are using django-storage replaces this app to django-storages-redux. You don't need the custom_storages.py file.
Keep things simple. This is enough.
from django.utils import six
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XxXxXxXxXxXxXxXxXxXxXxXxXxXxxXxX'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
AWS_AUTO_CREATE_BUCKET = False
AWS_QUERYSTRING_AUTH = False
AWS_EXPIRY = 60 * 60 * 24 * 7
AWS_HEADERS = {
'Cache-Control': six.b('max-age=%d, s-maxage=%d, must-revalidate' % (
AWS_EXPIRY, AWS_EXPIRY))
}
MEDIA_URL = 'https://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
STATICFILES_STORAGE = DEFAULT_FILE_STORAGE
STATIC_URL = MEDIA_URL

Related

Django static url with digitalocean spaces

I have an droplet on digitalocean which for the most part is working great, however i have enabled the cdn option in digitalocean spaces (similar to aws s3 storage) and in trying to load static files using the cdn url, i cannot seem to get the url working correctly so suspect i am missing the obvious?
For example when i change the STATIC_URL in settings to the cdn no change is seen in the web page source?
If i change the AWS_S3_ENDPOINT_URL and MEDIA_ENDPOINT_URL then the source does change but the files are not found and as one can guess a collectstatic no longer works, So i assume that the AWS_S3_ENDPOINT_URL & MEDIA_ENDPOINT_URL need to remain as is and i merely need to ensure the static_url is used?
I did read somewhere that it was not good practice to change the templates from {%static.... to {%static_url so have not done that, is this something i should update or not?
Settings:
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
MEDIA_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com/media/'
AWS_STORAGE_BUCKET_NAME = 'mysitestaging'
#STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# if False it will create unique file names for every uploaded file
AWS_S3_FILE_OVERWRITE = False
STATICFILES_STORAGE = 'mysite.settings.storage_backends.StaticStorage'
DEFAULT_FILE_STORAGE = 'mysite.settings.storage_backends.MediaStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
# the sub-directories of media and static files
STATIC_ROOT = 'static'
MEDIA_ROOT = 'media'
AWS_DEFAULT_ACL = 'public-read'
BUCKET_ROOT = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, STATIC_ROOT)
# the regular Django file settings but with the custom S3 URLs
STATIC_URL = '{}/{}/'.format('https://cdn.mysite.com', STATIC_ROOT)
MEDIA_URL = '{}/{}/'.format('https://cdn.mysite.com', MEDIA_ROOT)
Source view returns:
https://nyc3.digitaloceanspaces.com/mysitestaging/static/img/apple-touch-icon.png?AWSAccessKeyId=37FLLPUJLEUO5IG7R4GQ&Signature=eof5%2BZvHPo%2FRSzvKQsrobXkcOZ0%3D&Expires=1586789962
cdn.mysite.com
is an alias of
mysitestaging.nyc3.cdn.digitaloceanspaces.com.
My storage_backends.py:
import os
from storages.backends.s3boto3 import S3Boto3Storage
from my_site.settings import core_settings
class StaticStorage(S3Boto3Storage):
location = core_settings.STATIC_ROOT
class MediaStorage(S3Boto3Storage):
location = core_settings.MEDIA_ROOT
Ok figured this out by actually re-reading the docs:
https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
Adding the following worked immediately:
AWS_S3_CUSTOM_DOMAIN = 'cdn.mysite.com'
Hope it helps someone else.

Why can't I get images from my S3 bucket when all public access blocked? 403 Forbidden But static files load fine

My Django website allows users to upload photos. When the s3 bucket is on public the media content loads fine on the site. However once i block all public access the content no longer loads and a 403 forbidden error is shown. In my code I have added the values needed to allow for authenticate requests. There are no bucket policy's or any CORS configurations. I have tried several different suggestions from blogs and tutorials but nothing seems to work.
I have a user which has programmatic access and set the secret variables in the heroku environment and still get 403 error.
I have tried hard setting the secret variables and still no success.
settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
MEDIA_LOCATION = 'MediaStorage'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIA_LOCATION)
MEDIA_FILE_STORAGE = 'MediaStorage'
STATIC_ROOT = os.path.join(BASE_DIR, "static")
STATIC_LOCATION = 'static'
STATICFILES_LOCATION = 'StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATIC_LOCATION)
django_heroku.settings(locals())
storage_backends.py
from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3Boto3Storage):
location = settings.MEDIA_FILE_STORAGE
file_overwrite = False
custom_domain = False
All static files load fine but media files do not load at all.
I expect the files to load when displaying them in the view web page. however i just get a 403 forbidden error.
In your settings.py your have set?
AWS_DEFAULT_ACL = 'public-read'
Look this example is an public project for my state in Brazil.
Project
So after many iterations it came down to a line of code missing in the storage_backends.py file missing the line
custom_domain = False
I have updated the original post to match the correct storage_backends.py file

How to get the aws s3 object key using django-storages and boto3

I am using django-storage and boto3 for media and static files using aws s3. I need to get the object key of aws s3 bucket, so that I can generate a url for that object.
client = boto3.client('s3')
bucket_name = 'django-bucket'
key = ???
u = client.generate_presigned_url('get_object', Params = {'Bucket': bucket_name, 'Key': key,'ResponseContentType':'image/jpeg', 'ResponseContentDisposition': 'attachment; filename="your-filename.jpeg"'}, ExpiresIn = 1000)
These are in my settings:
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
STATICFILES_STORAGE = 'myproject.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'myproject.custom_storages.MediaStorage'
AWS_ACCESS_KEY_ID = "my_access_key_id"
AWS_SECRET_ACCESS_KEY = "my_secret_access_key"
AWS_STORAGE_BUCKET_NAME = "django-bucket"
AWS_QUERYSTRING_AUTH = False
AWS_S3_CUSTOM_DOMAIN = AWS_STORAGE_BUCKET_NAME + ".s3.amazonaws.com"
# static media settings
STATIC_URL = "https://" + AWS_STORAGE_BUCKET_NAME + ".s3.amazonaws.com/"
MEDIA_URL = STATIC_URL + "media/"
ADMIN_MEDIA_PREFIX = STATIC_URL + "admin/"
I can get the file path of the image file
ui = UserImage.objects.get(user=user_id, image=image_id)
url = ui.image.url
'https://django-bucket.s3.amazonaws.com/media/user_image/1497598249_49.jpeg'
But I don't know how to get the s3 object key so that I can generate a url for that object.
It would seem the prefix can be constructed from the file field 'storage' location value and the file 'name' (which is a path from the 'location' to the file - it includes what most of us think of as the file 'name').
Here's a quick demo of a function that should get the job done:
import os
def get_file_key(file_field):
return os.path.join(file_field.storage.location, file_field.name)
Use as follows:
prefix = get_file_key(my_model_instance.relevant_file_field)
Notes:
You'll probably want to implement error catching sanity checking in/around the function as appropriate for your needs - for example the function will raise an AttributeError if the file_field is None - and in most scenarios you almost certainly wouldn't want to end up with just the location if for some reason the field.name returned an empty string as a result of a bug or just because that was saved somehow (or '/' if the location wasn't set)
You can get s3 Object key by:
ui = UserImage.objects.get(user=user_id, image=image_id)
bucket_key = ui.image.file.obj.key
'media/user_image/1497598249_49.jpeg'

Configuring multiple subdomains to serve static content in django using cloud front

I have a website developed using django and hosted on AWS EC2.
I am using amazon cloud front as my content delivery network with origin server as amazon S3. I have created multiple cnames and added it to the cloud front distribution.
Using just one cname and using it as Cloudfront domain works as expected.
Now I want to extend it so that content can be downloaded in parallel. But AWS_S3_CUSTOM_DOMAIN setting takes a particular url which can be assigned to only one cname at a time.
How to assign multiple cnames for static contents. Or is there some other way to achieve the same?
Following is the relevant content from my django settings.py file
INSTALLED_APPS += ('s3_folder_storage',)
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
AWS_STORAGE_BUCKET_NAME = "mybucket"
CLOUDFRONT_DOMAIN = "*******.cloudfront.net"
CLOUDFRONT_CNAME = "data1.example.com"
MEDIA_ROOT = '/%s/' % DEFAULT_S3_PATH
MEDIA_URL = '//%s/%s/' % (CLOUDFRONT_DOMAIN, DEFAULT_S3_PATH)
STATIC_S3_PATH = "static"
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//%s/%s/' % (CLOUDFRONT_DOMAIN, STATIC_S3_PATH)
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
AWS_S3_CUSTOM_DOMAIN = CLOUDFRONT_CNAME #important: no "http://"
AWS_S3_SECURE_URLS = True #default, but must set to false if using an alias on cloudfront
AWS_ACCESS_KEY_ID = '******'
AWS_SECRET_ACCESS_KEY = '*****'

Django storages or avatar sets https on my urls so I get certificate errors

I have set my server up with django-storages and django-avatar. When I go to view my site, none of my css or images loads. When I inspect the url i see it is a https and that is causing a certificate error. If I remove the s to make it a normal http then it works fine. What setting have I messed up that is causing the issue?
Django 1.4.5
django-storages
django-avatar
import os
PROJECT_ROOT = os.path.dirname(__file__) + '/'
MEDIA_ROOT = PROJECT_ROOT + 'media/'
MEDIA_URL = 'http://static.XXXX.com.s3.amazonaws.com/'
STATIC_ROOT = PROJECT_ROOT + 'static/'
STATIC_URL = 'http://static.XXXX.com.s3.amazonaws.com/'
STATIC_DOC_ROOT = PROJECT_ROOT + 'static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXXX'
AWS_SECRET_ACCESS_KEY = 'XXXX'
AWS_STORAGE_BUCKET_NAME = 'static.XXXX.com'
AWS_QUERYSTRING_AUTH = False
AWS_S3_SECURE_URLS = False
#AVATAR
AVATAR_DEFAULT_SIZE = 80
AVATAR_THUMB_FORMAT = "PNG"
AVATAR_THUMB_QUALITY = 90
AVATAR_HASH_FILENAMES = False
AVATAR_HASH_USERDIRNAMES = False
AVATAR_GRAVATAR_BACKUP = False
AVATAR_DEFAULT_URL = MEDIA_URL + 'avatars/default.png'
I have used storages (on the same shared server) before with no issue so it leads me to believe that it is avatar that is the issue. Any pointers would be greatly appreciated.
EDIT:
linked files that are using {{ MEDIA_URL }} on the front end are fine
Images that were uploaded using the avatar are not showing due to the https
admin styles and images are not showing due to https
If it helps the images that are not showing are in a section that requires users to be logged in?
...I am just checking other images now
EDIT 2:
Other images (log & bg image) are ok , but are loaded via a css file on s3
I have fixed the issue. The way I did it was by replacing:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
with:
DEFAULT_FILE_STORAGE = 'storages.backends.s3.S3Storage'
from S3 import CallingFormat
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN