There are s3 from yandex cloud
https://cloud.yandex.com/docs/storage/tools/?utm_source=console&utm_medium=empty-page&utm_campaign=storage
How kan I configure django to use it ?
I) Install boto3 anf django-storages libs.
II) Add yandex_s3_storage.py file with the code below:
from storages.backends.s3boto3 import S3Boto3Storage
from sites.crm.settings import YOUR_YANDEX_BUCKET_NAME
class ClientDocsStorage(S3Boto3Storage):
bucket_name = YANDEX_CLIENT_DOCS_BUCKET_NAME
file_overwrite = False
III) Add the code below to settings.py:
INSTALLED_APPS = [
...
'storages',
...
]
...
# ----Yandex s3----
DEFAULT_FILE_STORAGE = 'yandex_s3_storage.ClientDocsStorage' # path to file we created before
YANDEX_CLIENT_DOCS_BUCKET_NAME = 'client-docs'
AWS_ACCESS_KEY_ID = env('AWS_ACCESS_KEY')
AWS_SECRET_ACCESS_KEY = env('AWS_SECRET_ACCESS_KEY')
AWS_S3_ENDPOINT_URL = 'https://storage.yandexcloud.net'
AWS_S3_REGION_NAME = 'storage'
IV) Add a file field to your model:
from sites.yandex_s3_storage import ClientDocsStorage
class ClientDocs(models.Model):
...
upload = models.FileField(storage=ClientDocsStorage())
...
I'm getting a very strange error that got me really stuck for many hours now.
I did my deploy with Heroku for the first time and am using S3 for Media files. First thing I noticed was that I never had a migration working successfully after installing boto and django-storages-redux, not sure why. Anyway I kept going.
My configurations seemed to be working in AWS, since my staticfiles did go to my bucket the first time I tried it, but I got some img files not coming through, so I decided to reinstall boto and django-registration-redux (to see if it would migrate properly).
In the end it never migrated as expected and now I'm getting the following error:
UnboundLocalError: local variable 'region_name' referenced before assignment
in my BOTO package when collectstatic.
I don't understand why migrate won't work and why I started to get this error when I re-installed boto and django-storages-redux.
terminal:
You have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
File "/Users/Alex/Desktop/Allugare/src/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
collected = self.collect()
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
handler(path, prefixed_path, storage)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 349, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 255, in delete_file
if self.storage.exists(prefixed_path):
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/storages/backends/s3boto.py", line 439, in exists
if self.entries:
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/storages/backends/s3boto.py", line 302, in entries
for entry in self.bucket.list(prefix=self.location))
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/storages/backends/s3boto.py", line 301, in <genexpr>
self._entries = dict((self._decode_name(entry.key), entry)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister
encoding_type=encoding_type)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/s3/bucket.py", line 473, in get_all_keys
'', headers, **params)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/s3/bucket.py", line 399, in _get_all
query_args=query_args)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/s3/connection.py", line 668, in make_request
retry_handler=retry_handler
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/connection.py", line 927, in _mexe
request.authorize(connection=self)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/connection.py", line 377, in authorize
connection._auth_handler.add_auth(self, **kwargs)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/auth.py", line 755, in add_auth
**kwargs)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/auth.py", line 574, in add_auth
string_to_sign = self.string_to_sign(req, canonical_request)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/auth.py", line 514, in string_to_sign
sts.append(self.credential_scope(http_request))
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/auth.py", line 496, in credential_scope
region_name = self.determine_region_name(http_request.host)
File "/Users/Alex/Desktop/allugare/lib/python3.5/site-packages/boto/auth.py", line 690, in determine_region_name
return region_name
UnboundLocalError: local variable 'region_name' referenced before assignment
settings.py:
...
INSTALLED_APPS = (
#DJANGO APPS
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
#THIRD PARTY APPS
'allauth',
'allauth.account',
'allauth.socialaccount',
#Social Authentications
'allauth.socialaccount.providers.facebook',
# 'allauth.socialaccount.providers.instagram',
# 'allauth.socialaccount.providers.twitter',
'crispy_forms',
'django_messages',
'storages',
#MY APPS
'lares',
'mensagens',
'profiles',
)
...
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, "static-live", "static")
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, "static-live", "media")
STATICFILES_DIRS = (
os.path.join(BASE_DIR, "static"),
)
AWS_ACCESS_KEY_ID = "***"
AWS_SECRET_ACCESS_KEY = "***"
AWS_FILE_EXPIRE = 200
AWS_PRELOAD_METADATA = True
AWS_QUERYSTRING_AUTH = True
DEFAULT_FILE_STORAGE = 'allugare.utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'allugare.utils.StaticRootS3BotoStorage'
# DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_STORAGE_BUCKET_NAME = 's3allugare'
S3DIRECT_REGION = 'sa-east-1'
S3_URL = '//%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = '//%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = MEDIA_URL
STATIC_URL = S3_URL + 'static/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
AWS_S3_HOST = 'sa-east-1.amazonaws.com'
import datetime
two_months = datetime.timedelta(days=61)
date_two_months_later = datetime.date.today() + two_months
expires = date_two_months_later.strftime("%A, %d %B %Y 20:00:00 GMT")
AWS_HEADERS = {
'Expires': expires,
'Cache-Control': 'max-age=%d' % (int(two_months.total_seconds()), ),
}
wsgi.py
"""
WSGI config for allugare project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "allugare.settings")
application = get_wsgi_application()
utils.py
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static') # S3 bucket name -> static
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media') # S3 bucket name -> static
Okay, here's the thing, if you're not using the default region of AWS you have to be careful when updating your region, since the S3's has different formats depending on the region.
I'm in South America, thus I'm using sa-east-1 (for São Paulo).
To me it worked when I changed the setting:
AWS_S3_HOST = 'sa-east-1.amazonaws.com'
to
AWS_S3_HOST = 's3-sa-east-1.amazonaws.com'
Enjoy!
I'm unable to create an S3 connection in my django environment via django-s3-storages middleware (I'm getting a 403 response from S3).
Boto doesn't seem to be able to pick up the environment settings, and I suspect this is the cause (the traceback isn't helping much). As a diagnosis in manage.py shell:
import boto
boto.connect_s3()
>>> boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
from django.conf import settings
boto.connect_s3(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
>>> S3Connection:s3.amazonaws.com
The docs (and other posts) indicate that these settings should work:
MEDIAFILES_LOCATION = 'media'
AWS_S3_CUSTOM_DOMAIN = 'my-bucket.s3-website-eu-west-1.amazonaws.com'
AWS_S3_HOST = 's3-website-eu-west-1.amazonaws.com'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'django_s3_storage.storage.StaticS3Storage'
#S3 settings from https://github.com/etianen/django-s3-storage
AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "yyy"
AWS_S3_BUCKET_NAME = "my-bucket"
AWS_S3_CALLING_FORMAT = "boto.s3.connection.OrdinaryCallingFormat"
# Make user uploaded files public
AWS_S3_BUCKET_AUTH = False
AWS_S3_MAX_AGE_SECONDS = 60*60*24*365 # 1 year
AWS_S3_GZIP = True
Why isn't boto able to connect?
The django storage middleware uses its own S3Storage class. That class has knowledge of django's settings.py and will use settings.AWS_ACCESS_KEY_ID and settings.AWS_SECRET_ACCESS_KEY if they're configured.
boto3 by itself has no knowledge of the django settings file, so it doesn't use anything you configure in that file. That's why you have to specify the key and secret to boto3 when trying to establish an S3 connection.
So when i try accessing a certain image on S3 from my browser everything works fine. But when python is doing it i get a SuspiciousOperation error.
My static folder is public on S3 so i really have no idea where this is coming from.
Publication.objects.get(id=4039).cover.url
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/django/db/models/fields/files.py", line 64, in _get_url
return self.storage.url(self.name)
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/queued_storage/backends.py", line 291, in url
return self.get_storage(name).url(name)
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/queued_storage/backends.py", line 115, in get_storage
elif cache_result is None and self.remote.exists(name):
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/storages/backends/s3boto.py", line 410, in exists
name = self._normalize_name(self._clean_name(name))
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/storages/backends/s3boto.py", line 341, in _normalize_name
name)
SuspiciousOperation: Attempted access to 'http:/s3-eu-west-1.amazonaws.com/xpto/static/images/default-image.png' denied.
My settings:
AWS_S3_SECURE_URLS = True # use http instead of https
S3_URL = 'http://s3-eu-west-1.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = 'media/'
STATIC_ROOT = '/static/'
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = S3_URL + '/' + MEDIA_ROOT
For now i can work around it, but that is not a long term solution. any ideas?
Danigosa's answer in this thread is the answer:
django-storages and amazon s3 - suspiciousoperation
Create a special storage class as outlined in this post:
Using Amazon S3 to store your Django sites static and media files.
Then override _normalize_name like this:
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
def _clean_name(self, name):
return name
def _normalize_name(self, name):
if not name.endswith('/'):
name += "/"
name = self.location + name
return name
class MediaStorage(S3Boto3Storage):
location = settings.MEDIAFILES_LOCATION
def _clean_name(self, name):
return name
def _normalize_name(self, name):
if not name.endswith('/'):
name += "/"
name = self.location + name
return name
Finally - (on Python 3 at least) DON'T use
{% load static from staticfiles %}
in your templates.
Stick with:
{% load static %}
I recently faced the same issue. Tried all the available solutions. Realised that the issue was with the '/' that I had added while calling static files. Like this, "{% static '/path/to/static/file' %}", changed this to "{% static 'path/to/static/file' %}"
I followed every QA suggestions found on SO and in different blogs, Everything works ok on my dev machine and nothing works on heroku.
here are my settings:
DEFAULT_FILE_STORAGE = 'arena.utils.MediaRootS3BotoStorage' # media files
# storage
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_PRELOAD_METADATA = True # necessary to fix manage.py collectstatic command to only upload changed files instead of all files
S3_URL = 'https://%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = S3_URL + '/media/'
STATIC_URL = S3_URL + '/static/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
COMPRESS_URL = STATIC_URL
COMPRESS_OFFLINE = True
COMPRESS_STORAGE = 'utils.CachedS3BotoStorage'
STATICFILES_STORAGE = COMPRESS_STORAGE
When i run collectstatic/compress everything is ok, i see the files being collected to S3 and put in proper places. I see the manifest file.
Loading any page with compressor tag, show an error OfflineGenerationError: You have offline compression enabled but key "d2a53169c44dec41ce3ee7da19b2b9d4" is missing from offline manifest. Running python manage.py compress again solves nothing. when i check the manifest file, indeed the key it looks for doesn't exist.
What is going wrong here?
Question i already checked:
How to configure django-compressor and django-staticfiles with Amazon's S3?
Django Compressor with S3 URL Heroku
Configuring django-compressor with remote storage (django-storage - amazon s3)
On my side I have very similar config, and I'm successfully using compressor for more than 2 years.
settings.py
COMPRESS_STORAGE = 'MyAwesomeApp.app.CachedS3BotoStorage.CachedS3BotoStorage'
AWS_ACCESS_KEY_ID = '#######'
AWS_SECRET_ACCESS_KEY = '########################+#########+BqoQ'
AWS_STORAGE_BUCKET_NAME = 'myAmazonS3cdn.myawesomewebsite.com'
AWS_S3_SECURE_URLS = False
AWS_QUERYSTRING_AUTH = False
COMPRESS_ROOT = 'MyAwesomeApp/static'
STATIC_ROOT = 'MyAwesomeApp/static/javascript'
COMPRESS_OUTPUT_DIR = 'compressed'
STATICFILES_STORAGE = COMPRESS_STORAGE
STATIC_URL = "http://myAmazonS3cdn.myawesomewebsite.com/"
COMPRESS_URL = STATIC_URL
COMPRESS_ENABLED = True
CachedS3BotoStorage.py
from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage
from django.core.files.base import File
class CachedS3BotoStorage(S3BotoStorage):
"""
S3 storage backend that saves the files locally, too.
"""
def __init__(self, *args, **kwargs):
super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
self.local_storage = get_storage_class("compressor.storage.CompressorFileStorage")()
def save(self, name, content):
name = super(CachedS3BotoStorage, self).save(name, content)
self.local_storage._save(name, content)
return name
I'm running python managep.py compress locally, and having manifest generated on my static files directory. Heroku only deals with the collecstatic and delivers the most recent manifest version to my cdn.
Regards,
I completed the above solution with some lines, to fix the problem that create many (multiples) manifest_%.json in Amazon S3
in setting.py:
STATICFILES_STORAGE = 'your_package.s3utils.CachedS3BotoStorage'
in s3utils.py:
from storages.backends.s3boto import S3BotoStorage
from django.core.files.storage import get_storage_class
class CachedS3BotoStorage(S3BotoStorage):
"""
S3 storage backend that saves the files locally, too.
"""
location = 'static'
def __init__(self, *args, **kwargs):
super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
self.local_storage = get_storage_class(
"compressor.storage.CompressorFileStorage")()
def url(self, name):
"""
Fix problem images admin Django S3 images
"""
url = super(CachedS3BotoStorage, self).url(name)
if name.endswith('/') and not url.endswith('/'):
url += '/'
return url
def save(self, name, content):
name = super(CachedS3BotoStorage, self).save(name, content)
self.local_storage._save(name, content)
return name
# HERE is secret to dont generating multiple manifest.json and to delete manifest.json in Amazon S3
def get_available_name(self, name):
if self.exists(name):
self.delete(name)
return name
I found a git repository that contains post_compile hooks to solve this problem. It runs compress after Heroku built the Django app (and also installs lessc if you need less in your compressor settings).
https://github.com/nigma/heroku-django-cookbook