Django amazon s3 SuspiciousOperation - django

So when i try accessing a certain image on S3 from my browser everything works fine. But when python is doing it i get a SuspiciousOperation error.
My static folder is public on S3 so i really have no idea where this is coming from.
Publication.objects.get(id=4039).cover.url
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/django/db/models/fields/files.py", line 64, in _get_url
return self.storage.url(self.name)
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/queued_storage/backends.py", line 291, in url
return self.get_storage(name).url(name)
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/queued_storage/backends.py", line 115, in get_storage
elif cache_result is None and self.remote.exists(name):
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/storages/backends/s3boto.py", line 410, in exists
name = self._normalize_name(self._clean_name(name))
File "/home/vagrant/.pyenv/versions/blook/lib/python2.7/site-packages/storages/backends/s3boto.py", line 341, in _normalize_name
name)
SuspiciousOperation: Attempted access to 'http:/s3-eu-west-1.amazonaws.com/xpto/static/images/default-image.png' denied.
My settings:
AWS_S3_SECURE_URLS = True # use http instead of https
S3_URL = 'http://s3-eu-west-1.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = 'media/'
STATIC_ROOT = '/static/'
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = S3_URL + '/' + MEDIA_ROOT
For now i can work around it, but that is not a long term solution. any ideas?

Danigosa's answer in this thread is the answer:
django-storages and amazon s3 - suspiciousoperation
Create a special storage class as outlined in this post:
Using Amazon S3 to store your Django sites static and media files.
Then override _normalize_name like this:
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
def _clean_name(self, name):
return name
def _normalize_name(self, name):
if not name.endswith('/'):
name += "/"
name = self.location + name
return name
class MediaStorage(S3Boto3Storage):
location = settings.MEDIAFILES_LOCATION
def _clean_name(self, name):
return name
def _normalize_name(self, name):
if not name.endswith('/'):
name += "/"
name = self.location + name
return name
Finally - (on Python 3 at least) DON'T use
{% load static from staticfiles %}
in your templates.
Stick with:
{% load static %}

I recently faced the same issue. Tried all the available solutions. Realised that the issue was with the '/' that I had added while calling static files. Like this, "{% static '/path/to/static/file' %}", changed this to "{% static 'path/to/static/file' %}"

Related

Django static files get from wrong url

I am migrating an old Django 2.2 project to Django 3.2.
I use AWS Cloudfront and S3.
I have done most of the tasks to do. But i have something weird in my static files serving.
When 'static' template tag is called, the rendered url is "https//<cloudfront_url>/static.." instead of "https://<cloudfront_url>/static..".
The comma disapears ! It obviously results to a 404 not found.
It worked without any problem on Django2.2.
So for the moment i dirty patched this by doing a '.replace("https//", "https://")' on django static template tag.
My settings relatives to static files are :
# settings.py
STATICFILES_LOCATION = 'static'
AWS_CLOUDFRONT_DOMAIN = <idcloudfront>.cloudfront.net
STATICFILES_STORAGE = 'app.custom_storages.StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_CLOUDFRONT_DOMAIN, STATICFILES_LOCATION)
AWS_STORAGE_STATIC_BUCKET_NAME = 'assets'
# app.custom_storages.py
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
bucket_name = settings.AWS_STORAGE_STATIC_BUCKET_NAME
def __init__(self, *args, **kwargs):
kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
super(StaticStorage, self).__init__(*args, **kwargs)
this seems to come from django storage since staticfiles_storage.url('css/a_css_file.css') return url without ":".
Oh ok after further investigations i see that come from storages.storages.backends.S3boto3.S3Boto3Storage from django-storages package.
https://github.com/jschneier/django-storages/blob/master/storages/backends/s3boto3.py#L577
url = '{}//{}/{}{}'.format(
self.url_protocol,
self.custom_domain,
filepath_to_uri(name),
'?{}'.format(urlencode(params)) if params else '',
)
":" seems to be missing.
Temporary fixed it by adding url_protocol = "https:" in my StaticStorage custom class and opened an issue on django-storage library.

Django - placing and accessing a js file in root directory

I am trying to implement Firebase cloud messaging push notifications in my Django project.
Firebase would look for a js file named firebase-messaging-sw.js in a project' root directory (no matter, what sort of project it is).
So, the problem is that I can't figure out what is my project's root directory (sorry, for being stupid), and how to make Firebase see this file. Just as an experiment, I copy-pasted the js file to each and every folder of my project, and still no success (Firebase service can't see the file).
Here's my settings.py file (with relevant content):
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ROOT_URLCONF = 'android_blend.urls'
WSGI_APPLICATION = 'android_blend.wsgi.application'
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))
STATIC_URL = '/static/'
if DEBUG:
MEDIA_URL = '/media/'
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR),"static","static-only")
#STATIC_ROOT = [os.path.join(BASE_DIR,"static-only")]
MEDIA_ROOT = os.path.join(os.path.dirname(BASE_DIR),"static","media")
#MEDIA_ROOT = [os.path.join(BASE_DIR,"media")]
STATICFILES_DIRS = (
#os.path.join(os.path.dirname(BASE_DIR),"static","static"),
os.path.join(BASE_DIR,"static"),
)
My Django project layout is like this:
android_blend
android_blend
settings.py
url.py
wsgi.py
app1
app2
...
appN
manage.py
So the question is, what should I do in order for an outside service app (Google Firebase) be able to see the javascript file ?
In my browser, I get the following error:
A bad HTTP response code (404) was received when fetching the script.
I was also facing the same problem.
I tried it in the following way:
Add the following line in urls.py
url(r'^firebase-messaging-sw.js', views.firebase_messaging_sw_js),
Now add the function in view.py file
#csrf_exempt
def firebase_messaging_sw_js(request):
filename = '/static/firebase-messaging-sw.js'
jsfile = open(absAppPath + filename, 'rb')
response = HttpResponse(content=jsfile)
response['Content-Type'] = 'text/javascript'
response['Content-Disposition'] = 'attachment; filename="%s"' % (absAppPath + filename)
return response
So localhost:8000/firebase-messaging-sw.js will work properly...
Hope this will help you...

'Suspicious Operation: Attempted access to "" denied' while loading static files

I've been through the ringer getting my Django app setup on Heroku using Amazon s3 to host the static and media files. I've been following this guide https://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/ and what seems like thousands of other resources, collectstatic has worked and heroku is deploying it - but displays a 400 error. When I try and run it locally i get more info:
Attempted access to '/css/reset.css' denied.
This is the line that gets highlighted:
<link rel="stylesheet" type="text/css" href="{% static '/css/reset.css' %}">
I can load the static files direct from the URL if I grab it from my s3 admin panel so I figured it wasn't a bucket policy issue, I've messed around with https / http options but no joy. So I figure it must be the wrong path is being called somehow in the code - i just can't see where!
Any help much appreciated, I don't think I've blinked for about 4 hours straight.
Traceback:
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/storages/backends/s3boto.py" in _normalize_name
358. return safe_join(self.location, name)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/storages/backends/s3boto.py" in safe_join
59. raise ValueError('the joined path is located outside of the base path'
During handling of the above exception (the joined path is located outside of the base path component), another exception occurred:
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
41. response = get_response(request)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/devtop/webdev/projects/intro/intro/profiles/views.py" in index
14. return render(request, 'home.html', {'welcome':welcome})
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/shortcuts.py" in render
30. content = loader.render_to_string(template_name, context, request, using=using)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/loader.py" in render_to_string
68. return template.render(context, request)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/backends/django.py" in render
66. return self.template.render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in render
207. return self._render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in _render
199. return self.nodelist.render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in render
990. bit = node.render_annotated(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in render_annotated
957. return self.render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/loader_tags.py" in render
177. return compiled_parent._render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in _render
199. return self.nodelist.render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in render
990. bit = node.render_annotated(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/template/base.py" in render_annotated
957. return self.render(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/templatetags/static.py" in render
105. url = self.url(context)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/templatetags/static.py" in url
102. return self.handle_simple(path)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/django/templatetags/static.py" in handle_simple
117. return staticfiles_storage.url(path)
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/storages/backends/s3boto.py" in url
487. name = self._normalize_name(self._clean_name(name))
File "/home/devtop/webdev/projects/intro/myvenv/lib/python3.5/site-packages/storages/backends/s3boto.py" in _normalize_name
361. name)
Exception Type: SuspiciousOperation at /
Exception Value: Attempted access to '/css/reset.css' denied.
settings.py
AWS_ACCESS_KEY_ID=os.environ.get('AWS_ACCESS_KEY_ID',None)
AWS_SECRET_KEY=os.environ.get('AWS_SECRET_KEY',None)
AWS_SECRET_ACCESS_KEY=os.environ.get('AWS_SECRET_KEY', None)
AWS_STORAGE_BUCKET_NAME = 'intro-story'
AWS_S3_HOST='s3.us-east-2.amazonaws.com'
AWS_S3_CUSTOM_DOMAIN = 's3.us-east-2.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
AWS_S3_SECURE_URLS = False
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
try:
from .local_settings import *
except ImportError:
pass
local_settings.py
AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "yyy"
custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
EDIT:
I managed to get it working by messing around with various values in the settings.py, but is still not right.
Here is all the stuff relating to static and media paths
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
import custom_storages
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
Here is custom_storage that I'm importing:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
Now, if i comment out
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
It will load the static files from S3 and all is fine, BUT collectstatic fails. If I uncomment that line, collectstatic works but it gives an error when I try and load the site. The error is:
# Ensure final_path starts with base_path and that the next character after
# the final path is '/' (or nothing, in which case final_path must be
# equal to base_path).
base_path_len = len(base_path)
if (not final_path.startswith(base_path) or
final_path[base_path_len:base_path_len + 1] not in ('', '/')):
raise ValueError('the joined path is located outside of the base path' ...
' component')
return final_path.lstrip('/')
So clearly something is up with that custom_storage part but I have no idea what :/
Sorted! As expected it was something annoyingly simple but perhaps this will help anyone else who hits the same wall.
This:
<link rel="stylesheet" type="text/css" href="{% static '/css/reset.css' %}">
Needed to be this:
<link rel="stylesheet" type="text/css" href="{% static 'css/reset.css' %}">
I.e. no leading slash in the href. Worked fine using local static files storage but broke the S3 link.

Mayan edms with s3

I am setting up the mayan edms with s3 according to 2nd message of Roberto Rosario on this link (https://groups.google.com/forum/#!topic/mayan-edms/tZjmn5u4y2A) but I am now having some errors.
Even though I added the s3 bucket settings on production.py, the s3.py from the storages/backends/ cannot load the settings.
The following is the error.
File "/home/proj/mayan-edms/lib/python2.7/site-packages/mayan/settings/production.py", line 42, in <module>
from storages.backends.s3 import S3Storage
File "/home/proj/mayan-edms/lib/python2.7/site-packages/storages/backends/s3.py", line 42, in <module>
class S3Storage(Storage):
File "/home/proj/mayan-edms/lib/python2.7/site-packages/storages/backends/s3.py", line 45, in S3Storage
def __init__(self, bucket=settings.AWS_STORAGE_BUCKET_NAME,
File "/home/proj/mayan-edms/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in __getattr__
return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'AWS_STORAGE_BUCKET_NAME'
And the settings on production.py looks like this.
from django.conf import settings
settings.INSTALLED_APPS += ('storages',)
AWS_ACCESS_KEY_ID = 'KEY_ID'
AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY'
AWS_STORAGE_BUCKET_NAME = 'BUCKET_NAME'
AWS_QUERYSTRING_AUTH = False
from storages.backends.s3 import S3Storage
DOCUMENTS_STORAGE_BACKEND=S3Storage
DEFAULT_FILE_STORAGE = 'storages.backends.s3.S3Storage'
Thanks in advance.
With a test install under virtualenv I was able to get storage to S3 working by adding the following to venv/lib/python2.7/site-packages/mayan/settings/local.py
INSTALLED_APPS += ('storages',)
DOCUMENTS_STORAGE_BACKEND = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = 'xxx'
AWS_SECRET_ACCESS_KEY = 'xxxxxxxx'
AWS_STORAGE_BUCKET_NAME = 'my-bucket-name'
AWS_S3_SIGNATURE_VERSION = 's3v4'
(I needed the AWS_S3_SIGNATURE_VERSION to use AWS4-HMAC-SHA256 authentication required at newer AWS regions like Frankfurt.)

How to setup django-compressor on heroku, offline compression to S3

I followed every QA suggestions found on SO and in different blogs, Everything works ok on my dev machine and nothing works on heroku.
here are my settings:
DEFAULT_FILE_STORAGE = 'arena.utils.MediaRootS3BotoStorage' # media files
# storage
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
AWS_PRELOAD_METADATA = True # necessary to fix manage.py collectstatic command to only upload changed files instead of all files
S3_URL = 'https://%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = S3_URL + '/media/'
STATIC_URL = S3_URL + '/static/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
COMPRESS_URL = STATIC_URL
COMPRESS_OFFLINE = True
COMPRESS_STORAGE = 'utils.CachedS3BotoStorage'
STATICFILES_STORAGE = COMPRESS_STORAGE
When i run collectstatic/compress everything is ok, i see the files being collected to S3 and put in proper places. I see the manifest file.
Loading any page with compressor tag, show an error OfflineGenerationError: You have offline compression enabled but key "d2a53169c44dec41ce3ee7da19b2b9d4" is missing from offline manifest. Running python manage.py compress again solves nothing. when i check the manifest file, indeed the key it looks for doesn't exist.
What is going wrong here?
Question i already checked:
How to configure django-compressor and django-staticfiles with Amazon's S3?
Django Compressor with S3 URL Heroku
Configuring django-compressor with remote storage (django-storage - amazon s3)
On my side I have very similar config, and I'm successfully using compressor for more than 2 years.
settings.py
COMPRESS_STORAGE = 'MyAwesomeApp.app.CachedS3BotoStorage.CachedS3BotoStorage'
AWS_ACCESS_KEY_ID = '#######'
AWS_SECRET_ACCESS_KEY = '########################+#########+BqoQ'
AWS_STORAGE_BUCKET_NAME = 'myAmazonS3cdn.myawesomewebsite.com'
AWS_S3_SECURE_URLS = False
AWS_QUERYSTRING_AUTH = False
COMPRESS_ROOT = 'MyAwesomeApp/static'
STATIC_ROOT = 'MyAwesomeApp/static/javascript'
COMPRESS_OUTPUT_DIR = 'compressed'
STATICFILES_STORAGE = COMPRESS_STORAGE
STATIC_URL = "http://myAmazonS3cdn.myawesomewebsite.com/"
COMPRESS_URL = STATIC_URL
COMPRESS_ENABLED = True
CachedS3BotoStorage.py
from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage
from django.core.files.base import File
class CachedS3BotoStorage(S3BotoStorage):
"""
S3 storage backend that saves the files locally, too.
"""
def __init__(self, *args, **kwargs):
super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
self.local_storage = get_storage_class("compressor.storage.CompressorFileStorage")()
def save(self, name, content):
name = super(CachedS3BotoStorage, self).save(name, content)
self.local_storage._save(name, content)
return name
I'm running python managep.py compress locally, and having manifest generated on my static files directory. Heroku only deals with the collecstatic and delivers the most recent manifest version to my cdn.
Regards,
I completed the above solution with some lines, to fix the problem that create many (multiples) manifest_%.json in Amazon S3
in setting.py:
STATICFILES_STORAGE = 'your_package.s3utils.CachedS3BotoStorage'
in s3utils.py:
from storages.backends.s3boto import S3BotoStorage
from django.core.files.storage import get_storage_class
class CachedS3BotoStorage(S3BotoStorage):
"""
S3 storage backend that saves the files locally, too.
"""
location = 'static'
def __init__(self, *args, **kwargs):
super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
self.local_storage = get_storage_class(
"compressor.storage.CompressorFileStorage")()
def url(self, name):
"""
Fix problem images admin Django S3 images
"""
url = super(CachedS3BotoStorage, self).url(name)
if name.endswith('/') and not url.endswith('/'):
url += '/'
return url
def save(self, name, content):
name = super(CachedS3BotoStorage, self).save(name, content)
self.local_storage._save(name, content)
return name
# HERE is secret to dont generating multiple manifest.json and to delete manifest.json in Amazon S3
def get_available_name(self, name):
if self.exists(name):
self.delete(name)
return name
I found a git repository that contains post_compile hooks to solve this problem. It runs compress after Heroku built the Django app (and also installs lessc if you need less in your compressor settings).
https://github.com/nigma/heroku-django-cookbook