django s3 storages does not recognise the bucket name - django

I am using django-s3-storage==0.11.2 and boto3==1.4.4. These are in the settings.py:
STATIC_URL = '/static/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, "static"),
]
STATIC_ROOT = os.path.join(BASE_DIR, 'static_cdn')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media_cdn')
AWS_S3_BUCKET_NAME = "my-bucket-name"
AWS_ACCESS_KEY_ID = 'test_id_x'
AWS_SECRET_ACCESS_KEY = 'test_id_x+test_id_x'
DEFAULT_FILE_STORAGE = "django_s3_storage.storage.S3Storage"
STATICFILES_STORAGE = "django_s3_storage.storage.StaticS3Storage"
AWS_S3_ADDRESSING_STYLE = "auto"
AWS_S3_BUCKET_AUTH_STATIC = False
AWS_S3_MAX_AGE_SECONDS_STATIC = 60 * 60 * 24 * 365 # 1 year.
AWS_S3_BUCKET_AUTH = False
AWS_S3_MAX_AGE_SECONDS = 60 * 60 * 24 * 365 # 1 year.
I have also ran these command:
manage.py s3_sync_meta django.core.files.storage.default_storage
But when I run collectstatic or this command
manage.py s3_sync_meta django.contrib.staticfiles.storage.staticfiles_storage
I get this error:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"
I have already created the bucket, and the bucket name is correct. Because this works and does not gives any error:
s3.meta.client.head_bucket(Bucket='my-bucket-name')
I don't know what am I missing here? Could you help me out please.

Alright, it looks confusing for me too.
Below are my observations -
1 . Bucket name pattern
Bucket name should not have '/' in them .
It would be good if you can update the AWS_S3_BUCKET_NAME from
"my-bucket-name" to the pattern which actually resembles with your
bucket name.
Source: https://github.com/boto/botocore/issues/680
2 . In the Django S3 Storage Documentation , it says
If your are updating a project that used django-storages
just for S3 file storage, migration is trivial.
Follow the installation instructions, replacing 'storages' in INSTALLED_APPS.
Be sure to scrutinize the rest of your settings file for changes,
most notably AWS_S3_BUCKET_NAME for AWS_STORAGE_BUCKET_NAME.
Can you please try to change AWS_S3_BUCKET_NAME_STATIC = bass-line-shop in your settings.py ?
Let me know, if it helps!

Related

Found another file with the destination path 'admin'

I am trying to deploy the app on Heroku but when I run the following command: "python manage.py collectstatic" it returns multiple "Found another file with the destination path 'admin...".
When I "git push heroku" the app, I can see the same error.
Next, the application works, but /admin page doesn't have any format, like it is not reading css, font, etc. I've checked staticfiles/admin and it has its css,font,img and js folder with files.
I've tried all the advices given here, but I'm not finding a solution. Here my settings (see my comments at the end of the code):
import os
import django_heroku
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
.
.
.
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.0/howto/static-files/
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles') #Heroku
#STATIC_ROOT = os.path.join(BASE_DIR, 'static') #ok for IBM Cloud
STATIC_URL = '/static/'
MEDIA_ROOT = os.path.join(STATIC_ROOT, 'media') # if I comment this line and
MEDIA_URL = '/media/' # this line, /admin page has correct css but I don't have images in the application
# Extra places for collectstatic to find static files.(extra heroku)
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'static'),
)
# Next line makes application not work, it shows database errors
django_heroku.settings(locals())
"""
so, I added "release: python manage.py migrate" to procfile, now the app works,
/admin shows format ok, but database seems to be empty, no information is displayed on
site pages, and admin users don't exist anymore (my database is sqlite3).
"""
And what I get when "git push heroku master" is:
remote: Found another file with the destination path 'admin/img/selector-icons.svg'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.
remote: Found another file with the destination path 'admin/img/calendar-icons.svg'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.
.
.
.
Please help

Django-compressor has a JS cache folder that is using an absurd amount of space

I woke up this morning to alerts from sentry that my production server has completely run out of space. It took some time figuring out the cause via ncdu, and the results were that my static folder had used over 60GB of space, specifically, CACHE/js that django-compressor is using.
I am not completely sure what is happening, or why there are over 500,000 js files where each file is following this format: output.<random string>.js. From my understanding, shouldn't there only be a small number of js files cached? My project doesn't even have that many scripts!
It seems to me that every user is getting their own output file, instead of the same cached files being shared to everyone.
Base settings:
# STATIC
# ------------------------------------------------------------------------------
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
# STATIC_ROOT = str(ROOT_DIR / "static")
STATIC_ROOT = os.path.join(BASE_DIR, "static/")
# https://docs.djangoproject.com/en/dev/ref/settings/#static-url
STATIC_URL = "/static/"
# https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
# STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]
# https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#staticfiles-finders
STATICFILES_FINDERS = [
"django.contrib.staticfiles.finders.FileSystemFinder",
"django.contrib.staticfiles.finders.AppDirectoriesFinder",
"compressor.finders.CompressorFinder",
]
COMPRESS_ENABLED = True
COMPRESS_PRECOMPILERS = (
('text/x-scss', 'django_libsass.SassCompiler'),
)
COMPRESS_FILTERS = {
"css": [
'compressor.filters.css_default.CssAbsoluteFilter',
# 'compressor.filters.cssmin.CSSMinFilter',
'core.CSSMinFilter.CSSMinFilter',
]
}
production settings:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
'LOCATION': '127.0.0.1:11211',
}
}
I originally installed django-compressor (https://django-compressor.readthedocs.io/en/stable/quickstart/) to fix issues where users would have to manually clear their browser cache to see correct changes to templates.
Any ideas on what is going on, or how to fix this?
edit:
I ran the following command rm -rf js to delete all the files, and within like a minute there's already 200 files created.

Django app on Azure incorrectly loading static files from an Azure Blob

This Django app is deployed on Azure as an App Service its static and media files are stored in an Azure storage account - blob.
The project used to work well in the past, but something has changed and now the problem is as following.
Relevant part of the app settings file:
STATIC_URL = 'https://myappstorage.blob.core.windows.net/static/'
MEDIA_URL = 'https://myappstorage.blob.core.windows.net/media/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
# any static paths you want to publish
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static')
]
so one would expect that for example a favicon, which is in the root directory of the storage could be found on https://myappstorage.blob.core.windows.net/static/favicon and it is indeed!
But all static files that the app on azure tries to load it tries to load from
https://myappstorage.myappstorage.blob.core.windows.net/static/
(note the duplication of myappstorage), same for media files.
This results in no static files being applied to the page as they are being loaded from the wrong url. When I run the app locally, it works fine. I destroyed it and recreated it, no success. Now I have two copies running, one deployed through FTP and startup command and on using Github action. Still the same problem.
I have also tried little hack-ish workarounds in the setting file, with no success.
Try with adding below in app setting
DEFAULT_FILE_STORAGE = 'backend.custom_azure.AzureMediaStorage'
STATICFILES_STORAGE = 'backend.custom_azure.AzureStaticStorage'
STATIC_LOCATION = "static"
MEDIA_LOCATION = "media"
AZURE_ACCOUNT_NAME = "djangoaccountstorage"
AZURE_CUSTOM_DOMAIN = f'{AZURE_ACCOUNT_NAME}.blob.core.windows.net'
STATIC_URL = f'https://{AZURE_CUSTOM_DOMAIN}/{STATIC_LOCATION }/'
MEDIA_URL = f'https://{AZURE_CUSTOM_DOMAIN}/{MEDIA_LOCATION}
refer this document for more details

How can i solve CertificateError by AWS -S3 in Django?

Here is the application hosted on heroku + aws s3 used for storing only media files.
When user upload's a images the error occures .
CertificateError at /create/
hostname 'shuboy.media.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'
Request Method: POST
Request URL: http://shuboy2015.herokuapp.com/create/
Django Version: 1.9.6
Exception Type: CertificateError
Exception Value:
hostname 'shuboy.media.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'
Exception Location: /app/.heroku/python/lib/python2.7/ssl.py in match_hostname, line 271
Python Executable: /app/.heroku/python/bin/python
Python Version: 2.7.10
Python Path:
['/app',
'/app/.heroku/python/bin',
'/app/.heroku/python/lib/python2.7/site-packages/setuptools-20.4-py2.7.egg',
'/app/.heroku/python/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg',
'/app',
'/app/.heroku/python/lib/python27.zip',
'/app/.heroku/python/lib/python2.7',
'/app/.heroku/python/lib/python2.7/plat-linux2',
'/app/.heroku/python/lib/python2.7/lib-tk',
'/app/.heroku/python/lib/python2.7/lib-old',
'/app/.heroku/python/lib/python2.7/lib-dynload',
'/app/.heroku/python/lib/python2.7/site-packages']
Server time: Tue, 7 Jun 2016 20:54:31 +0000
In settings.py file , I only want to store media files there no static files.
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_S3_SECURE_URLS = False
AWS_QUERYSTRING_AUTH = False
AWS_S3_ACCESS_KEY_ID = '*******'
AWS_S3_SECRET_ACCESS_KEY = '******'
AWS_STORAGE_BUCKET_NAME ='shuboy.media'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = '/static/'
MEDIA_URL = 'https://%s/media/' % AWS_S3_CUSTOM_DOMAIN
STATICFILES_DIRS = [
os.path.join(BASE_DIR, "appname/static"),
]
STATIC_ROOT = os.path.join(BASE_DIR, "STATIC_CDN")
MEDIA_ROOT = '%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
If anybody tell me why this error is occurring and how can I solve this issue ? Any further suggestions will be appreciable .
The above error is happening because your bucket name has a "." in it.
The work around for this issue is go to ~/.boto configuration file search for https_validate_certificates and set it to False it will work.
Or remove the "." from your bucket name for it to work correctly.
The boto package doesn't fully support S3 buckets with dots in the name, because of SSL certificate scope.
Either create and use a new bucket without a dot, like "shuboymedia", or add this monkey patch in your settings:
import ssl
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context

Django Admin CSS missing

I've been messing around with the new collectstatic command and have got it working for my normal pages. That is to say, I am able to load my css at this location http://localhost:8000/static/css/main.css. However, the css for my django admin doesn't seem to be showing up.
When I navigate to the admin css location at http://localhost:8000/static/admin/css/base.css, I'm getting a 404 page not found with the following error: /home/nai/GitProjects/cats/django-trunk/django/contrib/admin/media/css/base.css" does not exist. Looking in django-trunk, I never had the /home/nai/GitProjects/cats/django-trunk/django/contrib/admin/media/ folder to begin with.
Is that weird?
In any case, in my static folder, there is an admin folder with the accompanying css, img and js folders which was created when I ran collectstatic and the url of the base.css seems to be pointing to that location.
This is happening on my django development server. Here are some snippets to aid in the bug hunt:
urls
33 # In order for Dev Server to serve media files for the frontend site.
34 urlpatterns += staticfiles_urlpatterns()
35
36 try:
37 if settings.DEBUG: # defined in manage.py when the first arg is "runserver"
38 urlpatterns += patterns('',
39 (r'^media/(?P<path>.*)$', 'django.views.static.serve',{'document_root': settings.MEDIA_ROOT}),
40 (r'^media-admin/(?P<path>.*)$', 'django.views.static.serve',{'document_root': os.path.join(settings.MEDIA_ROOT, '..', settings.ADMIN_MEDIA_PREFIX)}),
41 )
42 except NameError:
43 pass
I think it might be something to do with line 40 in my URLS file but changing media-admin to static/admin didnt help.
settings
58 ADMIN_MEDIA_PREFIX = '/static/admin'
69 STATIC_ROOT = os.path.join(os.path.abspath(os.path.join(PROJECT_ROOT, '..', MEDIA_DIR, 'static')), '')
70
71 # URL prefix for static files.
72 # Example: "http://media.lawrence.com/static/"
73 STATIC_URL = '/static/'
74
75 # Additional locations of static files. Global files are stored in here
76 STATICFILES_DIRS = (
77 os.path.join(os.path.abspath(os.path.join(PROJECT_ROOT, '..', 'proj_public', 'static', 'proj')), ''),
78 )
79
Django recommends that you deploy static files with a web server other than wsgi.
In settings.py, set:
STATIC_ROOT = 'static'
Run python manage.py collectstatic, which will copy the Django admin static files to /path/to/project/static/
Configure your static file server. If you use Nginx, you could add this config:
location /static/ {
alias /path/to/project/static/;
expires modified +1w;
}
Reload your web server
You should now have access to the static files.
In Django 1.4 ADMIN_MEDIA_PREFIX is deprecated. Here are the steps I followed to catch up with these somewhat recent Django changes:
in settings.py, add django.contrib.staticfiles to INSTALLED_APPS
in settings.py define STATIC_URL — the staticfiles app won't run without it. While using runserver they'll get handled magically, but when you deploy, this needs to be a location where those resources can be fetched by a browser.
I think that's all there was to it.
I'm using Django 1.4.3
What did NOT work for me:
No matter how much I edited ADMIN_MEDIA_PREFIX in settings.py I noticed no change in the HTML generated for the Django Admin pages. It always says /media/admin/base.css when I view the source.
What DID work for me.
Copied the 'admin' folder from /django/contrib/admin/static/ and pasted it into my projects 'media' folder
Now it works great.
It seems dumb, but I actually had this exact issue and the solution was to set DEBUG=False to DEBUG=True on my local dev environment. When debug is set to False, it thinks it's in a production environment which relies on a place to put static files, such as /var/www/html/static, whereas the debug set to True just uses the local directory.
Also make sure that AppDirectoriesFinder is not commented, happens when you're trying to customize your own app structure. Unfortunatelly it's pointless to seek such information in official docs.
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]
just change one thing in settings.py
DEBUG = False # not working if you are not serve your static files for production.
just change
DEBUG = True
You need a trailing slash in your ADMIN_MEDIA_PREFIX setting.
Change to:
ADMIN_MEDIA_PREFIX = '/static/admin/'
I'm using chef to auto-build my django server on an AWS Ubuntu server. This post helped, but what I wound up doing was to add the directory to the package admin static pages in a local_setings.py:
https://github.com/jaycrossler/geoq-chef-repo/blob/master/cookbooks/geoq/templates/default/local_settings.py.erb#L16
(added to local_settings.py or to settings.py):
STATICFILES_DIRS = ('<%= node['geoq']['virtualenv']['location'] %>/local/lib/python2.7/site-packages/django/contrib/admin/static/',)
This resulted in local_settings.py having:
STATICFILES_DIRS = ('/var/lib/geoq/local/lib/python2.7/site-packages/django/contrib/admin/static/',)
Note, that if you have other items already in your STATICFILES_DIRS, you might want to append to the list, rather than overwriting it.
in my project, the solution is in settings.py, set:
DEBUG = False # debug false mod not working css
One solution might be to add your local IP to the ALLOWED_HOSTS list in settings.py
e.g.
CURRENT_IP = '192.168.0.123'
ALLOWED_HOSTS = ['127.0.0.1', 'localhost', '0.0.0.0', CURRENT_IP]
you need
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
I don't know how it was resolved, but I took these steps when I encountered the same problem. I simply add these line in settings.py.
ADMIN_MEDIA_PREFIX = '/static/admin/'
STATICFILES_FINDERS = ( 'django.contrib.staticfiles.finders.FileSystemFinder','django.contrib.staticfiles.finders.AppDirectoriesFinder','compressor.finders.CompressorFinder', )
i hope this will help you all.
In settings.py
Don't use tuple for the
STATICFILES_DIRS =(
os.path.join(BASE_DIR, 'static'),
)
you should use list,like this
STATICFILES_DIRS =[
os.path.join(BASE_DIR, 'static'),
]