I developed a Django app that I'm using VM's disk for saving and serving media and static files but in one of my models, I want to save my files in a FileField connected to my MinIO object storage. I set up the settings like this in the settings.py
AWS_ACCESS_KEY_ID = '###'
AWS_SECRET_ACCESS_KEY = '###'
AWS_S3_ENDPOINT_URL = '###'
and in my model I used S3Storage like this:
class CustomStorageBucket(S3Boto3Storage):
bucket_name = "files"
class Document(BaseModel):
document_file = models.ImageField(storage=CustomStorageBucket(),upload_to='documents')
with these codes, I can save my files into the storage but the URLs in the admin panel do not works properly because it points to the media files URL something like this :
http://localhost:8000/media/documents/file.jpg
but I want it to be like this ( presigned URL) :
https://object-storage.app/files/documents/file.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXX&X-Amz-Date=XXX&X-Amz-Expires=432000&X-Amz-SignedHeaders=host&X-Amz-Signature=XXX
Try to set MEDIA_URL variable
MEDIA_URL = 'https://object-storage.app/files/'
Related
I tried moving from local static files to S3 using django-storages. I followed the documentation carefully but still there is no access to the static files.
In the local environment I have:
STATIC_URL = '/static/'
in the settings.py and everything works fine.
when I add all the S3 params as the documentation shows:
STATIC_URL = 'https://django-main.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = 'https://django-main.s3.amazonaws.com/admin/' # tried with this and also without this
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
AWS_ACCESS_KEY_ID = '<AWS_ACCESS_KEY_ID>'
AWS_SECRET_ACCESS_KEY = '<AWS_SECRET_ACCESS_KEY>'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
I ran python manage.py collectstatic which seemed to work fine and uploaded the static files to the bucket.
but running the server and going to the admin page it looks like this:
which is because it doesn't have access to the static files. No error is thrown/shown
Any ideas?
EDIT:
So apperently I'm getting a forbbiden call:
but I changed my settings.py to:
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
and made sure that access is available (put a breakpoint and downloaded a css file using boto3 from that bucket using these exact environment variables and still no solution
The issue was that the bucket read permissions were not public. Changing the permissions worked
I have created a backend django app using AWS Beanstalk, and a frontend reactjs app deployed using cloudfront (plus S3)
I have a model in backend that does
class EnhancedUser(AbstractUser):
# some other attributes
picture = models.ImageField(blank=True)
my settings.py has
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '<my_elastic_beanstalk_domain>/media/'
Since I'm using cloudfront, if i just set the MEDIA_URL to /media/, it would just append /media/ to my cloudfront url, so I have to hardcode it to my backend url
and then, following the django docs, I added the static part to my urls.py
urlpatterns = [
path('admin/', admin.site.urls),
# some other urls
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Note that django doc does mention we can't use absolute url for MEDIA_URL, but I have no alternative solution at the moment
When I upload my image, it doesn't get stored in the right place, but I cannot open it with the url. It returns a 404 saying the img's url is not part of urls list
My question is:
How do I set it up so I can display the image
Since the images will be updated through users/admins, these will be stored in the EC2 instance created in beanstalk, so every time I deploy, I think they will be wiped. How do I prevent this?
Take a look at using django-storages to save your uploads. I use S3 for storing uploads of a django/docker/EB deployment, and include django settings that look something like this (I keep them in settings/deployment.py):
if 'AWS_ACCESS_KEY_ID' in os.environ:
# Use Amazon S3 for storage for uploaded media files
# Keep them private by default
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# Amazon S3 settings.
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
AWS_STORAGE_BUCKET_NAME = os.environ["AWS_STORAGE_BUCKET_NAME"]
AWS_S3_REGION_NAME = os.environ.get("AWS_S3_REGION_NAME", None)
AWS_S3_SIGNATURE_VERSION = 's3v4'
AWS_AUTO_CREATE_BUCKET = False
AWS_HEADERS = {"Cache-Control": "public, max-age=86400"}
AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = 'private'
AWS_QUERYSTING_AUTH = True
AWS_QUERYSTRING_EXPIRE = 600
AWS_S3_SECURE_URLS = True
AWS_REDUCED_REDUNDANCY = False
AWS_IS_GZIPPED = False
MEDIA_ROOT = '/'
MEDIA_URL = 'https://s3.{}.amazonaws.com/{}/'.format(
AWS_S3_REGION_NAME, AWS_STORAGE_BUCKET_NAME)
USING_AWS = True
I have an droplet on digitalocean which for the most part is working great, however i have enabled the cdn option in digitalocean spaces (similar to aws s3 storage) and in trying to load static files using the cdn url, i cannot seem to get the url working correctly so suspect i am missing the obvious?
For example when i change the STATIC_URL in settings to the cdn no change is seen in the web page source?
If i change the AWS_S3_ENDPOINT_URL and MEDIA_ENDPOINT_URL then the source does change but the files are not found and as one can guess a collectstatic no longer works, So i assume that the AWS_S3_ENDPOINT_URL & MEDIA_ENDPOINT_URL need to remain as is and i merely need to ensure the static_url is used?
I did read somewhere that it was not good practice to change the templates from {%static.... to {%static_url so have not done that, is this something i should update or not?
Settings:
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
MEDIA_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com/media/'
AWS_STORAGE_BUCKET_NAME = 'mysitestaging'
#STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# if False it will create unique file names for every uploaded file
AWS_S3_FILE_OVERWRITE = False
STATICFILES_STORAGE = 'mysite.settings.storage_backends.StaticStorage'
DEFAULT_FILE_STORAGE = 'mysite.settings.storage_backends.MediaStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
# the sub-directories of media and static files
STATIC_ROOT = 'static'
MEDIA_ROOT = 'media'
AWS_DEFAULT_ACL = 'public-read'
BUCKET_ROOT = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, STATIC_ROOT)
# the regular Django file settings but with the custom S3 URLs
STATIC_URL = '{}/{}/'.format('https://cdn.mysite.com', STATIC_ROOT)
MEDIA_URL = '{}/{}/'.format('https://cdn.mysite.com', MEDIA_ROOT)
Source view returns:
https://nyc3.digitaloceanspaces.com/mysitestaging/static/img/apple-touch-icon.png?AWSAccessKeyId=37FLLPUJLEUO5IG7R4GQ&Signature=eof5%2BZvHPo%2FRSzvKQsrobXkcOZ0%3D&Expires=1586789962
cdn.mysite.com
is an alias of
mysitestaging.nyc3.cdn.digitaloceanspaces.com.
My storage_backends.py:
import os
from storages.backends.s3boto3 import S3Boto3Storage
from my_site.settings import core_settings
class StaticStorage(S3Boto3Storage):
location = core_settings.STATIC_ROOT
class MediaStorage(S3Boto3Storage):
location = core_settings.MEDIA_ROOT
Ok figured this out by actually re-reading the docs:
https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
Adding the following worked immediately:
AWS_S3_CUSTOM_DOMAIN = 'cdn.mysite.com'
Hope it helps someone else.
I'm using Azure Storage account for storing my media files
Setting it is simple in settings.py like this:
DEFAULT_FILE_STORAGE = 'storages.backends.azure_storage.AzureStorage'
AZURE_ACCOUNT_NAME = 'my_account_name'
AZURE_ACCOUNT_KEY = 'my_account_key'
AZURE_CONTAINER = 'my-container'
However, I considered later that I want to use Azure CDN instead for serving my media files. How will I point it to the CDN URL instead? I tried setting it in the MEDIA_URL like
MEDIA_ROOT = os.path.join(BASE_DIR, 'upload')
MEDIA_URL = '//my-media.azureedge.net/my-container/'
However my storage-account blob is the one being shown as default URL which is 'xxxxx.blob.core.windows.net' instead of my MEDIA URL..
How will I show the MEDIA_URL instead of 'xxxxx.blob.core.windows.net'?
Thankfully I already have an answer for this one. You need to override the storage backend class of Azure on the 'storages' library replacing the blob hostname with the CDN hostname.
settings.py
MEDIA_URL = '//my-media.azureedge.net/my-container/'
storages.py
import re
from jaguar import settings
from storages.backends.azure_storage import AzureStorage
class AzureCDNURL(AzureStorage):
def url(self, name):
ret = super(AzureCDNURL, self).url(name)
_ret = re.sub('//[a-z.0-9A-Z]*/', settings.MEDIA_URL, ret)
return _ret
I use django-cleanup, sorl-thumbnail in my Django project.
I have a model like this:
from sorl.thumbnail import ImageField
class Variation(BaseModel):
image = ImageField(upload_to=draft_img_upload_location)
And I use signal for sorl-thumbnail like this (recommanded by https://github.com/un1t/django-cleanup):
from django_cleanup.signals import cleanup_pre_delete
def sorl_delete(**kwargs):
from sorl.thumbnail import delete
delete(kwargs['file'])
cleanup_pre_delete.connect(sorl_delete)
So, In local environment, belows work:
1. When I delete Variation model in ADMIN PAGE, it deletes BOTH image file and image cache(created by sorl-thumbnail).
2. When I change just image file with other image in ADMIN PAGE, it delete BOTH 'prior image file' and 'prior image cache file(created by sorl-thumbnail)'.
In production environment, I used AWS S3 and cloudfront for managing static and media file. So all sorl-thumbnail cache image files are stored in S3. But whenever I changed the image file with other image file in ADMIN PAGE, the prior image cache file(created by sorl-thumbnail) still remained.
Lets say that sorl-thumbnail image url is https://example.cloudfront.net/cache/da/75/abcdefg_i_am_cached_image.jpg (from Google development tool).
In this case, there were two image files exist in S3: abcdefg.jpg and /da/75/abcdefg_i_am_cached_image.jpg
I changed abcdefg.jpg with the other image. Then, it completely deleted abcdefg.jpg in S3 storage.
Now, I accessed https://example.cloudfront.net/cache/da/75/abcdefg_i_am_cached_image.jpg in my browser and guess what! It showed this sorl-thumbnail cached images in this time!
Strange thing happened in S3 storage. When I tried to check whether abcdefg_i_am_cached_image.jpg exists in path /da/75, there was no directory named 75 right below the da folder!
In short, abcdefg_i_am_cached_image.jpg still remained in my S3 storage!
I don't know why this happened only in production environment...
This is part of settings only for production mode.
settings.py
from .partials import *
DEBUG = False
ALLOWED_HOSTS = ["*", ]
STATICFILES_STORAGE = 'spacegraphy.storage.S3PipelineCachedStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY")
AWS_STORAGE_BUCKET_NAME = os.environ.get("AWS_STORAGE_BUCKET_NAME")
AWS_S3_CUSTOM_DOMAIN = os.environ.get("AWS_S3_CUSTOM_DOMAIN")
AWS_S3_URL_PROTOCOL = 'https'
AWS_S3_HOST = 's3-ap-northeast-1.amazonaws.com'
STATIC_URL = "https://this-is-my-example.cloudfront.net/"
INSTALLED_APPS += [
'storages',
'collectfast'
]
AWS_PRELOAD_METADATA = True
storage.py
from django.contrib.staticfiles.storage import CachedFilesMixin, ManifestFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage
class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
pass
class S3PipelineCachedStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
pass
After spending couple of hours debugging, I found out that sorl_delete signal is not called only in production environment!!.
I have no idea why this happened. I think that this one is a main problem.
And sorry for bad english (I'm not native). Really need your help. Thanks