Override the DEFAULT_FILE_STORAGE with MEDIA_URL to integrate Azure CDN - django

I'm using Azure Storage account for storing my media files
Setting it is simple in settings.py like this:
DEFAULT_FILE_STORAGE = 'storages.backends.azure_storage.AzureStorage'
AZURE_ACCOUNT_NAME = 'my_account_name'
AZURE_ACCOUNT_KEY = 'my_account_key'
AZURE_CONTAINER = 'my-container'
However, I considered later that I want to use Azure CDN instead for serving my media files. How will I point it to the CDN URL instead? I tried setting it in the MEDIA_URL like
MEDIA_ROOT = os.path.join(BASE_DIR, 'upload')
MEDIA_URL = '//my-media.azureedge.net/my-container/'
However my storage-account blob is the one being shown as default URL which is 'xxxxx.blob.core.windows.net' instead of my MEDIA URL..
How will I show the MEDIA_URL instead of 'xxxxx.blob.core.windows.net'?

Thankfully I already have an answer for this one. You need to override the storage backend class of Azure on the 'storages' library replacing the blob hostname with the CDN hostname.
settings.py
MEDIA_URL = '//my-media.azureedge.net/my-container/'
storages.py
import re
from jaguar import settings
from storages.backends.azure_storage import AzureStorage
class AzureCDNURL(AzureStorage):
def url(self, name):
ret = super(AzureCDNURL, self).url(name)
_ret = re.sub('//[a-z.0-9A-Z]*/', settings.MEDIA_URL, ret)
return _ret

Related

Static files - Django Azure

I am hosting a website on Microsoft Azure. I have updated all the configurations within Azure and my settings.py file looks as below -
DEFAULT_FILE_STORAGE = 'storages.backends.azure_storage.AzureStorage'
STATICFILES_STORAGE = 'custom_storage.custom_azure.AzureStaticStorage'
AZURE_ACCOUNT_NAME = os.environ.get('AZURE_ACCOUNT_NAME')
AZURE_STORAGE_KEY = os.environ.get('AZURE_STORAGE_KEY', False)
AZURE_MEDIA_CONTAINER = os.environ.get('AZURE_MEDIA_CONTAINER', 'media')
AZURE_STATIC_CONTAINER = os.environ.get('AZURE_STATIC_CONTAINER', 'static')
AZURE_CUSTOM_DOMAIN = f'{AZURE_ACCOUNT_NAME}.blob.core.windows.net/'
STATIC_URL = f'https://{AZURE_CUSTOM_DOMAIN}/{AZURE_STATIC_CONTAINER}/'
MEDIA_URL = f'https://{AZURE_CUSTOM_DOMAIN}/{AZURE_MEDIA_CONTAINER}/'
STATIC_ROOT = f'https://{AZURE_CUSTOM_DOMAIN}/{AZURE_STATIC_CONTAINER}/'
I have a custom_storage folder with a custom_azure.py file with following -
from django.conf import settings
from storages.backends.azure_storage import AzureStorage
class AzureMediaStorage(AzureStorage):
account_name = settings.AZURE_ACCOUNT_NAME
account_key = settings.AZURE_STORAGE_KEY
azure_container = settings.AZURE_MEDIA_CONTAINER
expiration_secs = None
class AzureStaticStorage(AzureStorage):
account_name = settings.AZURE_ACCOUNT_NAME
account_key = settings.AZURE_STORAGE_KEY
azure_container = settings.AZURE_STATIC_CONTAINER
expiration_secs = None
However I am now pulling a static admin file and not the static project file containing css, scss and js files..
As a work around the the install of whitenoise enables the static files when hosting on Azure. However does not solve the problem of storing the static files on the Azure platform.

How to make django uploaded images to display in CloudFront frontend + Beanstalk Backend

I have created a backend django app using AWS Beanstalk, and a frontend reactjs app deployed using cloudfront (plus S3)
I have a model in backend that does
class EnhancedUser(AbstractUser):
# some other attributes
picture = models.ImageField(blank=True)
my settings.py has
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '<my_elastic_beanstalk_domain>/media/'
Since I'm using cloudfront, if i just set the MEDIA_URL to /media/, it would just append /media/ to my cloudfront url, so I have to hardcode it to my backend url
and then, following the django docs, I added the static part to my urls.py
urlpatterns = [
path('admin/', admin.site.urls),
# some other urls
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Note that django doc does mention we can't use absolute url for MEDIA_URL, but I have no alternative solution at the moment
When I upload my image, it doesn't get stored in the right place, but I cannot open it with the url. It returns a 404 saying the img's url is not part of urls list
My question is:
How do I set it up so I can display the image
Since the images will be updated through users/admins, these will be stored in the EC2 instance created in beanstalk, so every time I deploy, I think they will be wiped. How do I prevent this?
Take a look at using django-storages to save your uploads. I use S3 for storing uploads of a django/docker/EB deployment, and include django settings that look something like this (I keep them in settings/deployment.py):
if 'AWS_ACCESS_KEY_ID' in os.environ:
# Use Amazon S3 for storage for uploaded media files
# Keep them private by default
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# Amazon S3 settings.
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
AWS_STORAGE_BUCKET_NAME = os.environ["AWS_STORAGE_BUCKET_NAME"]
AWS_S3_REGION_NAME = os.environ.get("AWS_S3_REGION_NAME", None)
AWS_S3_SIGNATURE_VERSION = 's3v4'
AWS_AUTO_CREATE_BUCKET = False
AWS_HEADERS = {"Cache-Control": "public, max-age=86400"}
AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = 'private'
AWS_QUERYSTING_AUTH = True
AWS_QUERYSTRING_EXPIRE = 600
AWS_S3_SECURE_URLS = True
AWS_REDUCED_REDUNDANCY = False
AWS_IS_GZIPPED = False
MEDIA_ROOT = '/'
MEDIA_URL = 'https://s3.{}.amazonaws.com/{}/'.format(
AWS_S3_REGION_NAME, AWS_STORAGE_BUCKET_NAME)
USING_AWS = True

Django static url with digitalocean spaces

I have an droplet on digitalocean which for the most part is working great, however i have enabled the cdn option in digitalocean spaces (similar to aws s3 storage) and in trying to load static files using the cdn url, i cannot seem to get the url working correctly so suspect i am missing the obvious?
For example when i change the STATIC_URL in settings to the cdn no change is seen in the web page source?
If i change the AWS_S3_ENDPOINT_URL and MEDIA_ENDPOINT_URL then the source does change but the files are not found and as one can guess a collectstatic no longer works, So i assume that the AWS_S3_ENDPOINT_URL & MEDIA_ENDPOINT_URL need to remain as is and i merely need to ensure the static_url is used?
I did read somewhere that it was not good practice to change the templates from {%static.... to {%static_url so have not done that, is this something i should update or not?
Settings:
AWS_S3_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com'
MEDIA_ENDPOINT_URL = 'https://nyc3.digitaloceanspaces.com/media/'
AWS_STORAGE_BUCKET_NAME = 'mysitestaging'
#STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# if False it will create unique file names for every uploaded file
AWS_S3_FILE_OVERWRITE = False
STATICFILES_STORAGE = 'mysite.settings.storage_backends.StaticStorage'
DEFAULT_FILE_STORAGE = 'mysite.settings.storage_backends.MediaStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
# the sub-directories of media and static files
STATIC_ROOT = 'static'
MEDIA_ROOT = 'media'
AWS_DEFAULT_ACL = 'public-read'
BUCKET_ROOT = '{}/{}/'.format(AWS_S3_ENDPOINT_URL, STATIC_ROOT)
# the regular Django file settings but with the custom S3 URLs
STATIC_URL = '{}/{}/'.format('https://cdn.mysite.com', STATIC_ROOT)
MEDIA_URL = '{}/{}/'.format('https://cdn.mysite.com', MEDIA_ROOT)
Source view returns:
https://nyc3.digitaloceanspaces.com/mysitestaging/static/img/apple-touch-icon.png?AWSAccessKeyId=37FLLPUJLEUO5IG7R4GQ&Signature=eof5%2BZvHPo%2FRSzvKQsrobXkcOZ0%3D&Expires=1586789962
cdn.mysite.com
is an alias of
mysitestaging.nyc3.cdn.digitaloceanspaces.com.
My storage_backends.py:
import os
from storages.backends.s3boto3 import S3Boto3Storage
from my_site.settings import core_settings
class StaticStorage(S3Boto3Storage):
location = core_settings.STATIC_ROOT
class MediaStorage(S3Boto3Storage):
location = core_settings.MEDIA_ROOT
Ok figured this out by actually re-reading the docs:
https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
Adding the following worked immediately:
AWS_S3_CUSTOM_DOMAIN = 'cdn.mysite.com'
Hope it helps someone else.

Django model FileField is set to "null" instead of url to the file when using Google Cloud Storage

I am running a File manager app on Local machine using Google Cloud Sql Proxy and storing the files in a Google Cloud Storage bucket.
The file is being saved in the bucket, but the FileField is set to "null". I want it to show the url by which I can access the file.
I am following this answer Configure Django and Google Cloud Storage?
I have set the Google Cloud Storage bucket to public.
Django Model:
class Document(models.Model):
docfile = models.FileField(upload_to='documents/%Y/%m/%d')
Setting.py:
#MEDIA_URL = "/media/"
#MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
GS_BUCKET_NAME = 'printhub-files'
GS_PROJECT_ID = 'preasy-53c43'
GS_MEDIA_BUCKET_NAME = 'printhub-files'
# GS_STATIC_BUCKET_NAME = '<name-of-static-bucket>'
# STATIC_URL = 'https://storage.googleapis.com/{}/'.format(GS_STATIC_BUCKET_NAME)
MEDIA_URL = 'https://storage.googleapis.com/{}/'.format(GS_MEDIA_BUCKET_NAME)
Expected Result:
{
"id": 13,
"docfile": "https://storage.googleapis.com/bucket/documents/2019/11/03/myfile.pdf",
}
Actual Result:
{
"id": 13,
"docfile": null,
}
If I change my Settings.py to (uncomment line 1,2. Comment line 4), the file is saved on my local machine media/ folder, and "docfile" is set to the bucket url:
MEDIA_URL = "/media/"
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
#DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
GS_BUCKET_NAME = 'printhub-files'
GS_PROJECT_ID = 'preasy-53c43'
GS_MEDIA_BUCKET_NAME = 'printhub-files'
# GS_STATIC_BUCKET_NAME = '<name-of-static-bucket>'
# STATIC_URL = 'https://storage.googleapis.com/{}/'.format(GS_STATIC_BUCKET_NAME)
MEDIA_URL = 'https://storage.googleapis.com/{}/'.format(GS_MEDIA_BUCKET_NAME)
I get the output:
{
"id": 13,
"docfile": "https://storage.googleapis.com/bucket/documents/2019/11/03/myfile.pdf",
}
It seems that you're not configuring the settings.py file correctly. Check this documentation to use Google Cloud Storage Bucket for Djanjo project.
As you can see in the documentation, settings.py should be like this:
settings.py
.....
GCS_ROOT = "https://storage.googleapis.com/{bucket_name}/".format(
bucket_name=os.environ.get("GCS_BUCKET")
)
MEDIA_PREFIX = "media"
MEDIA_URL = "{gcs_root}{prefix}/".format(
gcs_root=GCS_ROOT,
prefix=MEDIA_PREFIX,
)
......
Let me know if it works for you.

Amazon S3 for django in production

I am using django-s3-folder-storage for my static and media files storage. I followed all the instructions mentioned in the documentation, but still I am not able to serve my static files. I am able to serve my user uploaded files, but the static files are not displayed on the website even after running collectstatic command, which copied the files to the bucket. I haven't created any CNAME records in my domain (not sure whether this makes any difference as I am able to see user uploaded images).
My settings file:
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
AWS_ACCESS_KEY_ID = # omitted
AWS_SECRET_ACCESS_KEY = # omitted
AWS_STORAGE_BUCKET_NAME = # omitted
MEDIA_ROOT = '/%s/' % DEFAULT_S3_PATH
MEDIA_URL = '//s3.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
STATICFILES_DIRS = (
root('static'),
)
And my project directory:
website_name
static
website_name
settings.py
I have changed the STATIC_URL to like below and it is working fine now.
STATIC_URL = '//bucketname.s3.amazonaws.com/static/'
But MEDIA_URL is like mentioned in the documentation itself and MEDIA_URL works fine. It is only STATIC_URL which needs modifying.
Looks like the documentation is wrong.
Generally S3 can use either the subdomain or subfolder path format. The default for django-storages, which is used by this particular storage, is subdomain. So it's interesting that changing to subfolder solved the problem. The OP mentions using the European region in a different question, and I suspect the root problem here has to do with the endpoint being used, or one of S3/S3BotoStorage's region-affecting configuration settings (e.g. AWS_CALLING_FORMAT).