I have created a blog website in Django. I have posted multiple articles on the website. After deleting the article, the article gets deleted but the media files are not removed. I want to delete all media files which are not referred to the articles.
I know, I can create Django post-delete signal and delete media files from there. But it will applicable for only future use. I want to delete previous media files which are not in my database.
first install this:
pip install django-cleanup
then, add this in settings file inside your installed_apps:
'django_cleanup.apps.CleanupConfig'
It will delete your media files.
Just use this program to delete your unused media files.
This command deletes all media files from the MEDIA_ROOT directory which are no longer referenced by any of the models from installed_apps.
import os
from django.core.management.base import BaseCommand
from django.conf import settings
class Command(BaseCommand):
def handle(self, *args, **options):
physical_files = set()
db_files = set()
media_root = getattr(settings, 'MEDIA_ROOT', None)
if media_root is not None:
for relative_root, dirs, files in os.walk(media_root):
for file_ in files:
relative_file = os.path.join(os.path.relpath(relative_root,
media_root), file_)
physical_files.add(relative_file)
deletables = physical_files - db_files
if deletables:
for file_ in deletables:
os.remove(os.path.join(media_root, file_))
Related
I'm using Django 3.2. I've changed added this line to settings.py:
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
I then ran these commands:
$ python manage.py makemigrations
$ python manage.py migrate
The makemigrations command creates new migration files for my apps, not just the apps that I have created, but also in my dependencies. For example, I'm using django-allauth, and this file was created in my virtual environment (virtualenv):
.venv/lib/python3.8/site-packages/allauth/account/migrations/0003_auto_20210408_1526.py
This file is not shipped with django-allauth. When I deploy this application from git, this file is not included.
What should I do instead? How can I switch DEFAULT_AUTO_FIELD without the need to create new migration files for dependencies like django-allauth?
Ideally, your third party dependencies would include this line in the config found in apps.py:
from django.apps import AppConfig
class ExampleConfig(AppConfig):
default_auto_field = 'django.db.models.AutoField'
While waiting for upstream dependencies to update their apps.py or migration files, you can override the app config yourself. If it doesn't exist already, create an apps.py file in your main app directory (eg: project/apps.py), and override the config of a dependency. In this example, I'm overriding the config of django-allauth:
from allauth.account.apps import AccountConfig
from allauth.socialaccount.apps import SocialAccountConfig
class ModifiedAccountConfig(AccountConfig):
default_auto_field = 'django.db.models.AutoField'
class ModifiedSocialAccountConfig(SocialAccountConfig):
default_auto_field = 'django.db.models.AutoField'
Then modify INSTALLED_APPS in settings.py to look like this, replacing the old entries for django-allauth in this example:
INSTALLED_APPS = [
# ....
# replace: "allauth.account", with
"projectname.apps.ModifiedAccountConfig",
# replace: "allauth.socialaccount", with
"projectname.apps.ModifiedSocialAccountConfig",
]
If the dependency doesn't have an apps.py file to override, you can still create an AppConfig sub-class in project/apps.py like this:
from django.apps import AppConfig
class ModifiedExampleDependencyConfig(AppConfig):
name = 'exampledependency' # the python module
default_auto_field = 'django.db.models.AutoField'
Now when you run python manage.py makemigrations, no migration files should be created for the dependencies.
I work on a big project, we upgraded Django from 2.2. to 3.2 and then have got a need to create all new models with Big Integer (Int8) (PostgreSQL) field instead of default Integer (Int4).
When I defined it in settings.py:
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
I got the same problem, but with own apps - Django tried to make me to migrate 135 models I had, but I didn't want to do it. I only wanted to create new models with BigInt and manipuate olds manually.
I found the next solution. I changed the field to custom:
DEFAULT_AUTO_FIELD = 'project.db.models.CustomBigAutoField'
And then overrided its deconstruction:
from django.db import models
class CustomBigAutoField(models.BigAutoField):
"""Int8 field that is applied only for new models."""
def deconstruct(self):
name, path, args, kwargs = super().deconstruct()
if getattr(self, 'model', None):
path = 'django.db.models.AutoField'
return name, path, args, kwargs
As I discovered, fields of new models don't have a back reference to their models, so path wouldn't be overridden for them.
We override path because Django checks whether a model is changed by a key, that includes the path to this field. So we deceive Django and it thinks that existing model didn't changed.
I might not see the whole picture, but I tested it with different existing and new models and it worked for me. If someone tells me why this solution is bad, I'd be grateful.
I have a small Django app, and I'm trying to access data from a CSV file (static/blog/dat.csv, the static folder is at the same level as the templates folder and views.py; and everything is inside my blog app) so I can use it to plot a graph on the browser using Chart.js. Aside from not being able to do that, the app is working fine.
I know I'm gonna need to pass some sort of context to the view function, but I don't know how I'd do that. Also, I have a couple of similar csv files, and using them as static files in my app seems simpler and easier than to add everything to a database to access them that way.
# views.py
from django.shortcuts import render
from django.contrib.staticfiles.storage import staticfiles_storage
import csv
def rtest(request):
url = staticfiles_storage.url('blog/dat.csv')
with open(url, 'r') as csv_file:
csv_reader = csv.reader(csv_file)
for line in csv_reader:
context += line
return render(request, 'blog/r.html', context)
# urls.py
urlpatterns = [
# ...
path('r-test/', views.rtest, name='blog-r-test'),
]
Here is the error I'm getting:
FileNotFoundError at /r-test/
[Errno 2] No such file or directory: '/static/blog/dat.csv'
I'm sure this is not the only error.
I know that the way I'm using the context variable is wrong, but that's just to kind of show what I'm trying to do. If I could just print one cell from the csv, I would view this as a win. Please help, thank you!
------Edit1-------
After using staticfiles_storage.path() instead of staticfiles_storage.url()
ImproperlyConfigured at /r-test/
You're using the staticfiles app without having set the STATIC_ROOT setting to a filesystem path.
------Edit2------
I'm now able to find my csv file:
STATIC_URL = '/static/'
STATIC_ROOT = 'C:/Users/riccl/Documents/richie/Python/nuclear/main/book/static/book'
But my context variable still doesn't make any sense.
You need to use staticfiles_storage.path() to read the file. staticfiles_storage.url() will return the URL that a user will use to load the static file on your site
STATIC_ROOT is where all static files will be stored after you run collectstatic, most of the time this is set to <root of the project>/static/. This is also where staticfiles_storage.path() will look for static files.
You will also need to set STATICFILES_DIRS so that your file(s) can be found by collectstatic. I usually have a folder located at <root of the project>/<project name>/static/ which I add to STATICFILES_DIRS
I have uploading files to S3 working fine with my Wagtail/django application (both static and uploads). Now I'm trying to use ManifestStaticFilesStorage to enable cache busting. The urls are correctly being generated by the application and files are being copied with hashes to S3.
But each time I run collectstatic some files get copied twice to S3 - each with a different hash. So far the issue is ocurring for all CSS files.
file.a.css is loaded by the application and is the file referenced in staticfiles.json - however it is a 20.0B file in S3 (should be 6.3KB).
file.b.css has the correct contents in S3 - however it does NOT appear in the output generated by collectstatic.
# custom_storages.py
from django.conf import settings
from django.contrib.staticfiles.storage import ManifestFilesMixin
from storages.backends.s3boto import S3BotoStorage
class CachedS3Storage(ManifestFilesMixin, S3BotoStorage):
pass
class StaticStorage(CachedS3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
file_overwrite = False
Deps:
"boto==2.47.0",
"boto3==1.4.4",
"django-storages==1.5.2"
"Django==2.0.8"
Any pointers on where to look to track down this issue would be appreciated! :)
Edit:
Looking more carefully at all the files copied to S3 the issue is ONLY occurring for CSS files.
Disabling pushing assets to S3 and writing them to the local filesystem works as expected.
Edit 2:
Updated all the deps to the latest version - same behavior as above.
I eventually stumbled across this issue in django-storages issue tracker which then lead me to a very similar question on SO.
Between these two pages I managed to resolve the issue. I did the following to get django-storages + ManifestStaticFilesStorage + S3 to work together:
# custom_storages.py
from django.conf import settings
from django.contrib.staticfiles.storage import ManifestFilesMixin
from storages.backends.s3boto3 import S3Boto3Storage # note boto3!!
class PatchedS3StaticStorage(S3Boto3Storage):
def _save(self, name, content):
if hasattr(content, 'seek') and hasattr(content, 'seekable') and content.seekable():
content.seek(0)
return super()._save(name, content)
class CachedS3Storage(ManifestFilesMixin, PatchedS3StaticStorage):
pass
class StaticStorage(CachedS3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3Boto3Storage):
location = settings.MEDIAFILES_LOCATION
file_overwrite = False
Note that I had to use boto3 to get this to work django-storages must be >= 1.5 to use boto3. I removed boto as a dep. My final deps were:
"boto3==1.4.4",
"django-storages==1.7.1"
"Django==2.0.8"
I use django-cleanup, sorl-thumbnail in my Django project.
I have a model like this:
from sorl.thumbnail import ImageField
class Variation(BaseModel):
image = ImageField(upload_to=draft_img_upload_location)
And I use signal for sorl-thumbnail like this (recommanded by https://github.com/un1t/django-cleanup):
from django_cleanup.signals import cleanup_pre_delete
def sorl_delete(**kwargs):
from sorl.thumbnail import delete
delete(kwargs['file'])
cleanup_pre_delete.connect(sorl_delete)
So, In local environment, belows work:
1. When I delete Variation model in ADMIN PAGE, it deletes BOTH image file and image cache(created by sorl-thumbnail).
2. When I change just image file with other image in ADMIN PAGE, it delete BOTH 'prior image file' and 'prior image cache file(created by sorl-thumbnail)'.
In production environment, I used AWS S3 and cloudfront for managing static and media file. So all sorl-thumbnail cache image files are stored in S3. But whenever I changed the image file with other image file in ADMIN PAGE, the prior image cache file(created by sorl-thumbnail) still remained.
Lets say that sorl-thumbnail image url is https://example.cloudfront.net/cache/da/75/abcdefg_i_am_cached_image.jpg (from Google development tool).
In this case, there were two image files exist in S3: abcdefg.jpg and /da/75/abcdefg_i_am_cached_image.jpg
I changed abcdefg.jpg with the other image. Then, it completely deleted abcdefg.jpg in S3 storage.
Now, I accessed https://example.cloudfront.net/cache/da/75/abcdefg_i_am_cached_image.jpg in my browser and guess what! It showed this sorl-thumbnail cached images in this time!
Strange thing happened in S3 storage. When I tried to check whether abcdefg_i_am_cached_image.jpg exists in path /da/75, there was no directory named 75 right below the da folder!
In short, abcdefg_i_am_cached_image.jpg still remained in my S3 storage!
I don't know why this happened only in production environment...
This is part of settings only for production mode.
settings.py
from .partials import *
DEBUG = False
ALLOWED_HOSTS = ["*", ]
STATICFILES_STORAGE = 'spacegraphy.storage.S3PipelineCachedStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY")
AWS_STORAGE_BUCKET_NAME = os.environ.get("AWS_STORAGE_BUCKET_NAME")
AWS_S3_CUSTOM_DOMAIN = os.environ.get("AWS_S3_CUSTOM_DOMAIN")
AWS_S3_URL_PROTOCOL = 'https'
AWS_S3_HOST = 's3-ap-northeast-1.amazonaws.com'
STATIC_URL = "https://this-is-my-example.cloudfront.net/"
INSTALLED_APPS += [
'storages',
'collectfast'
]
AWS_PRELOAD_METADATA = True
storage.py
from django.contrib.staticfiles.storage import CachedFilesMixin, ManifestFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage
class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
pass
class S3PipelineCachedStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
pass
After spending couple of hours debugging, I found out that sorl_delete signal is not called only in production environment!!.
I have no idea why this happened. I think that this one is a main problem.
And sorry for bad english (I'm not native). Really need your help. Thanks
Before I explain, I have a question. I am making a blog and the posts have a heading image. I upload this image using admin when I create a post for my blog. Is this image 'static' or 'media'?
Anyway, I have my project set up as follows:
mysite is my project, blog is my app. I have a folder named static inside blog. Inside this static folder are four more folders- css, images, fonts, js.
In settings.py, I have this set up:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
My models.py looks like this:
class Post(models.Model):
...
header_image = models.ImageField('article_header_image', upload_to = 'blog/static/images/articleimages')
Now when I try to load this image in a template, it doesn't show up.
I am doing this the following way:
{% for post in posts %}
<img src="{{post.header_image.url}}">
{% endfor %}
Another thing, how can I use the upload_to argument to upload images into the app's static folder without explicitly stating it? I think thats where the issue is. If I use upload_to = 'static/images/articleimages', a new static folder is created in the mysite folder containing manage.py.
Any help is greatly appreciated!
You can import the STATIC_ROOT from settings file, and use that to biuld the path. In addition to that, you can use a callable, and build path there instead:
import os.path
from django.conf import settings
def generate_upload_path(instance, filename):
return os.path.join(settings.STATIC_ROOT, 'images/articleimages/')
class Post(models.Model):
...
header_image = models.ImageField('article_header_image', upload_to = generate_upload_path)
Ok, if you want to upload images to static directory of your app, you can use __path__ attribute of your app:
import blog
def generate_upload_path(instance, filename):
return os.path.join(blog.__path__, 'static/images/articleimages/')