I have a Django app on Heroku, with staging and production environments. Static files are hosted on S3. I'm streamlining my deployment process and plan to set up fabfiles once I have things working manually.
How can I configure collectstatic to push to multiple places? If I run it locally, it uses my dev settings (with a local STATIC_ROOT). If I run it on one of my Heroku apps (heroku run ./manage.py collectstatic), then it can't grab the files (since .slugignore ensures they're never pushed to Heroku). The same applies if I include collectstatic in my Procfile.
I'm also using django-pipeline, though it's not yet doing much since I'm stuck on the collectstatic bit.
UPDATE
In response to Marat's question, I tried passing a settings file as an option to collectstatic: ./manage.py collectstatic --settings=project.settings.prod, but got an error: Unknown command: 'collectstatic' I checked on the server though and Installed Apps does include django.contrib.staticfiles and I can also run collectstatic remotely, so I'm not sure what would cause that.
You can set the environment variable DJANGO_SETTINGS_MODULE so you don't need specify --settings everywhere:
heroku config:set DJANGO_SETTINGS_MODULE=project.settings.prod
First, if you are going to serve static via CloudFront, you can use custom origin and always use local STATIC_ROOT. Actually it has some advantages over S3 source, eg gzip support.
Another good thing you can do is to have environment dependent settings in a separate file and then import it in settings.py, eg:
local_settings.py (not in project repository, yet you can have local_settings.py.example):
#environment dependent settings
DATABASES = { .. }
CACHES = { .. }
STATIC_ROOT = 'your_path/static'
settings.py:
import local_settings
I've just replied a similar question on Upload Media from Heroku to Amazon S3. If you customise your settings to take in account environmental vars, you can use filesystem storage backends locally and S3 storage backends when pushing to Heroku. This will collect and upload your static files when your slug is compiled.
Related
I'm exploring using whitenoise to serve static files in a Django application that's packaged in a Docker container. In the documentation it says:
As part of deploying your application you’ll need to run ./manage.py collectstatic to put all your static files into STATIC_ROOT. (If you’re running on Heroku then this is done automatically for you.)
Is that really needed? I'm not running ./manage.py collectstatic and static files are still served.
If it's not needed, is it an optimization? I'm trying to avoid having needless steps in my deployment process.
As that quote states, Whitenoise serves files from STATIC_ROOT. collectstatic collects files from STATICFILES_DIRS and any app-specific static directories and puts them into STATIC_ROOT.
But there are two situations where it will serve files without running collectstatic. The first is if your files are already in STATIC_ROOT. They shouldn't be, but it's a common mistake to set the value of STATIC_ROOT to the directory containing the source files. However, in this case it won't for example find the admin files, which would need to be collected.
The other situation is if you are running with DEBUG=True. Whitenoise inherits this value for its USE_FINDERS setting, which makes it look in the same places as collectstatic itself to serve files. Clearly, you wouldn't want to run in production with DEBUG on, but you can set this setting explicitly. But as that docs link points out, doing so will disable the caching and compression features of Whitenoise.
So I was finally able to set up local + prod test project I'm working on.
# wsgi.py
from dj_static import Cling, MediaCling
application = Cling(MediaCling(get_wsgi_application()))
application = DjangoWhiteNoise(application)
I set up static files using whitenoise (without any problems) and media (file uploads) using dj_static and Postgres for local + prod. Everything works fine at first... static files, file uploads.
But after the Heroku dynos restart I lose all the file uploads. My question is, --- Since I'm serving the media files from the Django app instead of something like S3, does the dyno restart wipe all that out too?
PS: I'm aware I can do this with AWS, etc, but I just want to know if thats the reason I'm losing all the uploads.
Since I'm serving the media files from the Django app instead of something like S3, does the dyno restart wipe all that out too?
Yes!. That's right. According to the Heroku docs:
Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code.
See, also this answer and this answer.
Conclusion: For media files (the uploaded ones), you must use some external service (like S3 or something). whitenoise is just for static files. See here why whitenoise is not suitable for serving user-uploaded (media) files.
I put static files under a app directory which is related to the static files.
for example,
image.jpg is used for templates under exmapleapp. so I locate image.jpg file in a directory /project/exampleapp/static/image.jpg
In Debug=True settings, Dajngo finds static files automatically by django.contrib.staticfiles. If you put static files projoject directory /project/static/, you can set STATICFILES_DIRS = []in settings.py or add like urlspatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) in urls.py(if you want to serve static files manually)
so far, is nothing problem in development right? but you want to deploy, you need to do collectstatic. I am confused from now.
This is what I thought initially.
step1 : you work and test your code in local with DEBUG = True in setting. you continously save your code in git repo.
step2 : you are ready to deploy your code in server. you clone your repo and set apache server for running django framework properly(Alias static, WSGI Daemon process). you do python manage.py collectstatic to serve static file with apache server.
step3 : you keep work in local to improve your code and apply this improvement into your code in server with test.
I got confused and got questions.
Q1 : if you do collectstatic for deployment, there will be static folder(according to STATIC_ROOT settings) with all static files which is spread around in each app folders.
do you do collectstatic in local and send only the static folder which collects all to server? or do you do collectstatic in server according to Static Alias setting in Apache server?
Q2 : Do you change DEBUG = False and ALLOWED_HOSTS =[server IP] in local and save code in git repo and pull it in server??
well, now I am confused with working during development with github and deploying it to server with github with Django settings.
Tell me if you can't understand my question clearly please.
You are using static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) for serving static files in development mode. But in production, (means DEBUG = False) it will be empty list. (static returns empty list). Then collectstatic, in you can use development or production.Its intention is to copy all static files to STATIC_ROOT.So
Q1: Everyone do collectstatic on production to serve static files through webserver(Apache, Nginx)
Q2: For production and development write different settings file. You can see here for ex How to manage local vs production settings in Django?
I've followed the official Heroku docs on Django and Static Assets; I've installed dj-static and added it to my requirements.txt file, properly configured all the variables in my settings.py file:
STATIC_ROOT = os.path.join(CONFIG_ROOT, 'served/static/')
STATIC_URL = '/static/'
STATICFILES_DIRS = (
os.path.join(CONFIG_ROOT, 'static'),
)
And this is what my wsgi.py looks like:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_django_project.settings")
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())
The contents of Procfile:
web: gunicorn --bind 0.0.0.0:$PORT my_django_project.wsgi:application
In the docs, it says that "collectstatic is run automatically when it is configured properly." But when I navigate to my site there's clearly no css.
I've tried debugging using heroku run, but that just copies the static files as expected.
I've noticed that when I include the collectstatic command in my Procfile, i.e.
web: python my_django_project/manage.py collectstatic --noinput ; gunicorn -b 0.0.0.0:$PORT my_django_project.wsgi:application
...that works as expected, and the static files are served.
However what's strange about that is when I run heroku run bash and look inside the directory that STATIC_ROOT is pointing to, there's nothing there! In fact, the entire served/ directory is missing, and yet, the static files are still being served!
I'd still like to know why isn't collectstatic being run automatically though -- like mentioned in the docs -- when I deploy my Django app to Heroku.
It looks like you might be using a specific settings module for Heroku/production. Further, you've set the environment variable DJANGO_SETTINGS_MODULE to point to this settings module (and that way, when the app runs, Django knows to use that one and not, say, your default/development one). Finally, you've probably configured static asset settings in Heroku/production settings module (perhaps, STATIC_ROOT).
Okay, so if this is all correct, then here is the issue: heroku environment variables are only set at serve-time, not at compile-time. This is important because collectstatic is a compile-time operation for Heroku. (Heroku goes through 2 stages when you push: 1) compiling, which involves setting the application up (collectstatic, syncdb, etc) 2) serving, the normal operation of your application).
So, essentially, you've done everything correctly, but Heroku hasn't exposed your environment variables, including your specification of a different settings module, to collectstatic.
To have your environment variables set to compile-time, enable Heroku's user-env-compile lab feature like this:
heroku labs:enable user-env-compile
I think this is a silly thing to do by default, and would be interested in hearing why Heroku thought it was a good idea.
Have you tried adding the user_env_compile setting to your heroku config?
heroku labs:enable user-env-compile
With that enabled collectstatic should be run whenever you deploy to heroku automatically.
I am using the heroku python buildpack with dokku, and collectstatic was not being run because it had no execute permission. They fixed that in a recent commit (Dec 13, 2013), so it should work now.
I'm using django-storages for static files with S3 (and S3BotoStorage). When I do collectstatic from my local machine, the behaviour is as expected, where only modified files are pushed to S3. This process needs python-dateutils 1.5 to check for modified time.
However, doing the same on Heroku results in every file being pushed regardless, although the setup is the same. I then looked into the modified time of the files on Heroku itself, and it seems like, the os.stat(static_filename).st_mtime is the same as the time of the last push.
Is this expected behaviour? Does heroku copy around files even when there is no change from git?
Try setting DISABLE_COLLECTSTATIC=1 as an environment setting for your app - that should disable it from running on every push.
See this article for details - https://devcenter.heroku.com/articles/django-assets :
> Sometimes, you may not want Heroku to run collectstatic on your behalf.
> You can disable collectstatic by enabling user-env-compile as well:
$ heroku labs:enable user-env-compile
$ heroku config:set DISABLE_COLLECTSTATIC=1
I've found that simply setting the config will do - no need to also enable user-env-compile - it may be that that this has passed from labs into production?
NB the deployment is managed by the Heroku python buildpack, which you can see here - https://github.com/heroku/heroku-buildpack-python/
EDIT 1
I've just done a bunch of tests on this, and can confirm that DISABLE_COLLECTSTATIC does indeed disable collectstatic, irrespective of the user-env-compile setting - I think that's now in the main trunk (but that's speculation). Doesn't seem to care what the setting is - if DISABLE_COLLECTSTATIC exists as a config var it is used.
I strongly recommend using the collectfast package for any django static deployment to s3, whether local or from your heroku server. It ignores modified dates and utilizes md5 hashes, which the s3 api will provides very quickly, and (optional) caching to make your static deployments zoom. It took my static deployments from ~10-15 minutes to < 2 minutes and only deploys the files that have actually changed.
I've just had that exact same issue and contacted Heroku's support to find out what is going on. My question to them was
I've run into a funky issue doing some deployments. It appears that on each push the date modified on all files is updated to the time a new deploy/git push happens. Is this intended behaviour?
When considering that Django's collectstatic command only checks the modified date on files when evaluating if the file should be copied across to the final storage backend for static assets, it means that on each new push, all files are first removed from the remote storage (in this case S3) and then re-uploaded. This is both a very slow and wasteful process in terms of bandwidth consumed and requests made.
The answer I received today from "Caio", one of Heroku's support staff, was
Hi, that's how it currently works, yes. I'm routing your feedback to our runtime team to see if we can package files with their original dates.
As confirmed by Alen, Heroku changes the modified date of the files when it deploys. However, Amazon S3 also has an attribute called etag that is an md5 hash of the file content. It's possible to use this to check if the files have changed instead of the modified date, as implemented in this Django snippet.
I took that code, packaged it and fixed some errors I found and put it on Github as django-s3-collectstatic. It includes a new management command fasts3collectstatic that only uploads new files. Check the Github page for installation instructions.
Why not run collectstatic from local machine?
python manage.py collectstatic --noinput --settings=settings.[prod]
I agree this is annoying- there's a couple things you can do. I override the collectstatic command and wire it up in my production settings. Below is the command I use:
```
from django.core.management.base import BaseCommand
class Command(BaseCommand):
args = '< none >'
help = "disables collectstatic cmd in contrib"
def handle(self, *args, **kwargs):
print 'collectstatic disabled'
```
I keep this in mysite/disablecollectstatic/management/commands
Then in production settings:
INSTALLED_APPS += ('mysite.disablecollectstatic',)
Alternatively you could use the fact that Heroku does a dry run first before actually invoking the command. If it fails, it won't run it, which means you could contrive an error (by maybe deleting the static root in your settings, for example) but this approach makes me nervous:
https://devcenter.heroku.com/articles/django-assets#detection