I've setted up a Django project using settings modules, a base.py, development.py, and production.py but I'm a little confused on how to run the project locally and remotely.
So, I can run python manage.py runserver --settings=<project>.settings.development and everything goes alright but my doubt is: How can I run it through heroku local giving the settings option? It will also help to run it on remotely with production settings.
You can make use of Environment variables
In __init__.py of your settings directory add:
import os
from .base import *
environment = os.environ.get("ENV")
if environment == development:
from .development import *
else:
from .production import *
this will load the development settings module if the value of ENV is set to development, otherwise by default it will load the production settings module.
Configuring environment variables in Heroku
My Django project has multiple settings file for development, production and testing. And I am using supervisor to manage the celery worker. My question how to load the settings file for celery based on the environment I am in.
By using environment variables. Let's say you have the following setting files at the root of your repository.
config.settings.development.py
config.settings.production.py
...
The recommended way to have your celery instance defined is in your config like celery.py module:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
# os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.production')
app = Celery('proj')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
Instead of setting the DJANGO_SETTINGS_MODULE variable within the module (I have commented that out) make sure that those are present in the environment at the time that supervisord is started.
To set those variables in your staging, testing, and production system you can execute the following bash command.
E.g. on your production system:
$ export DJANGO_SETTINGS_MODULE=config.settings.production
$ echo $DJANGO_SETTINGS_MODULE
I would also suggest you load them from an .env file. In my opinion thats more convenient. You could do that with for example python-dotenv.
Update
The .env file is mostly unique on your different systems and is usually not under source/version control. By unique I mean for development you may have a more verbose LOG_LEVEL or a different SECRET_KEY, because you don't want them to show up in your source control or want to be able to adjust them without modifying your source files.
So, in your base.py (production.py and development.py are inheriting) you can load the variables from the file with for example:
import os
from dotenv import load_dotenv
load_dotenv() # .env file has to be in the same directory
# ...
import os
DJANGO_SETTINGS_MODULE = os.getenv("DJANGO_SETTINGS_MODULE")
print(DJANGO_SETTINGS_MODULE)
# ...
I personally don't use the package since I use docker, which has a declarative way of defining an .env file but the code above should give you an idea how it could work. There are similar packages out there like django-environ which is featured in the book Two Scoops of Django. So I would tend to use this instead of python-dotenv, a matter of taste.
You likely want to configure different settings files. From here you have two options. You can use the django-admin settings param at runtime
django-admin runserver --settings=thecelery.settings
Also, you may have the option to configure settings in the settings.py. If you have a 1 settings file currently, this would require you to setup additional settings files, and set environment variables on the instance. Then in your base settings file, you can do soemthing like this
import os
environment your_env = os.environ["environment"]
if your_env == "celery":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "thecelerysettings")
else:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "defaultsettings")
I have several customized django settings, this is basically my project structure:
MainProject/
manage.py
my_project/
settings/
base.py
dev.py
prod.py
I've created the __init__.py files inside the directories to identify them as packages.
I've exported the DJANGO_SETTINGS_MODULE to point to the chosen settings file.
The manage.py command seems to handle it pretty good, I never had problem with it.
The problem is that no matter what I do the django-admin.py is not able to find any settings file. I've tried several possible solution but nothing seems to work so far.
I've used the --settings=my_project.settings.dev
I've edited and hard-coded the manage.py to let it point to the dev.py file
I've created a settings file either inside the MainProject and my_project directories importing the dev file (that in turn imports the base.py).
I've created a settings file that let Django know which files should it use as settings
This is regarding the point 4:
from django.core.management import setup_environ
try:
import my_project.settings.dev as settings
except ImportError:
import sys
sys.stderr.write("Couldn't find the settings.py module.")
sys.exit(1)
setup_environ(settings)
Nothing seems to work so far.
====================================
SOLUTION:
I did not find the exact solution but thanks to a comment on the chosen answer I understood that you can basically use manage.py for everything that you could do in django-admin.py, I didn't know that! Since things DO work for me using manage.py I'm fine with it.
What I recommend doing:
Create a normal settings.py file and import one of the others in there. This avoids duplication of settings shared among the three scenarios + it is actually the recommended way of doing it according to the DRY principle.
Typically, you will only have the set the debug parameter, database settings and private keys in the specific settings files. All the other settings should be shared among all scenarios to avoid forgetting to update one and getting hard to debug errors.
Have you tried to import the dev settings inside the __init__.py from your settings module?
settings/_init_.py
from .dev import *
For the life of me, I have been looking for this everywhere and have not found the answer. I hope I am not posting a duplicate.
It is advised everywhere that you should keep your secret keys in a separate file from your general settings.py. Also, that you should never commit your "secret.py" file that contains keys such as SECRET_KEY, AWS_SECRET_KEY and so on.
My question is: In your production server, you need to reference your secret keys, that means that your "secret.py" settings file, should live somewhere around the server right? If so, how do you protect your secret keys in production?
I wanted to add a new answer because, as a beginner, the previous accepted answer didn't make a lot of sense to me (it was only one part of the puzzle).
So here's how I store my keys both LOCALLY and in PRODUCTION (Heroku, and others).
Note: You really only have to do this if you plan on putting your project online. If it's just a local project, no need.
I also made a video tutorial for people who prefer that format.
1) Install python-dotenv to create a local project environment to store your secret key.
pip install python-dotenv
2) Create a .env file in your base directory (where manage.py is).
YourDjangoProject
├───project
│ ├───__init__.py
│ ├───asgi.py
│ ├───settings.py
│ ├───urls.py
│ └───wsgi.py
├───.env
├───manage.py
└───db.sqlite3
If you have a Heroku project, it should look something like this:
YourDjangoProject
├───.git
├───project
│ ├───__init__.py
│ ├───asgi.py
│ ├───settings.py
│ ├───urls.py
│ └───wsgi.py
├───venv
├───.env
├───.gitignore
├───manage.py
├───Procfile
├───requirements.txt
└───runtime.txt
3) Add .env to your .gitignore file.
echo .env > .gitignore # Or just open your .gitignore and type in .env
This is how you keep your secret key more secure because you don't upload your .env file to git or heroku (or wherever else).
4) Add your SECRET_KEY from your settings.py file into the .env file like so (without quotes)
**Inside of your .env file**
SECRET_KEY=qolwvjicds5p53gvod1pyrz*%2uykjw&a^&c4moab!w=&16ou7 # <- Example key, SECRET_KEY=yoursecretkey
5) Inside of your settings.py file, add the following settings:
import os
import dotenv # <- New
# Add .env variables anywhere before SECRET_KEY
dotenv_file = os.path.join(BASE_DIR, ".env")
if os.path.isfile(dotenv_file):
dotenv.load_dotenv(dotenv_file)
# UPDATE secret key
SECRET_KEY = os.environ['SECRET_KEY'] # Instead of your actual secret key
or, thanks to #Ashkay Chandran's answer:
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
SECRET_KEY = os.environ['SECRET_KEY']
And now your secret key is successfully stored locally.
Update: I found out you can also use the config method from the package python-decouple that seems to be a bit easier:
from decouple import config
SECRET_KEY = config('SECRET_KEY')
Now you don't need to import os or use dotenv because it takes care of those parts for you AND will still use the .env file. I started using this in all of my projects.
6) Add the SECRET_KEY environment variable on your host (such as Heroku).
I work mostly with Heroku sites, so if you're wanting to use Heroku for a Django project, this part is for you.
This assumes that you already have a Heroku project setup and have Heroku CLI downloaded on your computer.
You have 2 options:
From Command Line / Terminal, you can enter the following command in your project's directory:
heroku config:set SECRET_KEY=yoursecretkey # Again, no quotes.
You can go to your Heroku dashboard, click on your app, go to your apps settings, and see the "Config Vars" section and click "Reveal Vars" or "Add Vars" and add your SECRET_KEY there.
Then, when you push your project to Heroku through git, it should be working properly without any issue.
and that's it! 🙂
This answer was targeted towards total beginners / intermediates to hopefully cut through any confusion (because it was definitely confusing for me).
See the Django deployment docs for a discussion on this.
There's quite a few options for production. The way I do it is by setting my sensitive data variables as environmental variables on the production environments. Then I retrieve the variables in the settings.py via os.environ like so:
import os
SECRET_KEY = os.environ['SECRET_KEY']
Another possible option is to copy in the secret.py file via your deploy script.
I'm sure there are also other specific options for different web servers.
You should store your settings in a modular way. By that I mean to spread your settings across multiple files.
For example, you can have base_settings.py to store all your base settings; dev_settings.py for your development server settings; and finally prod_base_settings.py for all production settings. All non-base settings files will import all the base settings and then only change whatever is necessary:
# base_settings.py
...
# dev_settings.py
from base_settings import *
DEBUG = TRUE
...
# prod_base_settings.py
from base_settings import *
DEBUG = FALSE
...
This approach allows you to have different settings from different setups. You can also commit all these files except then on the production server you can create the actual production settings file prod_settings.py where you will specify all the sensitive settings. This file should not be committed anywhere and its content kept secure:
# prod_settings.py
from prod_base_settings import *
SECRET_KEY = 'foo'
As for the file names you can use whatever filenames you feel are appropriate. Personally I actually create a Python package for the settings and then keep the various settings inside the package:
project/
project/
settings/
__init__.py
base.py
dev.py
...
app1/
models.py
...
app2/
models.py
...
Storing secrets in the environment still places them in the environment; which can be exploited if an unauthorized user gains access to the environment. It is a trivial effort to list environment variables, and naming one SECRET makes is all the more helpful and obvious to a bad actor an unwanted user.
Yet secrets are necessary in production, so how to access them while minimizing attack surface? Encrypt each secret in a file with a tool like git-secret, then allow authorized users to read in the file, as mentioned in django's docs. Then "tell" a non-root user the secret so it can be read-in during initialization.
(Alternatively, one could also use Hashicorp's Vault, and access the secrets stored in Vault via the HVAC python module.)
Once this non-root user is told, something like this is easy:
# Remember that './secret_key.txt' is encrypted until it's needed, and only read by a non-root user
with open('./secret_key.txt') as f:
SECRET_KEY = f.read().strip()
This isn't perfect, and, yes, an attacker could enumerate variables and access it -- but it's very difficult to do so during run-time, and Django does a good job of protecting its keys from such a threat vector.
This is a much safer approach than storing secrets in the environment.
I know it has been a long time, but I just opensourced a small Django app I am using to generate a new secret key if it does not exist yet. It is called django-generate-secret-key.
pip install django-generate-secret-key
Then, when provisioning / deploying a new server running my Django project, I run the following command (from Ansible):
python manage.py generate_secret_key
It simply:
checks if a secret key needs to be generated
generates it in a secretkey.txt file (can be customized)
All you need then is to have in your settings file:
with open('/path/to/the/secretkey.txt') as f:
SECRET_KEY = f.read().strip()
You can now benefit from a fully automated provisioning process without having to store a static secret key in your repository.
Instead of if/then logic you should use a tool designed for factoring out sensitive data. I use YamJam https://pypi.python.org/pypi/yamjam/ . It allows all the advantages of the os.environ method but is simpler -- you still have to set those environ variables, you'll need to put them in a script somewhere. YamJam stores these config settings in a machine config store and also allows a project by project ability to override.
from YamJam import yamjam
variable = yamjam()['myproject']['variable']
Is the basic usage. And like the os.environ method, it is not framework specific, you can use it with Django or any other app/framework. I've tried them all, multiple settings.py files, brittle logic of if/then and environment wrangling. In the end, I switched to yamjam and haven't regretted it.
Adding to zack-plauch's answer,
To get the path to the .env file, when using python-dotenv module, the find_dotenv method can be used,
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
SECRET_KEY = os.environ['SECRET_KEY']
The find_dotenv() looks for a ".env" file in the path, so it can be saved inside the same directory too,
Also, if a name is used for the .env file like "django-config.env", load_dotenv(find_dotenv("django-config.env"), will fetch and load that to host-machine environment variable mappings.
I am surprised that noone has talked about django-environ.
I usually create a .env file like this:
SECRET_KEY=blabla
OTHER_SECRET=blabla
This file should be added in .gitignore
You can checkin in git, an example file named .env.example just for others to know which env var they need. The content of .env.example file will look like this (just keys without any values)
SECRET_KEY=
OTHER_SECRETS=
Where to store SECRET_KEY DJANGO
Store your django SECRET_KEY in an environmental variable or separate file, instead of directly encoding In your configuration module settings.py
settings.py
#from an environment variable
import os
SECRET_KEY = os.environ.get('SECRET_KEY')
#from an file
with open('/etc/secret_key.txt') as f:
SECRET_KEY = f.read().strip()
How to generate Django SECRET_KEY manually:
$ python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
7^t+3on^bca+t7#)w%2pedaf0m&$_gnne#^s4zk3a%4uu5ly86
import string
import secrets
c = string.ascii_letters + string.digits + string.punctuation
secret_key = ''.join(secrets.choice(c) for i in range(67))
print(secret_key)
df&)ok{ZL^6Up$\y2*">LqHx:D,_f_of#P,~}n&\zs*:y{OTU4CueQNrMz1UH*mhocD
Make sure the key used in production is not used elsewhere and avoid sending it to source control.
Trying to use 2 different settings file for production and dev.
I set DJANGO_SETTINGS_MODULE='mysite.settings_production'
Works perfectly when running server with runserver
When I run it with apache though, apache doesn't seem to use the setting in the ~/.bash_profile and instead use os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings") in the wsgi.py file
I guess it's maybe because apache is running on different user , not mine..
ok.. so it seems it's running as www-data on my ec2 ubuntu.
So I have to create the /home/www-data/.bash_profile and set the env variable?
It seems like so much hassle to use a different settings file.
Is there an obviously easier way to do this?
(I don't want to change the wsgi.py file, because it's source controlled)
Using bash_profile is completely the wrong way to do this.
The correct way is to use the wsgi.py file. However, since you don't want to do this (although I don't understand what it being version-controlled has to do with anything) then you can set environment variables directly in your Apache configuration using SetEnv:
SetEnv DJANGO_SETTINGS_MODULE mysite.settings
Well, it's really wrong way. Common method to have separate settings for different environment is to store environment-dependent in local_settings.py (or whatever you name it) and imprort from settings.py
from local_settings.py import *
don't put local_settings.py under project repository as you will override it with each commit. If you want to keep a sample of local settings, put into a separate file, e.g. local_settings.py.example
You can import local settings at the beginning of settings.py (so settings.py settings would override local settings), or at the end, or have two local settings files for both cases.