I have a development app running locally. I have a production app running on my server. I would like to keep developing locally and push my changes. However, the dev version uses local postgres and the static and media files reside inside the project. The server version the static and media files are in a static public_html directory served by apache. Can I have a local static and media files as well as different postgres credentials on localhost than on the server? How do I accomplish that?
The python-dotenv package is designed specifically for the issue you are running into. Instead of using JSON files it uses .env files which follows the practice of 12 factor apps.
An example would be
settings.py
from dotenv import load_dotenv
load_dotenv()
DEBUG = os.getenv('DEBUG', 1) # with a default value
SECRET_KEY = os.getenv('DJANGO_SECRET_KEY')
DB_USERNAME = os.getenv('DB_USERNAME')
DB_PASSWORD = os.getenv('DB_PASSWORD')
LOGGING_FOLDER = os.getenv('LOGGING_FOLDER')
# Allowed hosts can be stored and split into a list
# Also the name in the separate settings file does not have to match the settings.py variable name
ALLOWED_HOSTS = os.getenv('HOSTS', '').split(' ')
# This will be ['locahost', 'localhost:8000']
# ...
.env
DJANGO_SECRET_KEY=somethingotherthanthis
DB_USERNAME=postgres
DB_PASSWORD=123456
LOGGING_FOLDER=/var/logging/app/
HOSTS=localhost localhost:8000
Of course you don't have to use the external package you can always use the JSON file and only change the following code in your settings. What you will want to be careful of in both circumstances is that the file you are saving your sensitive data and environment specific information (like media folders, logging folders, etc) is not checked into the version control system.
settings.py
import json
with open('settings.json') as fh:
file_settings = json.load(fh)
DEBUG = file_settings.get('DEBUG', 1)
# ...
Related
I'm trying to get my Django app to play audio files (supposedly) uploaded by users via a form. Said files are tied to a model :
# models.py
class Doc(models.Model):
# ...
tape = models.FileField()
The uploading and saving parts are working fine, and the files are stored where they should be :
- djangoproject
|
- docapp
|
- media <- here
So, in order to get where I want, I added these two lines to the settings.py file MEDIA_ROOT = os.path.join(BASE_DIR, 'docapp/media/') and MEDIA_URL = 'docapp/media/'.
I hoped to be able to link to the audio files thus:
# templates/docapp/index.html
...
<audio src='{{ doc.tape.url }}' controls></audio>
Unfortunately, this wasn't working because the link generated by doc.tap.url (http://localhost/docapp/media/filename.aac) was returning a 404 error.
After a lot of googling I found this answer, which I happily copy-pasted into my app ... and it worked. This is the code in question :
# urls.py
from django.conf import settings
from django.conf.urls.static import static
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
The problem is that I'm not comfortable with inserting code in my apps that I don't understand. I did some research about the static() function and all I could get is this :
Helper function to return a URL pattern for serving files in debug mode
Does this mean that the static function should not be used in production? If so, what should I be using in production? And what exactly does it do?
EDIT To be clear, the generated URL after injecting the solution is the same as the one generated without it. Yet, it only works when the static() function is present.
EDIT 2 I forgot to mention that the 404 errors persisted even after I chmoded the media folder to allow others to read-access it.
Thanks in advance!
You shouldn't do anything. No errors no problem. The docs write about development server and serving static files AND that it is for development only. In a production environment you configure your server (Apache, NGNIX or third party like S3) to serve the files. That's all.
Try to configure media files and access the file via it's url. If it works, try the {{ doc.tape.url }} template tag.
In your development environment your media may live in /media/ (route and directory). While on production it may be something like media.example.com. Running Django with the settings for that environment will change all static/media domains and paths to their correct locations.
You may split settings file into a settings file for each environment (production, acceptance, development). Like this:
project/
settings/
__init__.py
base.py
local.py
staging.py
test.py
production.py
You can run your project with a specific env: ./manage.py runserver --settings=project.settings.development. Do not repeat yourself and put development specific settings in development.py and from base import * so that base.py contains the default settings.
This project and settings layout is taken from the book Two Scoops of Django. It is just an example. Adjust to your own needs.
Yes, django.conf.urls.static.static is only for development and not for production. For production, you should just need to configure your MEDIA_URL and MEDIA_ROOT settings and have your webserver serve the MEDIA_ROOT directory in the MEDIA_URL path.
Basically adding that in the URL urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) will make the media files in the URL existing. Try removing that and visit your media file in the URL and you will receive a 404 not found. It's very similar to the concept that you are inserting a view for rendering the media files
I'm having trouble understanding the switchover from local to production settings for deploying Django projects. I'm using an Ubuntu virtual machine (VM) if it matters.
I understand how to configure my settings. I understand best practices for creating settings files (base.py, local.py, production.py, blah, blah). I know that in local development DEBUG=True, in production DEBUG=False, blah, blah.
But how do I implement this switchover in deployment? Do I get rid of the local.py? Do I create some kind of logic so that my VM only reads base.py and production.py?
What's the best approach?
I'm not sure about the best approach but what I do works...
At the foot of my settings.py, I have:
try:
from local_settings import *
except ImportError, e:
pass
I keep all my local development settings in local_settings.py which overide any production settings. I also make sure not to upload my local_settings.py file!
What you could do is to check in your settings py which environment is used at the moment.
To do this you can set an environment variable on your system that would have diffrent values on your development environment and the production environment.
you can set these environment variables by
sudo -H gedit /etc/environment
and add the following line in the file:
DEBUG="true"
(to make this changes available you will have to log out and log back in into your system)
in the production environment you would then set DEBUG="false".
then you can do this in your settings.py:
DEBUG = os.environ.get('DEBUG', 'true') != 'false'
and then you can set every setting that would be different depending on the environment that is used like this:
if DEBUG:
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
else:
STATICFILES_STORAGE = STATICFILES_STORAGE = 'custom_storages.StaticStorage'
(the setting above uses the local django server to serve staticfiles if it is in the development environment, and amazon s3 with boto if in the production environment (which is defined in the custom_storages module)
This way you can push your updates and always the right settings should be picked up depending on the environment.
How to change the default static file directory in django. currentyly i'm using this directory:-
MEDIA_ROOT = os.path.join(os.path.dirname(__file__), '..', 'static').replace('\\','/')
I want to change it in my directory in my server for example :-
/var/www/html/uploaded/
Sounds like your real question is how to use different settings in different environments (e.g. a DTAP architecture). There's an ongoing discussion within the django mailing list about settings if you're interested.
You could use a different python file in deployment, e.g.
settings.py
settings_deployment.py
Example of the latter's content
from settings import *
DEBUG = False
MEDIA_ROOT = "/var/www/static/"
Then be sure to load settings_deployment instead of settings on your server.
Another, more dynamic approach would be to overload settings based on an environment setting, its not very hard to fabricate something like that in plain python.
I'm using Django, django-storages with S3 (boto) in combination with easy-thumbnails. On my local machine, everything works as expected: if the thumbnail doesn't exists, it gets created and upload to S3 and saves in the easy-thumbnails database tables. But the problem is, when I push the code to my production server, it doesn't work, easy-thumbnails output an empty image SRC.
What I already noticed is that when I create the thumbnails on my local machine, the easy-thumbnail path uses backward slashes and my Linux server needs forwards slashes. If I change the slashes in the database, the thumbnails are showed on my Linux machine, but it is still not able to generate thumbnails on the Linux (production) machine.
The simple django-storages test fails:
>>> import django
>>> from django.core.files.storage import default_storage
>>> file = default_storage.open('storage_test', 'w')
Output:
django.core.exceptions.ImproperlyConfigured: Requested setting DEFAULT_FILE_STORAGE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
If I do:
>>> from base.settings import staging
>>> from django.conf import settings
>>> settings.configure(staging)
This works (I have a settings directory with 4 settings files: base.py, staging.py, development.py and production.py)
It seems that on my production server, the config file isn't loaded properly (however the rest of the website works fine). If I add THUMBNAIL_DEBUG = True to my settings file, but easy-thumbnails' debugging still doesn't work (it does work on my local machine).
What can be te problem? I've been debugging for 10+ hours already.
Try refactoring your settings to use a more object-oriented structure. A good example is outlined by [David Cramer from Disqus:
http://justcramer.com/2011/01/13/settings-in-django/
You'll put any server-specific settings in a local_settings.py file, and you can store a stripped-down version as example_local_settings.py within your repository.
You can still use separate settings files if you have a lot of settings specific to a staging or review server, but you wouldn't want to store complete database credentials in a code repo, so you'll have to customize the local_settings.py anyways. You can define which settings to include by adding imports at the top of local_settings.py:
from project.conf.settings.dev import *
Then, you can set your DJANGO_SETTINGS_MODULE to always point to the same place. This would be instead of calling settings.configure() as outlined in the Django docs:
https://docs.djangoproject.com/en/dev/topics/settings/#either-configure-or-django-settings-module-is-required
And that way, you know that your settings on your production server will definitely be imported, since local_settings.py is always imported.
first try to use:
python manage.py shell --settings=settings/staging
to load shell with correct settings file and then try to debug
For some reason, S3 and easy thumbnails in the templating language didn't seem to get along with each other ... some path problem which probably could be solved at some point.
My solution (read: workaround) was to move the thumbnail generation into the model inside the image field itseld, for example:
avatar = ThumbnailerImageField(upload_to = avatar_file_name, resize_source=dict(size=(125, 125), crop="smart"), blank = True)
For the sake of completeness:
def avatar_file_name(instance, filename):
path = "%s/avatar.%s" % (str(instance.user.username), filename.split('.')[1])
if os.path.exists(settings.MEDIA_ROOT + path):
os.remove(settings.MEDIA_ROOT + path)
return path
For the life of me, I have been looking for this everywhere and have not found the answer. I hope I am not posting a duplicate.
It is advised everywhere that you should keep your secret keys in a separate file from your general settings.py. Also, that you should never commit your "secret.py" file that contains keys such as SECRET_KEY, AWS_SECRET_KEY and so on.
My question is: In your production server, you need to reference your secret keys, that means that your "secret.py" settings file, should live somewhere around the server right? If so, how do you protect your secret keys in production?
I wanted to add a new answer because, as a beginner, the previous accepted answer didn't make a lot of sense to me (it was only one part of the puzzle).
So here's how I store my keys both LOCALLY and in PRODUCTION (Heroku, and others).
Note: You really only have to do this if you plan on putting your project online. If it's just a local project, no need.
I also made a video tutorial for people who prefer that format.
1) Install python-dotenv to create a local project environment to store your secret key.
pip install python-dotenv
2) Create a .env file in your base directory (where manage.py is).
YourDjangoProject
├───project
│ ├───__init__.py
│ ├───asgi.py
│ ├───settings.py
│ ├───urls.py
│ └───wsgi.py
├───.env
├───manage.py
└───db.sqlite3
If you have a Heroku project, it should look something like this:
YourDjangoProject
├───.git
├───project
│ ├───__init__.py
│ ├───asgi.py
│ ├───settings.py
│ ├───urls.py
│ └───wsgi.py
├───venv
├───.env
├───.gitignore
├───manage.py
├───Procfile
├───requirements.txt
└───runtime.txt
3) Add .env to your .gitignore file.
echo .env > .gitignore # Or just open your .gitignore and type in .env
This is how you keep your secret key more secure because you don't upload your .env file to git or heroku (or wherever else).
4) Add your SECRET_KEY from your settings.py file into the .env file like so (without quotes)
**Inside of your .env file**
SECRET_KEY=qolwvjicds5p53gvod1pyrz*%2uykjw&a^&c4moab!w=&16ou7 # <- Example key, SECRET_KEY=yoursecretkey
5) Inside of your settings.py file, add the following settings:
import os
import dotenv # <- New
# Add .env variables anywhere before SECRET_KEY
dotenv_file = os.path.join(BASE_DIR, ".env")
if os.path.isfile(dotenv_file):
dotenv.load_dotenv(dotenv_file)
# UPDATE secret key
SECRET_KEY = os.environ['SECRET_KEY'] # Instead of your actual secret key
or, thanks to #Ashkay Chandran's answer:
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
SECRET_KEY = os.environ['SECRET_KEY']
And now your secret key is successfully stored locally.
Update: I found out you can also use the config method from the package python-decouple that seems to be a bit easier:
from decouple import config
SECRET_KEY = config('SECRET_KEY')
Now you don't need to import os or use dotenv because it takes care of those parts for you AND will still use the .env file. I started using this in all of my projects.
6) Add the SECRET_KEY environment variable on your host (such as Heroku).
I work mostly with Heroku sites, so if you're wanting to use Heroku for a Django project, this part is for you.
This assumes that you already have a Heroku project setup and have Heroku CLI downloaded on your computer.
You have 2 options:
From Command Line / Terminal, you can enter the following command in your project's directory:
heroku config:set SECRET_KEY=yoursecretkey # Again, no quotes.
You can go to your Heroku dashboard, click on your app, go to your apps settings, and see the "Config Vars" section and click "Reveal Vars" or "Add Vars" and add your SECRET_KEY there.
Then, when you push your project to Heroku through git, it should be working properly without any issue.
and that's it! 🙂
This answer was targeted towards total beginners / intermediates to hopefully cut through any confusion (because it was definitely confusing for me).
See the Django deployment docs for a discussion on this.
There's quite a few options for production. The way I do it is by setting my sensitive data variables as environmental variables on the production environments. Then I retrieve the variables in the settings.py via os.environ like so:
import os
SECRET_KEY = os.environ['SECRET_KEY']
Another possible option is to copy in the secret.py file via your deploy script.
I'm sure there are also other specific options for different web servers.
You should store your settings in a modular way. By that I mean to spread your settings across multiple files.
For example, you can have base_settings.py to store all your base settings; dev_settings.py for your development server settings; and finally prod_base_settings.py for all production settings. All non-base settings files will import all the base settings and then only change whatever is necessary:
# base_settings.py
...
# dev_settings.py
from base_settings import *
DEBUG = TRUE
...
# prod_base_settings.py
from base_settings import *
DEBUG = FALSE
...
This approach allows you to have different settings from different setups. You can also commit all these files except then on the production server you can create the actual production settings file prod_settings.py where you will specify all the sensitive settings. This file should not be committed anywhere and its content kept secure:
# prod_settings.py
from prod_base_settings import *
SECRET_KEY = 'foo'
As for the file names you can use whatever filenames you feel are appropriate. Personally I actually create a Python package for the settings and then keep the various settings inside the package:
project/
project/
settings/
__init__.py
base.py
dev.py
...
app1/
models.py
...
app2/
models.py
...
Storing secrets in the environment still places them in the environment; which can be exploited if an unauthorized user gains access to the environment. It is a trivial effort to list environment variables, and naming one SECRET makes is all the more helpful and obvious to a bad actor an unwanted user.
Yet secrets are necessary in production, so how to access them while minimizing attack surface? Encrypt each secret in a file with a tool like git-secret, then allow authorized users to read in the file, as mentioned in django's docs. Then "tell" a non-root user the secret so it can be read-in during initialization.
(Alternatively, one could also use Hashicorp's Vault, and access the secrets stored in Vault via the HVAC python module.)
Once this non-root user is told, something like this is easy:
# Remember that './secret_key.txt' is encrypted until it's needed, and only read by a non-root user
with open('./secret_key.txt') as f:
SECRET_KEY = f.read().strip()
This isn't perfect, and, yes, an attacker could enumerate variables and access it -- but it's very difficult to do so during run-time, and Django does a good job of protecting its keys from such a threat vector.
This is a much safer approach than storing secrets in the environment.
I know it has been a long time, but I just opensourced a small Django app I am using to generate a new secret key if it does not exist yet. It is called django-generate-secret-key.
pip install django-generate-secret-key
Then, when provisioning / deploying a new server running my Django project, I run the following command (from Ansible):
python manage.py generate_secret_key
It simply:
checks if a secret key needs to be generated
generates it in a secretkey.txt file (can be customized)
All you need then is to have in your settings file:
with open('/path/to/the/secretkey.txt') as f:
SECRET_KEY = f.read().strip()
You can now benefit from a fully automated provisioning process without having to store a static secret key in your repository.
Instead of if/then logic you should use a tool designed for factoring out sensitive data. I use YamJam https://pypi.python.org/pypi/yamjam/ . It allows all the advantages of the os.environ method but is simpler -- you still have to set those environ variables, you'll need to put them in a script somewhere. YamJam stores these config settings in a machine config store and also allows a project by project ability to override.
from YamJam import yamjam
variable = yamjam()['myproject']['variable']
Is the basic usage. And like the os.environ method, it is not framework specific, you can use it with Django or any other app/framework. I've tried them all, multiple settings.py files, brittle logic of if/then and environment wrangling. In the end, I switched to yamjam and haven't regretted it.
Adding to zack-plauch's answer,
To get the path to the .env file, when using python-dotenv module, the find_dotenv method can be used,
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
SECRET_KEY = os.environ['SECRET_KEY']
The find_dotenv() looks for a ".env" file in the path, so it can be saved inside the same directory too,
Also, if a name is used for the .env file like "django-config.env", load_dotenv(find_dotenv("django-config.env"), will fetch and load that to host-machine environment variable mappings.
I am surprised that noone has talked about django-environ.
I usually create a .env file like this:
SECRET_KEY=blabla
OTHER_SECRET=blabla
This file should be added in .gitignore
You can checkin in git, an example file named .env.example just for others to know which env var they need. The content of .env.example file will look like this (just keys without any values)
SECRET_KEY=
OTHER_SECRETS=
Where to store SECRET_KEY DJANGO
Store your django SECRET_KEY in an environmental variable or separate file, instead of directly encoding In your configuration module settings.py
settings.py
#from an environment variable
import os
SECRET_KEY = os.environ.get('SECRET_KEY')
#from an file
with open('/etc/secret_key.txt') as f:
SECRET_KEY = f.read().strip()
How to generate Django SECRET_KEY manually:
$ python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
7^t+3on^bca+t7#)w%2pedaf0m&$_gnne#^s4zk3a%4uu5ly86
import string
import secrets
c = string.ascii_letters + string.digits + string.punctuation
secret_key = ''.join(secrets.choice(c) for i in range(67))
print(secret_key)
df&)ok{ZL^6Up$\y2*">LqHx:D,_f_of#P,~}n&\zs*:y{OTU4CueQNrMz1UH*mhocD
Make sure the key used in production is not used elsewhere and avoid sending it to source control.