I got a file .env with 4 values to hide sensitive data:
DATABASE_PASSWD=Password1
SECRET_KEY=Password2
VAR3=Password3
VAR4=Password4
All of above values are properly imported in Django code except the DATABASE_PASSWORD. When the DATABASES configuration is as follows:
# settings.py
from decouple import config
# ...
DB_PASSWORD=config('DATABASE_PASSWD')
SECRET_KEY=config('SECRET_KEY')
VAR3=config('VAR3')
VAR4=config('VAR4')
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': "database_name",
'USER': "database_test_admin",
'PASSWORD': DB_PASSWORD,
'HOST': "localhost",
'PORT': "5432",
}
}
The django outputs:
raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: DATABASE_PASSWD not found. Declare it as envvar or define a default value.
If I hardcode password that is just the same in .env the problem is gone - the password is correct since it's my private project. Other variables works well with the same config('VAR#') function in views for example. I have no clue what could be wrong here.
SOLUTION:
The official docs pushed me to try this:
from decouple import config, AutoConfig # <-- AutoConfig added
# ...
config = AutoConfig(search_path='.env') # <-- .env file located next to manage.py
SECRET_KEY = config('SECRET_KEY')
DB_PASSWORD = config('DB_PASSWORD')
VAR3 = config('VAR3')
VAR4 = config('VAR4')
When I provided empty string by the default value the code started to work, however no real string was passed by config() - just the default, which didi not solve anything but made django server running.
Related
Well, I had one setting file before and it was working perfectly both in local and AWS of course with both the database configuration in one setting file. When ever I use to push my code on AWS I would comment my local db configuration and then push. So for sure, that's annoying. I can't comment and uncomment constantly both the database configuration in one file.
Therefore, I decided to have 2 setting files. one for the local and the other for AWS.
Once I pulled the code into AWS server and run migrations
python manage.py migrate --settings=settings.staging
It worked and migrated. By the way, this is the setting file which resides my RDS configuration. Now the moment I hit the endpoints via postmant the output is as
OperationalError at /account/v1/login
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
means it is still reading the default settings file. How come I make the server run this particular staging setting file. Do I have to declare it on nginx, supervisor or gunicorn?
I'm using these 3 services as well.
Below is my settings file for staging.
from .base import *
# --------------- AWS RDS ---------------
DATABASES = {
'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db name here',
'HOST': 'RDS host here',
'USER': 'postgres',
'PASSWORD': 'pass here',
'PORT': '5432'
}
}
This is my complete staging.py file. which only have this while the rest of the setting is being imported from base.py which is a default setting file. Also it has the local settings.
Any recommendations?
This is also what I've tried.
import socket
hostname = socket.gethostname()
if hostname == "staging":
from settings.staging import *
Thank you
The way I solved this issue is using hostnames. We have 1 settings file with all the default settings. We then import specific files depending on the hostname. Of course you could also use things like IAM-instance roles or something.
We would have this in the default settings file:
import socket
DATABASE = {'default': {'ENGINE': 'django.db.backends.sqlite3', ...}}
hostname = socket.gethostname()
if hostname == "staging-blabla"
from staging import *
staging.py would contain the following:
DATABASE = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', ...}}
From the docs, it looks like the database needs to be present in the settings.py. Is there a way to pass the database details while running python manage.py runserver given the migrations have been completed?
Edit -
Use case - The database is not known beforehand so I can't have it hardcoded in settings.py. Also, there will be one database for the entire app.
Using a settings file other than settings.py in Django - This questions answers how to use a settings file different than settings.py. However, I am looking for ways in which I don't use settings file to specify the database credentials.
split your settings.py into two files
base_settings.py (add all the common settings here)
development.py
(add your database settings here)
like :
from .base_settings import *
DATABASES = {
'default': {
'ENGINE': 'XX.db.backends.postgresql',
'NAME': 'XX',
'USER': 'postgres',
'ATOMIC_REQUESTS':True,
'PASSWORD': '*****',
'HOST': '0.0.0.0',
'PORT': '5432',
}
}
and run your project using
python manage.py runserver 0.0.0.0:8002 --settings=django_project_name.development
I devoted today to migrating my app from my local environment to Heroku. It's been frustrating and fun at the same time, but now I'm very stuck.
I have a schema called DCPViews which I want the DB connections to default to when running queries. I've read all the relevant tutorials / posts / tips and here's where I am:
DB hierarchy
postgres (default system database)
-> DCP (app database)
-> DCP (base tables schema)
-> DCPViews (views layer schema)
settings.py
import django_heroku
import dj_database_url
...
DATABASES = {}
# DATABASE_URL = 'postgres://<user>:<pass>#<host>:<port>/<db_name>?currentSchema=<schema>'
DATABASE_URL = 'postgres://' + \
config('DB_USER') + ':' + \
config('DB_PASSWORD') + '#' + \
config('DB_HOST') + ':' + \
config('DB_PORT') + '/' + \
config('DB_NAME') + \
'?currentSchema=' + config('DB_SCHEMA_NAME')
DATABASES['default'] = dj_database_url.config(default=DATABASE_URL, ssl_require=True)
...
# Configure Django App for Heroku
django_heroku.settings(locals())
The problem
I've tried everything to get Heroku to use the correct DATABASE_URL (with currentSchema = DCPViews), but no luck. I don't have DB permissions to create new roles nor set a search path for Heroku's default DB user. It also won't let me export DATABASE_URL manually and doesn't seem to accept the value I pass in the settings.py file. The heroku config -s command always returns the same DATABASE_URL value.
Everything works fine in my local environment, but this is a major snag. How can I get Heroku to use the correct search_path or default to the DCPViews schema when running queries?
Update
I'm send send the DB credentials and currentSchema I want to use in my settings.py file, but Heroku seems to ignore them and overwrites them when it creates the DATABASES['default'] key. Here's the Django debug when I load a page in my browser:
DATABASES
{'default': {'ATOMIC_REQUESTS': False,
'AUTOCOMMIT': True,
'CONN_MAX_AGE': 600,
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': '<host_name>',
'NAME': '<db_name>',
'OPTIONS': {'sslmode': 'require'},
'PASSWORD': '********************',
'PORT': 5432,
'TEST': {'CHARSET': None,
'COLLATION': None,
'MIRROR': None,
'NAME': None},
'TIME_ZONE': None,
'USER': '<user_name>'}}
DATABASE_URL
'postgres://<user>:<pass>#<host>:<port>/<db_name>?currentSchema=<schema>'
I cannot comment yet, but since your settings file is using an environment file, you can just go to your heroku app in heroku.com and go to your database that is connected to your app. Then go to settings and click reveal config vars. Change your DATABASE_URL to whatever your need it to be (mind you it wil probably be just an extension off of your actual heroku postgres database url)
Well, I was on the right path. I added the search_path to the ROLE and that did the trick:
ALTER ROLE <role_name> SET search_path = <go, pirates>
I am using django-nose to run my unit tests in django (1.4).
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
Creating the database takes a long time.
So I found out putting this in settings.py:
os.environ['REUSE_DB'] = "1"
should do the trick.
actually django itsellve gives this suggestion:
To reuse old database "<path not very interesting>/var/sqlite/unittest.db" for speed, set env var REUSE_DB=1.
of course you need to run it once (or after every database change) with this flag =0
However, when you set the flag to 0, my tests end with the remark:
Destroying test database for alias 'default'...
So when I want to run it with reuse.... there is nothing to reuse... and I will get errors saying the table does not exist
DatabaseError: no such table: <and than a table name>
The test runs perfectly when set the reuse_db to 0
I am using the test database alias in my development settings:
DATABASES = {
'default': {
'NAME': os.path.join(BUILDOUT_DIR, 'var', 'sqlite', 'development.db'),
'TEST_NAME': os.path.join(BUILDOUT_DIR, 'var', 'sqlite', 'unittest.db'),
'ENGINE': 'django.db.backends.sqlite3',
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': '',
}
}
I am not using the in-memory sqllite database for testing because I read somewhere this doesn't work well with django-nose.
So how can I reuse the DB when it is destroying the databse in the end...
according to this https://docs.djangoproject.com/en/1.4/topics/testing/#the-test-database django is doing this, but it does not show how to prevent this (if I can), or how to use the reuse_db option.
should I use other settings?
If I have understood correctly, you don't know how to create the test database first time (in order to reuse it later).
NoseTestSuiteRunner should create it automatically if DB does not exist even if you set REUSE_DB = 0.
If you want to create test DB manually you can create the following file:
test_db_settings.py
in which you specify:
from settings import *
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
# TestRunner uses a database with name that consists
# of prefix test_ and your database name if your database
# name is db then the test database name has to be test_db
'NAME': 'test_db',
'USER': 'postgres_user',
'PASSWORD': 'postgres_user',
'HOST': 'localhost',
'PORT': '5432',
}
}
after that create test_db:
createdb -U postgres_user -h localhost test_db # if you use postgres
python manage.py syncdb --settings test_db_settings.py
python manage.py migrate --settings test_db_settings.py (only if you use South)
Now we have DB that is used by TestRunner. We may run test:
REUSE_DB=1 python manage.py test
Updated
Are you sure that you use NoseTestSuiteRunner? Here is some code from django_nose.NoseTestSuiteRunner. As we can see if option REUSE_DB is set then teardown_database is disabled. If you want you can debug it for example set here a breakpoint to check you really use its Runner etc.
def teardown_databases(self, *args, **kwargs):
"""Leave those poor, reusable databases alone if REUSE_DB is true."""
if not _reusing_db():
return super(NoseTestSuiteRunner, self).teardown_databases(
*args, **kwargs)
# else skip tearing down the DB so we can reuse it next time
I accepted the answer of Andrey Kaygorodov, because he lead me to the solution... reading his answer I thought (how easy, and how stupid of me...)
Anyway, putting this settings_test_db.py next to the settings with this content:
(filling in the name of your project)
from <myproject>.settings import *
DATABASES = {
'default': {
'NAME': os.path.join(BUILDOUT_DIR, 'var', 'sqlite', 'unittest.db'),
'ENGINE': 'django.db.backends.sqlite3',
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': '',
}
}
And for sqlite you do not have to create a db.
just run
python manage.py syncdb --settings <myproject>.settings_test_db
(note: use the projectname, and without .py)
and to run the migrations
python manage.py migrate --settings <myproject>.settings_test_db
(only if you use South)
and set
in your settings.py:
os.environ['REUSE_DB'] = "1"
so you can use
python manage.py test
by the way, in stead of using a separate test settings file...
you can define a test anme in the same settings:
DATABASES = {
'default': {
'NAME': os.path.join(BUILDOUT_DIR, 'var', 'sqlite', 'test.db'),
'TEST_NAME': os.path.join(BUILDOUT_DIR, 'var', 'sqlite', 'unittest.db'),
'ENGINE': 'django.db.backends.sqlite3',
'USER': '',
'PASSWORD': '',
'HOST': '', # empty string for localhost.
'PORT': '', # empty string for default.
}
}
I'm trying to get my local dev django app to work after following these instructions on adding env database settings.
https://devcenter.heroku.com/articles/django-injection
I followed the instructions but get the following error when my app tries to access the local database
Request Method: GET
Request URL: http://localhost:8000
Django Version: 1.4
Exception Type: ImproperlyConfigured
Exception Value:
You need to specify NAME in your Django settings file.
My database settings originally,
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db', # Or path to database file if using sqlite3.
'USER': 'foo', # Not used with sqlite3.
'PASSWORD': 'bar', # Not used with sqlite3.
'HOST': 'localhost',
'PORT': '5432',
}
}
the heroku article says to add the following to the settings file
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
how do I get dj_database_url.config to use my my dev settings when the DATABASE_URL is not available in dev?
You can just add your dev settings to the default values like this...
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://foo:bar#localhost:5432/db')}
Use this in your settings.py:
DATABASES = {'default': dj_database_url.config(default=os.environ['DATABASE_URL'])}
and in your .env file have this:
DATABASE_URL=postgres://localhost/yourdbname
when you launch with "foreman start" it will look at the .env file and create all those environment variables, just like running on Heroku itself. Type "heroku config" to confirm that you have a DATABASE_URL set, which you should if you added the postgres database addon.
Just set an environment variable on your operating system and check weither or not it's set. For instance, with a UNIX system:
# In ~/.bash_profile
export LOCAL_DEV=true
# In settings.py
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
if bool(os.environ.get('LOCAL_DEV', False)):
# Override DATABASES['default'] with your local database configuration
Also, if you need to set an environment variable on your heroku space:
heroku config:add MY_VAR='my_value'
I just tried this and here is my code:
import dj_database_url
local_db = 'postgres://django_login:123456#localhost/django_db'
DATABASES = {'default': dj_database_url.config(default=local_db)}
My database name is "django_db", user name is "django_login", password is "123456".
My code can run both in local machine and heroku.
import dj_database_url
DATABASES = {'default':
dj_database_url.config(default='postgres://yourusername:yourpassword#yourhosturl:5432/yourdbname')}
** Replace bold string with your database settings
if you are using local database then replace yourhosturl by localhost