I devoted today to migrating my app from my local environment to Heroku. It's been frustrating and fun at the same time, but now I'm very stuck.
I have a schema called DCPViews which I want the DB connections to default to when running queries. I've read all the relevant tutorials / posts / tips and here's where I am:
DB hierarchy
postgres (default system database)
-> DCP (app database)
-> DCP (base tables schema)
-> DCPViews (views layer schema)
settings.py
import django_heroku
import dj_database_url
...
DATABASES = {}
# DATABASE_URL = 'postgres://<user>:<pass>#<host>:<port>/<db_name>?currentSchema=<schema>'
DATABASE_URL = 'postgres://' + \
config('DB_USER') + ':' + \
config('DB_PASSWORD') + '#' + \
config('DB_HOST') + ':' + \
config('DB_PORT') + '/' + \
config('DB_NAME') + \
'?currentSchema=' + config('DB_SCHEMA_NAME')
DATABASES['default'] = dj_database_url.config(default=DATABASE_URL, ssl_require=True)
...
# Configure Django App for Heroku
django_heroku.settings(locals())
The problem
I've tried everything to get Heroku to use the correct DATABASE_URL (with currentSchema = DCPViews), but no luck. I don't have DB permissions to create new roles nor set a search path for Heroku's default DB user. It also won't let me export DATABASE_URL manually and doesn't seem to accept the value I pass in the settings.py file. The heroku config -s command always returns the same DATABASE_URL value.
Everything works fine in my local environment, but this is a major snag. How can I get Heroku to use the correct search_path or default to the DCPViews schema when running queries?
Update
I'm send send the DB credentials and currentSchema I want to use in my settings.py file, but Heroku seems to ignore them and overwrites them when it creates the DATABASES['default'] key. Here's the Django debug when I load a page in my browser:
DATABASES
{'default': {'ATOMIC_REQUESTS': False,
'AUTOCOMMIT': True,
'CONN_MAX_AGE': 600,
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': '<host_name>',
'NAME': '<db_name>',
'OPTIONS': {'sslmode': 'require'},
'PASSWORD': '********************',
'PORT': 5432,
'TEST': {'CHARSET': None,
'COLLATION': None,
'MIRROR': None,
'NAME': None},
'TIME_ZONE': None,
'USER': '<user_name>'}}
DATABASE_URL
'postgres://<user>:<pass>#<host>:<port>/<db_name>?currentSchema=<schema>'
I cannot comment yet, but since your settings file is using an environment file, you can just go to your heroku app in heroku.com and go to your database that is connected to your app. Then go to settings and click reveal config vars. Change your DATABASE_URL to whatever your need it to be (mind you it wil probably be just an extension off of your actual heroku postgres database url)
Well, I was on the right path. I added the search_path to the ROLE and that did the trick:
ALTER ROLE <role_name> SET search_path = <go, pirates>
Related
Well, I had one setting file before and it was working perfectly both in local and AWS of course with both the database configuration in one setting file. When ever I use to push my code on AWS I would comment my local db configuration and then push. So for sure, that's annoying. I can't comment and uncomment constantly both the database configuration in one file.
Therefore, I decided to have 2 setting files. one for the local and the other for AWS.
Once I pulled the code into AWS server and run migrations
python manage.py migrate --settings=settings.staging
It worked and migrated. By the way, this is the setting file which resides my RDS configuration. Now the moment I hit the endpoints via postmant the output is as
OperationalError at /account/v1/login
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
means it is still reading the default settings file. How come I make the server run this particular staging setting file. Do I have to declare it on nginx, supervisor or gunicorn?
I'm using these 3 services as well.
Below is my settings file for staging.
from .base import *
# --------------- AWS RDS ---------------
DATABASES = {
'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db name here',
'HOST': 'RDS host here',
'USER': 'postgres',
'PASSWORD': 'pass here',
'PORT': '5432'
}
}
This is my complete staging.py file. which only have this while the rest of the setting is being imported from base.py which is a default setting file. Also it has the local settings.
Any recommendations?
This is also what I've tried.
import socket
hostname = socket.gethostname()
if hostname == "staging":
from settings.staging import *
Thank you
The way I solved this issue is using hostnames. We have 1 settings file with all the default settings. We then import specific files depending on the hostname. Of course you could also use things like IAM-instance roles or something.
We would have this in the default settings file:
import socket
DATABASE = {'default': {'ENGINE': 'django.db.backends.sqlite3', ...}}
hostname = socket.gethostname()
if hostname == "staging-blabla"
from staging import *
staging.py would contain the following:
DATABASE = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', ...}}
From the docs, it looks like the database needs to be present in the settings.py. Is there a way to pass the database details while running python manage.py runserver given the migrations have been completed?
Edit -
Use case - The database is not known beforehand so I can't have it hardcoded in settings.py. Also, there will be one database for the entire app.
Using a settings file other than settings.py in Django - This questions answers how to use a settings file different than settings.py. However, I am looking for ways in which I don't use settings file to specify the database credentials.
split your settings.py into two files
base_settings.py (add all the common settings here)
development.py
(add your database settings here)
like :
from .base_settings import *
DATABASES = {
'default': {
'ENGINE': 'XX.db.backends.postgresql',
'NAME': 'XX',
'USER': 'postgres',
'ATOMIC_REQUESTS':True,
'PASSWORD': '*****',
'HOST': '0.0.0.0',
'PORT': '5432',
}
}
and run your project using
python manage.py runserver 0.0.0.0:8002 --settings=django_project_name.development
I have been struggling for this issue for the whole days while no solutions at all. so I post it here.
I am trying to set up a blog website in Heroku via Django 1.8 which uses Python 3.4.3. I follows the instructions from Heroku website here.
I use "foreman start" to run Django project in my Mac and I already installed all dependence.
Part of my setting.py file involving the database initially looks like:
import dj_database_url
DATABASES = {}
DATABASES['default'] = dj_database_url.config()
Then I got error: ImproperlyConfigured at /settings.DATABASES is improperly configured. Please supply the ENGINE value.
Then I modify the files by adding one line supplying the ENGINE value:
import dj_database_url
DATABASES = {}
DATABASES['default'] = dj_database_url.config()
DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
Based on this post answered by Or Arbel, it should work. But I got another error: ImproperlyConfigured at /settings.DATABASES is improperly configured. Please supply the NAME value.
What should I do next? Actually my Django project is very simple and does not involve any database operations(may need in the future). I just want to make it works on Heroku. Thanks!
Do I need to create a database to continue? I just want to make the webpage works.
Thanks for your guys help, specially souldeux.
Update:
I have fixed the issue by using souldeux's method by providing more informations about the database. Here I want to emphasis that it seems the code from the original Heroku tutorial does not work for Django 1.8:
import dj_database_url ####not working for my case
DATABASES = {}
DATABASES['default'] = dj_database_url.config()
Initially I did not create a database because I think it is not necessary for simple projects, based on my understanding obtained from Heroku tutorial. Actually it does need to create a database in Heroku to make it works. The tutorial is here. You need run "heroku config -s | grep HEROKU_POSTGRESQL" to get the database information. The format is like:
scheme://username:password#host:port/database
So you can get 'database', 'username', 'password', etc.
Afterwards, modify the 'settings.py' according to souldeux, then run following codes:
git add .
git commit -m "Ready to go to Heroku"
git push heroku master
heroku run python manage.py syncdb
Now it works. But other issues arise like my webpages do not show images... Anyway it solved. Please confirm my solutions, thanks.
I think you need to add more information to your database definition. For instance, here's what my own database entry looks like in my settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'django',
'USER': 'redacted',
'PASSWORD': 'redacted',
'HOST': '127.0.0.1',
'PORT': '5432',
}
}
If you don't have a database user and password to enter in the fields marked redacted then you need to make sure you have the actual database created and psycopg2 installed.
I put a Django app on Heroku recently. The home page looks fine, but when I try to go to a page that involves making a query (e.g. p = Photo.objects.get(title=title)), I get this error:
could not connect to server: Connection refused
Is the server running on host "localhost" and accepting
TCP/IP connections on port 5432?
In accordance with this answer, I did $ heroku pg:promote HEROKU_POSTGRESQL_GREEN_URL
Then in my settings.py:
DATABASES = {'default': dj_database_url.config(default=os.environ['DATABASE_URL'])}
Still got the same error, so I tried looking at the results of this (as this answer suggests):
$ heroku run python manage.py shell
>>> from django.conf import settings
>>> print settings.DATABASES['default']
{'TIME_ZONE': 'UTC', 'TEST_MIRROR': None, 'NAME': 'snorthway', 'OPTIONS': {},
'HOST': 'localhost', 'TEST_NAME': None, 'PASSWORD': '******', 'ENGINE':
'django.db.backends.postgresql_psycopg2', 'PORT': '', 'USER': 'snorthway',
'TEST_COLLATION': None, 'TEST_CHARSET': None}
At which point I realized I don't know what I should even be looking for in that. I still don't understand what the error means, so I am unsure how to go about debugging it.
You have not configured your django database correctly in settings.py. It thinks your database is on localhost. Sounds like you have a heroku postgres database so your host should be something like:
df3-64-304-50-250.compute-1.amazonaws.com
Heroku exposes a special database URL through an environment variable called:
DATABASE_URL
There is a very cool python package here called dj_database_url: https://github.com/kennethreitz/dj-database-url it converts that environment variable to what django expects.
you can install it with:
$pip install dj-database-url
I use the following in my settings.py
import dj_database_url
DATABASES = {
'default': dj_database_url.config()
}
I'm trying to get my local dev django app to work after following these instructions on adding env database settings.
https://devcenter.heroku.com/articles/django-injection
I followed the instructions but get the following error when my app tries to access the local database
Request Method: GET
Request URL: http://localhost:8000
Django Version: 1.4
Exception Type: ImproperlyConfigured
Exception Value:
You need to specify NAME in your Django settings file.
My database settings originally,
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db', # Or path to database file if using sqlite3.
'USER': 'foo', # Not used with sqlite3.
'PASSWORD': 'bar', # Not used with sqlite3.
'HOST': 'localhost',
'PORT': '5432',
}
}
the heroku article says to add the following to the settings file
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
how do I get dj_database_url.config to use my my dev settings when the DATABASE_URL is not available in dev?
You can just add your dev settings to the default values like this...
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://foo:bar#localhost:5432/db')}
Use this in your settings.py:
DATABASES = {'default': dj_database_url.config(default=os.environ['DATABASE_URL'])}
and in your .env file have this:
DATABASE_URL=postgres://localhost/yourdbname
when you launch with "foreman start" it will look at the .env file and create all those environment variables, just like running on Heroku itself. Type "heroku config" to confirm that you have a DATABASE_URL set, which you should if you added the postgres database addon.
Just set an environment variable on your operating system and check weither or not it's set. For instance, with a UNIX system:
# In ~/.bash_profile
export LOCAL_DEV=true
# In settings.py
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
if bool(os.environ.get('LOCAL_DEV', False)):
# Override DATABASES['default'] with your local database configuration
Also, if you need to set an environment variable on your heroku space:
heroku config:add MY_VAR='my_value'
I just tried this and here is my code:
import dj_database_url
local_db = 'postgres://django_login:123456#localhost/django_db'
DATABASES = {'default': dj_database_url.config(default=local_db)}
My database name is "django_db", user name is "django_login", password is "123456".
My code can run both in local machine and heroku.
import dj_database_url
DATABASES = {'default':
dj_database_url.config(default='postgres://yourusername:yourpassword#yourhosturl:5432/yourdbname')}
** Replace bold string with your database settings
if you are using local database then replace yourhosturl by localhost