My Django settings.py file contains the following configuration options:
# Caches
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.redis.RedisCache',
'LOCATION': 'redis://redis:6379',
}
}
# Queues
RQ_QUEUES = {
'default': {
'HOST': 'redis',
'PORT': 6379,
'DB': 0,
'DEFAULT_TIMEOUT': 360,
},
}
Both CACHES and RQ_QUEUES contain configuration details that point to a redis server.
Is it possible to reconfigure these settings to point to an instance of fakeredis instead ?
I have reviewed the fakeredis documentation and so far I have only seen examples where the redis connection is manually over-ridden, every time a call to redis is made.
It seems to me that when running tests, it would be much more convenient to simply point the Django CACHE location directly to fakeredis.
Is this possible?
Related
I am trying to connect AstraDB with my application served by gcloud AppEngine. I am using Django and therefore have used django_cassandra_engine. I want to keep Postgres as my default DB and use cassandra as a second DB.
Everything works as expected on localhost,but when I deploy to gcloud, I receive 502 Bad gateway error and in the logs it says:
cassandra.cqlengine.CQLEngineException: Connection name 'cassandra'
doesn't exist in the registry.
I am using:
Django==4.1
django-cassandra-engine==1.7.0
cassandra-driver==3.25.0
My secure_connect_bundle (ZIP file) is in the same folder where manage.py is located.
This is my settings.py:
# [START db_setup]
# [START gaestd_py_django_database_config]
# Use django-environ to parse the connection string
DATABASES = {
"default": env.db(),
'cassandra': {
'ENGINE': 'django_cassandra_engine',
'NAME': 'the_keyspace',
'TEST_NAME': 'test_db',
'OPTIONS': {
'connection': {
'auth_provider': PlainTextAuthProvider(username=env("ASTRA_CLIENT_ID"),password=env("ASTRA_SECRET")),
'cloud': {
'secure_connect_bundle': os.path.join(BASE_DIR, "secure-connect-db.zip")
}
}
}
}
}
# If the flag as been set, configure to use proxy
if os.getenv("USE_CLOUD_SQL_AUTH_PROXY", None):
DATABASES["default"]["HOST"] = "127.0.0.1"
DATABASES["default"]["PORT"] = 5432
# [END gaestd_py_django_database_config]
# [END db_setup]
# Use a in-memory sqlite3 database when testing in CI systems
# TODO(glasnt) CHECK IF THIS IS REQUIRED because we're setting a val above
if os.getenv("TRAMPOLINE_CI", None):
DATABASES = {
"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": os.path.join(BASE_DIR, "db.sqlite3"),
}
}
Just a thought, but I don't think you need the extra cassandra group in the DATABASES section. Try this:
DATABASES = {
"default": {
'ENGINE': 'django_cassandra_engine',
'NAME': 'brondau_keyspace',
We (DataStax) actually ran a workshop showing Django and Astra DB in December: https://www.youtube.com/watch?v=tUF_dX0lgt0
The Git repo can be found here: Build A Blog in Python with Django and Astra DB NoSQL Database
This section of the repo should help with your current issue: https://github.com/tomitokko/django-blog-with-astradb/blob/ddc1c2d608e8fc0d6569796775cdfaf503afab94/techblog/settings.py#L92
Anyway, give my suggestion a try and check the video and/or repo for additional help.
So the problem was that secure_connect_bundle (ZIP file) couldn't be extracted on AppEngine.
Setting 'use_default_tempdir' to True in cloud_config solves the issue.
If you are using python driver to establish connection, it should look like this:
cloud_config= {
'secure_connect_bundle': ASTRADB_CONNECT_BUNDLE,
'use_default_tempdir': True
}
Or if you use django_cassandra_engine:
'OPTIONS': {
'connection': {
'auth_provider': PlainTextAuthProvider(username=env("ASTRA_CLIENT_ID"),password=env("ASTRA_SECRET")),
'cloud': {
'secure_connect_bundle': 'secure-brondau-db.zip',
'use_default_tempdir': True
}
}
}
I'm trying to run the tests for my Django project. I wrote this project some time ago, I had different settings then and tests were passing. Now I changed settings and deployed it on Heroku with Heroku Postgres database. Everything works fine already except I can't run tests. I've tried many different settings and nothing worked. Most of the time I'm getting this error: permission denied to create database
My last setting is following the instruction from this article on medium
Basically I have added 2nd Heroku Postgres database, add settings like below (but with valid variables of my heroku databases):
if 'test' in sys.argv:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'd7osdssadag0ugv5',
'USER': 'lhwwasadqlgjra',
'PASSWORD': '1524f48a2ce41177c4ssdadasd3a11680b735302d14979d312ff36',
'HOST': 'ec2-54-75-2326-118.eu-west-1.compute.amazonaws.com',
'PORT': 5432,
'TEST': {
'NAME': 'd7osdssadag0ugv5',
}
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'd7hasadas9hqts5',
'USER': 'nybkduadsdgqot',
'PASSWORD': 'bb535b9cdsfsdfdsfdsfac54851f267444dd8cc230b2a786ab9f446',
'HOST': 'ec2-54-247-132-38.eu-west-1.compute.amazonaws.com',
'PORT': 5432,
'TEST': {
'NAME': 'd7hasadas9hqts5',
}
}
}
Then run python manage.py test --keepdb in my venv. Then I get an error:
RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the first PostgreSQL database instead.
RuntimeWarning
Got an error creating the test database: permission denied to create database
I have also tried what is advised in this article
Do you have any ideas what I could do about this error? I don't know Django well. I play with it from time to time.
I'm using: Python 3.6.9, Django 3.0.3, Heroku Postgresql Hobby Dev
EDIT:
I'm not sure if this is now an issue with my settings DATABASES.
Now when I commented out all my settings concerning DATABASES and I run python manage.py runserver my development server starts as normal and I have access to a database I set before (even after restarting a computer). This looks like actual settings don't have effect (??) Any thoughts?
Django version 3.0.3, using settings 'forumproject.settings'
Starting development server at http://127.0.0.1:8000/
Ok, I found out what it was. My database settings were not taken into account, even I had DEBUG=True because I had this line on the end of the settings:
# Activate Django-Heroku.
django_heroku.settings(locals())
After commenting this out the error permission denied to create database
goes away and I can run tests with
python manage.py test --keepdb
I'm surprised how always posting a question on stack overflow help me to find answer immediately after. I was running in circles
I am trying to run two identical Django projects on different databases, one for production using a certain port number (say, 80) and the other for testing using another port number (say, 8000). I also used Nginx and Gunicorn as the reverse proxy server and the application server, with Nginx listening to ports 80 and 8000 and forwarding to gunicorn of ports 8001 and 8002, respectively.
The problem is: how do I know the port number of the request in Django's settings.py so that the project can choose different databases?
The standard practice for doing this in django is to create a local_settings.py file
Put this at the top of the local_settings.py file:
try:
from settings import *
except ImportError:
pass
Now in local_settings.py you must override the following variable:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'dbname',
'USER': 'dbuser',
'PASSWORD': 'dbpassword',
'HOST': '127.0.0.1',
'PORT': '5432',
}
}
With different values for both projects
Then when running Django you need to set the following environment variable:
export DJANGO_SETTINGS_MODULE="appname.local_settings"
One way to bunch this all together is to create a run.sh file that first sets this variable, then runs gunicorn
To sum it up, settings.py is common between both projects, local_settings.py overrides those variables that are different between different projects
I have a Django website with Postgresql backend, for which I'm utilizing pgbouncer for db connection pooling (transaction mode).
The application and the DB reside on separate servers (1 server each). I have installed pgbouncer on the application server. My question is: what should the config be in settings.py? Note that I'm using Unix sockets for connecting to pgbouncer.
My current settings.py contains:
DATABASE_URL = 'postgres://user1:pass1#xx.xxx.xxx.xxx:5432/db1'
DATABASES = {
'default': dj_database_url.config(default=DATABASE_URL)
}
Relevant sections of pgbouncer.ini are:
[databases]
db1 = host=xx.xxx.xxx.xxx port=5432 dbname=db1
listen_addr = *
listen_port = 6432
auth_type = md5
unix_socket_dir = /var/run/postgresql
pool_mode = transaction
max_client_conn = 200
default_pool_size = 300
userlist.txt contains:
"user1" "pass1"
Note: One answer is here, but doesn't work for me since the DB isn't available locally in my case. I need to set the DATABASE_URL environment variable, instead of using default = '...'.
One suggestions seems to be to treat pgbouncer as a database in settings.py. In that case, would something like the following work?
if PRODUCTION == '1':
#PRODUCTION is set to '1' if in production environment
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'pgbouncer',
'USER': 'user1',
'PASSWORD': 'pass1',
'HOST': '/var/run/postgresql',
'PORT': '6432',
}
From the docs:
pgbouncer is a PostgreSQL connection pooler. Any target application
can be connected to pgbouncer as if it were a PostgreSQL server, and
pgbouncer will create a connection to the actual server, or it will
reuse one of its existing connections.
Also,
Have your application (or the psql client) connect to pgbouncer
instead of directly to PostgreSQL server.
The configurations:
pgbouncer.ini: An example pgbouncer.ini with comments about defaults
[databases]
db1 = host=xx.xxx.xxx.xxx port=5432 dbname=db1
[pgbouncer]
listen_addr = *
listen_port = 6432
auth_type = md5
auth_file = userlist.txt
unix_socket_dir = /var/run/postgresql
pool_mode = transaction
max_client_conn = 100
default_pool_size = 20
userlist.txt:
"user1" "pass1"
to put in settings.py:
if PRODUCTION == '1':
#PRODUCTION is set to '1' if in production environment
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db1',
'USER': 'user1',
'PASSWORD': 'pass1',
'HOST': '/var/run/postgresql',
# 'PORT': '6432',
}
Extra:
In case not using unix socket - you can set HOST : '127.0.0.1' or 'localhost' if pgbouncer is running locally, or whatever the IP of server pgbouncer will be running on.
From the docs:
If you’re using PostgreSQL, by default (empty HOST), the connection to
the database is done through UNIX domain sockets (‘local’ lines in
pg_hba.conf). If your UNIX domain socket is not in the standard
location, use the same value of unix_socket_directory from
postgresql.conf. If you want to connect through TCP sockets, set HOST
to ‘localhost’ or ‘127.0.0.1’ (‘host’ lines in pg_hba.conf). On
Windows, you should always define HOST, as UNIX domain sockets are not
available.
In case of postgreSQL For ENGINE you can use postgresql or postgresql_psycopg2 - there's difference between the both given your Django version - postgresql_psycopg2 vs posgresql.
All of your DB settings in settings.py should be identical to the settings in your pgbouncer config, except the host in settings.py will point to pgbouncer. You probably need to change 'NAME': 'pgbouncer' to 'NAME': 'db1'. Since you're using a unix socket the port shouldn't matter.
I've been stuck with this problem for a couple of days.
I developed an application for appengine using Django and I'd like to use Google Cloud SQL for my database. Everything works fine until I want to apply migrations on the development server when it fails with the following error:
django.db.utils.OperationalError: (1045, "Access denied for user 'MY_DB_USER'#'MY_IP' (using password: YES)")
What I've done is as follows:
I followed the instructions in the Django Support page to
develop my application.
In order to create a 1st generation Cloud SQL instance I followed
the steps outlined here, using the Cloud SDK.
I then created a new user following the instructions here and assigned it a password.
I deployed the application using the following command line:
gcloud preview app deploy MY-APP-DIR/app.yaml --version 0-1-0
I authorized my IP and my AppEngine Application ID. They are both listed in the ''Authorization'' section under ''Access Control'' in my SQL instance.
Finally, I tried to apply migrations using the following command line:
SETTINGS_MODE='prod' MY-APP-DIR/manage.py migrate
settings.py
The relevant portion of my settings.py looks as follows:
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine'):
DEBUG = False
# Running on production App Engine, so use a Google Cloud SQL database.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST' : '/cloudsql/[MY-PROYECT-ID]:[MY-CLOUD-SQL-INSTANCE]',
'NAME': '[MY-DB-NAME]',
'USER': 'root',
}
}
elif os.getenv('SETTINGS_MODE') == 'prod':
DEBUG = False
# Running in development, but want to access the Google Cloud SQL instance
# in production.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': '[MY-DB-NAME]', # db name.
'USER': '[MY-DB-USER]',
'PASSWORD' : '[MY-DB-USER-PASSWORD]',
'HOST' : '[IPV4 ASSIGNED IN GOOGLE CONSOLE]',
'PORT': '3306',
}
}
else:
# Running in development, so use a local MySQL database.
DEBUG = True
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': '[MY-LOCAL-DB]',
'USER': 'root',
'PASSWORD': 'root',
}
}
Any idea as to what might be causing the problem?
Thank you!
I finally figured out what was the problem.
The proper way to grant a user database access, in order to apply migrations, is the following:
White list your IP. It should be shown under 'Authorized Networks'
Create a new database user account, but do not choose the 'Allow any host (%)' wildcard, instead select the "Restrict host by name, address, or address range" option and assign your IP ( The one you just whitelisted ).
You should now be able to run migrations with the command: SETTINGS_MODE='prod' PROJECT_DIR/manage.py migrate
As a side note, make sure the root user whose host is localhost doesn't have a password, else your App Engine application won't be able to connect to the database.
Hope this helps someone else!