Can't connect DataStax AstraDB with Django app on Google Cloud - django

I am trying to connect AstraDB with my application served by gcloud AppEngine. I am using Django and therefore have used django_cassandra_engine. I want to keep Postgres as my default DB and use cassandra as a second DB.
Everything works as expected on localhost,but when I deploy to gcloud, I receive 502 Bad gateway error and in the logs it says:
cassandra.cqlengine.CQLEngineException: Connection name 'cassandra'
doesn't exist in the registry.
I am using:
Django==4.1
django-cassandra-engine==1.7.0
cassandra-driver==3.25.0
My secure_connect_bundle (ZIP file) is in the same folder where manage.py is located.
This is my settings.py:
# [START db_setup]
# [START gaestd_py_django_database_config]
# Use django-environ to parse the connection string
DATABASES = {
"default": env.db(),
'cassandra': {
'ENGINE': 'django_cassandra_engine',
'NAME': 'the_keyspace',
'TEST_NAME': 'test_db',
'OPTIONS': {
'connection': {
'auth_provider': PlainTextAuthProvider(username=env("ASTRA_CLIENT_ID"),password=env("ASTRA_SECRET")),
'cloud': {
'secure_connect_bundle': os.path.join(BASE_DIR, "secure-connect-db.zip")
}
}
}
}
}
# If the flag as been set, configure to use proxy
if os.getenv("USE_CLOUD_SQL_AUTH_PROXY", None):
DATABASES["default"]["HOST"] = "127.0.0.1"
DATABASES["default"]["PORT"] = 5432
# [END gaestd_py_django_database_config]
# [END db_setup]
# Use a in-memory sqlite3 database when testing in CI systems
# TODO(glasnt) CHECK IF THIS IS REQUIRED because we're setting a val above
if os.getenv("TRAMPOLINE_CI", None):
DATABASES = {
"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": os.path.join(BASE_DIR, "db.sqlite3"),
}
}

Just a thought, but I don't think you need the extra cassandra group in the DATABASES section. Try this:
DATABASES = {
"default": {
'ENGINE': 'django_cassandra_engine',
'NAME': 'brondau_keyspace',
We (DataStax) actually ran a workshop showing Django and Astra DB in December: https://www.youtube.com/watch?v=tUF_dX0lgt0
The Git repo can be found here: Build A Blog in Python with Django and Astra DB NoSQL Database
This section of the repo should help with your current issue: https://github.com/tomitokko/django-blog-with-astradb/blob/ddc1c2d608e8fc0d6569796775cdfaf503afab94/techblog/settings.py#L92
Anyway, give my suggestion a try and check the video and/or repo for additional help.

So the problem was that secure_connect_bundle (ZIP file) couldn't be extracted on AppEngine.
Setting 'use_default_tempdir' to True in cloud_config solves the issue.
If you are using python driver to establish connection, it should look like this:
cloud_config= {
'secure_connect_bundle': ASTRADB_CONNECT_BUNDLE,
'use_default_tempdir': True
}
Or if you use django_cassandra_engine:
'OPTIONS': {
'connection': {
'auth_provider': PlainTextAuthProvider(username=env("ASTRA_CLIENT_ID"),password=env("ASTRA_SECRET")),
'cloud': {
'secure_connect_bundle': 'secure-brondau-db.zip',
'use_default_tempdir': True
}
}
}

Related

Using fakeredis in a Django development settings file?

My Django settings.py file contains the following configuration options:
# Caches
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.redis.RedisCache',
'LOCATION': 'redis://redis:6379',
}
}
# Queues
RQ_QUEUES = {
'default': {
'HOST': 'redis',
'PORT': 6379,
'DB': 0,
'DEFAULT_TIMEOUT': 360,
},
}
Both CACHES and RQ_QUEUES contain configuration details that point to a redis server.
Is it possible to reconfigure these settings to point to an instance of fakeredis instead ?
I have reviewed the fakeredis documentation and so far I have only seen examples where the redis connection is manually over-ridden, every time a call to redis is made.
It seems to me that when running tests, it would be much more convenient to simply point the Django CACHE location directly to fakeredis.
Is this possible?

post request and url not working on google app engine django project

I am hosting my blogging app on google app engine in a standard environment. the website deploys successfully. but when I try to go to any other URL like anirudhmalik.in to anirudhmalik/blog/list/ it shows me 500 server error on the server side I am using PostgreSQL instance and the logs message only show me the error 
projects/anirudhmalik-274008/logs/appengine.googleapis.com%2Frequest_log" 
but I couldn't get any hint from this message and none of the post requests isworking on the website and everything is working fine in a local server or my local machine 
you can check the website on anirudhmalik.in 
and you can also give suggestion is google app engine is good to host your Django project and other cheap college student type hosting service  thankyou 
main.py 
from annyportfolio.wsgi import application
app=application
app.yaml file
 
handlers:# This configures Google App Engine to serve the files in the app's static# directory.- - url: /static
static_dir: static-storage/
# This handler routes all requests not caught above to your main app. It is# required when static routes are defined, but can be omitted (along with# the entire handlers section) when there are no static files defined.
- url: /.*
secure: always redirect_http_response_code: 301
script: auto
env_variables:
DJANGO_SETTINGS_MODULE: annyportfolio.settings ```
I am coonecting to database by using the below code
if os.getenv('GAE_APPLICATION', None):
# Running on production App Engine, so connect to Google Cloud SQL using
# the unix socket at /cloudsql/<your-cloudsql-connection string>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST':'<instance address>',
'NAME': '<server name >',
'USER': '<server user>',
'PASSWORD': '<password of user>',
}
}
I can see in your code some issues, you need to put the name of the schema (not the server name) on the property 'name' of your default dictionary and the 'host' property need to follow the Cloud SQL connection string pattern, App Engine Standard not use the public IP to reach your PostgreSQL instance
For example:
# Running on production App Engine, so connect to Google Cloud SQL using
# the unix socket at /cloudsql/<your-cloudsql-connection string>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST':'/cloudsql/my-project:us-central1:my-sql-instance',
'NAME': 'mydjangoschema',
'USER': '<server user>',
'PASSWORD': '<password of user>',
}
To create the correct host string please follow this steps
Run this gcloud sdk command
gcloud sql instances describe [YOUR_INSTANCE_NAME]
In the output, note the value shown for [CONNECTION_NAME].
the host string must follow this pattern /cloudsql/[CONNECTION_NAME] in my example is /cloudsql/my-project:us-central1:my-sql-instance

Django settings when using pgbouncer

I have a Django website with Postgresql backend, for which I'm utilizing pgbouncer for db connection pooling (transaction mode).
The application and the DB reside on separate servers (1 server each). I have installed pgbouncer on the application server. My question is: what should the config be in settings.py? Note that I'm using Unix sockets for connecting to pgbouncer.
My current settings.py contains:
DATABASE_URL = 'postgres://user1:pass1#xx.xxx.xxx.xxx:5432/db1'
DATABASES = {
'default': dj_database_url.config(default=DATABASE_URL)
}
Relevant sections of pgbouncer.ini are:
[databases]
db1 = host=xx.xxx.xxx.xxx port=5432 dbname=db1
listen_addr = *
listen_port = 6432
auth_type = md5
unix_socket_dir = /var/run/postgresql
pool_mode = transaction
max_client_conn = 200
default_pool_size = 300
userlist.txt contains:
"user1" "pass1"
Note: One answer is here, but doesn't work for me since the DB isn't available locally in my case. I need to set the DATABASE_URL environment variable, instead of using default = '...'.
One suggestions seems to be to treat pgbouncer as a database in settings.py. In that case, would something like the following work?
if PRODUCTION == '1':
#PRODUCTION is set to '1' if in production environment
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'pgbouncer',
'USER': 'user1',
'PASSWORD': 'pass1',
'HOST': '/var/run/postgresql',
'PORT': '6432',
}
From the docs:
pgbouncer is a PostgreSQL connection pooler. Any target application
can be connected to pgbouncer as if it were a PostgreSQL server, and
pgbouncer will create a connection to the actual server, or it will
reuse one of its existing connections.
Also,
Have your application (or the psql client) connect to pgbouncer
instead of directly to PostgreSQL server.
The configurations:
pgbouncer.ini: An example pgbouncer.ini with comments about defaults
[databases]
db1 = host=xx.xxx.xxx.xxx port=5432 dbname=db1
[pgbouncer]
listen_addr = *
listen_port = 6432
auth_type = md5
auth_file = userlist.txt
unix_socket_dir = /var/run/postgresql
pool_mode = transaction
max_client_conn = 100
default_pool_size = 20
userlist.txt:
"user1" "pass1"
to put in settings.py:
if PRODUCTION == '1':
#PRODUCTION is set to '1' if in production environment
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db1',
'USER': 'user1',
'PASSWORD': 'pass1',
'HOST': '/var/run/postgresql',
# 'PORT': '6432',
}
Extra:
In case not using unix socket - you can set HOST : '127.0.0.1' or 'localhost' if pgbouncer is running locally, or whatever the IP of server pgbouncer will be running on.
From the docs:
If you’re using PostgreSQL, by default (empty HOST), the connection to
the database is done through UNIX domain sockets (‘local’ lines in
pg_hba.conf). If your UNIX domain socket is not in the standard
location, use the same value of unix_socket_directory from
postgresql.conf. If you want to connect through TCP sockets, set HOST
to ‘localhost’ or ‘127.0.0.1’ (‘host’ lines in pg_hba.conf). On
Windows, you should always define HOST, as UNIX domain sockets are not
available.
In case of postgreSQL For ENGINE you can use postgresql or postgresql_psycopg2 - there's difference between the both given your Django version - postgresql_psycopg2 vs posgresql.
All of your DB settings in settings.py should be identical to the settings in your pgbouncer config, except the host in settings.py will point to pgbouncer. You probably need to change 'NAME': 'pgbouncer' to 'NAME': 'db1'. Since you're using a unix socket the port shouldn't matter.

How to properly grant access to a database user in Google Cloud SQL?

I've been stuck with this problem for a couple of days.
I developed an application for appengine using Django and I'd like to use Google Cloud SQL for my database. Everything works fine until I want to apply migrations on the development server when it fails with the following error:
django.db.utils.OperationalError: (1045, "Access denied for user 'MY_DB_USER'#'MY_IP' (using password: YES)")
What I've done is as follows:
I followed the instructions in the Django Support page to
develop my application.
In order to create a 1st generation Cloud SQL instance I followed
the steps outlined here, using the Cloud SDK.
I then created a new user following the instructions here and assigned it a password.
I deployed the application using the following command line:
gcloud preview app deploy MY-APP-DIR/app.yaml --version 0-1-0
I authorized my IP and my AppEngine Application ID. They are both listed in the ''Authorization'' section under ''Access Control'' in my SQL instance.
Finally, I tried to apply migrations using the following command line:
SETTINGS_MODE='prod' MY-APP-DIR/manage.py migrate
settings.py
The relevant portion of my settings.py looks as follows:
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine'):
DEBUG = False
# Running on production App Engine, so use a Google Cloud SQL database.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST' : '/cloudsql/[MY-PROYECT-ID]:[MY-CLOUD-SQL-INSTANCE]',
'NAME': '[MY-DB-NAME]',
'USER': 'root',
}
}
elif os.getenv('SETTINGS_MODE') == 'prod':
DEBUG = False
# Running in development, but want to access the Google Cloud SQL instance
# in production.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': '[MY-DB-NAME]', # db name.
'USER': '[MY-DB-USER]',
'PASSWORD' : '[MY-DB-USER-PASSWORD]',
'HOST' : '[IPV4 ASSIGNED IN GOOGLE CONSOLE]',
'PORT': '3306',
}
}
else:
# Running in development, so use a local MySQL database.
DEBUG = True
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': '[MY-LOCAL-DB]',
'USER': 'root',
'PASSWORD': 'root',
}
}
Any idea as to what might be causing the problem?
Thank you!
I finally figured out what was the problem.
The proper way to grant a user database access, in order to apply migrations, is the following:
White list your IP. It should be shown under 'Authorized Networks'
Create a new database user account, but do not choose the 'Allow any host (%)' wildcard, instead select the "Restrict host by name, address, or address range" option and assign your IP ( The one you just whitelisted ).
You should now be able to run migrations with the command: SETTINGS_MODE='prod' PROJECT_DIR/manage.py migrate
As a side note, make sure the root user whose host is localhost doesn't have a password, else your App Engine application won't be able to connect to the database.
Hope this helps someone else!

Configure databases section in settings.py for Travis CI

I have a locally running Django 1.7 app with some tests, connecting
to MySQL
I configured Travis CI with this repo
Question:
I want to have to have a separate database for Travis , that is different from the one I use for development.
I tried adding separate settings in settings.py : default (for use with tests) and development (for use in dev boxes); and thought .travis.xml would use the 'default' when it ran migrate tasks.
But Travis CI errors out with the error : django.db.utils.OperationalError: (1045, "Access denied for user 'sajay'#'localhost' (using password: YES)")
I have no idea why it is trying to access my development db settings? I checked django1.7 docs, googled around but no luck.
Appreciate any help,
Thanks
My settings.py database section looks like the below :
DATABASES = {
'default': {
'ENGINE':'django.db.backends.mysql',
'NAME':'expenses_db',
'USER':'root',
'PASSWORD':'',
'HOST':'127.0.0.1',
'PORT':'3306',
},
# 'development': {
# 'ENGINE':'django.db.backends.mysql',
# 'NAME':'myapp_db',
# 'USER':'sajay',
# 'PASSWORD':'secret',
# 'HOST':'127.0.0.1',
# 'PORT':'3306',
# },
}
Note : When the 'development' section is commented, Travis CI build is green
My .travis.yml is pasted below:
language: python
services:
- mysql
python:
- "2.7"
env:
- DJANGO_VERSION=1.7 DB=mysql
install:
- pip install -r requirements.txt
- pip install mysql-python
before_script:
- mysql -e 'create database IF NOT EXISTS myapp_db;' -uroot
- mysql -e "GRANT ALL PRIVILEGES ON *.* TO 'root'#'localhost';" -uroot
- python manage.py migrate
script:
- python manage.py test
The problem you are getting is because you haven't got the right database name and settings for Travis CI. First you will need to separate out your settings between Travis and your project. To do that I use an environment variable called BUILD_ON_TRAVIS (alternatively you can use a different settings file if you prefer).
settings.py:
import os
#Use the following live settings to build on Travis CI
if os.getenv('BUILD_ON_TRAVIS', None):
SECRET_KEY = "SecretKeyForUseOnTravis"
DEBUG = False
TEMPLATE_DEBUG = True
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'travis_ci_db',
'USER': 'travis',
'PASSWORD': '',
'HOST': '127.0.0.1',
}
}
else:
#Non-travis DB configuration goes here
Then in your .travis.yml file in the before_script sections you will need to use the same database name as in the DATABASES settings. We then just have to set the environment variable in the .travis.yml file like so:
env:
global:
- BUILD_ON_TRAVIS=true
matrix:
- DJANGO_VERSION=1.7 DB=mysql
EDIT:
There is now an environment variable set by default when building on Travis.
Using this environment variable we can more simply solve the problem:
settings.py:
import os
#Use the following live settings to build on Travis CI
if os.getenv('TRAVIS', None):
#Travis DB configuration goes here
else:
#Non-Travis DB configuration goes here
Doing it this way is preferable as we no longer have to define the environment variable ourselves in the .travis.yml file.
Followed the approach suggested in http://www.slideshare.net/jacobian/the-best-and-worst-of-django to separate the settings for different environments
I checked out How to manage local vs production settings in Django? but did not get a clear answer. Thanks