I have a Django 4 app with PostgreSQL. I connected the SQL instance vie google-proxy correctly, however when I upload it into App Engine it fails to connect.
settings.py
if os.getenv('GAE_APPLICATION', None):
# Running on production App Engine, so connect to Google Cloud SQL using
# the unix socket at /cloudsql/<your-cloudsql-connection string>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': '/cloudsql/instancename',
'USER': 'user',
'PASSWORD': '********',
'NAME': 'database',
'PORT': '5432',
}
}
else:
DATABASES = {
'default': {
"ENGINE": "django.db.backends.postgresql",
'NAME': 'database',
'USER': 'user',
'PASSWORD': '******',
'HOST': '127.0.0.1',
'PORT': '5432',
}
}
The error that on the screen was related with psycopg2:
could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/cloudsql/instancename/.s.PGSQL.5432"?
/layers/google.python.pip/pip/lib/python3.9/site-packages/psycopg2/__init__.py, line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
I updated the requirements.txt because locally with the proxy everything was correct.
asgiref==3.5.0
Django==4.0
django_better_admin_arrayfield==1.4.2
djangorestframework==3.13.1
django-cors-headers==3.7.0
django-filter==2.4.0
django-ratelimit==3.0.1
gunicorn==20.1.0
django-cors-headers==3.7.0
psycopg2==2.8.6
psycopg2-binary==2.9.3
pytz==2020.4
sqlparse==0.4.1
google-api-core==1.26.0
google-api-python-client==1.12.8
google-auth==1.27.0
google-auth-httplib2==0.0.4
google-cloud-core==1.6.0
google-cloud-storage==1.36.1
google-crc32c==1.1.2
google-resumable-media==1.2.0
googleapis-common-protos==1.53.0
I also added all the APIs necessaries according to the google docs
Lastly, I read the logs in App Engine and the first error is about IAM
rpc error: code = PermissionDenied desc = IAM permission denied for service account services#service.iam.gserviceaccount.com. "
I added all the possible permisions to that service account, however it continuous to happend.
I don't know what else to do.
Thanks you all
Related
I'm trying to tunnel my db connection in a django application through a jump server but can't seem to get it working because django manage.py handles & process the connections.
here's that I have in the settings.py
#process ssh_key first
ssh_key= os.getenv('SSH_KEY', '').encode('utf8').decode('unicode_escape')
server ={}
with sshtunnel.open_tunnel(
(os.environ.get('SSH_HOST'),int(os.getenv('SSH_PORT'))),
ssh_pkey=paramiko.RSAKey.from_private_key(io.StringIO(ssh_key)),
ssh_username= os.environ.get('SSH_USERNAME'),
remote_bind_address=(os.environ.get('DB_HOST'), int(os.getenv('DB_PORT'))),
) as ssh_proxy_host:
server={
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': 'localhost',
'PORT': ssh_proxy_host.local_bind_port,
'NAME': os.environ.get('DB_NAME'),
'USER': os.environ.get('DB_USER'),
'PASSWORD': os.environ.get('DB_PASS'),
}
# here's where I should have the connection function to db, but don't know if django has that option available
The true remote host is specified upon tunnel creation. The HOST used for the db connection should be localhost because you need it to find your end of the tunnel, not the other end of it.
Here is Django setting.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'rulings',
'USER': 'postgres',
'PASSWORD': '******',
'HOST': '',
'PORT': '',
}
}
In AWS, when I run a server, an error message occurs.
File "/home/app_admin/venv_ruling/lib64/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError:
FATAL: Peer authentication failed for user "postgres"
How can I run the server with the user name of postgres??
When I use another user name it will run without a problem, but I want to use "postgres" as a user.
I don't think "peer" authentication can even be configured on RDS, so you must be running your own PostgreSQL server on EC2? Since you apparently want to use password authentication, change your pg_hba to specify "md5", rather than "peer".
Also, it is a bad practise to run an app as "postgres". That user is supposed to be reserved for maintenance operations.
I'm deploying a django project using zappa to aws-lambda and using mongodb atlas as my database.
I'm tring to connect to the database using djongo.
I set my django_setting in the zappa_settings.json to my project's django settings.
The connection to the database with this settings works just fine in localhost. when deploying, it fails to connect to the server and I suspect that it tries to connect to a default local db (the db sent to mongo_client.py isnt valid or something and it needs to connect to default HOST).
The actual error I get is:
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
djongo.sql2mongo.SQLDecodeError: FAILED SQL: SELECT
If anyone has an idea I'd would love to hear.
attaching the settings with some fields unset (but set at my settings)
Django settings (database part):
DATABASES = {
'default': {
'ENGINE': 'djongo',
'NAME': 'db',
'HOST': 'mongodb://<username>:<password>#<>
'USER': 'username',
'PASSWORD': 'password',
}
}
zappa_settings:
{
"dev":
{
"aws_region": "eu-west-1",
"django_settings": settings,
"profile_name": "default",
"project_name": name,
"runtime": "python3.6",
"s3_bucket": bucket,
"timeout_seconds": 900,
"manage_roles": false,
"role_name": name,
"role_arn": arn,
"slim_handler": true
}
}
Try this
'default': {
'ENGINE': 'djongo',
'CLIENT': {
'host': 'mongodb+srv://url',
'username': '<username>',
'password': '<password>',
'name':'<db_name>'
}
}
I'm trying to connect Django to the Google cloud SQL, working with python 2.7 and django 1.5 under windows. I went through the instructions on this page: https://developers.google.com/appengine/docs/python/cloud-sql/django
My settings.py file has basic database settings of the form:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'INSTANCE': 'my_project:instance1',
'NAME': 'my_database',
}
}
With of course a proper could SQL instance and a database created through the SQL prompt of the google apis console
When I try to run manage.py syncdb for the first time in order to obtain an OAuth2 token, I get this:
OperationalError: (2003, "Can't connect to MySQL server on 'localhost'
(10061)")
Before you ask, I did make sure that both the django and google packages are in my PYTHONPATH, as well as "C:\Program Files (x86)\Google\google_appengine\lib\django-1.5"
Any help would be really welcome!
That database configuration only makes sense when connecting from AppEngine. If you want to access your CloudSQL database from your local machine using django, you should use the google.appengine.ext.django.backends.rdbms engine.
You can see the different configuration options here:
https://developers.google.com/appengine/docs/python/cloud-sql/django#development-settings
EDIT: The google.appengine.ext.django.backends.rdbms engine has been deprecated. If you want to connect to Google Cloud SQL from your local machine you should use IP connectivity. You can use the Cloud SQL instance IP (IPv4 or IPv6) and connect using the standard django.db.backends.mysql engine.
Example connection to Google Cloud SQL in Django:
AppEngine Standard, Python 2.7:
try:
import MySQLdb # noqa: F401
except ImportError:
import pymysql
pymysql.install_as_MySQLdb()
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine'):
# Running on production App Engine, so connect to Google Cloud SQL using
# the unix socket at /cloudsql/<your-cloudsql-connection string>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '/cloudsql/<your-cloudsql-connection-string>',
'NAME': '<your-database-name>',
'USER': '<your-database-user>',
'PASSWORD': '<your-database-password>',
}
}
else:
# Running locally so connect to either a local MySQL instance or connect to
# Cloud SQL via the proxy. To start the proxy via command line:
#
# $ cloud_sql_proxy -instances=[INSTANCE_CONNECTION_NAME]=tcp:3306
#
# See https://cloud.google.com/sql/docs/mysql-connect-proxy
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '127.0.0.1', # DB's IP address
'PORT': '3306',
'NAME': '<your-database-name>',
'USER': '<your-database-user>',
'PASSWORD': '<your-database-password>',
}
}
Source: GCP Python Django Samples AppEngine Standard Python 2.7
AppEngine Standard, Python 3.7:
# Install PyMySQL as mysqlclient/MySQLdb to use Django's mysqlclient adapter
# See https://docs.djangoproject.com/en/2.1/ref/databases/#mysql-db-api-drivers
# for more information
import pymysql # noqa: 402
pymysql.install_as_MySQLdb()
if os.getenv('GAE_APPLICATION', None):
# Running on production App Engine, so connect to Google Cloud SQL using
# the unix socket at /cloudsql/<your-cloudsql-connection string>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '/cloudsql/[YOUR-CONNECTION-NAME]',
'USER': '[YOUR-USERNAME]',
'PASSWORD': '[YOUR-PASSWORD]',
'NAME': '[YOUR-DATABASE]',
}
}
else:
# Running locally so connect to either a local MySQL instance or connect to
# Cloud SQL via the proxy. To start the proxy via command line:
#
# $ cloud_sql_proxy -instances=[INSTANCE_CONNECTION_NAME]=tcp:3306
#
# See https://cloud.google.com/sql/docs/mysql-connect-proxy
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '127.0.0.1', # DB's IP address
'PORT': '3306',
'NAME': '[YOUR-DATABASE]',
'USER': '[YOUR-USERNAME]',
'PASSWORD': '[YOUR-PASSWORD]',
}
}
Source GCP Python Django Samples AppEngine Standard Python 3.7
AppEngine Flexible:
DATABASES = {
'default': {
# If you are using Cloud SQL for MySQL rather than PostgreSQL, set
# 'ENGINE': 'django.db.backends.mysql' instead of the following.
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'polls',
'USER': '<your-database-user>',
'PASSWORD': '<your-database-password>',
# For MySQL, set 'PORT': '3306' instead of the following. Any Cloud
# SQL Proxy instances running locally must also be set to tcp:3306.
'PORT': '5432',
}
}
# In the flexible environment, you connect to CloudSQL using a unix socket.
# Locally, you can use the CloudSQL proxy to proxy a localhost connection
# to the instance
DATABASES['default']['HOST'] = '/cloudsql/<your-cloudsql-connection-string>'
if os.getenv('GAE_INSTANCE'):
pass
else:
DATABASES['default']['HOST'] = '127.0.0.1' # DB's IP address
Source GCP Python Django Samples AppEngine Flexible
I'm using pgbouncer with Django. I've added test_foo database to its config to be able to run tests, because apparently Django can't use a different port for the test DB. Now the test run but at the end, when Django tries to drop the test DB, I receive
django.db.utils.DatabaseError: database "test_foo" is being accessed by other users
DETAIL: There are 1 other session(s) using the database.
I suppose that is caused by the open connection stored by pgbouncer. What can I do?
This is not the perfect solution, but it does the trick. You can force Django to use different database settings when running unit tests by adding to your settings.py:
if 'test' in sys.argv or 'test_coverage' in sys.argv:
# Use 5432 as db port (avoid going through pgbouncer, can't delete test DB).
DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': 'xxx',
'USER': 'xxx',
'PASSWORD': 'xxx',
'HOST': '',
'PORT': '5432'
},
}