I am developing a django app and trying to run it inside docker. I have an issue that I could not understand so far. while running the app with docker-compose, it seems that the web app cannot connect to the database when i use these configurations:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'my_db',
'USER': 'my_user',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '5432',
}
but once I change the host to postgres, it works. like this
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'my_db',
'USER': 'my_user',
'PASSWORD': '',
'HOST': 'postgres',
'PORT': '5432',
}
what is the difference between postgres and localhost. One is running without and issue inside docker and not in development environment in my mac and the other one is the opposite.
# docker-compose.yml
version: '3'
services:
db:
image: postgres
expose:
- "5432"
web:
build: .
command: python3 manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
Docker Compose actually add the hostnames of all your linked containers to each other.
On you machine, the postgres database is actually running in localhost, that why you have the localhost hostname.
In Compose, it's running in the postgres container, with the hostname postgres, that's why you have the postgres hostname.
If you want, you can create an entry in your host file to redirect postgres to localhost, you will then just have to use postgres everywhere.
Each docker container comes with it's own networking namespace by default. That namespace includes it's own private loopback interface, aka localhost. And they are also attached to networks inside of docker where they have their own internal DNS entry and can talk to other containers on that same network.
When you run your application inside a container with a bridge network, localhost will point to the container, not the docker host you are running on. The hostname to use depends on your scenario:
To talk to other containers, use the container name in DNS.
If it's started by docker-compose, use the service name to talk to one of the containers in that service using DNS round robin.
If it's started inside of swarm mode, you can use the service name there to go to a VIP that round robin load balances to all containers providing that service.
And if you need to talk to the docker host itself, use a non-loopback IP address of the docker host.
Related
I am surprised that I cannot get this to work, as I have run a django applications with a postgres DB in docker containers multiple times, however, this time on my Linux machine the django application cannot connect to the postgres docker container, whereas on my Mac the setup works without a problem.
I am getting the standard error:
django.db.utils.OperationalError: could not connect to server: Host is unreachable
Is the server running on host "postgresdb" (172.27.0.2) and accepting
TCP/IP connections on port 5432?
The postgres container is running and should be reachable. I check the IP address of the docker container and it seems correct. My docker-compose file looks like this:
version: "3.8"
services:
app:
container_name: data-management
restart: always
build: ./
ports:
- '3000:3000'
links:
- 'postgresdb'
volumes:
- ./:/usr/local/share/src/app
env_file:
- .dev.env
postgresdb:
container_name: be-postgres
restart: always
image: postgres:13.4-alpine
ports:
- '5432:5432'
volumes:
- postgres-db:/var/lib/postgresql/data
env_file:
- .dev.env
volumes:
postgres-db:
The configurations are stored in an environment file, which is used in both containers to avoid any mismatches.
POSTGRES_PASSWORD=devpassword123
POSTGRES_USER=devuser
POSTGRES_SERVICE=postgresdb
POSTGRES_PORT=5432
POSTGRES_DB=data_management
The django DB settings are as standard as they could be:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': os.environ['POSTGRES_SERVICE'],
'NAME': os.environ['POSTGRES_DB'],
'PORT': os.environ['POSTGRES_PORT'],
'USER': os.environ['POSTGRES_USER'],
'PASSWORD': os.environ['POSTGRES_PASSWORD'],
}
}
I am out of ideas as what could be causing this as this happens only on my Fedora machine and there are no issues when running on my Mac. I have already had SELinux problems with some docker volumes in the past, however, after googling I could not find any suggestions that SELinux could be the culprit again.
I have deployed django application in app engine flexible.
I'm able to run migrations using cloud_sql_proxy. But i want to add migrate step as a part of deployment. Where do i specify that in app.yaml file ?
Also tried
gcloud beta app gen-config --custom
Which creates docker file. on adding migration command in docker file, recieved the following error:
could not connect to server: Cannot assign requested address
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
Settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'test',
'USER': 'test',
'PASSWORD': 'password',
'PORT': '5432',
'HOST': 'connection-name',
}}
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT wsgi
beta_settings:
cloud_sql_instances: connection-name
runtime_config:
python_version: 3
Please suggest approach to add migrate command.
You can't add migrate command as part of the deployment process. The app.yaml file just for app engine related configuration. So that what you can do is to connect with your Google Cloud SQL Instance in the local machin and run the migrate command.
The other options is to setup continuous integration. You can visit this link to get the idea about how to setup CI/CD on Travis.
I am trying to dockerize my Django project. For this purpose, I am trying to divide the whole project into 2 parts
The whole web related things in one container.
Database i.e Postgres in another
I am creating the Postgres database container using command:
docker run --name postgres -it -e POSTGRES_USER=username -e POSTGRES_PASSWORD=mysecretpassword postgres
When this postgres instance started running I entered the shell postgres instance using:
docker exec -it postgres /bin/bash
root#ae052fbce400:/# psql -U psql
Inside Psql shell that i got, I am creating the Database named DBNAME and granted all the privileges to username;
Database settings inside the webapp container is:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'DBNAME',
'USER': 'username',
'PASSWORD': 'mysecretpassword',
'HOST': 'postgres',
'PORT': 5432
}
}
Here is my docker-compose.yml file
services:
web:
image: 1ce04167758d #image build of webapp container
command: python manage.py runserver
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- postgres
postgres:
image: postgres
env_file:
- .env
expose:
- "5432"
When I ran docker-compose up I am getting the following error:-
web_1 | django.db.utils.OperationalError: FATAL: password authentication failed for user "username"
I tried various steps but this is the one which solved my problem.
docker stop $(docker ps -qa) && docker system prune -af --volumes
docker-compose up
This is because you created two database services. One manually via docker run and one via docker-compose. Unfortunately both unusable, meaning they'd have to be reconfigured in order to cooperate.
Scenario 1 - using a separate DB
You should remove the database definition from compose file - so that it looks like this:
version: '3'
services:
web:
image: 1ce04167758d
command: python manage.py runserver
volumes:
- .:/code
ports:
- "8000:8000"
And in your config you should change postgres to your host machine - for example 192.168.1.2
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'itoucan',
'USER': 'mannu',
'PASSWORD': 'mysecretpassword',
'HOST': '192.168.1.2',
'PORT': 5432
}
}
Then, run a separate database service just like you did, via the run command, but exposing a port publicly.
docker run --name postgres -it -p 5432:5432 -e POSTGRES_USER=mannu -e POSTGRES_PASSWORD=mysecretpassword postgres
When it finished initializing and when you finish adding databases and users you can fire up your Django app and it'll connect.
further reading on postgres env variables
Scenario 2 - using composed database
There's a lot of explaining here, as you have to set up a entrypoint that will wait until the DB is fully initialized. But I've already written a step by step answer on how to do it here on stack
Your situation is basically the same except for the DB service. You leave your compose nearly as it is now, with a little changes:
version: '3'
services:
web:
image: 1ce04167758d
command: python manage.py runserver
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- postgres
entrypoint: ["./docker-entrypoint.sh"]
postgres:
image: postgres
env_file:
- .env
I've added a entrypoint that is supposed to wait until your DB service completes initialization (for instructions on how to set it up you should refer to the link I provided earlier on).
I can see you've defined a entrypoint already - I'd suggest removing this entrypoint from Dockerfile, move it to the compose file and merge it with what I've described in the referred link. It's a common practice in commercial/bigger environments, as you might have many entrypoints, or/and as your entrypoint might not be intended to run while building - like the one I suggest is.
I've removed DB port mapping as you shouldn't expose services if there's no need - if only the web service is supposed to use the DB, then we shouldn't expose the DB for other possibilities.
With the above configuration, your Django configuration would be perfectly fine.
edit from comments
The 0.0.0.0 IP provided for postgres states that the server will listen on all incoming connections. It means that in settings.py you should specify not the 0.0.0.0 address but a address of the host on which your service runs - in your case I guess it's run on your computer - so simply running:
$ ifconfig
on your host will give your your local ip address ( 192.x.x.x or 10.x.x.x ) and this IP is what you specify in settings
I stocked at postgresql/django issue. After attempt of running the localhost server got the following error
django.db.utils.OperationalError: FATAL: no pg_hba.conf entry for host "172.17.0.1", database "closerdb", SSL off
According to many similar issues and answers last lines at pg_hba.conf look this way:
host all all 0.0.0.0/0 md5
local replication postgres trust
host replication postgres 127.0.0.1/32 trust
host replication postgres 172.17.0.1 trust
host replication postgres ::1/128 trust
I have restarted postgres after editing .conf file, but error remains. I supposed that it was cause by docker installation, because normally django project has to run on loclahost(127.0.0.1) and 172.17.0.1 is a default docker ip.
How to fix all that and avoid such problems in future?
EDIT: django database settings. worked fine previously before installing docker:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'closerdb',
# 'HOST': '/tmp/mysql.sock',
# 'PORT': '8000',
# 'HOSTNAME': '',
'USER': 'thekotik',
'PASSWORD': '',
'HOST': 'localhost',
'PORT': '',
}
}
I assume your django just wants to connect to a database called closerdb. In this case the corresponding pg_hba.conf entry should look like that:
host closerdb postgres 172.17.0.1 trust
From the postgresql docs:
The value replication specifies that the record matches if a replication connection is requested (note that replication connections do not specify any particular database). Otherwise, this is the name of a specific PostgreSQL database
I also want to point out that you should use trust carefully depending on your environment, because:
Allow the connection unconditionally. This method allows anyone that can connect to the PostgreSQL database server to login as any PostgreSQL user they wish, without the need for a password or any other authentication.
I am trying to deploy my django app on heroku server,i followed the instructions from this website https://devcenter.heroku.com/articles/getting-started-with-python#introduction .it worked fine till , "heroku open" command.When i came to the part where i need to host my database using " heroku run python manage.py syncdb" command , it failed showing the mesage "OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?". I tried lots of fixes including the one suggested here Deploying Django app's local postgres database to heroku? and http://www.cyberciti.biz/tips/postgres-allow-remote-access-tcp-connection.html .I tried all the solutions including editing the "listen_address" = '*' and tcpip_socket='true' in postgresql.conf and editing the ipv4 and v6 values in pg_hba.conf to
host all all 127.0.0.1 255.255.0.1 trust
host all all 10.0.0.99/32 md5
host all all 0.0.0.0/0 .
But none of them worked .I am guessing the problem arises because heroku can not connect to my local postgres server.This is strange because i'm able to access the postgres server via pgadmin.
And also in the django settings.py looks like this
DATABASES =
{
'default':
{
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'django_test',
'USER': 'postgres',
'PASSWORD': '******',
'HOST': 'localhost', # Or an IP Address that your DB is hosted on
'PORT': '5432',
}
}
Do i need to change this and use heroku's database settings instead??
localhost on the server points to the server not your local machine. The reason why is because the server running your django code will try and resolve the dns name localhost and it has a pointer to 127.0.0.1 which is local to the server resolving that name. That will NOT point to your computer you are working on.
You need to get an instance of postgres on heroku and change HOST: 'xxx.xxx.xxx.xxx to the IP address of your new postgres instance in your django settings.