I have 3 docker containers web(django) , nginx, db(postgresql)
When I run the following command
docker-compose -f docker-compose.prod.yml exec web python manage.py migrate --noinput
The exact error is:
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Address not available
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
docker-compose.prod.yml
version: '3.7'
services:
db:
image: postgres:12.0-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
env_file:
- ./.envs/.db
web:
build:
context: ./tubscout
dockerfile: Dockerfile.prod
command: gunicorn hello_django.wsgi:application --bind 0.0.0.0:8000
volumes:
- .static_volume:/home/app/web/staticfiles
expose:
- 8000
env_file:
- ./.envs/.prod
depends_on:
- db
nginx:
build: ./nginx
volumes:
- .static_volume:/home/app/web/staticfiles
ports:
- 1337:80
depends_on:
- web
volumes:
postgres_data:
static_volume:
Dockerfile.prod
###########
# BUILDER #
###########
# pull official base image
FROM python:3.8.3-alpine as builder
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apk update \
&& apk add postgresql-dev gcc python3-dev musl-dev
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps -w /usr/src/app/wheels -r requirements.txt
#########
# FINAL #
#########
# pull official base image
FROM python:3.8.3-alpine
# create directory for the app user
RUN mkdir -p /home/app
# create the app user
RUN addgroup -S app && adduser -S app -G app
# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/staticfiles
WORKDIR $APP_HOME
# install dependencies
RUN apk update && apk add libpq
COPY --from=builder /usr/src/app /wheels
COPY --from=builder /usr/src/app/requirements.txt .
RUN pip install --no-cache /wheels/wheels/*
# copy entrypoint.sh
COPY ./entrypoint.sh $APP_HOME
# copy project
COPY . $APP_HOME
# chown all the files to the app user
RUN chown -R app:app $APP_HOME
# change to the app user
USER app
# run entrypoint.prod.sh
ENTRYPOINT ["/home/app/web/entrypoint.sh"]
settings.py
DATABASES = {
'default': {
"ENGINE": os.environ.get("SQL_ENGINE", "django.db.backends.sqlite3"),
"NAME": os.environ.get("SQL_DATABASE", os.path.join(BASE_DIR, "db.sqlite3")),
"USER": os.environ.get("SQL_USER", "user"),
"PASSWORD": os.environ.get("SQL_PASSWORD", "password"),
"HOST": os.environ.get("SQL_HOST", "localhost"),
"PORT": os.environ.get("SQL_PORT", "5432"),
}
}
./.envs/.db
POSTGRES_USER=postgres
POSTGRES_PASSWORD=123456789
POSTGRES_DB=tubscoutdb_prod
./.envs/.prod
DEBUG=0
SECRET_KEY='#yinppohul88coi7*f+1^_*7#o9u#kf-sr*%v(bb7^k5)n_=-h'
DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
SQL_ENGINE=django.db.backends.postgresql
SQL_DATABASE=tubscoutdb_prod
SQL_USER=postgres
SQL_PASSWORD=123456789
SQL_HOST=localhost
SQL_PORT=5432
DATABASE=postgres
Change SQL_HOST to db in your .envs/.prod file. This will let the Web container reach the DB container and perform the migration.
Docker compose containers can be accessed with their service name from other containers.
Related
I'm trying to build Django + Docker + PostgreSQL
Dockerfile
FROM python:3.10.0-alpine
WORKDIR /usr/src/app
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN pip install --upgrade pip
RUN apk update
RUN apk add postgresql-dev gcc python3-dev musl-dev
COPY ./requirements.txt .
RUN pip install -r requirements.txt
COPY ./entrypoint.sh .
RUN sed -i 's/\r$//g' /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
COPY . .
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]
docker-compose.yml
version: '3.9'
services:
web:
build: .
command: gunicorn pets.wsgi:application --bind 0.0.0.0:8000
expose:
- 8000
env_file:
- .env
depends_on:
- db
db:
image: postgres:12.0-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
env_file:
- .env
nginx:
build: ./nginx
ports:
- 1337:80
depends_on:
- web
volumes:
postgres_data:
entrypoint.sh
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $POSTGRES_HOST $POSTGRES_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
cd pets
python manage.py flush --no-input
python manage.py migrate
exec "$#"
So, when the container starts i use docker exec -it <container_id> python manage.py createsuperuser to create a superuser and then add some data to database.
But, when i use docker-compose stop and then running again with the same id the container forgets about the written data and everything has to be repeated.
Shouldn't volume solve this issue?
I am trying to deploy my app to Heroku, am using amazon s3 bucket for static but the static icon is not showing on the website, and I need help setting up a worker for celery and celery beat to work on Heroku.
This is my Dockerfile:
# pull the official base image
FROM python:3.8.3-alpine as builder
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apk update \
&& apk add postgresql-dev gcc python3-dev musl-dev
RUN apk add zlib libjpeg-turbo-dev libpng-dev \
freetype-dev lcms2-dev libwebp-dev \
harfbuzz-dev fribidi-dev tcl-dev tk-dev
# lint
RUN pip install --upgrade pip
RUN pip install flake8
COPY . .
# install dependencies
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt
# pull official base image
FROM python:3.8.3-alpine
# create directory for the app user
RUN mkdir -p /home/app
# create the app user
RUN addgroup -S app && adduser -S app -G app
# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/static
RUN mkdir $APP_HOME/media
WORKDIR $APP_HOME
# install dependencies
RUN apk update && apk add libpq
RUN apk add zlib libjpeg-turbo-dev libpng-dev \
freetype-dev lcms2-dev libwebp-dev \
harfbuzz-dev fribidi-dev tcl-dev tk-dev
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /usr/src/app/requirements.txt .
RUN pip install --no-cache /wheels/*
# copy entrypoint.sh
COPY ./entrypoint.sh $APP_HOME
# copy project
COPY . $APP_HOME
# chown all the files to the app user
RUN chown -R app:app $APP_HOME
# change to the app user
USER app
# run entrypoint.prod.sh
ENTRYPOINT ["/home/app/web/entrypoint.prod.sh"]
CMD gunicorn my_proj.wsgi:application --bind 0.0.0.0:$PORT ```
This is my docker-compose file
```version: "3.8"
services:
web:
build:
context: .
dockerfile : Dockerfile
container_name: django
command: gunicorn my_proj.wsgi:application --bind 0.0.0.0:8000
volumes:
- static_volume:/home/app/web/static
- media_volume:/home/app/web/media
expose:
- 8000
depends_on:
- pgdb
- redis
celery-worker:
build: .
command: celery -A my_proj worker -l INFO
volumes:
- .:/usr/src/app
environment:
- DEBUG=1
- DJANGO_ALLOWED_HOSTS=['localhost', '127.0.0.1', 'app_name.herokuapp.com']
- CELERY_BROKER=redis://redis:6379/0
- CELERY_BACKEND=redis://redis:6379/0
depends_on:
- web
- redis
celery-beat:
build: .
command: celery -A my_proj beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
volumes:
- .:/usr/src/app
environment:
- DEBUG=1
- DJANGO_ALLOWED_HOSTS=['localhost', '127.0.0.1', 'app_name.herokuapp.com']
- CELERY_BROKER=redis://redis:6379/0
- CELERY_BACKEND=redis://redis:6379/0
depends_on:
- web
- redis
- celery-worker
pgdb:
image: postgres
container_name: pgdb
environment:
- POSTGRES_DB=databasename
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=password
volumes:
- pgdata:/var/lib/postgresql/data/
redis:
image: "redis:alpine"
nginx:
build: ./nginx
volumes:
- static_volume:/home/app/web/static
- media_volume:/home/app/web/media
ports:
- 1337:80
depends_on:
- web
volumes:
pgdata:
static_volume:
media_volume: ```
Here are the steps I followed to deploy on Heroku
heroku container:login
docker build -t registry.heroku.com/app name/web .
docker push registry.heroku.com/app name/web
heroku container:release -a app name web
Also, I ran this below and it is starting but I want it to work on its own.
heroku run celery -A my_proj worker -l INFO -a <app name>
heroku run celery -A my_proj beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler -a <app name>
Issue:
How do I set up celery to work as a worker.
How do I also set up a worker for celery-beat
Please I need help, in development, I usually use docker-compose up -d --build, and all images are built and working together. I feel it's only the Django container that is just working, and others were not built.
Okay, so this is what I did that worked for me, so for any new developer like me coming up on the Django web framework.
The best way I used in deploying the app on Heroku was using the Heroku build manifest. I created a heroku.yml file on the root directory of my project.
#This is the Build Manifest for creating web and worker.
build:
docker:
web: Dockerfile
worker: Dockerfile
run:
web: gunicorn my_proj.wsgi:application --bind 0.0.0.0:$PORT
worker: celery -A my_proj worker -l INFO --beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
release:
image: web
command:
- python manage.py collectstatic --noinput
For the Database, I used the Heroku Postgres add-on also for the Redis message broker, I used Heroku Redis.
I didn't Use Nginx to serve Staticfiles instead I used Whitenoise, pretty easy to set up but nasty bug when you turn off Debug. There are lots of help on here to help you fix that.
Best you create a static and staticfiles folder in your root directory and in your settings.py file
add these, worked for me
INSTALLED_APP = [
...,
'whitenoise.runserver_nostatic',
#BEFORE THE 'django.contrib.staticfiles',
]
STATIC_URL = "/static/"
STATIC_ROOT = os.path.join(BASE_DIR, "staticfiles")
STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"
STATICFILES_DIRS = [
os.path.join(BASE_DIR, "static"),
]
WHITENOISE_MANIFEST_STRICT = False
This is how I learned and resolved the issue.
Any better implementation am always open to learning.
Try setting up CORS in AWS S3 bucket permission to get the fonts working.
[
{
"AllowedHeaders": [
"Authorization"
],
"AllowedMethods": [
"GET",
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": [],
"MaxAgeSeconds": 3000
}
]
I'm trying to create a Django app in a docker container. The app would use a postgres db with postgis extension, which I have in another database. I'm trying to solve this using docker-compose but can not get it working.
I can get the app working without the container with the database containerized just fine. I can also get the app working in a container using a sqlite db (so a file included without external container dependencies). Whatever I do, it can't find the database.
My docker-compose file:
version: '3.7'
services:
postgis:
image: kartoza/postgis:12.1
volumes:
- postgres_data:/var/lib/postgresql/data/
ports:
- "${POSTGRES_PORT}:5432"
environment:
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_DB=${POSTGRES_DB}
env_file:
- .env
web:
build: .
# command: sh -c "/wait && python manage.py migrate --no-input && python /code/app/manage.py runserver 0.0.0.0:${APP_PORT}"
command: sh -c "python manage.py migrate --no-input && python /code/app/manage.py runserver 0.0.0.0:${APP_PORT}"
# restart: on-failure
ports:
- "${APP_PORT}:8000"
volumes:
- .:/code
depends_on:
- postgis
env_file:
- .env
environment:
WAIT_HOSTS: 0.0.0.0:${POSTGRES_PORT}
volumes:
postgres_data:
name: ${POSTGRES_VOLUME}
My Dockerfile (of the app):
# Pull base image
FROM python:3.7
LABEL maintainer="yb.leeuwen#portofrotterdam.com"
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install dependencies
# RUN pip install pipenv
RUN pip install pipenv
RUN mkdir /code/
COPY . /code
WORKDIR /code/
RUN pipenv install --system
# RUN pipenv install pygdal
RUN apt-get update &&\
apt-get install -y binutils libproj-dev gdal-bin python-gdal python3-gdal postgresql-client
## Add the wait script to the image
ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.7.3/wait /wait
RUN chmod +x /wait
# set work directory
WORKDIR /code/app
# RUN python manage.py migrate --no-input
# CMD ["python", "manage.py", "migrate", "--no-input"]
# RUN cd ${WORKDIR}
# If we want to run docker by itself we need to use below
# but if we want to run from docker-compose we'll set it there
EXPOSE 8000
# CMD /wait && python manage.py migrate --no-input
# CMD ["python", "manage.py", "migrate", "--no-input"]
# CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
My .env file:
# POSTGRES
POSTGRES_PORT=25432
POSTGRES_USER=username
POSTGRES_PASSWORD=pass
POSTGRES_DB=db
POSTGRES_VOLUME=data
POSTGRES_HOST=localhost
# GEOSERVER
# DJANGO
APP_PORT=8000
And finally my in my settings.py of the django app:
DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': os.getenv('POSTGRES_DBNAME'),
'USER': os.getenv('POSTGRES_USER'),
'PASSWORD': os.getenv('POSTGRES_PASS'),
'HOST': os.getenv("POSTGRES_HOST", "localhost"),
'PORT': os.getenv('POSTGRES_PORT')
}
}
I've tried quite a lot of things (as you see in some comments). I realized that docker-compose doesn't seem to wait until postgres is fully up, spinning and accepting requests so I tried to build in a waiting function (as suggested on the website). I first had migrations and running the server inside the Dockerfile (migrations in the build process and runserver as the startup command), but that requires postgres and as it wasn't waiting for it it didn't function. I finally took it all out to the docker-compose.yml file but still can't get it working.
The error I get:
web_1 | Is the server running on host "localhost" (127.0.0.1) and accepting
web_1 | TCP/IP connections on port 25432?
web_1 | could not connect to server: Cannot assign requested address
web_1 | Is the server running on host "localhost" (::1) and accepting
web_1 | TCP/IP connections on port 25432?
Does anybody have an idea why this isn't working?
I see that in your settings.py of the django app, you are connecting to Postgres via
'HOST': os.getenv("POSTGRES_HOST", "localhost"),
While in .env you are setting the value of to POSTGRES_HOST to localhost. This means that the web container is trying to reach the Postgres server postgis at localhost which should not be the case.
In order to solve this problem, simply update your .env file to be like this:
POSTGRES_PORT=5432
...
POSTGRES_HOST=postgis
...
The reason is that in your case, the docker-compose brings up 2 containers: postgis and web inside the same Docker network and they can reach each other via their DNS name i.e. postgis and web respectively.
Regarding the port, web container can reach postgis at port 5432 but not 25432 while your host machine can reach the database at port 25432 but not 5432
you can not use localhost for the docker containers, it will be pointing to the container itself, not to the host of the containers. Instead switch to use the service name.
to fix the issue, change your env to
# POSTGRES
POSTGRES_PORT=5432
POSTGRES_USER=username
POSTGRES_PASSWORD=pass
POSTGRES_DB=db
POSTGRES_VOLUME=data
POSTGRES_HOST=postgis
# DJANGO
APP_PORT=8000
and you compose file to
version: '3.7'
services:
postgis:
image: kartoza/postgis:12.1
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_DB=${POSTGRES_DB}
env_file:
- .env
web:
build: .
# command: sh -c "/wait && python manage.py migrate --no-input && python /code/app/manage.py runserver 0.0.0.0:${APP_PORT}"
command: sh -c "python manage.py migrate --no-input && python /code/app/manage.py runserver 0.0.0.0:${APP_PORT}"
# restart: on-failure
ports:
- "${APP_PORT}:8000"
volumes:
- .:/code
depends_on:
- postgis
env_file:
- .env
environment:
WAIT_HOSTS: postgis:${POSTGRES_PORT}
volumes:
postgres_data:
name: ${POSTGRES_VOLUME}
I am developing on a windows and trying to run the Django application in Linux container with Gunicorn and Nginx to deploy it to Linux machine in production.
I mostly used this Connect docker python to SQL server with pyodbc post as a guide but I think I have tried every solution found online about this error.
If I ping DB server from container it gets connected so the port 1433 is open and everything should be good to go. But for some reason I'm getting error django.db.utils.ProgrammingError: ('42000', "[42000] [FreeTDS][SQL Server]Login failed for user
Django settings.py
DATABASES = {
'default': {
'ENGINE': "sql_server.pyodbc",
'NAME': 'database1',
'HOST': '123.45.6.78',
'PORT':'1433',
'USER': "user",
'PASSWORD': "pswd",
'OPTIONS': {
"driver": "FreeTDS",
"host_is_server": True,
"unicode_results": True,
"extra_params": "tds_version=7.3",
}
}
}
Dockerfile
# start from an official image
FROM python:3
# arbitrary location choice: you can change the directory
RUN mkdir -p /opt/services/djangoapp/src
WORKDIR /opt/services/djangoapp/src
#Install FreeTDS and dependencies for PyODBC
RUN apt-get update \
&& apt-get install unixodbc -y \
&& apt-get install unixodbc-dev -y \
&& apt-get install freetds-dev -y \
&& apt-get install freetds-bin -y \
&& apt-get install tdsodbc -y \
&& apt-get install --reinstall build-essential -y
# populate "ocbcinst.ini"
RUN echo "[FreeTDS]\n\
TDS_Version = '7.3'\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini
# modify "freetds.conf"
RUN echo "[mssql]\n\
host = 172.30.2.18\n\
port = 1433\n\
tds version = 7.3" >> /etc/freetds/freetds.conf
RUN echo MinProtocol = TLSv1.0 >> /etc/ssl/openssl.cnf
RUN echo CipherString = DEFAULT#SECLEVEL=1 >> /etc/ssl/openssl.cnf
# install our dependencies
# we use --system flag because we don't need an extra virtualenv
COPY Pipfile Pipfile.lock /opt/services/djangoapp/src/
RUN pip install pipenv && pipenv install --system
# copy our project code
COPY . /opt/services/djangoapp/src
# expose the port 8000
EXPOSE 8000
# define the default command to run when starting the container
CMD ["gunicorn", "--chdir", "app", "--bind", ":8000", "config.wsgi:application"]
docker-compose.yml
version: '3'
services:
djangoapp:
build: .
volumes:
- .:/opt/services/djangoapp/src
- /static:/static
networks: # <-- here
- nginx_network
nginx:
image: nginx:1.13
ports:
- 8000:80
volumes:
- ./config/nginx/conf.d:/etc/nginx/conf.d
- /static:/static
depends_on:
- djangoapp
networks: # <-- here
- nginx_network
networks: # <-- and here
nginx_network:
driver: bridge
Pipfile
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
django = "==2.1.0"
pyodbc = "==4.0.28"
django-pyodbc-azure = "*"
django-datatables-view = "*"
gunicorn = "*"
whitenoise = "*"
[requires]
python_version = "3.8"
I finally solved it myself. Had to add port 1433 to nginx in .yml file.
Like this
nginx:
image: nginx:1.13
ports:
- 8000:80
- 1433:1433
volumes:
- ./config/nginx/conf.d:/etc/nginx/conf.d
- /static:/static
depends_on:
- djangoapp
networks: # <-- here
- nginx_network
I run the following yml file with $docker-compose up command. I'm developing a REST API with django (Udemy "Build a Backend REST API with Python & Django - Advanced").
System: OS X 10.10.5
Docker version: 18.03.0-ce
docker-compose version: 1.20.1
Q: I am unable to access localhost with 127.0.0.1
docker-compose.yml
version: "2.2"
services:
app:
build:
context: .
ports:
- "8000:8000"
volumes:
- ./app:/app
command: >
sh -c "python manage.py wait_for_db &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8000"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=supersecretpassword
depends_on:
- db
db:
image: postgres:10-alpine
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
Dockerfile
FROM python:3.7-alpine
#python unbuffered environmental variable
ENV PYTHONUNBUFFERED 1
#copy requirements.txt from the directory adjacent to
#the Dockerfile into dockerimage/requirments.txt
COPY ./requirements.txt /requirements.txt
RUN apk add --update --no-cache postgresql-client
RUN apk add --update --no-cache --virtual .tmp-build-deps \
gcc libc-dev linux-headers postgresql-dev
RUN pip install -r /requirements.txt
RUN apk del .tmp-build-deps
#create an empty folder named app in docker image
RUN mkdir /app
WORKDIR /app
COPY ./app /app
RUN adduser -D user
USER user
Some suggest to add CMD ["python", "manage.py", "runserver",
"0.0.0.0:8000"] in Dockerfile; didn't work in this case. In django
settings.py, ALLOWED_HOSTS = ['192.168.99.100']
Console output (part of: see bold)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py",
line 178, in get_new_connection connection = Database.connect(conn_params)
File "/usr/local/lib/python3.7/site-packages/psycopg2/init.py",
line 130, in connect conn = _connect(dsn, connection_factory=connection_factory, kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused **
Is the server running on host "db" (172.19.0.2) and accepting
TCP/IP connections on port 5432?
I'm super new to this stuff, appreciate your help and suggestions.