I want to insert environmental variables from an .env file into my containerized Django application, so I can use it to securely set Django's settings.py.
However on $docker-compose up I receive part of an UserWarning which apparently originates in the django-environ package (and breaks my code):
/usr/local/lib/python3.9/site-packages/environ/environ.py:628: UserWarning: /app/djangoDocker/.env doesn't exist - if you're not configuring your environment separately, create one. web | warnings.warn(
The warning breaks at that point and (although all the container claim to be running) I can neither stop them from that console (zsh, Ctrl+C) nor can I access the website locally. What am I missing? Really appreciate any useful input.
Dockerfile: (located in root)
# pull official base image
FROM python:3.9.5
# set environment variables, grab via os.environ
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
# set work directory
WORKDIR /app
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# add entrypoint script
COPY ./entrypoint.sh ./app
# run entrypoint.sh
ENTRYPOINT ["./entrypoint.sh"]
# copy project
COPY . /app
docker-compose.yml (located in root; I've tried either using env_file or environment as in the comments)
version: '3'
services:
web:
build: .
container_name: web
command: gunicorn djangoDocker.wsgi:application --bind 0.0.0.0:8000
volumes:
- .:/app
ports:
- "8000:80"
env_file:
- .env
# environment:
# BASE_URL: ${BASE_URL}
# SECRET_KEY: ${SECRET_KEY}
# ALLOWED_HOSTS: ${ALLOWED_HOSTS}
# DEBUG: ${DEBUG}
# SQL_ENGINE: ${SQL_ENGINE}
# SQL_DATABASE: ${SQL_DATABASE}
# SQL_USER: ${SQL_USER}
# SQL_PASSWORD: ${SQL_PASSWORD}
# SQL_HOST: ${SQL_HOST}
# SQL_PORT: ${SQL_PORT}
# EMAIL_HOST_USER: ${EMAIL_HOST_USER}
# EMAIL_HOST_PASSWORD: ${EMAIL_HOST_PASSWORD}
# TEMPLATE_DIR: ${TEMPLATE_DIR}
depends_on:
- pgdb
pgdb:
image: postgres
container_name: pgdb
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- pgdata:/var/lib/postgresql/data/
nginx:
build: ./nginx
volumes:
- .:/app
links:
- web:web
ports:
- "80:80"
depends_on:
- web
volumes:
pgdata:
.env. (also located in root)
BASE_URL=localhost
SECRET_KEY=mySecretKey
ALLOWED_HOSTS=localhost,127.0.0.1,0.0.0.0
DEBUG=True
SQL_ENGINE=django.db.backends.postgresql
SQL_DATABASE=postgres
SQL_USER=postgres
SQL_PASSWORD=postgres
SQL_HOST=pgdb
SQL_PORT=5432
EMAIL_HOST_USER=my#mail.com
EMAIL_HOST_PASSWORD=myMailPassword
TEMPLATE_DIR=frontend/templates/frontend/
Terminal Output after running $docker-compose up in the root
pgdb is up-to-date
Recreating web ... done
Recreating djangodocker_nginx_1 ... done
Attaching to pgdb, web, djangodocker_nginx_1
nginx_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1 | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
web | [2021-06-04 14:58:09 +0000] [1] [INFO] Starting gunicorn 20.0.4
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web | [2021-06-04 14:58:09 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
web | [2021-06-04 14:58:09 +0000] [1] [INFO] Using worker: sync
web | [2021-06-04 14:58:09 +0000] [8] [INFO] Booting worker with pid: 8
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: using the "epoll" event method
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: nginx/1.20.1
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: OS: Linux 4.19.121-linuxkit
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: start worker processes
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: start worker process 23
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: start worker process 24
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: start worker process 25
nginx_1 | 2021/06/04 14:58:09 [notice] 1#1: start worker process 26
pgdb |
pgdb | PostgreSQL Database directory appears to contain a database; Skipping initialization
pgdb |
pgdb | 2021-06-04 14:34:00.119 UTC [1] LOG: starting PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
pgdb | 2021-06-04 14:34:00.120 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
pgdb | 2021-06-04 14:34:00.120 UTC [1] LOG: listening on IPv6 address "::", port 5432
pgdb | 2021-06-04 14:34:00.125 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
pgdb | 2021-06-04 14:34:00.134 UTC [27] LOG: database system was shut down at 2021-06-04 14:21:19 UTC
pgdb | 2021-06-04 14:34:00.151 UTC [1] LOG: database system is ready to accept connections
web | /usr/local/lib/python3.9/site-packages/environ/environ.py:628: UserWarning: /app/djangoDocker/.env doesn't exist - if you're not configuring your environment separately, create one.
web | warnings.warn(
requirements.txt
Django==3.2
gunicorn==20.0.4
djoser==2.1.0
django-environ
psycopg2-binary~=2.8.0
django-cors-headers==3.5.0
django-templated-mail==1.1.1
djangorestframework==3.12.2
djangorestframework-simplejwt==4.7.0
Let me know in case any further information is required.
Until now I do not know what caused the error, but in case anyone else has the same problem: switching to python-decouple instead of django-environ fixed it. Of course you have to adapt everything in settings.py accordingly, f.e. add from decouple import config and DEBUG = config('DEBUG', default=False, cast=bool).
Related
My system is Ubuntu 22.04.1 LTS.
I was going through the Django for professionals book and follow all line by line but somehow when i dockerize my django project with postgres it just doesn't run localhost properly, with sqlite all works fine.
Dockerfile
# Pull base image
FROM python:3.10.6-slim-bullseye
# Set environment variables
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Set work directory
WORKDIR /code
# Install dependencies
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# Copy project
COPY . .
docker-compose.yml
version: "3.9"
services:
web:
build: .
command: python /code/manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- 8000:8000
depends_on:
- db
db:
image: postgres:13
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- "POSTGRES_HOST_AUTH_METHOD=trust"
volumes:
postgres_data:
django.settings.py
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "postgres",
"USER": "postgres",
"PASSWORD": "postgres",
"HOST": "db", # set in docker-compose.yml
"PORT": 5432, # default postgres port
}
}
When I run docker-compose up that shows and seems like all must be fine, but somehow it just doesn't work:
Attaching to ch3-postgresql_web_1, ch3-postgresql_db_1
web_1 | Watching for file changes with StatReloader
web_1 | Performing system checks...
web_1 |
web_1 | System check identified no issues (0 silenced).
db_1 |
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1 |
db_1 | 2022-12-05 11:58:21.572 UTC [1] LOG: starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
db_1 | 2022-12-05 11:58:21.574 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2022-12-05 11:58:21.575 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2022-12-05 11:58:21.577 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2022-12-05 11:58:21.580 UTC [26] LOG: database system was shut down at 2022-12-05 11:58:17 UTC
db_1 | 2022-12-05 11:58:21.584 UTC [1] LOG: database system is ready to accept connections
And when I go to the url 127.0.0.1:8000:
This site can’t be reached
The webpage at http://127.0.0.1:8000/ might be temporarily down or it may have moved permanently to a new web address.
ERR_SOCKET_NOT_CONNECTED
If i wait some time and then check docker-compose logs, this messages appears
psycopg2.OperationalError: connection to server at "db" (172.21.0.2), port 5432 failed: Connection timed out
web_1 | Is the server running on that host and accepting TCP/IP connections?
I spend a lot of hours of searching for info about this problem but nothing did help. I try even to reinstall different versions of Postgres and docker.
Simply try this:
change this:
volumes:
- postgres_data:/var/lib/postgresql/data/
To this:
volumes:
- postgres_data:/var/lib/postgresql/data #removed forward slash from here.
Try above thing and see if it solves that problem.
Note: after making above changes don't forget to re-build docker-compsoe
You will need to add more environment variables to make the PostgreSQL work.
Please refer to https://hub.docker.com/_/postgres "Environment Variables"
The only variable required is POSTGRES_PASSWORD, the rest are optional.
db:
image: postgres:13
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
- POSTGRES_HOST_AUTH_METHOD=trust
And, you are encouraged to make your docker compose file more secure:
db:
image: postgres:13
volumes:
- postgres_data:/var/lib/postgresql/data/
env_file:
- .env.dev
.env.dev
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=postgres
POSTGRES_HOST_AUTH_METHOD=trust
I have currently implemented websocket connections via django channels using a redis layer.
I'm new to docker and not sure where I might have made a mistake. After the docker-compose up -d --build the "static files, media, database and gunicorn wsgi" all function, but redis won't connect. even though it is running in the background.
Before trying to containerize the application with docker, it worked well with:
python manage.py runserver
with the following settings.py setction for the redis layer:
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("0.0.0.0", 6379)],
},
},
}
and by calling a docker container for the redis layer:
docker run -p 6379:6379 -d redis:5
But after the trying to containerize the entire application it was unable to find the websocket
The new setup for the docker-compose is as follows:
version: '3.10'
services:
web:
container_name: web
build:
context: ./app
dockerfile: Dockerfile
command: bash -c "gunicorn core.wsgi:application --bind 0.0.0.0:8000"
volumes:
- ./app/:/usr/src/app/
- static_volume:/usr/src/app/staticfiles/
- media_volume:/usr/src/app/media/
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- db
networks:
- app_network
redis:
container_name: redis
image: redis:5
ports:
- 6379:6379
networks:
- app_network
restart: on-failure
db:
container_name: db
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- ./.env.psql
ports:
- 5432:5432
networks:
- app_network
volumes:
postgres_data:
static_volume:
media_volume:
networks:
app_network:
with this settings.py:
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("redis", 6379)],
},
},
}
After building successfully the container and running docker-compose logs -f:
Attaching to web, db, redis
db | The files belonging to this database system will be owned by user "postgres".
db | This user must also own the server process.
db |
db | The database cluster will be initialized with locale "en_US.utf8".
db | The default database encoding has accordingly been set to "UTF8".
db | The default text search configuration will be set to "english".
db |
db | Data page checksums are disabled.
db |
db | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db | creating subdirectories ... ok
db | selecting dynamic shared memory implementation ... posix
db | selecting default max_connections ... 100
db | selecting default shared_buffers ... 128MB
db | selecting default time zone ... Etc/UTC
db | creating configuration files ... ok
db | running bootstrap script ... ok
db | performing post-bootstrap initialization ... ok
db | initdb: warning: enabling "trust" authentication for local connections
db | You can change this by editing pg_hba.conf or using the option -A, or
db | --auth-local and --auth-host, the next time you run initdb.
db | syncing data to disk ... ok
db |
db |
db | Success. You can now start the database server using:
db |
db | pg_ctl -D /var/lib/postgresql/data -l logfile start
db |
db | waiting for server to start....2022-06-27 16:18:30.303 UTC [48] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
db | 2022-06-27 16:18:30.310 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db | 2022-06-27 16:18:30.334 UTC [49] LOG: database system was shut down at 2022-06-27 16:18:29 UTC
db | 2022-06-27 16:18:30.350 UTC [48] LOG: database system is ready to accept connections
db | done
db | server started
db | CREATE DATABASE
db |
db |
db | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db |
db | 2022-06-27 16:18:31.587 UTC [48] LOG: received fast shutdown request
db | waiting for server to shut down....2022-06-27 16:18:31.596 UTC [48] LOG: aborting any active transactions
db | 2022-06-27 16:18:31.601 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
db | 2022-06-27 16:18:31.602 UTC [50] LOG: shutting down
db | 2022-06-27 16:18:31.650 UTC [48] LOG: database system is shut down
db | done
db | server stopped
db |
db | PostgreSQL init process complete; ready for start up.
db |
db | 2022-06-27 16:18:31.800 UTC [1] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
db | 2022-06-27 16:18:31.804 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db | 2022-06-27 16:18:31.804 UTC [1] LOG: listening on IPv6 address "::", port 5432
db | 2022-06-27 16:18:31.810 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db | 2022-06-27 16:18:31.818 UTC [62] LOG: database system was shut down at 2022-06-27 16:18:31 UTC
db | 2022-06-27 16:18:31.825 UTC [1] LOG: database system is ready to accept connections
redis | 1:C 27 Jun 2022 16:18:29.080 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis | 1:C 27 Jun 2022 16:18:29.080 # Redis version=5.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
redis | 1:C 27 Jun 2022 16:18:29.080 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis | 1:M 27 Jun 2022 16:18:29.082 * Running mode=standalone, port=6379.
redis | 1:M 27 Jun 2022 16:18:29.082 # Server initialized
redis | 1:M 27 Jun 2022 16:18:29.082 * Ready to accept connections
web | Waiting for postgres...
web | PostgreSQL started
web | Waiting for redis...
web | redis started
web | [2022-06-27 16:18:33 +0000] [1] [INFO] Starting gunicorn 20.1.0
web | [2022-06-27 16:18:33 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
web | [2022-06-27 16:18:33 +0000] [1] [INFO] Using worker: sync
web | [2022-06-27 16:18:33 +0000] [8] [INFO] Booting worker with pid: 8
web | [2022-06-27 16:19:18 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
web | [2022-06-27 18:19:18 +0200] [8] [INFO] Worker exiting (pid: 8)
web | [2022-06-27 16:19:18 +0000] [9] [INFO] Booting worker with pid: 9
web | Not Found: /ws/user_consumer/1/
web | Not Found: /ws/accueil/accueil/
web | Not Found: /ws/user_consumer/1/
web | Not Found: /ws/accueil/accueil/
And the docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb3e489e0831 dermatology-project_web "/usr/src/app/entryp…" 35 minutes ago Up 35 minutes 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp web
aee14c8665d0 postgres "docker-entrypoint.s…" 35 minutes ago Up 35 minutes 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp db
94c29591b352 redis:5 "docker-entrypoint.s…" 35 minutes ago Up 35 minutes 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp redis
The build Dockerfile:
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apt-get update
RUN apt-get install -y libpq-dev python3-pip python-dev postgresql postgresql-contrib netcat
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# copy entrypoint.sh
COPY ./entrypoint.sh .
RUN sed -i 's/\r$//g' /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# create the appropriate directories for staticfiles
# copy project
COPY . .
# staticfiles
RUN python manage.py collectstatic --no-input --clear
# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]
and the entrypoint that checks the connections:
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
if [ "$CHANNEL" = "redis" ]
then
echo "Waiting for redis..."
while ! nc -z $REDIS_HOST $REDIS_PORT; do
sleep 0.1
done
echo "redis started"
fi
#python manage.py flush --no-input
#python manage.py migrate
exec "$#"
I have also tried to run the redis container separate like before and maintain the working containers, but that doesn´t work either. I have also tried it while running daphne on a different port and passing the asgi:application (daphne -p 8001 myproject.asgi:application) and it also didn't work.
Thank you
Managed a solution eventually
To make it work I needed to run the wsgi and asgi servers separately from from each other, each with its own container. Also, the previous service "web" that exposed the ports to the applications needed to be run twice for each container as well, with nginx proxies that upstreamed to each respective port.
This was all thanks to this genius of a man:
https://github.com/pplonski/simple-tasks
Here he explains what I needed and more. He also uses celery workers to manage the asynchronous task queue/job queue based on distributed message passing, which was a bit overkill for my project but beautiful.
New docker-compose:
version: '2'
services:
nginx:
container_name: nginx
restart: always
build: ./nginx
ports:
- 1337:80
volumes:
- static_volume:/usr/src/app/staticfiles/
- media_volume:/usr/src/app/media/
depends_on:
- wsgiserver
- asgiserver
postgres:
container_name: postgres
restart: always
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data/
ports:
- 5433:5432
expose:
- 5432
environment:
- ./.env.db
redis:
container_name: redis
image: redis:5
restart: unless-stopped
ports:
- 6378:6379
wsgiserver:
build:
context: ./app
dockerfile: Dockerfile
container_name: wsgiserver
command: gunicorn core.wsgi:application --bind 0.0.0.0:8000
env_file:
- ./.env.dev
volumes:
- ./app/:/usr/src/app/
- static_volume:/usr/src/app/staticfiles/
- media_volume:/usr/src/app/media/
links:
- postgres
- redis
expose:
- 8000
asgiserver:
build:
context: ./app
dockerfile: Dockerfile
container_name: asgiserver
command: daphne core.asgi:application -b 0.0.0.0 -p 9000
env_file:
- ./.env.dev
volumes:
- ./app/:/usr/src/app/
links:
- postgres
- redis
expose:
- 9000
volumes:
static_volume:
media_volume:
postgres_data:
New entrypoint.sh:
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
#python manage.py flush --no-input
#python manage.py migrate
exec "$#"
New nginx
nginx.conf:
server {
listen 80;
# gunicon wsgi server
location / {
try_files $uri #proxy_api;
}
location #proxy_api {
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Url-Scheme $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://wsgiserver:8000;
}
# ASGI
# map websocket connection to daphne
location /ws {
try_files $uri #proxy_to_ws;
}
location #proxy_to_ws {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;
proxy_pass http://asgiserver:9000;
}
# static and media files
location /static/ {
alias /usr/src/app/staticfiles/;
}
location /media/ {
alias /usr/src/app/media/;
}
}
Dockerfile for nginx:
FROM nginx:1.21
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d
Note
If anyone is using this as reference, this is not a production container, there are further steps needed.
This article explains the other step:
https://testdriven.io/blog/dockerizing-django-with-postgres-gunicorn-and-nginx/#conclusion
, as well as securing the application with AWS with Docker and Let's Encrypt, in the conclusion link.
I'm trying to get celery to work with Django and docker and the building works well but I celery won't run. Any ideas?
Here is are the docker-compose logs -f errors
Starting django-celery_redis_1 ... done
Starting django-celery_db_1 ... done
Starting django-celery_flower_1 ... done
Starting django-celery_celery_beat_1 ... done
Starting django-celery_celery_worker_1 ... done
Starting django-celery_web_1 ... done
Attaching to django-celery_db_1, django-celery_redis_1, django-celery_celery_worker_1, django-celery_flower_1, django-celery_celery_beat_1, django-celery_web_1
celery_beat_1 | standard_init_linux.go:219: exec user process caused: exec format error
db_1 | 2021-03-28 18:18:15.611 UTC [1] LOG: starting PostgreSQL 12.0 on x86_64-pc-linux-musl, compiled by gcc (Alpine 8.3.0) 8.3.0, 64-bit
db_1 | 2021-03-28 18:18:15.613 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2021-03-28 18:18:15.616 UTC [1] LOG: listening on IPv6 address "::", port 5432
celery_worker_1 | standard_init_linux.go:219: exec user process caused: exec format error
db_1 | 2021-03-28 18:18:15.648 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
redis_1 | 1:C 28 Mar 2021 18:18:15.425 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 28 Mar 2021 18:18:15.425 # Redis version=5.0.12, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 28 Mar 2021 18:18:15.425 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 28 Mar 2021 18:18:15.427 * Running mode=standalone, port=6379.
redis_1 | 1:M 28 Mar 2021 18:18:15.427 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 28 Mar 2021 18:18:15.427 # Server initialized
redis_1 | 1:M 28 Mar 2021 18:18:15.427 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 28 Mar 2021 18:18:15.428 * DB loaded from disk: 0.000 seconds
redis_1 | 1:M 28 Mar 2021 18:18:15.428 * Ready to accept connections
flower_1 | standard_init_linux.go:219: exec user process caused: exec format error
web_1 | standard_init_linux.go:219: exec user process caused: exec format error
db_1 | 2021-03-28 18:18:15.777 UTC [19] LOG: database system was shut down at 2021-03-28 18:16:52 UTC
db_1 | 2021-03-28 18:18:15.791 UTC [1] LOG: database system is ready to accept connections
django-celery_celery_worker_1 exited with code 1
django-celery_flower_1 exited with code 1
django-celery_celery_beat_1 exited with code 1
django-celery_web_1 exited with code 1
UPDATED: added docker-compose.yml file for better references on the stack problem. The build succeeds but when running docker-compose up it doesn't and throws the celery error
docker-compose.yml
version: '3.8'
services:
web:
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
image: django_celery_example_web
# '/start' is the shell script used to run the service
command: /start
# this volume is used to map the files and folders on the host to the container
# so if we change code on the host, code in the docker container will also be changed
volumes:
- .:/app
ports:
- 8010:8000
# env_file is used to manage the env variables of our project
env_file:
- ./.env/.dev-sample
depends_on:
- redis
- db
db:
image: postgres:12.0-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_DB=hello_django_dev
- POSTGRES_USER=hello_django
- POSTGRES_PASSWORD=hello_django
redis:
image: redis:5-alpine
celery_worker:
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
image: django_celery_example_celery_worker
command: /start-celeryworker
volumes:
- .:/app
env_file:
- ./.env/.dev-sample
depends_on:
- redis
- db
celery_beat:
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
image: django_celery_example_celery_beat
command: /start-celerybeat
volumes:
- .:/app
env_file:
- ./.env/.dev-sample
depends_on:
- redis
- db
flower:
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
image: django_celery_example_celey_flower
command: /start-flower
volumes:
- .:/app
env_file:
- ./.env/.dev-sample
ports:
- 5557:5555
depends_on:
- redis
- db
volumes:
postgres_data:
I'm not sure it applies to your case, but in docker i start celery with this command:
command: celery -A my_proj worker -l DEBUG
Since the error it gives you is "exec format error" it might just be this.
I have a dockerized Django application using Postgresql. Everything runs find until I add volumes to the docker-compose file so I can have persistent data storage for the DB.
docker-compose.yml :
version: "3"
services:
app:
build:
context: .
ports:
- "8000:8000"
volumes:
- ./app:/app
command: >
sh -c "python manage.py wait_for_db &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8000"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=dbpassword
depends_on:
- db
db:
image: postgres:10-alpine
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=dbpassword
ports:
- "5432:5432"
Error Message here:
django.db.utils.OperationalError: could not connect to server: Connection refused
app_1 | Is the server running on host "db" (172.26.0.2) and accepting
app_1 | TCP/IP connections on port 5432?
app_1 |
recipe-api_app_1 exited with code 1
db_1 | 2019-08-31 22:52:35.969 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2019-08-31 22:52:35.969 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2019-08-31 22:52:35.972 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-08-31 22:52:36.042 UTC [18] LOG: database system was shut down at 2019-08-31 22:41:04 UTC
db_1 | 2019-08-31 22:52:36.055 UTC [1] LOG: database system is ready to accept connections
Once you add the volumes for the docker-compose I keep getting this error and without volumes everything runs fine.
EDIT : wait_for_db command :
def handle(self, *args, **options):
"""Handle the command"""
self.stdout.write('Waiting for database...')
db_conn = None
while not db_conn:
try:
db_conn = connections['default']
except OperationalError:
self.stdout.write('Database unavailable, waiting 1 second...')
time.sleep(1)
self.stdout.write(self.style.SUCCESS('Database available!'))
I do
docker-compose up
I get
$ docker-compose up
Starting asynchttpproxy_postgres_1
Starting asynchttpproxy_web_1
Attaching to asynchttpproxy_postgres_1, asynchttpproxy_web_1
postgres_1 | LOG: database system was interrupted; last known up at 2017-
05-01 18:52:29 UTC
postgres_1 | LOG: database system was not properly shut down; automatic
recovery in progress
postgres_1 | LOG: invalid record length at 0/150F410: wanted 24, got 0
postgres_1 | LOG: redo is not required
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
web_1 | Performing system checks...
web_1 |
web_1 | System check identified no issues (0 silenced).
My Dockerfile:
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/
My docker-compose.yml
postgres:
image: postgres:latest
volumes:
- ./code/
env_file:
- .env
volumes:
- /usr/src/app/static
expose:
- '5432'
web:
build: .
command: python3 manage.py runserver 0.0.0.0:8000
env_file:
- .env
volumes:
- .:/code
links:
- postgres
expose:
- '8000'
As you can see, the django server wont start. What am I doing wrong? Thanks in advance.
First, try to run on another terminal
docker ps
to check if you server really did not start.
And check if you postgres database setup is ready when your django application start, if not try to run the an bash script to see if the connection is setup at postgress to initialize the django container.
wait-bd.sh
#!/bin/bash
while true; do
COUNT_PG=`psql postgresql://username:password#localhost:5432/name_db -c '\l \q' | grep "name_db" | wc -l`
if ! [ "$COUNT_PG" -eq "0" ]; then
break
fi
echo "Waiting Database Setup"
sleep 10
done
and in docker-compose.yml add the tag command in the django container:
web:
build: .
command: /bin/bash wait-bd.sh && python3 manage.py runserver 0.0.0.0:8000
This script gonna wait you database setup, so will run the django setup container.