Django Celery Beat cannot connect to postgres database from inside docker container - django

Question

My docker contianer django server, postgres, celery worker and celery beat. Running the application by itself there is no issue with connecting to the database however when the timed task to update the database runs I get an error
celery_1 | django.db.utils.OperationalError: could not connect to server: No such file or directory
celery_1 | Is the server running locally and accepting
celery_1 | connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
below are my files
docker-compose.yaml
version: "3"
services:
redis:
image: redis:alpine
db:
image: postgres:10-alpine
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
ports:
- "5432:5432"
volumes:
- "db:/var/lib/postgresql/data"
pgadmin:
image: dpage/pgadmin4
depends_on:
- db
ports:
- "5555:80"
environment:
- PGADMIN_DEFAULT_EMAIL=dev#localhost
- PGADMIN_DEFAULT_PASSWORD=password
dj-dividend:
container_name: dividend
build: backend
command: sh -c "python manage.py wait_for_db && python manage.py makemigrations && python manage.py migrate && python manage.py runserver 0.0.0.0:80"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=supersecretpassword
volumes:
- ./backend:/code
ports:
- "80:80"
depends_on:
- db
- redis
celery:
build: backend
command: celery -A backend worker -l info
volumes:
- ./backend:/code
depends_on:
- db
- redis
celery-beat:
build: backend
command: celery -A backend beat -l info
volumes:
- ./backend:/code
depends_on:
- db
- redis
- celery
volumes:
db:
settings.py
import os
from celery.schedules import crontab
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '1a0#6lsfxtfrt0n=#q1&up6hf$px-$ug^n%^v!n)^iwp_n*7c%'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['*']
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'dividends',
'rest_framework',
'django_celery_beat',
'core'
]
CORS_ORIGIN_ALLOW_ALL = True
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'corsheaders.middleware.CorsMiddleware',
]
ROOT_URLCONF = 'backend.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'backend.wsgi.application'
# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': os.environ.get('DB_HOST'),
'NAME': os.environ.get('DB_NAME'),
'USER': os.environ.get('DB_USER'),
'PASSWORD': os.environ.get('DB_PASS'),
}
}
# Password validation
# https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/2.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.2/howto/static-files/
STATIC_URL = '/static/'
#Celery
CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_BEAT_SCHEDULE = {
"get_stocks": {
'task': 'dividends.tasks.get_stocks',
'schedule': crontab(minute="*/1") #every minute
}
}
Dockerfile
FROM python:3.7
# Set environment variables
ENV PYTHONUNBUFFERED 1
COPY requirements.txt /
# Install dependencies.
RUN pip install -r /requirements.txt
# Set work directory.
RUN mkdir /code
WORKDIR /code
# Copy project code.
COPY . /code/
EXPOSE 80
tasks.py
from celery import shared_task
from .utils import get_stock_data
from .models import Stock
#shared_task(default_retry_delay=5, max_retries=3)
def get_stocks():
for stock in Stock.objects.all().values_list('ticker'):
try:
get_stock_data(stock[0])
except:
continue
celery.py
from celery import Celery
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings')
app = Celery('backend')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

Answer 1

you need to add environment variables of the postgresql to the celery-beat

Answer 2

In your database container you set :
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
Then you need to copy and paste these lines in your celery_beat container so that it can connect to your database.
You'd then get:
celery-beat:
build: backend
command: celery -A backend beat -l info
volumes:
- ./backend:/code
depends_on:
- db
- redis
- celery
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword