Docker run image_celery not able to detect redis - django

Question

I have a django application i want to run redis and celery using docker run command
after I build images using docker-compose file
I run two commands on different windows powershell
docker run -it -p 6379:6379 redis
docker run -it image_celery
my celery powershell is not able to detect redis
[2020-02-08 13:08:44,686: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/1: Error -2 connecting to redis:6379. Name or service not known..
Trying again in 2.00 seconds...
version: '3'
services:
the-redis:
image: redis:3.2.7-alpine
ports:
- "6379:6379"
volumes:
- ../data/redis:/data
celery_5:
build:
context: ./mltrons_backend
dockerfile: Dockerfile_celery
volumes:
- ./mltrons_backend:/code
- /tmp:/code/static
depends_on:
- the-redis
deploy:
replicas: 4
resources:
limits:
memory: 25g
restart_policy:
condition: on-failure
volumes:
db_data:
external: true
Dockerfile_celery
FROM python:3.6
ENV PYTHONUNBUFFERED 1
# Install Java
RUN apt-get -y update && \
apt install -y openjdk-11-jdk && \
apt-get install -y ant && \
apt-get clean && \
rm -rf /var/lib/apt/lists/ && \
rm -rf /var/cache/oracle-jdk11-installer;
ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code
ENV REDIS_HOST=redis://the-redis
ENV REDIS_PORT=6379
RUN pip install --upgrade 'sentry-sdk==0.7.10'
ENTRYPOINT celery -A mlbot_webservices worker -c 10 -l info
EXPOSE 8102
celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mlbot_webservices.settings')
app = Celery('mltrons_training')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
settings.py
CELERY_BROKER_URL = 'redis://the-redis:6379/'
CELERY_RESULT_BACKEND = 'redis://the-redis:6379/'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

Answer 1

It is expected because when you start container as you do (docker run IMAGE), the containers use the default bridge network of Docker.
You can check it by inspecting that network : docker network inspect bridge.
The default bridge doesn't accept network resolution of the containers by container name as you do (redis).
Besides the default name of a container is not the image name but a generated name by docker.
That's why you get that error at runtime :
Cannot connect to redis://redis:6379/1
Note that you can still reference containers belonging to the default bridge by their ip addresses, but that is generally undesirable because that hard code them from the client side.
That works with Docker compose because :
By default Compose sets up a single network for your app. Each
container for a service joins the default network and is both
reachable by other containers on that network, and discoverable by
them at a hostname identical to the container name.
To be able to communicate by container name with docker run, you need :
- to add these containers in the same network but not the default one provided by Docker
- to give an explicit name to the container that you want to reference (while doing it for both container is good to monitor/manager it more simply) by the other.
For example create a user-defined bridge network and add the containers to that when you start them :
docker network create -d bridge my-bridge-network
docker run -it -p 6379:6379 --network=my-bridge-network --name=redis redis
docker run -it --network=my-bridge-network --name=celery image_celery