I have the following heroku.yml. The 'containers' share the same Dockerfile:
build:
docker:
web: Dockerfile
celery: Dockerfile
celery-beat: Dockerfile
release:
image: web
command:
- python manage.py migrate users && python manage.py migrate
run:
web: python manage.py runserver 0.0.0.0:$PORT
celery: celery --app=my_app worker --pool=prefork --concurrency=4 --statedb=celery/worker.state -l info
celery-beat: celery --app=my_app beat -l info
I intended to have three containers, but it turns out that Heroku accepts only one web and the other should be workers.
So what do I modify at heroku.yml to have celery and celery-beat containers as worker?
UPDATE
I've changed the heroku.yml to the following, but Heroku keeps only the last worker (i.e. celery beat) and ignores the first worker:
build:
docker:
web: Dockerfile
release:
image: web
command:
- python manage.py migrate users && python manage.py migrate
run:
web: python manage.py runserver 0.0.0.0:$PORT
worker:
command:
- celery --app=my_app worker --pool=prefork --concurrency=4 --statedb=celery/worker.state -l info
image: web
worker:
command:
- celery --app=my_app beat -l info
image: web
What am I missing?
The name worker isn't really important:
No process types besides web and release have special properties
So just give them different names:
run:
web: python manage.py runserver 0.0.0.0:$PORT
celery_worker:
command:
- celery --app=my_app worker --pool=prefork --concurrency=4 --statedb=celery/worker.state -l info
image: web
celery_beat:
command:
- celery --app=my_app beat -l info
image: web
When you scale those processes, use the names celery_worker and celery_beat.
A better option is to combine the celery worker and beat in a single worker / command : (can only be done on Linux os)
run:
web: python manage.py runserver 0.0.0.0:$PORT
celery_worker:
command:
- celery --app=my_app worker --pool=prefork --concurrency=4 --statedb=celery/worker.state -l info --beat -l info
image: web
Related
Question
I am a beginner with docker; this being the first project I have set up with it and don't particularly know what I am doing. I would very much appreciate if someone could give me some advice on what the best way to get migrations from a dockerized django app to store locally
What I have tried so far
I have a local django project setup with the following file structure:
Project
.docker
-Dockerfile
project
-data
-models
- __init__.py
- user.py
- test.py
-migrations
- 0001_initial.py
- 0002_user_role.py
...
settings.py
...
manage.py
Makefile
docker-compose.yml
...
In the current state the migrations for the test.py model have not been run; so I attempted to do so using docker-compose exec main python manage.py makemigrations. This worked successfully returning the following:
Migrations for 'data':
project/data/migrations/0003_test.py
- Create model Test
But produced no local file. However, if I explore the file system of the container I can see that the file exists on the container itself.
Upon running the following:
docker-compose exec main python manage.py migrate
I receive:
Running migrations:
No migrations to apply.
Your models in app(s): 'data' have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
I was under the impression that even if this did not create the local file it would at least run the migrations on the container.
Regardless, my intention was that when I run docker-compose exec main python manage.py makemigrations it store the file locally in the project/data/migrations folder and then I just run migrate manually. I can't find much documentation on how to do this; the only post I have seen suggested bind mounts (Migrations files not created in dockerized Django) which I attempted by adding the following to my docker-compose file:
volumes:
- type: bind
source: ./data/migrations
target: /var/lib/migrations_test
but I was struggling to get it to work and following from this I had no idea how to run commands through this volume using docker-compose and I was questioning whether this was even a good idea as I had read somewhere it was not best practice to use bind mounts.
Project setup:
The docker-compose.yml file looking like so:
version: '3.7'
x-common-variables: &common-variables
ENV: 'DEV'
DJANGO_SETTINGS_MODULE: 'project.settings'
DATABASE_NAME: 'postgres'
DATABASE_USER: 'postgres'
DATABASE_PASSWORD: 'postgres'
DATABASE_HOST: 'postgres'
CELERY_BROKER_URLS: 'redis://redis:6379/0'
volumes:
postgres:
services:
main:
command:
python manage.py runserver 0.0.0.0:8000
build:
context: ./
dockerfile: .docker/Dockerfile
target: main
environment:
<<: *common-variables
ports:
- '8000:8000'
env_file:
- dev.env
networks:
- default
postgres:
image: postgres:13.6
volumes:
- postgres:/var/lib/postgresql/data
ports:
- '25432:5432'
environment:
POSTGRES_PASSWORD: 'postgres'
command: postgres -c log_min_messages=INFO -c log_statement=all
wait_for_dependencies:
image: dadarek/wait-for-dependencies
environment:
SLEEP_LENGTH: '0.5'
redis:
image: redis:latest
ports:
- '16379:6379'
worker:
build:
context: .
dockerfile: .docker/Dockerfile
target: main
command: celery -A project worker -l INFO
environment:
<<: *common-variables
volumes:
- .:/code/delegated
env_file:
- dev.env
networks:
- default
beat:
build:
context: .
dockerfile: .docker/Dockerfile
target: main
command: celery -A project beat -l INFO
environment:
<<: *common-variables
volumes:
- .:/code/delegated
env_file:
- dev.env
networks:
- default
networks:
default:
Makefile:
build: pre-run
build:
docker-compose build --pull
dev-deps: pre-run
dev-deps:
docker-compose up -d postgres redis
docker-compose run --rm wait_for_dependencies postgres:5432 redis:6379
migrate: pre-run
migrate:
docker-compose run --rm main python manage.py migrate
setup: build dev-deps migrate
up: dev-deps
docker-compose up -d main
Dockerfile:
FROM python:3.10.2 as main
ENV PYTHONUNBUFFERED 1
COPY ./requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
RUN mkdir -p /code
WORKDIR /code
ADD . ./
RUN useradd -m -s /bin/bash app
RUN chown -R app:app .
USER app
EXPOSE 8000
Follow up based on diptangsu-goswami's response
I tried adding the following:
volumes:
- type: bind
source: C:\dev\Project\project
target: /code/
This creates an empty directory in my Project folder; named C:\dev\Project\project but the app doesn't run as it cannot find the manage.py file... I assumed this was because it was in the parent directory Project and tried again with:
volumes:
- type: bind
source: C:\dev\Project
target: /code/
But the same problem occured. Why is it creating the empty directory? surely it should just be binding the existing directory with the container directory? Also using this method, would I need to change my Dockerfile to not copy the codebase to the container in the first place and just mount it on instead?
I managed to fix it by adding the following to my 'main' service in my docker compose:
volumes:
- .:/code:delegated
I have the following heroku.yml file for containers deployment:
build:
docker:
release:
dockerfile: Dockerfile
target: release_image
web: Dockerfile
config:
PROD: "True"
release:
image: web
command:
- python manage.py collectstatic --noinput && python manage.py migrate users && python manage.py migrate
run:
# web: python manage.py runserver 0.0.0.0:$PORT
web: daphne config.asgi:application --port $PORT --bind 0.0.0.0 -v2
celery:
command:
- celery --app=my_app worker --pool=prefork --concurrency=4 --statedb=celery/worker.state -l info
image: web
celery_beat:
command:
- celery --app=my_app beat -l info
image: web
When I deploy I get the following warning, which does not make any sense to me:
Warning: You have declared both a release process type and a release section. Your release process type will be overridden.
My Dockerfile is composed of two stages and I want to keep only the release_image stage:
FROM python:3.8 as builder
...
FROM python:3.8-slim as release_image
...
According to the docs the proper way to choose release_image is to use the target section within build step.
But it also mentions that I can run my migrations within a release section.
So what am I supposed to do to get rid of this warning? I could only live with it if it was clear that both my migrations and target are being considered during deploy.Thanks in advance!
I want to keep only the release_image stage
Assuming this is true for your web process as well, update your build section accordingly:
build:
docker:
web:
dockerfile: Dockerfile
target: release_image
config:
PROD: "True"
Now you only have one process type defined and it targets the build stage you want to use.
Since you can run your migrations from the web container there's no need to build a whole container just for your Heroku release process. (And since your release section uses the web image the release process defined in build wouldn't have been for anything used anyway.)
I'm using the following with my Django application:
Django channels
Celery with both regular and periodic tasks
Deployed on Heroku
My Procfile looks like this:
web: daphne artist_notify.asgi:channel_layer --port $PORT --bind 0.0.0.0 -v2
worker: python manage.py migrate --noinput && python manage.py runworker -v2
celerybeat: celery -A artist_notify beat -l info
celeryworker: celery -A artist_notify worker -l info
This combination seems to be expensive, and I'm wondering if I can make it better. I tried combining celerybeat and celeryworker (with &&) into one dyno called celery, like so:
celery: celery -A artist_notify beat -l info && celery -A artist_notify worker -l info
But that doesn't work, although other && combinations do work. I'm wondering if I can maybe combine the workers from worker and celeryworker?
How one deploys the following stack on Heroku platform ?
Django
Django Channels
Celery
The limitation surely is on the Procfile.
To deploy Django with Celery it would be something like:
web: gunicorn project.wsgi:application
worker: celery worker --app=project.taskapp --loglevel=info
While deploying Django with Channels:
web: daphne project.asgi:channel_layer --port $PORT --bind 0.0.0.0 -v2
worker: python manage.py runworker -v2
The web process can use ASGI, but the worker process will be used by Channels and I don't see how Celery can be started alongside it.
You can have as many entries in the Procfile as you like. The only one that is special is "web", because that's the one Heroku expects to receive web requests, and the only one it will start automatically for you. You can use your own names for the rest:
web: gunicorn project.wsgi:application
celeryworker: celery worker --app=project.taskapp --loglevel=info
channelsworker: python manage.py runworker -v2
Now you can do heroku ps:scale celeryworker=1 and heroku ps:scale channelsworker=1 to start the other two processes.
See the Heroku Procfile docs for more information.
I am trying to deploy the simplest possible "hello world" celery configuration on heroku for my Django app. My Procfile is as follows:
web: gunicorn myapp.wsgi
worker: celery -A myapp worker -l info -B -b amqp://XXXXX:XXXXX#red-thistle-3.bigwig.lshift.net:PPPP/XXXXX
This is the RABBITMQ_BIGWIG_RX_URL that I'm giving to the celery worker. I have the corresponding RABBITMQ_BIGWIG_TX_URL in my settings file as the BROKER_URL.
If I use these broker URLs in my local dev environment, everything works fine and I can actually use the Heroku RabbitMQ system. However, when I deploy my app to Heroku it isn't working.
This Procfile seems to work (although Celery is giving me memory leak issues).
web: gunicorn my_app.wsgi
celery: celery worker -A my_app -l info --beat -b amqp://XXXXXXXX:XXXXXXXXXXXXXXXXXXXX#red-thistle-3.bigwig.lshift.net:PPPP/XXXXXXXXX