running django worker and daphne in docker container - django

I have django application that run in docker container. Recently i figured out that i'm going to need to add websockets interface to my application. I'm using channels with daphne behind nginx and redis as a cache. The problem is that i have to run django workers and daphne in 1 container.
Script that is running on container startup:
#!/usr/bin/env bash
python wait_for_postgres.py
python manage.py makemigrations
python manage.py migrate
python manage.py collectstatic --no-input
python manage.py runworker --only-channels=http.* --only-channels=websocket.* -v2
daphne team_up.asgi:channel_layer --port 8000 -b 0.0.0.0
But it hangs on running a worker. I tried nohup but it seems to not work. If i run daphne directly from container with docker exec everything works just fine.

This is an old question, but I figured I will answer it anyway, because I recently faced the same issue and thought I can shed some light on this.
How Django channels work
Django Channels is another layer on top of Django and it has two process types:
One that accepts HTTP/Websockets
One that runs Django views, Websocket handlers, background tasks, etc
Basically, when a request comes in, it first hits the interface server (Daphne), which accepts the HTTP/Websocket connection and puts it on the Redis queue. The worker (consumer) then sees it, takes it off the queue and runs the view logic (e.g. Django views, WS handlers, etc).
Why it didn't work for you
Because you only run the worker (consumer) and it's blocking the execution of the interface server (producer). Meaning, that no connections will be accepted and worker is just staring at an empty redis queue.
How I made it work
I run Daphne, redis and workers as separate containers for easy scaling. DB migrations, static file collection, etc are executed only in Daphne container. This container will only have one instance running to ensure that there are no parallel db migrations running.
Workers on the other hand can be scaled up and down to deal with the incoming traffic.
How you could make it work
Split your setup into at least two containers. I wouldn't recommend running everything in one container (using Supervisor for example). Why? Because when the time comes to scale the setup there's no easy way to do it. You could scale your container to two instances, but that just creates another supervisor with daphne, redis, django in it... if you split the worker from daphne, you could easily scale the worker container to deal with growing incoming requests.
One container could run:
#!/usr/bin/env bash
python wait_for_postgres.py
python manage.py migrate
python manage.py collectstatic --no-input
daphne team_up.asgi:channel_layer --port 8000 -b 0.0.0.0
while the other one:
#!/usr/bin/env bash
python wait_for_postgres.py
python manage.py runworker --only-channels=http.* --only-channels=websocket.* -v2
The 'makemigrations' command
There is no need to run the command in the script you provided, if anything it could block the whole thing because of some question it is awaiting input for (e.g. "Did you rename column X to Y?").
Instead, you can execute it in a running container like this:
docker exec -it <container_name> python manage.py makemigrations

Related

why would you separate a celery worker and django container?

I am building a django app with celery. I tried composing a docker-compose without a container for the worker. In my Dockerfile for django, an entrypoint running the celery worker and django app:
...
python manage.py migrate
celery -A api worker -l INFO --detach
python manage.py runserver 0.0.0.0:8000
The celery will run using this order but not django runserver. I have seen in tutorials that they separated the django container from woker container or vice-versa. I do not see the explanation for this separation. I also observed that the two python container (django, worker) has the same volume. How can celery add tasks if it has a different environment with django? In my mind there would be two django apps (the same volume) for two containers only 1 running the runserver, and the other one running the celery worker. I do not understand the separation.
You should aim to set up your containers to run only a single foreground process in each container, and no background processes. Even in this simple example, there are two obvious advantages: if the Celery worker fails, you can restart a standalone container, but it's invisible to Docker as a background process; and you can separately read the docker logs of the Web server and background worker without having them intertwined. At larger scale you can imagine wanting to run different numbers of Django and Celery containers depending on your load.
To make this work it's important that the entrypoint script not run the program directly. It is passed the (possibly overridden) container command as arguments, and you can use a special shell construct to run that
#!/bin/sh
./manage.py migrate
exec "$#"
In the Dockerfile, declare both the ENTRYPOINT and a default CMD to run, say, the Web server
ENTRYPOINT ["./entrypoint.sh"] # probably unchanged, must be JSON array syntax
CMD ["./manage.py", "runserver", "0.0.0.0:8000"]
In a Compose setup, you can run multiple containers off the same image, but override the command: for a Celery worker.
version: '3.8'
services:
web:
build: .
ports: ['8000:8000']
environment:
REDIS_HOST: redis
worker:
build: .
command: celery -A api worker -l INFO
environment:
REDIS_HOST: redis
redis:
image: redis
The main application communicates with the worker via a queue in Redis (or another store), so there's no need for them to be in the same container.
As Celery documentation mentions:
Celery communicates via messages, usually using a broker to mediate
between clients and workers. To initiate a task the client adds a
message to the queue, the broker then delivers that message to a
worker.
Meaning the communication between the Client (Django) and Worker (Celery) are done through a message queue. Hence it does not matter if the workers and clients in separate containers or even separate machines. If the Client can access the message queue (for example using Redis or RabbitMQ) and worker can pop tasks from that queue, it will always work.
About the docker-compose part, there is no ideal standard for keeping or separating Celery and Django. You can put them in same container or not, it is up to you and what are the requirements of the project. If you are using two containers, then they need to share volumes because of the source code and any other data which are needed for executing tasks.

How to keep Celery running in Django (drf) + Redis + WSGI (EC2)

I don't think its a very new question. I just could not find the right answer. I am trying to use Celery for background tasks while implementing a backend with the Django Rest Framework. I have a Redis server.
Celery is working as expected with
celery worker -A my_project --loglevel=info
However, it does not work if I sop this command. How do I keep that running? I have found a blog with supervisor. I just want to know what is the standard (as well as easier) to do this.
What you should do is go for docker and use docker-compose for your services. But if you're just testing stuff:
$ nohup celery worker -A my_project --loglevel=info &
& is used to take the process to the background, you can recall it using fg, suspend it to bg using Ctrl + Z, nohup makes sure that celery will remain functioning even if you close the ssh session.
Edit: The only drawback using this method, is that if the process exits, then you'll have to invoke it again. In a production environment, you should go for docker with docker-compose.

Run django migrate in docker

I am building a Python+Django development environment using docker. I defined Dockerfile files and services in docker-compose.yml for web server (nginx) and database (postgres) containers and a container that will run our app using uwsgi. Since this is a dev environment, I am mounting the the app code from the host system, so I can easily edit it in my IDE.
The question I have is where/how to run migrate command.
In case you don't know Django, migrate command creates the database structure and later changes it as needed by the project. I have seen people run migrate as part of the compose command directive command: python manage.py migrate && uwsgi --ini app.ini, but I do not want migrations to run on every container restart. I only want it to run once when I create the containers and never run again unless I rebuild.
Where/how would I do that?
Edit: there is now an open issue with the compose team. With any luck, one time command containers will get supported by compose. https://github.com/docker/compose/issues/1896
You cannot use RUN because as you mentioned in the comments your source is mounted during running of the container.
You cannot use CMD either since you don't want it to run everytime you restart the container.
I recommend using docker exec manually after running the container. I do not think there is a way to automate this inside a dockerfile or docker-compose because of the two reasons I gave above.
It sounds like what you need is a tool for managing project tasks. dobi is a tool designed to handle these tasks (disclaimer: I am the author of this tool).
You can see an example of how to run a migration here: https://github.com/dnephin/dobi/tree/master/examples/init-db-with-rails. The example uses rails, but it's basically the same idea as django.
You could setup a task called migrate which would run the command in a container and write the data to a volume. Then when you start your docker-compose containers, use that volume as the source for your database service.
https://github.com/docker/compose/issues/1896 is finally resolved now by the new service profiles introduced with docker-compose 1.28.0. With profiles you can mark services to be only started in specific profiles:
services:
nginx:
# ...
postgres:
# ...
uwsgi:
# ...
migrations:
profiles: ["cli-only"] # profile name chosen freely
# ...
docker-compose up # start only your app services, no migrations
docker-compose run migrations # run migrations on-demand
docker exec -it container-name bash
Then you will be inside the container and you can run any command you normally do when you develop without using docker.

Running migrations when deploying django app to heroku with codeship

I'm trying to set up a continous integration pipeline for my python 3.5.1 / django 1.9.7 project.
The project is running fine on heroku, and the codeship deployment pipeline for heroku works well as long as my database is unchanged.
If I want to run migrations, I have to do so manually by entering heroku run python manage.py migrate on my computer which I would like to avoid.
I added a "Custom Script" in my codeship deployment pipeline after the "heroku"-pipeline containing heroku run python manage.py migrate, but when coedship attempts to execute it, it fails with the
Cannot run more than 1 Free size dynos.
message. I assume this is because the server is already up and running and I don't have more worker processes available? (please correct me if I'm wrong)
EDIT: This is where I was wrong - I had an additional process running (see answer)
Is there any way to include the database migration step in the heroku deployment pipeline? Or did I do something wrong?
Ifound the answer here: Heroku: Cannot run more than 1 Free size dynos
My assumption about theweb server beeing the blocking dyno was wrong, I had a zombie process (createsuperuser) running I did not know about.
I used heroku ps to show all running prcesses. Output was:
=== web (Free): gunicorn my_app.wsgi --log-file - (1)
web.1: idle 2016/06/07 17:09:06 +0200 (~ 13h ago)
=== run: one-off processes (1)
run.7012 (Free): up 2016/06/07 15:19:13 +0200 (~ 15h ago): python manage.py createsuperuser
I killed the process by typing
heroku ps:stop run.7012
and afterwards my migration via codeship custom script worked as expected.

Running Docker-Compose Commands With Fabric

I have a Django site running in Docker containers, which uses docker-compose to manage the various containers (database, nginx, etc.). There are a few Django tasks that I use for site maintenance using the Django manage.py command. They commands take the form of:
manage.py updateflickr --settings=mysite.myproj.prod
Running under docker-compose, they look like:
docker-compose run --rm app manage.py updateflickr --settings=mysite.myproj.prod
My problem is that when I try to run these same commands using Fabric, it appears that the settings file I am specifying is not being used. Django is returning database connection errors, which typically mean that it is not getting the correct database information, or in this case the connection specified in mysite.myprod.prod
My Fabric file looks like:
import os
from fabric.api import *
env.hosts = ['myserver.com']
env.user = "myuser"
env.key_filename = '~/.ssh/do_rsa'
env.shell = '/bin/bash -c'
#task
def updateflickr():
run('docker-compose run --rm app python manage.py updateflickr --settings=mysite.myproj.prod')
I have also expirimented with setting the DJANGO_SETTINGS_MODULE environment variable in my docker-compose.yml but am getting the same results. Finally, the last thing I tried was wrapping the command in a shell script. Same results - if I run on the server, it runs fine. If I run the shell script from Fabric, I get database connection issues.
UPDATE
I am not so sure this is so much a question about Fabric, then a question about how docker-compose runs. If I try the following:
ssh -t me#myserver.com 'docker-compose run --rm app python manage.py updateflickr --settings=mysite.myproj.prod'
I still get the same results. There must be something different about loading up an interactive shell with just sending a command. I have tried using ssh with and without a -t flag, because docker-compose might need a pty active.