Docker containers exiting without identifiable cause (Django web application) - django

Question

I've taken over the maintenance of a live web project that utilizes docker containers. Immediately, I've noticed that the web app goes down after a couple of hours, and docker ps -a shows me:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b02f1352f15 nginx:latest "nginx -g 'daemon off" 9 weeks ago Exited (1) 14 hours ago 80/tcp, 443/tcp, 0.0.0.0:80->8000/tcp ng01
8079b3d3b398 webapp_web "gunicorn --error-log" 9 weeks ago Exited (1) 14 hours ago 8000/tcp webapp_web_1
564fe0b72fa6 d0f5f9c3d3a6 "/bin/sh -c 'apt-get " 12 weeks ago Exited (0) 12 weeks ago modest_perlman
6cddbfcfa8f6 d0f5f9c3d3a6 "/bin/sh -c 'apt-get " 12 weeks ago Exited (0) 12 weeks ago backstabbing_goldwasser
7460be4f4451 postgres "/docker-entrypoint.s" 4 months ago Exited (1) 14 hours ago 5432/tcp webapp_db_1
Notice the 3 containers that exited 14 hours ago - those relate to the web app. How do I diagnose/fix this problem? Being a beginner, I'm struggling here. Thanks in advance! Following are some diagnostics I tried to run.
I used docker logs on the errant containers to see what could be going wrong.
docker logs 9b02f1352f15 (nginx) is empty.
docker logs 8079b3d3b398 (application server - gunicorn) shows many incidences of:
Exception in thread Thread-725265:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/site-packages/unirest/__init__.py", line 97, in __request
response = urllib2.urlopen(req, timeout=_timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/local/lib/python2.7/site-packages/poster/streaminghttp.py", line 142, in http_open
return self.do_open(StreamingHTTPConnection, req)
File "/usr/local/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 101] Network is unreachable>
docker logs 7460be4f4451 (postgresql backend) shows many incidences of:
LOG: database system was interrupted; last known up at 2017-01-22 12:42:46 UTC
LOG: database system was not properly shut down; automatic recovery in progress
LOG: invalid record length at 0/17AAD28
LOG: redo is not required
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
In case it matters, doing tail -f docker.log at /var/run/upstart/docker.log gives the output:
INFO[0000] Firewalld running: false
time="2017-01-23T14:38:47.142345718Z" level=error msg="devmapper: Error unmounting device 3da5c7e87cc8969249d7ed8b15c9cea9296feaefeba62fde534b4c183e4edbd4: Device is Busy"
time="2017-01-23T14:38:47.142542018Z" level=error msg="Error unmounting container 8079b3d3b3988a793537c4116bd12c70823415cb84068021d25658a316d8f568: Device is Busy"
INFO[0000] Firewalld running: false
INFO[0000] Firewalld running: false
time="2017-01-23T14:40:20.694963580Z" level=error msg="devmapper: Error unmounting device 6fd51632808dede3fea81b4f19fb84f1a13d93c38917e2845d0776de6a2ef941: Device is Busy"
time="2017-01-23T14:40:20.695010680Z" level=error msg="Error unmounting container 9b02f1352f15447acb7669bff918db0eeed58dc832fff565f4b2a4236474db1f: Device is Busy"
INFO[0000] Firewalld running: false
time="2017-01-23T14:41:27.307003457Z" level=error msg="devmapper: Error unmounting device 7c5a35dc9fc1929e57de43f5efc8e05a9442782c7496713d313c95fe62910f7b: Device is Busy"
time="2017-01-23T14:41:27.307059457Z" level=error msg="Error unmounting container 7460be4f445102274dd4aba4f113db23c17f58b9f771fc5c474d174766ae593c: Device is Busy"
I also tried docker inspect on all three. Following are the results relating to state - there seems to be nothing wrong here.
docker inspect 9b02f1352f15(nginx):
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2017-01-22T12:42:47.142236155Z",
"FinishedAt": "2017-01-22T23:38:46.7038628Z"
},
docker inspect 8079b3d3b398 (application server - gunicorn):
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2017-01-22T12:42:42.602662338Z",
"FinishedAt": "2017-01-22T23:38:46.5945186Z"
},
docker inspect 7460be4f4451 (postgresql backend):
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2017-01-22T12:42:34.413283342Z",
"FinishedAt": "2017-01-22T23:38:46.5102334Z"
},
docker-compose.yml is simply:
version: '2'
services:
db:
image: postgres
web:
build: .
command: gunicorn --error-logfile err.log myapp.wsgi:application -b 0.0.0.0:8000
volumes:
- .:/code
expose:
- "8000"
depends_on:
- db
nginx:
image: nginx:latest
container_name: ng01
ports:
- "80:8000"
volumes:
- .:/src
- ./config/nginx:/etc/nginx/conf.d
depends_on:
- web