Why is the Django app I moved to Heroku crashing inexplicably? - django

I have a locally running Django App which I'm trying to move into Heroku. I'm following and adapting these instructions to do so. However, when I push my code to Heroku, the app is unavailable via Web Browser. When I tail the logs, I see many lines such as these:
2014-09-25T17:02:21.927991+00:00 app[web.1]: 2014-09-25 17:02:21 [2] [CRITICAL] WORKER TIMEOUT (pid:10)
2014-09-25T17:02:23.050632+00:00 app[web.1]: 2014-09-25 17:02:23 [11] [INFO] Booting worker with pid: 11
2014-09-25T17:02:54.021864+00:00 app[web.1]: 2014-09-25 17:02:54 [2] [CRITICAL] WORKER TIMEOUT (pid:11)
2014-09-25T17:02:55.039070+00:00 app[web.1]: 2014-09-25 17:02:55 [12] [INFO] Booting worker with pid: 12
2014-09-25T17:03:26.061225+00:00 app[web.1]: 2014-09-25 17:03:26 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
2014-09-25T17:03:27.081593+00:00 app[web.1]: 2014-09-25 17:03:27 [13] [INFO] Booting worker with pid: 13
Why am I getting this indecipherable error and how can I fix it?
What other things should I look at to troubleshoot this?
Here is the contents of my Procfile:
web: gunicorn MyProject.wsgi --log-file -
EDIT: FYI: When I run this same app locally using foreman start, it works.

Related

Why AWS Cloudwatch don't show me all the logs?

I deployed my neural network via AWS ECS and I want to see what happen with it, but all what I can see is:
[2022-03-14 15:23:10 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2022-03-14 15:23:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2022-03-14 15:23:10 +0000] [1] [INFO] Using worker: threads
[2022-03-14 15:23:10 +0000] [10] [INFO] Booting worker with pid: 10
[2022-03-14 15:38:44 +0000] [1] [INFO] Handling signal: term
And only when I kill task I see other logs:
[2022-03-14 15:23:10 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2022-03-14 15:23:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2022-03-14 15:23:10 +0000] [1] [INFO] Using worker: threads
[2022-03-14 15:23:10 +0000] [10] [INFO] Booting worker with pid: 10
[2022-03-14 15:38:44 +0000] [1] [INFO] Handling signal: term
Traceback (most recent call last):
File "/app/app.py", line 98, in cartoonize
last_line = file.readlines()[-1]
IndexError: list index out of range
Traceback (most recent call last):
File "/app/app.py", line 72, in cartoonize
aligned_face = align_face(pil_image)
File "/app/util.py", line 134, in align_face
lm = get_landmark(np_img, predictor)
File "/app/util.py", line 112, in get_landmark
assert len(dets) > 0, "Face not detected, try another face image"
AssertionError: Face not detected, try another face image
Traceback (most recent call last):
File "/app/app.py", line 72, in cartoonize
aligned_face = align_face(pil_image)
File "/app/util.py", line 134, in align_face
lm = get_landmark(np_img, predictor)
File "/app/util.py", line 112, in get_landmark
assert len(dets) > 0, "Face not detected, try another face image"
AssertionError: Face not detected, try another face image
[2022-03-14 15:38:45 +0000] [10] [INFO] Worker exiting (pid: 10)
Loading e4e over the pSp framework from checkpoint: models/e4e_ffhq_encode.pt
None
RuntimeError: The session is unavailable because no secret key was set. Set the secret_key on the application to something unique and secret.
RuntimeError: The session is unavailable because no secret key was set. Set the secret_key on the application to something unique and secret.
None
None
[2022-03-14 15:38:45 +0000] [1] [INFO] Shutting down: Master
How can I see all the logs in real time?

Is it possible to get a stack trace when a gunicorn worker hits a timeout?

Specifically, I'm running a Flask app with default workers in gunicorn. I'm trying to figure out how to debug / trace what is happening when a worker is killed due to timeout while serving a request. Is there a way to get a stack trace or profile the request to debug this?

Honestly a very good question, I was not sure if it is even possible to achieve the same or not. But as I started digging I found lot of interesting threads
Showing the stack trace from a running Python application
Get stacktrace from stuck python process
https://github.com/khamidou/lptrace
https://gist.github.com/reywood/e221c4061bbf2eccea885c9b2e4ef496
So first I created a simple flask app with below code
app.py
from flask import Flask
app = Flask(__name__)
import time
def a():
b()
def b():
c ()
def c():
i = 0
while i < 900:
time.sleep(1)
i += 1
#app.route('/', defaults={'path': ''})
#app.route('/<path:path>')
def catch_all(path):
a()
return 'You want path: %s' % path
if __name__ == '__main__':
app.run()
wsgi.py
from app import app
if __name__ == "__main__":
app.run()
Now running the app like below and doing curl localhost:8000/abc
$ gunicorn wsgi:app
[2019-08-01 08:19:06 +0000] [26825] [INFO] Starting gunicorn 19.9.0
[2019-08-01 08:19:06 +0000] [26825] [INFO] Listening at: http://127.0.0.1:8000 (26825)
[2019-08-01 08:19:06 +0000] [26825] [INFO] Using worker: sync
[2019-08-01 08:19:06 +0000] [26828] [INFO] Booting worker with pid: 26828
[2019-08-01 08:19:40 +0000] [26825] [CRITICAL] WORKER TIMEOUT (pid:26828)
[2019-08-01 08:19:40 +0000] [26828] [INFO] Worker exiting (pid: 26828)
[2019-08-01 08:19:40 +0000] [26832] [INFO] Booting worker with pid: 26832
Now what we need is a hook which can be called before the worked is killed. gunicorn supports server events in the configuration file
So now we create a config file
gunicorn_config.py
timeout = 3
def worker_abort(worker):
pid = worker.pid
print("worker is being killed - {}".format(pid))
And our output is now
$ gunicorn -c gunicorn_config.py wsgi:app
[2019-08-01 08:22:17 +0000] [26837] [INFO] Starting gunicorn 19.9.0
[2019-08-01 08:22:17 +0000] [26837] [INFO] Listening at: http://127.0.0.1:8000 (26837)
[2019-08-01 08:22:17 +0000] [26837] [INFO] Using worker: sync
[2019-08-01 08:22:17 +0000] [26840] [INFO] Booting worker with pid: 26840
[2019-08-01 08:22:22 +0000] [26837] [CRITICAL] WORKER TIMEOUT (pid:26840)
worker is being killed - 26840
[2019-08-01 08:22:22 +0000] [26840] [INFO] Worker exiting (pid: 26840)
[2019-08-01 08:22:22 +0000] [26844] [INFO] Booting worker with pid: 26844
This is good, now we need mix our previous knowledge on pyrasite and this to get the stack. So we updated the config file like below
gunicorn_config.py
timeout = 3
__code_dump_stack__ = """
import sys, traceback
for thread, frame in sys._current_frames().items():
print('Thread 0x%x' % thread)
traceback.print_stack(frame)
print()
"""
def dump_stack_for_process(pid):
import pyrasite
ipc = pyrasite.PyrasiteIPC(pid)
ipc.connect()
print(ipc.cmd(__code_dump_stack__))
ipc.close()
def worker_abort(worker):
pid = worker.pid
print("worker is being killed - {}".format(pid))
dump_stack_for_process(pid)
And now our output is
$ [2019-08-01 08:25:29 +0000] [26848] [INFO] Starting gunicorn 19.9.0
[2019-08-01 08:25:29 +0000] [26848] [INFO] Listening at: http://127.0.0.1:8000 (26848)
[2019-08-01 08:25:29 +0000] [26848] [INFO] Using worker: sync
[2019-08-01 08:25:29 +0000] [26851] [INFO] Booting worker with pid: 26851
[2019-08-01 08:25:38 +0000] [26848] [CRITICAL] WORKER TIMEOUT (pid:26851)
worker is being killed - 26851
Thread 0x7ff0a7a4b700
File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "<string>", line 72, in run
File "<string>", line 92, in on_command
File "<string>", line 6, in <module>
Thread 0x7ff0ac512700
File "/home/vagrant/.local/bin/gunicorn", line 11, in <module>
sys.exit(run())
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 61, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/app/base.py", line 223, in run
super(Application, self).run()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 203, in run
self.manage_workers()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
self.spawn_workers()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
self.spawn_worker()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.run()
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 124, in run
self.run_for_one(timeout)
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 68, in run_for_one
self.accept(listener)
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 30, in accept
self.handle(listener, client, addr)
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135, in handle
self.handle_request(listener, req, client, addr)
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/home/vagrant/.local/lib/python3.5/site-packages/flask/app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "/home/vagrant/.local/lib/python3.5/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/home/vagrant/.local/lib/python3.5/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/home/vagrant/.local/lib/python3.5/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/vagrant/remotedebug/app.py", line 20, in catch_all
a()
File "/home/vagrant/remotedebug/app.py", line 6, in a
b()
File "/home/vagrant/remotedebug/app.py", line 9, in b
c ()
File "/home/vagrant/remotedebug/app.py", line 14, in c
time.sleep(1)
File "/home/vagrant/.local/lib/python3.5/site-packages/gunicorn/workers/base.py", line 195, in handle_abort
self.cfg.worker_abort(self)
File "gunicorn_config.py", line 23, in worker_abort
dump_stack_for_process(pid)
File "gunicorn_config.py", line 17, in dump_stack_for_process
print(ipc.cmd(__code_dump_stack__))
File "/home/vagrant/.local/lib/python3.5/site-packages/pyrasite/ipc.py", line 161, in cmd
return self.recv()
File "/home/vagrant/.local/lib/python3.5/site-packages/pyrasite/ipc.py", line 174, in recv
header_data = self.recv_bytes(4)
File "/home/vagrant/.local/lib/python3.5/site-packages/pyrasite/ipc.py", line 187, in recv_bytes
chunk = self.sock.recv(n - len(data))
[2019-08-01 08:25:38 +0000] [26851] [INFO] Worker exiting (pid: 26851)
[2019-08-01 08:25:38 +0000] [26862] [INFO] Booting worker with pid: 26862
The stack trace is large but it gives us what we need

Just wanted to add to the accepted answer that there's, also, another way to get similar results by using the built-in Python 3 module faulthandler introduced in v3.3 https://docs.python.org/dev/library/faulthandler.html
You can add the following two lines to wsgi.py mentioned in the accepted answer:
import faulthandler
faulthandler.enable()
When timeout occurs the following output will be printed to stderr:
[2020-12-24 13:38:32 +0000] [31304] [INFO] Starting gunicorn 20.0.4
[2020-12-24 13:38:32 +0000] [31304] [INFO] Listening at: http://0.0.0.0:8888 (31304)
[2020-12-24 13:38:32 +0000] [31304] [INFO] Using worker: sync
[2020-12-24 13:38:32 +0000] [31307] [INFO] Booting worker with pid: 31307
[2020-12-24 13:38:55 +0000] [31304] [CRITICAL] WORKER TIMEOUT (pid:31307)
Fatal Python error: Aborted
Current thread 0x00007f411d781700 (most recent call first):
File "/tmp/app.py", line 14 in c
File "/tmp/app.py", line 9 in b
File "/tmp/app.py", line 6 in a
File "/tmp/app.py", line 20 in catch_all
File "/tmp/venv/lib/python3.5/site-packages/flask/app.py", line 1936 in dispatch_request
File "/tmp/venv/lib/python3.5/site-packages/flask/app.py", line 1950 in full_dispatch_request
File "/tmp/venv/lib/python3.5/site-packages/flask/app.py", line 2447 in wsgi_app
File "/tmp/venv/lib/python3.5/site-packages/flask/app.py", line 2464 in __call__
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 175 in handle_request
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 134 in handle
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 29 in accept
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 67 in run_for_one
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 123 in run
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 140 in init_process
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 583 in spawn_worker
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 616 in spawn_workers
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 545 in manage_workers
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 202 in run
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/app/base.py", line 72 in run
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/app/base.py", line 228 in run
File "/tmp/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 58 in run
File "/tmp/venv/bin/gunicorn", line 8 in <module>
[2020-12-24 13:38:55 +0000] [31307] [INFO] Worker exiting (pid: 31307)
[2020-12-24 13:38:55 +0000] [31503] [INFO] Booting worker with pid: 31503

How to fix "server error(500)" of Heroku caused by setting DEBUG to False in Django?

I know this issue has been discussed a thousand times here, but I couldn't stop putting it again as I'd a difficult time finding the cause of this problem and a working solution.So, my problem is when I set DEBUG=True on heroku, everything works, and if it is set to False, it gives server error(500). Please help me!
Things I've tried and didn't work:
run collectstatic locally and then tried to add, commit and push to Heroku, but didn't work.
done ALLOWED_HOSTS = ["herokuapp.com"]
Anyhow when nothing worked, switched the database from sqlite to postgres and it is working fine.But somehow, the error still persists!
My wsgi.py file looks like this:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "website.settings")
application = get_wsgi_application()
The heroku logs I see:
2019-01-05T17:41:17.844144+00:00 app[web.1]: [2019-01-05 17:41:17 +0000] [10] [INFO] Booting worker with pid: 10
2019-01-05T17:41:17.857502+00:00 app[web.1]: [2019-01-05 17:41:17 +0000] [11] [INFO] Booting worker with pid: 11
2019-01-05T17:41:18.234497+00:00 app[web.1]: base dir path /app
2019-01-05T17:41:18.240629+00:00 app[web.1]: local settings failed
2019-01-05T17:41:18.357015+00:00 app[web.1]: base dir path /app
2019-01-05T17:41:18.358185+00:00 app[web.1]: local settings failed
2019-01-05T17:41:19.010564+00:00 heroku[web.1]: State changed from starting to up
2019-01-05T17:41:21.230071+00:00 app[web.1]: 10.28.83.116 - - [05/Jan/2019:17:41:21 +0000] "GET / HTTP/1.1" 500 27 "https://dashboard.heroku.com/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0"
2019-01-05T17:41:21.230836+00:00 heroku[router]: at=info method=GET path="/" host=obscure-dusk-29347.herokuapp.com request_id=b5fb4554-fa37-4e39-b710-fae1ac8c6651 fwd="103.208.75.78" dyno=web.1 connect=1ms service=184ms status=500 bytes=234 protocol=https
2019-01-05T17:41:21.670528+00:00 heroku[router]: at=info method=GET path="/favicon.ico" host=obscure-dusk-29347.herokuapp.com request_id=0d271d43-17a0-4b37-aff1-51767d192d35 fwd="103.208.75.78" dyno=web.1 connect=1ms service=20ms status=404 bytes=266 protocol=https

I did it, finally!
The DEBUG variable in my settings.py was like DEBUG = config('DEBUG', default=False, cast=bool) . So I just changed it to be DEBUG = config('DEBUG') and it worked.

Sentry ignores --config path, but only for "start" command

I've installed Sentry into a virtualenv located at /www/sentry/ and I have a config file /www/sentry/sentry.conf.py. I am able to run the following commands successfully:
$ sentry --config=/www/sentry/sentry.conf.py celery worker -B
$ sentry --config=/www/sentry/sentry.conf.py upgrade
I can even run sentry --config=/www/sentry/sentry.conf.py shell and then once in the Django shell, check that the settings module imported from django.conf has the custom settings I added in my sentry.conf.py file.
However, when I try to spin up the included Gunicorn server I get the following:
$ sentry --config=/www/sentry/sentry.conf.py start
Performing upgrade before service startup...
Loading help page organizations.md
Loading help page sampling.md
Loading help page tagging.md
Loading help page quotas.md
Loading help page teams_and_projects.md
Running service: 'http'
[2015-02-20 19:47:01 +0000] [19199] [INFO] Starting gunicorn 19.2.1
[2015-02-20 19:47:01 +0000] [19199] [INFO] Listening at: http://0.0.0.0:9000 (19199)
[2015-02-20 19:47:01 +0000] [19199] [INFO] Using worker: sync
[2015-02-20 19:47:01 +0000] [19219] [INFO] Booting worker with pid: 19219
[2015-02-20 19:47:01 +0000] [19219] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/www/sentry/lib/python2.7/site-packages/gunicorn/arbiter.py", line 503, in spawn_worker
worker.init_process()
File "/www/sentry/lib/python2.7/site-packages/gunicorn/workers/base.py", line 116, in init_process
self.wsgi = self.app.wsgi()
File "/www/sentry/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/www/sentry/lib/python2.7/site-packages/sentry/services/http.py", line 34, in load
import sentry.wsgi
File "/www/sentry/lib/python2.7/site-packages/sentry/wsgi.py", line 20, in <module>
configure()
File "/www/sentry/lib/python2.7/site-packages/sentry/utils/runner.py", line 399, in configure
initializer=initialize_app,
File "/www/sentry/lib/python2.7/site-packages/logan/runner.py", line 89, in configure_app
raise ValueError("Configuration file does not exist at %r" % (config_path,))
ValueError: Configuration file does not exist at '/www/.sentry/sentry.conf.py'
...etc...
I tried creating a /www/.sentry/ directory, and then copying my config file into it and then the server loads without a problem:
$ mkdir /www/.sentry/
$ cp /www/sentry/sentry.conf.py /www/.sentry/sentry.conf.py
$ sentry --config=/www/sentry/sentry.conf.py start
Performing upgrade before service startup...
Loading help page organizations.md
Loading help page sampling.md
Loading help page tagging.md
Loading help page quotas.md
Loading help page teams_and_projects.md
Running service: 'http'
[2015-02-20 19:50:12 +0000] [19653] [INFO] Starting gunicorn 19.2.1
[2015-02-20 19:50:12 +0000] [19653] [INFO] Listening at: http://0.0.0.0:9000 (19653)
[2015-02-20 19:50:12 +0000] [19653] [INFO] Using worker: sync
[2015-02-20 19:50:12 +0000] [19673] [INFO] Booting worker with pid: 19673
[2015-02-20 19:50:12 +0000] [19674] [INFO] Booting worker with pid: 19674
[2015-02-20 19:50:12 +0000] [19675] [INFO] Booting worker with pid: 19675
This seems silly and unnecessary however. Can anyone point me in the correct direction?

I had the same issue, I found this bug report:
https://github.com/getsentry/sentry/issues/1438
This commit fixed it for me: https://github.com/getsentry/sentry/commit/7629de1102973e4a3930487a3bf126a2f13c6850

Heroku NameError: name 'application' is not defined

I'm trying to deploy a django app to heroku and it keeps crashing. Does anyone have any idea what I'm doing wrong?
Here is my Procfile:
web: python app/manage.py collectstatic --noinput; gunicorn --workers=4 --bind=0.0.0.0:$PORT app.settings
And here is a snippet from my heroku logs:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 100, in init_process
File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 456, in spawn_worker
self.wsgi = self.app.wsgi()
app = eval(obj, mod.__dict__)
File "<string>", line 1, in <module>
File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 101, in wsgi
File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 24, in load
self.callable = self.load()
return util.import_app(self.app_uri)
NameError: name 'application' is not defined
2014-07-04 17:58:23 [4] [INFO] Starting gunicorn 0.13.4
2014-07-04 17:58:23 [10] [ERROR] Exception in worker process:
worker.init_process()
File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 250, in import_app
2014-07-04 17:58:23 [9] [INFO] Booting worker with pid: 9
2014-07-04 17:58:23 [10] [INFO] Worker exiting (pid: 10)
2014-07-04 17:58:23 [4] [INFO] Listening at: http://0.0.0.0:36148 (4)
2014-07-04 17:58:23 [7] [INFO] Booting worker with pid: 7

It says application not found. well try changing gunicorn
web: gunicorn YourProject.wsgi

I solved my own issue after doing several things, including upgrading my django install and changing my procfile to this:
web: gunicorn app.wsgi --pythonpath app --log-file -