We have a Google Cloud run application that runs at 1am every morning to ingest, process, and output some data files. Currently, we have max concurrency, as timing is important.
Over the past weeks we have noticed several errors generated by the platform.
The Cloud Run logs themselves show Shutting down: Master
At a similar timestamp, a logging sink picks up
The request failed because either the HTTP response was malformed or connection to the instance had an error.
We have modified the instances and concurrency, as well as memory and vCPUs available, and they just show the same error but generated over a longer period of time.
There are no other errors or logs generated to give us any further info.
Any ideas where we may be going wrong?
Edit:
The full trace from the Cloud Run log is below
[INFO] Starting gunicorn 20.0.4
[1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[1] [INFO] Using worker: sync
[4] [INFO] Booting worker with pid: 4
[5] [INFO] Booting worker with pid: 5
[6] [INFO] Booting worker with pid: 6
[1] [INFO] Handling signal: term
[6] [INFO] Worker exiting (pid: 6)
[5] [INFO] Worker exiting (pid: 5)
[4] [INFO] Worker exiting (pid: 4)
[1] [INFO] Shutting down: Master
My situation was the same.
I sloved it by stop the firewall.
Related
I just re-deployed an application in the new Eco Dyno and provisioning a new PostgreSQL datbase, however after deploying successfully, when I try to open the application, this HTTP error is returned ERR_CONNECTION_TIMED_OUT and the application never opens. I also updated the stack from heroku-20 to heroku-22 and it is not working on both. Is there any additional configuration that I need to complete?
This is the log from the latest deploy
2023-01-30T22:13:55.000000+00:00 app[api]: Build started by user farmacia#aselsi.org
2023-01-30T22:14:32.846408+00:00 app[api]: Deploy c6f350bc by user farmacia#aselsi.org
2023-01-30T22:14:32.846408+00:00 app[api]: Release v23 created by user farmacia#aselsi.org
2023-01-30T22:14:33.126404+00:00 heroku[web.1]: Restarting
2023-01-30T22:14:33.141734+00:00 heroku[web.1]: State changed from up to starting
2023-01-30T22:14:34.066230+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2023-01-30T22:14:34.109732+00:00 app[web.1]: [2023-01-30 22:14:34 +0000] [4] [INFO] Handling signal: term
2023-01-30T22:14:34.109979+00:00 app[web.1]: [2023-01-30 22:14:34 +0000] [9] [INFO] Worker exiting (pid: 9)
2023-01-30T22:14:34.110205+00:00 app[web.1]: [2023-01-30 22:14:34 +0000] [10] [INFO] Worker exiting (pid: 10)
2023-01-30T22:14:34.310286+00:00 app[web.1]: [2023-01-30 22:14:34 +0000] [4] [INFO] Shutting down: Master
2023-01-30T22:14:34.461478+00:00 heroku[web.1]: Process exited with status 0
2023-01-30T22:14:35.372879+00:00 heroku[web.1]: Starting process with command `gunicorn aselsi.wsgi`
2023-01-30T22:14:36.441571+00:00 app[web.1]: [2023-01-30 22:14:36 +0000] [4] [INFO] Starting gunicorn 20.1.0
2023-01-30T22:14:36.441910+00:00 app[web.1]: [2023-01-30 22:14:36 +0000] [4] [INFO] Listening at: http://0.0.0.0:54892 (4)
2023-01-30T22:14:36.441956+00:00 app[web.1]: [2023-01-30 22:14:36 +0000] [4] [INFO] Using worker: sync
2023-01-30T22:14:36.447323+00:00 app[web.1]: [2023-01-30 22:14:36 +0000] [9] [INFO] Booting worker with pid: 9
2023-01-30T22:14:36.521146+00:00 app[web.1]: [2023-01-30 22:14:36 +0000] [10] [INFO] Booting worker with pid: 10
2023-01-30T22:14:36.588676+00:00 heroku[web.1]: State changed from starting to up
2023-01-30T22:14:37.000000+00:00 app[api]: Build succeeded
2023-01-30T22:17:56.524340+00:00 app[api]: Starting process with command `python manage.py migrate` by user farmacia#aselsi.org
2023-01-30T22:17:59.955485+00:00 heroku[run.7847]: State changed from starting to up
2023-01-30T22:18:00.056845+00:00 heroku[run.7847]: Awaiting client
2023-01-30T22:18:00.078452+00:00 heroku[run.7847]: Starting process with command `python manage.py migrate`
2023-01-30T22:18:05.101096+00:00 heroku[run.7847]: Process exited with status 0
2023-01-30T22:18:05.175678+00:00 heroku[run.7847]: State changed from up to complete
I re deployed from github making minor changes on the code to be sure that the most recent changes are being deployed properly. I updated the heroku stack to make sure that was not the problem. I also ran migrations after the application is deployed and they are applied successfully so the only problem is that the site is not reachable for some reason.
Ive deployed a few heroku webapps. But for some reason, this one is giving me pain.
with
git push heroku main
my app starts, and is deployed. No issues whatsoever. But when I click the link, I get a this site cant be reached. ____ took too long to respond
Upon checking the logs, there seems to be no errors:
2022-09-01T17:46:08.937248+00:00 heroku[web.1]: State changed from down to starting
2022-09-01T17:46:12.101512+00:00 heroku[web.1]: Starting process with command `gunicorn wsgi:app`
2022-09-01T17:46:13.204452+00:00 app[web.1]: [2022-09-01 17:46:13 +0000] [4] [INFO] Starting gunicorn 20.1.0
2022-09-01T17:46:13.204789+00:00 app[web.1]: [2022-09-01 17:46:13 +0000] [4] [INFO] Listening at: http://0.0.0.0:42097 (4)
2022-09-01T17:46:13.204842+00:00 app[web.1]: [2022-09-01 17:46:13 +0000] [4] [INFO] Using worker: sync
2022-09-01T17:46:13.207772+00:00 app[web.1]: [2022-09-01 17:46:13 +0000] [9] [INFO] Booting worker with pid: 9
2022-09-01T17:46:13.268559+00:00 app[web.1]: [2022-09-01 17:46:13 +0000] [10] [INFO] Booting worker with pid: 10
2022-09-01T17:46:13.737219+00:00 heroku[web.1]: State changed from starting to up
What is going on?
It runs completely fine when I run the wsgi.py file locally. Im so confused why its taking too long to respond because It's just rendering an HTML template.
I have been working on a side project using django and django rest framework. There is even an active version running in the cloud currently without any issue. I wanted to make some minor changes on the code base but while using the django admin page it crashes silently. I haven't changed anything in the code until now.
[2022-08-31 13:39:10 +0200] [8801] [INFO] Starting gunicorn 20.1.0
[2022-08-31 13:39:10 +0200] [8801] [INFO] Listening at: http://127.0.0.1:8000 (8801)
[2022-08-31 13:39:10 +0200] [8801] [INFO] Using worker: sync
[2022-08-31 13:39:10 +0200] [8802] [INFO] Booting worker with pid: 8802
[2022-08-31 13:39:18 +0200] [8801] [WARNING] Worker with pid 8802 was terminated due to signal 11
[2022-08-31 13:39:18 +0200] [8810] [INFO] Booting worker with pid: 8810
[2022-08-31 13:39:23 +0200] [8801] [WARNING] Worker with pid 8810 was terminated due to signal 11
[2022-08-31 13:39:23 +0200] [8814] [INFO] Booting worker with pid: 8814
Same happens with python manage.py runserver command.
I'm using a virtual environment with python 3.9
Signal 11 (SIGSEGV, also known as segmentation violation) means that the program accessed a memory location that was not assigned to it.
That's usually a bug in a program. So if you're writing your own program, that's the most likely cause.
It can also commonly occur with some hardware malfunctions.
I have working django REST API docker image with following dependencies:
python 3.5.2, django 1.10.6, djangorestframework 3.6.2, gevent 1.2.2
In my dockerfile, port 5000 is exposed.
docker command:
/usr/local/bin/gunicorn --log-level=DEBUG --worker-class gevent --timeout=300 config.wsgi -w 4 -b :5000
In the ECS task definition, 5000 container port is forwarded to port 80 of the host. The security group has an inbound rule allowing everyone at port 80.
When I ran the ECS task with this ECS task definition, following are the application logs, which seem fine.
[2017-09-13 16:45:34 +0000] [9] [INFO] Starting gunicorn 19.6.0
[2017-09-13 16:45:34 +0000] [9] [INFO] Listening at: http://0.0.0.0:5000 (9)
[2017-09-13 16:45:34 +0000] [9] [INFO] Using worker: gevent
[2017-09-13 16:45:34 +0000] [12] [INFO] Booting worker with pid: 12
[2017-09-13 16:45:34 +0000] [13] [INFO] Booting worker with pid: 13
[2017-09-13 16:45:35 +0000] [15] [INFO] Booting worker with pid: 15
[2017-09-13 16:45:35 +0000] [16] [INFO] Booting worker with pid: 16
But I am unable to access the service endpoints using the EC2 instance's public IP/Public DNS address.
I tried to get into the running container and curl the application url curl localhost:5000. Following are the logs that I see (the connections are closed)
[2017-09-13 17:42:42 +0000] [14] [DEBUG] GET /
[2017-09-13 17:42:42 +0000] [14] [DEBUG] Closing connection.
[2017-09-13 17:42:56 +0000] [12] [DEBUG] GET /
[2017-09-13 17:42:56 +0000] [12] [DEBUG] Closing connection.
[2017-09-13 17:53:20 +0000] [14] [DEBUG] GET /users/get_mfatype/
[2017-09-13 17:53:20 +0000] [14] [DEBUG] Closing connection.
The same docker image is working as expected when I run locally. I even tried running the same docker image inside EC2 instance, which is working fine.
I am not able to find the root cause why the application is not running as ECS task.
Am I missing anything?
In my Django settings and on my machine I have utc+3 configured time so the expectations were to get all logs in utc+3, but turned out, that actually they are pretty messy:
[2017-08-08 10:29:22 +0000] [1] [INFO] Starting gunicorn 19.7.1
[2017-08-08 10:29:22 +0000] [1] [DEBUG] Arbiter booted
[2017-08-08 10:29:22 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000
[2017-08-08 10:29:22 +0000] [1] [INFO] Using worker: sync
[2017-08-08 10:29:22 +0000] [7] [INFO] Booting worker with pid: 7
[2017-08-08 10:29:23 +0000] [1] [DEBUG] 1 worker
[2017-08-08 13:29:26 +0300] [7] [INFO] [dashboard.views:9] Displaying menu
Settings:
TIME_ZONE = 'Europe/Moscow'
USE_TZ = True
Maybe you can provide some hints/information how to configure or debug it?
For a moment I thought that this is a gunicorn's problem, but it uses Django settings soo I have no idea what's wrong :/
Gunicorn logging time do not relay on Django timezone, but in the local machine one, so to get the right timezone you should configure your local machine and how to do it depends in what OS is running on it.
For Debian/Ubuntu:
sudo dpkg-reconfigure tzdata
Follow the directions in the terminal.
The timezone info is saved in /etc/timezone - which can be edited or used below
If you are using CentOS you can check it in this article.
For other options, check it in Google.
Hope that it helps.
So, the timestamps were correct, but different because of my company proxy settings. Also it turned out that is best way to handle different time zones is just use utc everywhere except presentation to user.