I have a django app which I communicate with, using a python script.
Sometime (about 1/500) I get strange errors.
https <-> nginx <-> uwsgi <-> django
Rare errors in the python script:
http.client.BadStatusLine: ''
urllib.error.URLError:
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
I cannot see any trace of those in either:
appLog
uwsgi log
nginx log
How can I begin to trace this issue?
Related
I am currently running a web server through two containers:
NGINX Container: Serves HTTPS requests and redirects HTTP to HTTPS. HTTPS requests are passed through uwsgi to the django app.
Django Container: Runs the necessary Django code.
When running docker-compose up --build, everything compiles correctly until uWSGI raises a Segmentation Fault.
....
django3_1 | Python main interpreter initialized at 0x7fd7bce0d190
django3_1 | python threads support enabled
django3_1 | your server socket listen backlog is limited to 100 connections
django3_1 | your mercy for graceful operations on workers is 60 seconds
django3_1 | mapped 145840 bytes (142 KB) for 1 cores
django3_1 | *** Operational MODE: single process ***
django3_1 | !!! uWSGI process 7 got Segmentation Fault !!!
test_django3_1 exited with code 1
Would appreciate if there's any advice, as I'm not able to see into the container for debugging purposes when it is starting up, therefore I don't know where this segmentation fault is occurring.
The SSL certificates have been correctly set up.
The django3 container was running on a python:3.9-alpine image. This installs Python 3.9.2 on the container. There appear to be some issue between uWSGI and Python dependencies on this version. Rolling the container back to python:3.8-alpine resolved the dependency version mismatch.
I'm new in a company that have this project on Google Cloud PLatform, that I've never used. In my first day I've got this 502 Bad Gateway error. Lookin the log on the Google Cloud Platform I've got the following:
[error] 33#33: *285 upstream prematurely closed connection while reading response header from upstream, client: 172.217.172.212, server: , request: "POST /product/fast_appraisal/result/ HTTP/1.1", upstream: "http://172.17.0.1:8080/product/fast_appraisal/result/", host: "avalieidjango.appspot.com", referrer: "https://avalieidjango.appspot.com/product/fast_appraisal/search/"
I've tried to edit the app.yaml adding --timeout and --graceful-timeout parameters to it like the following:
# [START runtime]
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT mysite.wsgi --timeout=90 --graceful-timeout=10
beta_settings:
cloud_sql_instances: avalieidjango:southamerica-east1:avaliei
runtime_config:
python_version: 3
handlers:
- url: /.*
script: manage.py
secure: always
redirect_http_response_code: 301
# [END runtime]
In the settings.py file DEBUG variable is setted to False
Looking for answers on the internet I found a few cases, but no one looks like mine exactly.
Locally I'm running the project on Windows 7, so the error only occurs when I deploy it to GCP. I'm new on GCP and gunicorn.
Edit
After these days I've passed through a lot of forums, and added a few new configurations to my app.yaml trying to work with threads and workers to solve the question.
The entrypoint line looks like this:
entrypoint: gunicorn -b :$PORT --worker-class=gevent --worker-connections=1000 --workers=3 mysite.wsgi --timeout 90
This project consists in search a Postgres database on GCP gathering information about properties and running an AI to show some predictions about its values.
I've tried threads and process, but even with just my requests the application is too slow, even a simple page take some time to render.
Local tests run better, but in production it isn't working at all.
The AI wasn't developed for me and it uses a large joblib file.
The project doesn't use containers like Docker. Maybe it could help in some way if I "dockerize" the project?
I've stoped seeing this error changing CONN_MAX_AGE value to None, which keeps the database connetion time undefined. However this may cause some security issues that must be evaluated before deploy your application. If you change it, stay tuned on Google Cloud Logs looking for strange connection attempts.
I am running Django on nginix but one of my functions is returning this error:
upstream prematurely closed connection while reading response header from upstream
While other methods are working fine.
I have traied increasing server timeout in nginx.conf file but still it gives me the same error.
This methods runs fine whe i run my django independently with
python manage.py runserver 0.0.0.0:8000
Can someone help me to resolve it?
I know that I can use logging to log all the requests that my django server gets. However I would like to log the requests along with headers and body that my django server makes to other servers. How can I access this info?
What type of logs do you really want?
1 - in development mode, you can get the request as .txt using: python manage.py runserver > logs.txt - source: stackoverflow
2 - The Apache2 webserver provides two important logs: error.log or access.log. If you are using debian/ubuntu, you can get it in?
/var/log/apache2/error.log or access.log
I am working on a flask-socketio server which is getting stuck in a state where only 504s (gateway timeout) are returned. We are using AWS ELB in front of the server. I was wondering if anyone wouldn't mind giving some tips as to how to debug this issue.
Other symptoms:
This problem does not occur consistently, but once it begins happening, only 504s are received from requests. Restarting the process seems to fix the issue.
When I run netstat -nt on the server, I see many entries with rec-q's of over 100 stuck in the CLOSE_WAIT state
When I run strace on the process, I only see select and clock_gettime
When I run tcpdump on the server, I can see the valid requests coming into the server
AWS health checks are coming back succesfully
EDIT:
I should also add two things:
flask-socketio's server is used for production (not gunicorn or uWSGI)
Python's daemonize function is used for daemonizing the app
It seemed that switching to gunicorn as the wsgi server fixed the problem. This legitimately might be an issue with the flask-socketio wsgi server.