Nginx responds with 502 error - django

While trying to deploy my app to Digital ocean I did everything according to this tutorial: How To Deploy a Local Django App to a VPS.
While Gunicorn is working perfectly and http://95.85.34.87:8001/ opens my app, Nginx, however, does not work, http://95.85.34.87 or http://95.85.34.87/static causes a 502 error.
Nginx log says, that :
2014/04/19 02:43:52 [error] 896#0: *62 connect() failed (111: Connection refused) while connecting to upstream, client: 78.62.163.9, server: 95.85.34.87, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "95.85.34.87"
My nginx configuration file looks like this:
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
server_name 95.85.34.87;
access_log off;
location /static/ {
alias /opt/myenv/static/;
}
location / {
proxy_pass http://127.0.0.1:8001;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
}
In Django.settings I have ALLOWED_HOSTS set to '[*]'
Nginx is listening to port 80:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 894/nginx
tcp6 0 0 :::80 :::* LISTEN 894/nginx
I think that the point is that Nginx does not point user to Gunicorn for some reason...
EDIT: I changed the proxy_pass http://127.0.0.1:8001; line under location / to my servers IP address (instead of loccalhost) and everything worked. I am not sure if it's good decission or not.

I see the instructions tell you to use this to start Gunicorn:
$ gunicorn_django --bind yourdomainorip.com:8001
If you start it like this then Gunicorn will listen only on the interface that is bound to yourdomainorip.com. So it won't listen on the loopback interface and won't receive anything sent to 127.0.0.1. Rather than changing nginx's configuration like you mention in your edit, you should do:
$ gunicorn_django --bind localhost:8001
This would cause Gunicorn to listen on the loopback. This is preferable because if you bind Gunicorn to an external interface people can access it without going through nginx.
With this setup the interaction between nginx and your Django app is like this:
nginx is the entry point for all HTTP requests. It listens on 95.85.34.87:80.
When an request is made to a URL that should be forwarded to your application, nginx forwards it by connecting on localhost:8001 (same as 127.0.0.1:8001).
Your Django application is listening on localhost:8001 to receive forwards from nginx.
By the way, gunicorn_django is deprecated.
And another thing: don't set ALLOWED_HOSTS to serve all domains. If you do so you are opening yourself to cache poisoning. Set it only to the list of domains that your Django project is meant to serve. See the documentation for details.

Related

Deploying Django with Nginx, Gunicorn and Supervisor

I'm trying to deploy my Django app with Nginx and Gunicorn by following this tutorial, but I modified some steps so I can use Conda instead of ViritualEnv.
The setup looks like this:
Nginx replies with my Vue app
Requests from Vue are made to api.example.com
Nginx listens to api.example.com and directs requests to Gunicorn's unix socket
Things I've checked:
I can see the Vue requests in Nginx's access.log.
I can also see those requests with journalctl -f -u gunicorn, in the supervisor.log, and gunicorn's access.log
When my Django app starts, it's creates a log file, so I can see that Gunicorn starts it. But Django is not responding to requests from the unix socket.
I can see a response from Django when I ssh in and run the following command:
curl --no-buffer -XGET --unix-socket /var/www/example/run/gunicorn.sock http://localhost/about. This command only gives a response when any of my ALLOWED_HOSTS are used in place of localhost.
My Nginx, Supervisor and Gunicorn configurations all use the full path to gunicorn.sock.
Should I see Django running on port 8000 or anything if I do something like nmap localhost?
I saw another post mention that Nginx should point to port 8000 and that gunicorn should be run with either:
gunicorn --bind 0.0.0.0:8000 <djangoapp>.wsgi --daemon
gunicorn <djangoapp>.wsgi:application --bind <IP>:8000 --daemon
gunicorn <djangoapp>.wsgi:application --bind=unix:/var/www/example/run/gunicorn.sock
But doesn't exposing port 8000 defeat the purpose of using Nginx as a reverse proxy and Gunicorn's unix socket? Doesn't exposing 8000 also increase the surface area for attack vectors? Or is it best practice to expose port 8000? I'm a bit confused why I would use both expose that port and use both Nginx and Gunicorn.
My main problem: Why can I get responses from Django via the unix socket with curl, but not via requests from Vue? Why aren't Vue's requests making it from Gunicorn to Django via the unix socket?
I'm really stuck. Any suggestions?
Frontend Nginx config
server {
listen 80 default_server;
listen [::]:80 default_server;
# server_name example.com;
# server_name myIP;
root /var/www/example/frontend/dist;
server_name example.com www.example.com;
location =/robots.txt {
root /opt/example;
}
location /thumbnail/ {
alias /opt/example/static/img/thumbnail/;
}
location /bg/ {
alias /opt/example/static/img/bg/;
}
location / {
try_files $uri $uri/ /index.html;
}
}
API Nginx config
upstream backend_server {
server unix:/var/www/example/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name api.example.com
client_max_body_size 4G;
access_log /var/log/nginx/api-access.log;
error_log /var/log/nginx/api-error.log;
location / {
include proxy_params;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 128;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://backend_server;
}
}
}
Gunicorn config
#!/bin/bash
NAME=”backend”
DJANGODIR=/var/www/example/backend
SOCKFILE=/var/www/example/run/gunicorn.sock
USER=django
GROUP=example
NUM_WORKERS=3
DJANGO_SETTINGS_MODULE=backend.settings
DJANGO_WSGI_MODULE=backend.wsgi
CONDA_SRC=/home/justin/anaconda3/etc/profile.d/conda.sh
GUNICORN=/home/justin/anaconda3/envs/production/bin/gunicorn
echo “starting backend”
cd $DJANGODIR
source $CONDA_SRC
conda activate production
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
exec $GUNICORN
${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE \
--log-level=debug \
--log-file=- \
--error-logfile=/var/www/example/backend/logs/gunicorn-error.log \
--access-logfile=/var/www/example/backend/logs/gunicorn-access.log
Gunicorn access.log
- - [08/Sep/2020:01:51:24 -0400] "OPTIONS /c/about/ HTTP/1.0" 200 0 "http://example.com/c/about" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Mobile Safari/537.36"
- - [08/Sep/2020:01:51:24 -0400] "POST /c/about/ HTTP/1.0" 400 143 "http://example.com/c/about" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Mobile Safari/537.36"
But doesn't exposing port 8000 defeat the purpose of using Nginx as a reverse proxy and Gunicorn's unix socket?
In gunicorn, you have to expose 8000 port on localhost like this gunicorn --bind 127.0.0.1:8000 <djangoapp>.wsgi --daemon. Exposing it on 0.0.0.0 will obviously be a security vulnerability considering your nginx in on the same server.
Doesn't exposing 8000 also increase the surface area for attack vectors? Or is it best practice to expose port 8000? I'm a bit confused why I would use both expose that port and use both Nginx and Gunicorn.
You don't need to expose port 8000 you can expose any port but you need to tell gunicon to listen on at least a single port so that nginx can pass requests to it.
And regarding using both nginx and gunicorn, they both are really different and handle very different use case/functions of an application.
Nginx uses "event‑driven" approach to handle requests so a single worker of nginx can handle 1000s of req simultaneously. But Gunicorn on the other hand mostly(by default) uses sync worker which means a request will remain with a worker till it is processed. (posted this twice today :p)
So you need both if you remove nginx all your requests will return 50X except which are currently handled by gunicorn until the worker is free. And also gunicorn is not made to handle user traffic or in bigger application things like load balancing can only be done by nginx. So nginx has it's own purpose in an application.
After neeraj9194 pointed out the 400, I did more searching for issues relating to Nginx, Gunicorn 400 and Django and I came across a ton of similar issues. Looks like it's mainly an Nginx issue. The answer in this blog fixed my issue.
I replaced the location block in my API Nginx config with:
location / {
proxy_set_header Host $host;
proxy_pass http://backend_server;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
}

Django Channels Nginx production

I have a django project and recently added channels to use websockets. This seems to all work fine, but the problem I have is to get the production ready.
My setup is as follows:
Nginx web server
Gunicorn for django
SSL enabled
Since I have added channels to the mix. I have spent the last day trying to get it to work.
On all the turtotials they say you run daphne on some port then show how to setup nginx for that.
But what about having gunicorn serving django?
So now I have guncorn running this django app on 8001
If I run daphne on another port, lets say 8002 - how should it know its par of this django project? And what about run workers?
Should Gunicorn, Daphne and runworkers all run together?
This question is actually addressed in the latest Django Channels docs:
It is good practice to use a common path prefix like /ws/ to
distinguish WebSocket connections from ordinary HTTP connections
because it will make deploying Channels to a production environment in
certain configurations easier.
In particular for large sites it will be possible to configure a
production-grade HTTP server like nginx to route requests based on
path to either (1) a production-grade WSGI server like Gunicorn+Django
for ordinary HTTP requests or (2) a production-grade ASGI server like
Daphne+Channels for WebSocket requests.
Note that for smaller sites you can use a simpler deployment strategy
where Daphne serves all requests - HTTP and WebSocket - rather than
having a separate WSGI server. In this deployment configuration no
common path prefix like is /ws/ is necessary.
In practice, your NGINX configuration would then look something like (shortened to only include relevant bits):
upstream daphne_server {
server unix:/var/www/html/env/run/daphne.sock fail_timeout=0;
}
upstream gunicorn_server {
server unix:/var/www/html/env/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name _;
location /ws/ {
proxy_pass http://daphne_server;
}
location / {
proxy_pass http://gunicorn_server;
}
}
(Above it is assumed that you are binding the Gunicorn and Daphne servers to Unix socket files.)
I have created an example how to mix Django Channels and Django Rest Framework. I set nginx routing that:
websockets connections are going to daphne server
HTTP connections (REST API) are going to gunicorn server
Here is my nginx configuration file:
upstream app {
server wsgiserver:8000;
}
upstream ws_server {
server asgiserver:9000;
}
server {
listen 8000 default_server;
listen [::]:8000;
client_max_body_size 20M;
location / {
try_files $uri #proxy_to_app;
}
location /tasks {
try_files $uri #proxy_to_ws;
}
location #proxy_to_ws {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_redirect off;
proxy_pass http://ws_server;
}
location #proxy_to_app {
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Url-Scheme $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app;
}
}
I recently answered a similiar question, have a look there for an explanation on how django channels work.
Basically, you don't need gunicorn anymore. You have daphne which is the interface server that accepts HTTP/Websockets and you have your workers that run django views. Then obviously you have your channel backend that glues everything together.
To make it work you have to configure CHANNEL_LAYERS in settings.py and also run the interface server: $ daphne my_project.asgi:channel_layer
and your worker:
$ python manage.py runworker
NB! If you chose redis as the channel backend, pay attention to file sizes you're serving. If you have large static files make sure NGINX serves them or otherwise clients will experience cryptic errors that may occur due to redis running out of memory.

Django gunicorn nginx (111: Connection refused) while connecting to upstream

A Django application is running on the AWS instance, configured via gunicorn and nginx, it is running well for more than a year, but suddenly, I got 502 bad gateway error, then I saw the below mentioned message in the nginx error log,
2017/05/17 16:18:35 [error] 1040#0: *7460 connect() to unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock failed (111: Connection refused) while connecting to upstream, client: xx.xxxx.xx.xxx, server: xx.xx.xx.xxx, request: "GET / HTTP/1.1", upstream: "http://unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock:/", host: "xx.xx.xx.xxx", referrer: "http://xx.xx.xx.xxx"
my nginx configuration:
server {
client_max_body_size 200M;
listen 80;
listen [::]:80 ipv6only=on;
server_name xx.xx.xx.xxx;
listen 443 ssl;
ssl_certificate /etc/nginx/ssl/myserver.crt;
ssl_certificate_key /etc/nginx/ssl/myserver.key;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/ubuntu/webapps/myproject/myproject;
}
location / {
include proxy_params;
proxy_pass http://unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock;
proxy_set_header X-Forwarded-Protocol $scheme;
}
if ($scheme = http){
return 301 https://xx.xx.xx.xxx$request_uri;
}
if ($http_host = pas-cash.com){
return 303 https://xx.xx.xx.xxx$request_uri;
}
}
my gunicorn.conf
description "Gunicorn application server handling myproject"
start on runlevel [6789]
stop on runlevel [!6789]
respawn
setuid ubuntu
setgid www-data
chdir /home/ubuntu/webapps/myproject/myproject
exec /home/ubuntu/webapps/myproject/venv/bin/gunicorn --workers 3 --bind unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock myproject.wsgi:application
After that I restarted the nginx by following command
sudo service nginx restart
After restarting, the application is running well, I can't find what will be the specific reason behind this error, I googled it for this, but I got different types of answer but nothing suitable for me, can you guys please help me out for, why this happened, is there any thing missing in my configuration or what will the common/general reason behind this behavior. It will be very helpful for me, Thanks in advance.
This is caused "suddenly" not because of an nginx error but rather an error with gunicorn or your app (code error, packages not installed etc.). It should be relatively easy to log and fix though.
First try running your app from the server python manage.py runserver and see if you get any issues. The same for ... migrate. Often the issue that production does not work but local does is because of missing packages or missing migrations. Create a requirements.txt file on local and install it on production.
If the error is still there check the gunicorn logs with gunicorn --log-file=- YourApp.wsgi:application . Once all those errors have been corrected run
sudo systemctl status gunicorn.socket
sudo systemctl status gunicorn
And you want to have both active and running. If you start getting a 400 error that is a good sign as it is now a Django error (usually allowed hosts). Turn debug=True to see the exact error from django.
Remember whenever any changes are made to code run
sudo systemctl daemon-reload
sudo systemctl restart gunicorn
Just FYI if none of the above work then you can always check your nginx logs with
sudo tail -30 /var/log/nginx/error.log
try to remove http:// from the proxy_pass in the nginx configuration:
server {
client_max_body_size 200M;
listen 80;
listen [::]:80 ipv6only=on;
server_name xx.xx.xx.xxx;
listen 443 ssl;
ssl_certificate /etc/nginx/ssl/myserver.crt;
ssl_certificate_key /etc/nginx/ssl/myserver.key;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/ubuntu/webapps/myproject/myproject;
}
location / {
include proxy_params;
proxy_pass unix:/home/ubuntu/webapps/myproject/myproject/myproject.sock;
proxy_set_header X-Forwarded-Protocol $scheme;
}
if ($scheme = http){
return 301 https://xx.xx.xx.xxx$request_uri;
}
if ($http_host = pas-cash.com){
return 303 https://xx.xx.xx.xxx$request_uri;
}
}
The reason is that gunicorn is listening on a unix socket (the --bind argument). Then nginx should forward traffic to this socket. http:// stands for a TCP socket in a regular IP:PORT, which is not your case.
There are several reasons for this particular issue, in my case it was because the service was not enabled. After running sudo systemctl enable gunicorm.service fixed it.

docker nginx connection refused while connecting to upstream

I use shiny server to build a web-app on port 3838, when i use nginx in my server it works well. But when I stop nginx on my server and try to use docker nginx, I find the site comes to a '502-Bad Gate Way' error and nginx log shows:
2016/04/28 18:51:15 [error] 8#8: *1 connect() failed (111: Connection refused) while connecting to upstream, ...
I install docker-nginx by this command:
sudo docker pull nginx
My docker command line is something like (for clear i add some indent):
sudo docker run --name docker-nginx -p 80:80
-v ~/docker-nginx/default.conf:/etc/nginx/conf.d/default.conf
-v /usr/share/nginx/html:/usr/share/nginx/html nginx
I create a folder name 'docker-nginx' in my home dir, move my nginx conf file in this folder, and then remove my original conf in etc/nginx dir just in case.
My nginx conf file looks like this:
server {
listen 80 default_server;
# listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.html index.htm;
# Make site accessible from http://localhost/
server_name localhost;
location / {
proxy_pass http://127.0.0.1:3838/;
proxy_redirect http://127.0.0.1:3838/ $scheme://$host/;
auth_basic "Username and Password are required";
auth_basic_user_file /etc/nginx/.htpasswd;
# enhance the performance
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
You have to define upstream directly. Currently your nginx can not proxy to your web application.
http://nginx.org/en/docs/http/ngx_http_upstream_module.html
upstream backend {
server backend1.example.com weight=5;
server backend2.example.com:8080;
server unix:/tmp/backend3;
server backup1.example.com:8080 backup;
server backup2.example.com:8080 backup;
}
server {
location / {
proxy_pass http://backend;
}
}
My situation was running 3 containers, a nginx container and two containerized services. I was using the Nginx container as a reverse proxy for my go services.
Issue is the nginx container was looking for microservice ports in its own container environment. I didn't realize that at the time and I didn't use the docker-compose.yml then. When using docker-compose.yml file you specify a network and that's that.
So when running the containers you should use --net=host.
Info on that: What does --net=host option in Docker command really do?
This worked for me, I hope it saves someone the pain :):
docker run --net=host nginx:someTag
docker run --net=host service1:someTag

nginx can't listen on port 80

I'm trying to set nginx with gunicorn but I keep getting the "Welcome to nginx!" page. I am able to successfully listen to other ports (like 8080) but port 80 does not work at all.
server {
listen 80;
server_name host.ca www.host.ca;
access_log /var/log/nginx/example2.log;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://127.0.0.1:8000;
}
}
I'm running the server as root.
I can't seem to see anything running in port 80.
Diagnosing the Problem
Make sure to check your logs (likely /var/log/nginx or some variant).
Check to see what might be hogging port 80
netstat -nlp | grep 80
Sites-enabled, port hogging
Then, make sure you have the Django site enabled in sites-enabled. Delete any old symlinks if you created one first.
rm /etc/nginx/sites-enabled/django
ln -s /etc/nginx/sites-available/django /etc/nginx/sites-enabled/django
Double check your /etc/nginx/nginx.conf to make sure it's loading sites-enabled and not loading some other default.
http {
...
include /etc/nginx/sites-enabled/*;
}
After you do all this, shut down and restart the nginx service.
Either service nginx restart or service nginx stop && service nginx start