How to fix 504 error caused by Docker image update - django

I have Django project. It works with nginx, uwsgi and google cloud run.
This project is using docker which python:3.9 image. I have got this error since 17,Aug.
2021-10-13 17:22:29.654 JSTGET504717 B899.9 sGoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring) https://xxxx/
The request has been terminated because it has reached the maximum request timeout. To change this limit, see https://cloud.google.com/run/docs/configuring/request-timeout
and also this error occur on all my pages. However when I open my pages myself, I can see my pages. It means I can't see 504 error and I can only check that it happens from server log.
I added a line in admin.py at 17, Aug. I didn't think this line is no related with this error. Because this change is only effect in admin page. I had rollback my code before the error. Now I'm still can't fix this error.
Builded docker image is different size before after error. And Vulnerability has decreased. I think this is caused by some small change on python image. In this case, how can I solve this problem?
What I did
I changed docker image to python:3.8 and python:3.9.6-buster. I couldn't fix the error.

I solved this problem. I changed socket to port connection.
This is my settings.
uwsgi.ini
[uwsgi]
# this config will be loaded if nothing specific is specified
# load base config from below
ini = :base
# %d is the dir this configuration file is in
http = 127.0.0.1:8000
master = true
processes = 4
max-requests = 1000 ; Restart workers after this many requests
max-worker-lifetime = 3600 ; Restart workers after this many seconds
reload-on-rss = 512 ; Restart workers after this much resident memory
threaded-logger = true
[dev]
ini = :base
# socket (uwsgi) is not the same as http, nor http-socket
socket = :8001
[local]
ini = :base
http = :8000
# set the virtual env to use
home = /Users/you/envs/env
[base]
# chdir to the folder of this config file, plus app/website
chdir = %dapp/
# load the module from wsgi.py, it is a python path from
# the directory above.
module = website.wsgi:application
# allow anyone to connect to the socket. This is very permissive
chmod-socket = 666
nginx-app.conf
# the upstream component nginx needs to connect to
upstream django {
# server unix:/code/app.sock; # for a file socket
server 127.0.0.1:8000; # for a web port socket (we'll use this first)
}
# configuration of the server
server {
# the port your site will be served on, default_server indicates that this server block
# is the block to use if no blocks match the server_name
listen 8080;
# the domain name it will serve for
server_name MY_DOMAIN.COM; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 10M; # adjust to taste
# set timeout
uwsgi_read_timeout 900;
proxy_read_timeout 900;
# Django media
location /media {
alias /code/app/media; # your Django project's media files - amend as required
}
location /static {
alias /code/app/static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
include /code/uwsgi_params; # the uwsgi_params file you installed
}
}

Related

Nginx: 403 Forbidden nginx/1.12.1 (Ubuntu)

I've never before configured any production server, I'm trying to configure nginx and keep getting the 403 Forbidden error. I can't figure out the reason why it's happening.
Here is a complete error report:
[crit] 25145#25145: *1 connect() to unix:/home/albert/deploy_test/django_env
/run/gunicorn.sock failed (13: Permission denied) while connecting to
upstream, client: 192.168.1.118, server: 192.168.1.118, request: "GET /
HTTP/1.1", upstream: "http://unix:/home/albert/deploy_test/django_env
/run/gunicorn.sock:/", host: "192.168.1.118"
Here is my /etc/nginx/sites-available/deployproject.conf:
(I removed the default config and created a symlink as follows: sudo ln -s /etc/nginx/sites-available/deployproject.conf /etc/nginx/sites-enabled/deployproject.conf)
upstream sample_project_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/albert/deploy_test/django_env/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name 192.168.1.118;
client_max_body_size 4G;
access_log /home/albert/logs/nginx-access.log;
error_log /home/albert/logs/nginx-error.log;
location /static/ {
alias /home/albert/static/;
}
location /media/ {
alias /home/albert/media/;
}
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://sample_project_server;
break;
}
}
# Error pages
error_page 500 502 503 504 /500.html;
location = /500.html {
root /home/albert/static/;
}
}
Here is the complete tutorial I'm using to deploy my app. Here I'm just trying to deploy the most primitive,default django app but in my real app I'm using django as a serverside, so there seems to be no need for nginx to serve static and all that.
File Permissions. Incorrect file permissions are another cause of the "403 Forbidden" error. The standard setting of 755 for directories and 644 for files is recommended for use with NGINX. The NGINX user also needs to be the owner of the files
Try to change the permissions on your web dir
sudo chown -R albert:www-data /webdirectory
sudo chmod -R 0755 /webdirectory
Move all your sites inside the webdirectory do not leave the dir and files in your root home.
Have you taken a look at the gunicorn docs here which has example of how to configure nginx
http://docs.gunicorn.org/en/stable/deploy.html
Can you try running gunicorn via TCP instead of unix socket, in your upstream sample_project_server replace server with:
server 192.168.0.7:8000 fail_timeout=0;
What are the settings in gunicorn? You can bind to localhost via TCP with the following, to check that it isn't a problem with your unix socket:
--bind 127.0.0.1:8000

Doesn't Nginx support Django admin static files

My django site user-end is running good with the static files but don't know why all the admin panel static files is not working. While it's working normally but not with linux any idea ??
nginx .conf file
upstream sample_project_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/me/SPEnv/run/gunicorn.sock fail_timeout=0;
}
server {
listen 800;
server_name <your domain name>;
client_max_body_size 4G;
access_log /home/me/logs/nginx-access.log;
error_log /home/me/logs/nginx-error.log;
location /static {
root /home/me/DjangoProjects/SP/SP;
}
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://sample_project_server;
break;
}
}
# Error pages
error_page 500 502 503 504 /500.html;
location = /500.html {
root /home/me/DjangoProjects/SP/SP;
}
}
and settings.py
Static files (CSS, JavaScript, Images)
https://docs.djangoproject.com/en/1.8/howto/static-files/
STATIC_URL = '/static/'
STATIC_ROOT = '/home/me/DjangoProjects/SP/SP/static/'
STATICFILES_DIRS = (
# os.path.join(BASE_DIR, 'SP','static/admin'),
'/home/me/DjangoProjects/SP/SP/static/',
'/home/me/SPEnv/lib/python2.7/site-packages/django/contrib/admin/static'
)
You need to add the directory for the admin static files to your STATICFILES_DIRS list before you run collectstatic. Something like this:
STATICFILES_DIRS = [
'/project/src/static',
'/usr/local/lib/python3.6/site-packages/django/contrib/admin/static',
]
Where the first entry is the path to your app's static files and the second is the location of the django admin package.
I have the same problem. My nginx server on Centos 7.6 can't access to static folder in path /home/user/app/mysyte/static/. In /var/log/nginx/error.log same error
open() "/home/user/app/mysyte/static/*.css" failed (13: Permission denied)
For solving and understanding this problem :=*
run command getenforce
if enforcing - cat /var/log/audit/audit.log | grep nginx
for me string with errrors looks like
type=AVC msg=audit(1558033633.723:201): avc: denied { read } for pid=7758 comm="nginx" name="responsive.css" dev="dm-0" ino=17312394 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1558033633.723:201): arch=c000003e syscall=2 success=no exit=-13 a0=564f710dd55d a1=800 a2=0 a3=68632f656d6f682f items=0 ppid=7757 pid=7758 auid=4294967295 uid=998 gid=996 euid=998 suid=998 fsuid=998 egid=996 sgid=996 fsgid=996 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)
copy id of audit msg 1558033633.723:201
run command grep yours_audit_id /var/log/audit/audit.log | audit2why
output for me
[root#uwsgi ~]# grep 1558034479.384:221 /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1558034479.384:221): avc: denied { read } for pid=7758 comm="nginx" name="responsive.css" dev="dm-0" ino=17312394 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
Was caused by:
The boolean httpd_read_user_content was set incorrectly.
Description:
Allow httpd to read user content
Allow access by executing:
# setsebool -P httpd_read_user_content 1
So as you can see answer here setsebool -P httpd_read_user_content 1 when you run this command you see your static content

How can I configure nginx with multiple uwsgi vassals (with websockets) in emperor mode?

I'm having issues running 2 vassals in emperor mode (for a main app + websocket) behind an nginx server. Everything seems to be running well, but all the websocket requests return error 502 bad gateway. The websocket app is running django-websocket-redis. Any ideas where I went wrong?
Running from upstart
exec /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals/ --logto /var/log/uwsgi.log
With /etc/uwsgi/vassals/dashdb_websocket.ini:
# dashdb_websocket.ini file
[uwsgi]
umask = 002
home = /opt/dashdb/venv/dashdb-3.5
plugin = python35
chdir = /opt/dashdb
master = true
no-orphans = true
die-on-term = true
memory-report = true
env = DJANGO_SETTINGS_MODULE=dashdb.settings.opener
socket = /var/run/uwsgi/dashdb_ws.sock
pythonpath = /opt/dashdb
module = dashdb.wsgi_websocket
threads = 1
processes = 1
http-websockets = true
gevent = 1000
# Log file location
daemonize = /var/log/uwsgi/dashdb_websocket.log
With /etc/uwsgi/vassals/dashdb.ini:
# dashdb.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = /opt/%n
# Django's wsgi file
module = dashdb.wsgi
# the virtualenv (full path)
home = /opt/dashdb/venv/%n-3.5
plugin = python35
# settings location
env = DJANGO_SETTINGS_MODULE=dashdb.settings.opener
# Process-related settings
# master
master = true
# maximum number of worker processes
processes = 3
# the socket (use the full path to be safe)
socket = /var/run/uwsgi/%n.sock
# ... with appropriate permissions - may be needed
chmod-socket = 666
# clear environment on exit
vacuum = true
# Log file location
daemonize = /var/log/uwsgi/%n.log
and nginx config:
upstream django_dashdb {
server unix:/var/run/uwsgi/dashdb.sock;
}
server {
listen *:80;
server_name MY_DOMAIN;
server_tokens off;
root /opt/dashdb;
charset utf-8;
# Increase this if you want to upload large attachments
# Or if you want to accept large git objects over http
client_max_body_size 250m;
# Individual nginx logs for this GitLab vhost
access_log /var/log/dashdb/dashdb_access.log;
error_log /var/log/dashdb/dashdb_error.log;
# Django media
location /media {
alias /opt/dashdb/media;
}
# Django static
location /static {
alias /opt/dashdb/static;
}
location / {
uwsgi_read_timeout 180;
uwsgi_pass django_dashdb;
include /opt/dashdb/uwsgi_params;
}
location /ws/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://unix:/var/run/uwsgi/dashdb_ws.sock;
}
error_page 502 /502.html;
}
Errors in the logs:
2017/03/20 23:27:51 [error] 3586#3586: *2759 upstream prematurely closed connection while reading response header from upstream, client:
10.31.32.22, server: MY_DOMAIN...com, request: "GET /ws/builds?subscribe-broadcast&echo HTTP/1.1", upstream: "http://unix:/var/run/uwsgi/dashdb_ws.sock:/ws/builds?subscribe-broadcast&echo", host: "MY_DOMAIN...com"
I needed to set
http-socket = /var/run/uwsgi/dashdb_ws.sock
instead of
socket = /var/run/uwsgi/dashdb_ws.sock
in the websocket uwsgi config ini

How to deploy django on VPS with external subdomain.?

Good day.
I have a web app that I have developed using django. I tested fine on my local, and I'm happy with how it works.
However I'm facing an issue bringing it online I used those two guides to reach my deployment:
https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04
and
http://michal.karzynski.pl/blog/2013/06/09/django-nginx-gunicorn-virtualenv-supervisor/
However my page is giving me a forbidden page.
I suspect my issue is with the way I'm handling the subdomain. So the site . has been developed using php, and I have worked on my part with django and been provided with a subdomain which is member.domain.com, So I'm deploying it on the VPS and have to make it use the subdomain.
This is how my allowed hosts looks in the settings.py
ALLOWED_HOSTS = ['member.domain.com']
and
in my nginx:
upstream app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/path/project/run/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name member.domain.com;
client_max_body_size 4G;
access_log /home/path/project/logs/nginx-access.log;
error_log /home/path/project/logs/nginx-error.log;
location /static/ {
alias /home/path/project/src/static/;
}
location /media/ {
alias /home/path/project/src/media/;
}
location / {
# an HTTP header important enough to have its own Wikipedia entry:
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if and only if you use HTTPS, this helps Rack
# set the proper protocol for doing redirects:
# proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client right along so redirects
# can be set properly within the Rack application
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
# set "proxy_buffering off" *only* for Rainbows! when doing
# Comet/long-poll stuff. It's also safe to set if you're
# using only serving fast clients with Unicorn + nginx.
# Otherwise you _want_ nginx to buffer responses to slow
# clients, really.
# proxy_buffering off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
# Error pages
error_page 502 503 504 /500.html;
location = /500.html {
root /home/path/project/src/static/;
}
}
I'm not sure what I am doing wrong.
I will appreciate any help
To respond to 'example.com' and any subdomains, start the domain with a dot
ALLOWED_HOSTS = ['.example.com', '203.0.113.5']
I didn't even try how to run django on subdomains, but from article link you shared, you missed some configuration in your settings.py
ALLOWED_HOSTS = ['member.domain.com']
Changed
ALLOWED_HOSTS = ['.domain.com']
Hope this will solve your problem

Permission denied Accessing static files Nginx +uwsgi +Django

I have setup Django project on CentOS 6.5 with Nginx and uwsgi.
I am Getting error while accessing static content as below (/var/log/nginx/error.log)-
2015/11/02 19:05:37 [error] 29701#0: *52 open() "/home/amar/workspace/myproj/config/static/rest_framework/js/default.js" failed (13: Permission denied), client: 172.29.100.104, server: myapi.dev, request: "GET /static/rest_framework/js/default.js HTTP/1.1", host: "myapi.dev", referrer: "http://myapi.dev/api/v1/datasets/"
My /etc/nginx/conf.d/virtual.conf is as shown below -
# mysite_nginx.conf
# the upstream component nginx needs to connect to
upstream django {
server unix:///tmp/uwsgi.sock; # for a file socket
#server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}
# configuration of the server
#
#API
#
server {
# the port your site will be served on
listen 80;
# the domain name it will serve for
server_name myapi.dev; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 75M; # adjust to taste
location /static {
autoindex on;
alias /home/amar/workspace/myproj/config/static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
include /etc/nginx/uwsgi_params; # the uwsgi_params file you installed
}
}
Here is my uwsgi.ini file :
[uwsgi]
chdir = /home/amar/workspace/myproj
#home = %(base)/.virtualenvs/myproj
module = config.wsgi:application
home = /home/amar/.virtualenvs/myproj
master = true
processes = 3
socket = /tmp/uwsgi.sock
chmod-socket = 777
vacuum = true
Could someone point me in the right direction?
It took time but I've fixed the problem myself. Changed the user from amar to root and set static folder permission to 666. Hope it helps someone in future.
Probably related to SELinux. You will need to allow HTTPD scripts and modules to connect to the network.
setsebool httpd_can_network_connect on -P