uWSGI not releasing memory - django

I tried my hands with an extremely small django app that serves mainly html+static content with no db operations. The app is on nginx and uwsgi. I also have postgres installed, but for this issue, i did not do any DB operations.
I find that memory is not getting released by the uwsgi process. In this chart from newrelic, you will find that the memory occupied by the uwsgi process remains stagnant at ~100MB , though during that stagnancy there have been absolutely no activity with the website/app.
Also FYI: The app/uwsgi process when it started consumed only 56MB. I reached this ~100MB when i was testing with ab(apache benchmark) and was hitting it with -n 1000 -c 10 or around that range.
Nginx Conf
server
{
listen 80;
server_name <ip_address>;
root /var/www/mywebsite.com/;
access_log /var/www/logs/nginx_access.log;
error_log /var/www/logs/nginx_error.log;
charset utf-8;
default_type application/octet-stream;
tcp_nodelay off;
gzip on;
location /static/
{
alias /var/www/mywebsite.com/static/;
expires 30d;
access_log off;
}
location /
{
include uwsgi_params;
uwsgi_pass unix:/var/www/mywebsite.com/django.sock;
}
}
app_uwsgi.ini
[uwsgi]
plugins = python
; define variables to use in this script
project = myapp
base_dir = /var/www/mywebsite.com
app=reloc
uid = www-data
gid = www-data
; process name for easy identification in top
procname = %(project)
no-orphans = true
vacuum = true
master = true
harakiri = 30
processes = 2
processes = 2
pythonpath = %(base_dir)/
pythonpath = %(base_dir)/src
pythonpath = %(base_dir)/src/%(project)
logto = /var/www/logs/uwsgi.log
chdir = %(base_dir)/src/%(project)
module = reloc.wsgi:application
socket = /var/www/mywebsite.com/django.sock
chmod-socket = 666
chown-socket = www-data
Update 1: So it looks like, its not uwsgi, but the python processes that retain certain datastructures for faster processing.

It is common for web frameworks to load their code up into memory. This is not generally a problem, but it is not a bad idea to put a cap on your worker's total memory consumption as, over the course of several requests, an individual worker's memory consumption may grow.
When the worker reaches or exceeds the cap, it will restart itself once the request is served. This is done via the reload_on_rss flag
what you want to set it to depends on the memory available on your server and the number of workers you are running.

You might also limit the maximum number of requests per worker with the max-requests option in your .ini file. This will kill the worker that has handled the specified amount of max-requests and spawn a new one.

Related

why is uwsgi cursing and removing instances?

I am trying to test-run a django project on my VPS for the first time. I followed a step-by-step tutorial on a blog (thanks to a nice guy on #django channel at liberachat). The setup involves uWSGI and nginx. The django project files are at /srv/www/site/*. uwsgi configuration is at /etc/uwsgi.d/mysite.com.ini, and nginx configuration is at /etc/nginx/conf.d/mysite.com.conf.
Here is what happens when I start uwsgi (systemctl start uwsgi):
Jul 29 14:35:09 mysite.com uwsgi[6998]: [uWSGI] getting INI configuration from mysite.com.ini
Jul 29 14:35:09 mysite.com uwsgi[6318]: Fri Jul 29 14:35:09 2022 - [emperor] curse the uwsgi instance mysite.com.ini (pid: 6998)
Jul 29 14:35:09 mysite.com uwsgi[6318]: Fri Jul 29 14:35:09 2022 - [emperor] removed uwsgi instance mysite.com.ini
How do I interpret and fix that?
contents of /etc/uwsgi.d/mysite.com.ini:
procname-master = demosite
# Now paths can be specified relative to here.
chdir = /srv/www/
# allow nginx
uid = 981
gid = 981
chmod-socket = 644
socket = server.sock
# Task management
; Max 4 processes
processes = 2
; Each running 4 threads
threads = 2
; Reduce to 1 process when quiet
cheaper = 1
; Save some memory per thread
thread-stack-size = 512
# Logging
plugin = logfile
; Log request details here
req-logger = file:logs/request.log
; Log other details here
logger = file:logs/error.log
log-x-forwarded-for = true
# Python app
plugin = python3
; Activate this virtualenv
virtualenv = venv/
; Add this dir to PYTHONPATH so Python can find our code
pythonpath = site/
; The WSGI module to load
module = mysite.wsgi
# Don't load the app in the Master - saves memory in quiet times
lazy-apps = true
contents of /etc/nginx/conf.d/mysite.conf:
# Allow gzip compression
gzip_types text/css application/json application/x-javascript;
gzip_comp_level 6;
gzip_proxied any;
# Look for files with .gz to serve pre-compressed data
gzip_static on;
server {
listen 80;
# The hostname(s) of my site
server_name mysite.com;
# Where to look for content (static and media)
root /srv/www/html/;
# Defines the connection for talking to our Django app service
location #proxy {
# Pass other requests to uWSGI
uwsgi_pass unix://srv/www/server.sock;
include uwsgi_params;
}
# nginx docs recommend try_files over "if"
location / {
# Try to serve existing files first
#try_files $uri #proxy =404;
root /srv/www/html/;
}
}
What did I miss?

AWS ec2 Nginx + Django server easily gets to 100% CPU utilisation without much traffic

Hi I am using AWS EC2 instance (c5.18xlarge (72vcpu and 144 Gib ram)) with Nginx + Django and postgresql.
The problem is it gets to 100% CPU utilisation in 400-500 concurrent users and slows down.
Here are my settings:
postgresql.conf
max_connections = 800
shared_buffers = 50000MB
Nginx.conf
user www-data;
worker_processes 128;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
worker_rlimit_nofile 50000;
events {
worker_connections 10000;
#multi_accept on;
}
sysctl.conf
net.core.somaxconn = 65000
net.core.netdev_max_backlog = 65535
fs.file-max = 90000
net.ipv4.ip_forward = 1
kernel.shmmax=1006632960
gunicorn_start.bash
NAME="django_app" # Name of the application
DJANGODIR=/home/ubuntu/project # Django project directory
SOCKFILE=/home/ubuntu/env/run/gunicorn.sock # we will communicte using this unix socket
USER=ubuntu # the user to run as
GROUP=ubuntu # the group to run as
NUM_WORKERS=128 # how many worker processes should Gunicorn spaw
TIMEOUT=120
DJANGO_SETTINGS_MODULE=project.settings # which settings file should Django use
DJANGO_WSGI_MODULE=project.wsgi # WSGI module name
echo "Starting $NAME as `whoami`"
I know the server can't overload on the current usage of the app. Please suggest how can this be solved. Or how can it be debugged to reach the problem.
I am also attaching the top screenshot:

Nginx not serving django 3 through wsgi

I am trying to serve a Django 3 build from nginx through wsgi and I cannot seem to get the last stage. I can browse to the Django site by a runserver and launching the uwsgi.ini, but, when I try to browse to the site through nginx I get a 502 error (bad gateway)(firefox). If I try from a remote site it renders the nginx home page.
I have a build of Apache on the same server. I had to spoof an IP and use a unique port to get them running side by side.
the nginx/error.log does not register any problem.
Below is the uwsgi.ini.
[uwsgi]
# variables
projectname = website
base = /opt/website/
# configuration
master = true
http = :8000
uid = nginx
virtualenv = /opt/website/djangoprojectenv
pythonpath = %(base)
chdir = %(base)
env = DJANGO_SETTINGS_MODULE=%(projectname).settings.pro
#module = %(projectname).wsgi:application
module = website.wsgi:application
socket = /tmp/%(projectname).new.sock
chown-socket = %(uid):nginx
chmod-socket = 666
And below is the conf file from nginx/conf.d
server {
listen 192.168.1.220:81;
server_name willdoit.com;
access_log off;
error_log /var/log/nginx_error.log;
location / {
uwsgi_pass unix:/tmp/website.sock;
include /etc/nginx/uwsgi_params;
uwsgi_read_timeout 300s;
client_max_body_size 32m;
}
}
The /tmp/website.sock file is owned by nginx:nginx.
If there are additional details I need to post, please advise. Any help you can provide would be welcome.
The socket was defined incorrectly. Closing.

uwsgi ImportError: No module named os

I'm teaching myself how to setup an Ubuntu Server to run my Django application. I want to use Nginx + uwsgi. I know that this question can be very easy for experts but I've spent 6 days looking for it over the internet without getting it (in any case, forgive me if there is any link with the answer). I've followed a lot of tutorials and posts but I didn't found a solution.
I describe my file structure below:
My django project is located in /usr/local/projects/myproject
My virtualenv is in /root/.virtualenvs/myproject
My uwsgi config file myproject.ini is in /etc/uwsgi/apps-available/ and correctly symbolic linked in /etc/uwsgi/apps-enabled/
[uwsgi]
plugins = python
socket = /tmp/myproject.sock
chmod-socket = 644
uid = www-data
gid = www-data
master = true
enable-threads = true
processes = 2
no-site=true
virtualenv = /root/.virtualenvs/myproject
chdir = /usr/local/projects/myproject
module = myproject.wsgi:application
pidfile = /usr/local/projects/myproject/myproject.pid
logto = /var/log/uwsgi/myproject_uwsgi.log
vacuum = true
My nginx config file myproject.conf is in /etc/nginx/sites-available/ and correctly symbolic linked in /etc/nginx/sites-enabled/
# the upstream component nginx needs to connect to
upstream django {
server unix:///tmp/myproject.sock; # for a file socket
}
server {
listen 80;
server_name dev.myproject.com www.dev.myproject.com;
access_log /var/log/nginx/myproject_access.log;
error_log /var/log/nginx/myproject_error.log;
location / {
uwsgi_pass unix:///tmp/myproject.sock;
include /etc/nginx/uwsgi_params;
uwsgi_param UWSGI_SCRIPT myproject.wsgi;
}
location /media/ {
alias /usr/local/projects/myproject/media/;
}
location /static/ {
alias /usr/local/projects/myproject/static/;
}
}
When I try to access to dev.myproject.com I get an Internal Server Error. Then I take a look to my uwsgi log:
Traceback (most recent call last):
File "./myproject/wsgi.py", line 9, in <module>
import os
ImportError: No module named os
Sat Jul 26 17:39:16 2014 - unable to load app 0 (mountpoint='') (callable not found or import error)
Sat Jul 26 17:39:16 2014 - --- no python application found, check your startup logs for errors ---
[pid: 8559|app: -1|req: -1/8] 79.148.138.10 () {40 vars in 685 bytes} [Sat Jul 26 17:39:16 2014] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 1 headers in 57 bytes (0 switches on core 0)
I need your help because I'm not able to find a solution despite the posibility of being very simple.
If you need to know something else let me know and I will update my question as soon as possible.
I finally found a solution.
I followed kchan's suggestion about not putting any of the contents in the /root/ directory. Basically, I did some small changes in my myproject.conf file and my myproject.ini file. I created a user and structured everything like below:
uwsgi config file myproject.ini in /etc/uwsgi/apps-available/ and correctly symbolic linked in /etc/uwsgi/apps-enabled/
[uwsgi]
plugins = python
socket = /tmp/myproject.sock
chmod-socket = 644
uid = www-data
gid = www-data
master = true
enable-threads = true
processes = 2
virtualenv = /home/user/.virtualenvs/myproject
chdir = /home/user/projects/myproject
module = myproject.wsgi:application
pidfile = /home/user/projects/myproject/myproject.pid
daemonize = /var/log/uwsgi/myproject_uwsgi.log
vacuum = true
nginx config file myproject.conf in /etc/nginx/sites-available/ and correctly symbolic linked in /etc/nginx/sites-enabled/
# the upstream component nginx needs to connect to
upstream django {
server unix:///tmp/myproject.sock; # for a file socket
}
server {
listen 80;
server_name dev.myproject.com www.dev.myproject.com;
access_log /var/log/nginx/myproject_access.log;
error_log /var/log/nginx/myproject_error.log;
location / {
uwsgi_pass django;
include /etc/nginx/uwsgi_params;
}
location /media/ {
alias /home/user/projects/myproject/media/;
}
location /static/ {
alias /home/user/projects/myproject/static/;
}
}
I must say that I think the real problem was to try to setup my DB configuration in the postactivate file of my virtualenv. Hope to help someone else.

Nginx + uWSGI basic configuration

I'm new to both, I got to run 2 Django skeleton apps (just shows the "It works!" page) using Emperor, but I want to try it without Emperor. (to better understand how it works)
My nginx.conf:
# snipped...
server {
listen 92;
server_name example.com;
access_log /home/john/www/example.com/logs/access.log;
error_log /home/john/www/example.com/logs/error.log;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:8001;
}
}
# snipped...
And I start uWSGI by:
$ uwsgi --ini /home/john/www/example.com/uwsgi.ini
With uwsgi.ini being:
[uwsgi]
http = :8001
chdir = /home/john/www/example.com/example
module = example.wsgi
master = True
home = /home/john/Envs/example.com
Once uwsgi and nginx are running, I can access localhost:8001, but not localhost:92.
What am I missing?
Thanks in advance.
You are telling the uwsgi process to serve applications using the http protocol. This feature is meant mainly for developer convenience. You should instead tell it to use the uwsgi protocol:
[uwsgi]
protocol = uwsgi
socket = 127.0.0.1:8001
chdir = /home/john/www/example.com/example
module = example.wsgi
master = True
home = /home/john/Envs/example.com