"[Errno 12] Cannot allocate memory" on Geoip2() in Django with uWSGI - django

The following code runs successfully in manage.py shell:
from django.contrib.gis.geoip2 import GeoIP2
g = GeoIP2()
If I start the server manually with manage.py runserver
and put the code in my Django view, it also runs fine.
My Django application is hosted with uWSGI and Nginx as reverse proxy.
Both uWSGI and Nginx run with www-data user.
Here is the exception that I get when running on uWSGI:
...
File "/home/myuser/Env/virtenv/myproject/index/views.py" in index
28. g = GeoIP2()
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/django/contrib/gis/geoip2/base.py" in __init__
95. self._city = geoip2.database.Reader(city_db, mode=cache)
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/geoip2/database.py" in __init__
82. self._db_reader = maxminddb.open_database(filename, mode)
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/maxminddb/__init__.py" in open_database
37. return maxminddb.reader.Reader(database, mode)
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/maxminddb/reader.py" in __init__
52. db_file.fileno(), 0, access=mmap.ACCESS_READ)
Exception Type: error at /
Exception Value: [Errno 12] Cannot allocate memory
Local variables are:
Variable Value
database '/home/myuser/Env/virtenv/myproject/geoip/GeoLite2-City.mmdb'
db_file <closed file '/home/myuser/Env/virtenv/myproject/geoip/GeoLite2-City.mmdb', mode 'rb' at 0x7f8c5cf5d390>
mode 0
self <maxminddb.reader.Reader object at 0x7f8c5cf5f550>
I use Virtualbox and my guest OS is Ubuntu 16.04. I have 4GB swap file there.
If I free up some RAM the problem persists.
This shouldn't be a memory issue on OS level, though,
as I can create the GeoIP2 object in the shell and also if I run the server manually.
Next thing I checked is that the geoip directory in my project is owned by www-data and has 775.
Both files inside (GeoLite2-City.mmdb and GeoLite2-Country.mmdb) are also owned by www-data and have 774.
/etc/systemd/system/uwsgi.service contains:
[Unit]
Description=uWSGI Emperor service
[Service]
ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown www-data:www-data /run/uwsgi'
ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target
/etc/uwsgi/sites/my_site.ini contains:
[uwsgi]
project = myproject
base = /home/myuser
home = %(base)/Env/virtenv/%(project)
binary-path = /usr/local/bin/uwsgi
chdir = %(home)
chmod-socket = 660
chown-socket = www-data:www-data
#emperor = true
#enable-threads = true
gid = www-data
limit-as = 1024
logto = /tmp/uwsgi.log
master = true
module = myproject.wsgi:application
pidfile = /opt/logs/uwsgi/master.pid
# number of cores on machine
processes = 2
python-path = %(home)
py-autoreload = 2
socket = /run/uwsgi/%(project).sock
uid = www-data
vacuum = true
virtualenv = %(base)/Env/virtenv
vhost = true
workers = 4
env = AWS_KEY=***************
env = AWS_SECRET=***************
env = DJANGO_SETTINGS_MODULE=myproject.settings.local
env = GMAIL_PASS=***************
env = PSQL_PASS=***************
env = SECRET_KEY=*********************************************
I am wondering where the limitation of uWSGI is?

limit-as = 512 in uWSGI's ini file was the culprit. Doubling it to 1024 mb resolves the problem.
From the official uWSGI 2 documentation here:
limit-as
Argument: number
Limit process address space (vsz) (in megabytes).
Limits the address space usage of each uWSGI (worker) process using
POSIX/UNIX setrlimit(). For example, limit-as 256 will disallow uWSGI
processes to grow over 256MB of address space. Address space is the
virtual memory a process has access to. It does not correspond to
physical memory. Read and understand this page before enabling this
option: http://en.wikipedia.org/wiki/Virtual_memory

Related

Touch command kill uWSGI service

I'm getting a little issue with uWSGI and my Django application on production server. I have a FreeBSD jail which has only one Django application. When I made code improvements, I do a touch on settings file in order to take into account modifications.
However, touch kills my uWSGI service each time. So I need to start uWSGI manually else I get a 502 Bad Gateway issue with my browser.
Environment:
Django version : 1.11.20
uWSGI version : 2.0.15
Python version : 3.6.2
uWSGI.ini file:
This is my uwsgi.ini file :
[uwsgi]
pythonpath=/usr/local/www/app/src/web
virtualenv = /usr/local/www/app/venv
module=main.wsgi:application
env = DJANGO_SETTINGS_MODULE=main.settings.prod
env = no_proxy=*.toto.fr
env = LANG=en_US.UTF-8
master=true
processes=2
vaccum=true
chmod-socket=660
chown-socket=www:www
socket=/tmp/uwsgi.sock
socket-timeout = 60
post-buffering = 8192
max-requests = 5000
buffer-size = 32768
offload-threads = 1
uid=www
gid=www
logdate=true
log-maxsize = 20000000
manage-script-name=true
touch-reload = /usr/local/www/app/src/web/main/settings/prod.py
Issue:
When I make a deployment, once it's done, I do :
touch /usr/local/www/app/src/web/main/settings/prod.py
Then I have my uWSGI service out.
This is the last log I have :
Thank you very much !

Superset in production

I've been trying to work out how best to productionise superset, or at least getting it running in a daemon. I created a SystemD service with the following:
[Unit]
Description=Superset
[Service]
Type=simple
WorkingDirectory=/home/XXXX/Documents/superset/venv
ExecStart=/home/XXXX/Documents/superset/venv/bin/superset runserver
[Install]
WantedBy=multi-user.target
And the last error I got to was gunicorn cannot be found. I don't know what else I am missing or is there another way to set it up?
I was able to set it up, after a bunch of searching and trial and error with supervisor, which is a python 2 program, but you can run any command (including other python version in other virtual environments).
I'm running it on an ubuntu 16 VPS. After creating an environment and installing supervisor, you create a configuration file and mine looks like this:
[supervisord]
logfile = %(ENV_HOME)s/sdaprod/supervisor/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = %(ENV_HOME)s/sdaprod/supervisor/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
identifier = supervisor
directory = %(ENV_HOME)s/sdaprod/supervisor
nocleanup = true
childlogdir = %(ENV_HOME)s/sdaprod/supervisor
strip_ansi = false
[unix_http_server]
file=/tmp/supervisor.sock
chmod = 0777
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:superset]
command = %(ENV_HOME)s/miniconda3/envs/superset/bin/superset runserver
directory = %(ENV_HOME)s/sdaprod
environment = PATH='%(ENV_PATH)s:%(ENV_HOME)s/miniconda3/envs/superset/bin',PYTHONPATH='%(ENV_PYTHONPATH)s:%(ENV_HOME)s/sdacore:%(ENV_HOME)s/sdaprod'
And then you just run supervisord from an environment that has it installed
The %(ENV_<>)s are environment variables. This is my first time doing this, so I absolutely can not vouch for this approach's efficiency, but it does work.

start: Job failed to start UWSGI,

Trying to configure ubuntu+nginx+uwsgi+django
upstart script /etc/init/uwsgi.conf
description "uWSGI application server in Emperor mode"
start on runlevel [2345]
stop on runlevel [!2345]
setuid voxa
setgid www-data
exec /usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
uwsgi configuration
[uwsgi]
project = project
base = /home/user
chdir = %(base)/%(project)
home = home/user/Env/project_env
module = %(project).wsgi:application
master = true
processes = 5
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
But after running the command sudo service uwsgi start I get an error
start: Job failed to start
What should I check to handle it?
UPD:
with virtualenv enabled app successfully runs with uwsgi command
uwsgi --http :8000 --module project.wsgi
uWSGI doesn't have permissions to create socket file in specified directory. To solve that, you can run emperor as root and drop privileges in vassal, after creating socket. Simply add to your vassal config:
username = voxa
gropuname = www-data
And remove setuid and setgid from your upstart config file.
If you're worried that someone will abuse that and use other user/group, you can use emperor tyrant mode, by adding --emperor-tyrant to uwsgi start line in upstart config. That will disallow change of username and groupname to other than owner of vassal config file.

cant run uwsgi with ini file in emperror mode - Permission denied

I cant run uwsgi in emperror mode with my ini file. I have installed uwsgi with sudo pip install uwsgi.
When I try to start uwsgi, I get error:
sudo /etc/init.d/uwsgi start
/etc/init.d/uwsgi: line 72: /usr/share/uwsgi/init/snippets: No such file or directory
/etc/init.d/uwsgi: line 73: /usr/share/uwsgi/init/do_command: No such file or directory
[....] Starting app server(s): uwsgi/etc/init.d/uwsgi: line 80: do_command: command not found
failed!
I'm using Debian. Firstly I tried to follow django and nginx docs, but when it did not work I googled a lot.
This is my ebook_uwsgi.ini file:
# ebook_uwsgi.ini file
[uwsgi]
emperor = /etc/uwsgi/vassals
#plugins = python #if uwsgi installed with pip, dont have to use this line
# Django-related settings
# the base directory (full path)
chdir = /var/www/django/ebook/ebook/wsgi/ebook/
# Django's wsgi file
module = controller.wsgi:application
# the virtualenv (full path)
home = /var/www/django/ebook
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe
socket = /var/uwsgi/ebook.sock
# ... with appropriate permissions - may be needed
chmod-socket = 664
uid = www-data
gid = www-data
# clear environment on exit
vacuum = true
no-site = True
wsgi-file = /var/www/django/ebook/ebook/wsgi/ebook/controller/wsgi.py
env = DJANGO_SETTINGS_MODULE=controller.settings # set an environment variable
And here is my ebook_nginx.conf:
# mysite_nginx.conf
# the upstream component nginx needs to connect to
upstream django {
server unix:///var/uwsgi/ebook.sock; # for a file socket
#server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}
# configuration of the server
server {
# the port your site will be served on
listen 8001;
# the domain name it will serve for
server_name IP #.bookdownloading.com; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /var/www/django/ebook/ebook/data; # your Django project's media files - amend as required
}
location /static {
alias /var/www/django/ebook/ebook/wsgi/static/; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
include /var/www/django/ebook/ebook/uwsgi_params; # the uwsgi_params file you installed
}
}
Where IP stands for servers real IP.
When I start uwgi with my ini file I get Permission denied:
uwsgi --ini ebook_uwsgi.ini
[uWSGI] getting INI configuration from ebook_uwsgi.ini
*** Starting uWSGI 2.0.10 (64bit) on [Tue Apr 14 17:11:32 2015] ***
compiled with version: 4.7.2 on 14 April 2015 16:47:40
os: Linux-2.6.32-042stab104.1 #1 SMP Thu Jan 29 12:58:41 MSK 2015
nodename: zoltan
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /var/www/django/ebook/ebook
detected binary path: /usr/local/bin/uwsgi
chdir() to /var/www/django/ebook/ebook/wsgi/ebook/
your processes number limit is 2062113
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
*** starting uWSGI Emperor ***
bind(): Permission denied [core/socket.c line 230]
I am using socked in /var/uwsgi/ebook.sock where I have set www-data as owner of uwsgi directory:
ls -la /var
total 128
drwxr-xr-x 13 root root 15 apr 14 16:29 .
drwxr-xr-x 23 root root 23 apr 12 19:46 ..
drwxr-xr-x 2 root root 19 apr 13 06:25 backups
...
drwxr-xr-x 2 www-data www-data 2 apr 14 16:36 uwsgi
Why do I get Permission denied?
//EDIT:
I was missing some config settings. So here is my new ini file:
# ebook_uwsgi.ini file
[uwsgi]
#emperor = /etc/uwsgi/vassals
#plugins = python #if uwsgi installed with pip, dont have to use this line
# Django-related settings
# the base directory (full path)
chdir = /var/www/django/ebook/ebook/wsgi/ebook/
# Django's wsgi file
module = controller.wsgi:application
# the virtualenv (full path)
home = /var/www/django/ebook
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 1
# the socket (use the full path to be safe
socket = /var/uwsgi/ebook.sock
# ... with appropriate permissions - may be needed
chmod-socket = 664
uid = www-data
gid = www-data
# clear environment on exit
vacuum = true
no-site = True
wsgi-file = /var/www/django/ebook/ebook/wsgi/ebook/controller/wsgi.py
#env = DJANGO_SETTINGS_MODULE=controller.settings # set an environment variable
and nginx settings:
# mysite_nginx.conf
# the upstream component nginx needs to connect to
upstream ebook {
server unix:///var/uwsgi/ebook.sock; # for a file socket
#server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}
# configuration of the server
server {
# the port your site will be served on
listen 8001 default_server;
# the domain name it will serve for
server_name IP; #.bookdownloading.com; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /var/www/django/ebook/ebook/data; # your Django project's media files - amend as required
}
location /static {
alias /var/www/django/ebook/ebook/wsgi/static/; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass ebook;
include /var/www/django/ebook/ebook/uwsgi_params; # the uwsgi_params file you installed
}
}
No matter if I run uwsgi or not, I get 502 Bad Gateway. Error nginx log:
2015/05/12 16:22:08 [crit] 3002#0: *1 connect() to unix:///var/uwsgi/ebook.sock failed (13: Permission denied) while connecting to upstream, client: 78.45.37.119, server: IP, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///var/uwsgi/ebook.sock:", host: "IP:8001"
Hm, even though I have set owner of stocket file to www-data, it was created as my user and nginx could not read it.
I've read your post and you're getting the permission issue because maybe www-data don't have write permissions on the folder uwsgi, you have two ways of testing it:
Log in with the www-data user and try to create a file inside /var/uwsgi/
To log in with www-data user su - www-data
Change the folder permissions to 775 (chmod 775 /var/uwsgi/)
Another option could be try to change the path of the sockets to the tmp folder. I tried your configuration in a VM using tmp folder and it works perfectly
ebook_uwsgi.ini:
socket = /tmp/ebook.sock
ebook_nginx.conf:
upstream django {
server unix:///tmp/ebook.sock;
}
I too faced the same.
/usr/share/uwsgi/init/snippets and /usr/share/uwsgi/init/do_commands
Above two files are missing with pip install uwsgi
But apt-get install uwsgi adds above two files and resolves issue
Somehow, when I run uwsgi from my linux user, uwsgi ignored my ini file and created ebook.sock with owner and group of my user, not specified www-data. When I changed owner and group manually to www-data, web loaded.
Problem was with permissions for file uwsgi_params. After using nginx file it worked all right.

Create uWSGI Socket in other folder?

I want to create uWsgi socket in my project folder and not in /tmp/
Here's my uWSGI config
[uwsgi]
socket = /tmp/uwsgi.sock #I want this in any other folder
#say in /home/me/Desktop/myDjangoApp/
chmod-socket = 666
processes = 1
master = true
vhost = true
no-site = true
But whenever I restart uWSGI with the socket created in my folder, it [fails].
Can't I create the uwsgi.sock in other folder?
I use the following uwsgi config (reduced to relevant parts):
[uwsgi]
uid = moin
gid = www-data
socket = /var/run/moin/uwsgi.sock
hook-as-root = exec:mkdir -vp /var/run/moin/; chown -v moin:www-data /var/run/moin/
Note the hook which creates the /var/run/wiki/ directory with user permissions before the server drops privileges. Call the directory whatever you want, moin was the name of my wiki engine.
On newer distros, /var/run points to a tmpfs location so that any manually created directory gets dropped after a reboot. The hook definition in this file keeps your configuration compact, compared to adding/modifying another init script.
Well, I found an alternative solution for the same. I created the socket in localhost
Here is my uwsgi file
[uwsgi]
uid = www-data
gid = www-data
master = 1
workers = 2
plugins = python
socket = 127.0.0.1:3100
enable-threads = true
processes = 2
pythonpath = <>
wsgi-file = <>
chdir = <>
unix sockets must obey to file permission schemes.
In the second config you're seting uid=www-data and gid=www-data.
The socket file must be writable by www-data and nginx must be able to read/write /tmp/uwsgi.sock
On the other hand if you find that difficult, using host:port (tcp sockets) will work too and you've seem to be able to do it like that.