Superset in production - apache-superset

I've been trying to work out how best to productionise superset, or at least getting it running in a daemon. I created a SystemD service with the following:
[Unit]
Description=Superset
[Service]
Type=simple
WorkingDirectory=/home/XXXX/Documents/superset/venv
ExecStart=/home/XXXX/Documents/superset/venv/bin/superset runserver
[Install]
WantedBy=multi-user.target
And the last error I got to was gunicorn cannot be found. I don't know what else I am missing or is there another way to set it up?

I was able to set it up, after a bunch of searching and trial and error with supervisor, which is a python 2 program, but you can run any command (including other python version in other virtual environments).
I'm running it on an ubuntu 16 VPS. After creating an environment and installing supervisor, you create a configuration file and mine looks like this:
[supervisord]
logfile = %(ENV_HOME)s/sdaprod/supervisor/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = %(ENV_HOME)s/sdaprod/supervisor/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
identifier = supervisor
directory = %(ENV_HOME)s/sdaprod/supervisor
nocleanup = true
childlogdir = %(ENV_HOME)s/sdaprod/supervisor
strip_ansi = false
[unix_http_server]
file=/tmp/supervisor.sock
chmod = 0777
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:superset]
command = %(ENV_HOME)s/miniconda3/envs/superset/bin/superset runserver
directory = %(ENV_HOME)s/sdaprod
environment = PATH='%(ENV_PATH)s:%(ENV_HOME)s/miniconda3/envs/superset/bin',PYTHONPATH='%(ENV_PYTHONPATH)s:%(ENV_HOME)s/sdacore:%(ENV_HOME)s/sdaprod'
And then you just run supervisord from an environment that has it installed
The %(ENV_<>)s are environment variables. This is my first time doing this, so I absolutely can not vouch for this approach's efficiency, but it does work.

Related

How to setup uWSGI vassal names for better log reference?

INFO:
Framweork: Django 2.X < 3.x ;
Services: supervisord; uWSGI
Host: CentOS Linux 7
Hello, i am currently testing how to deploy multiple django apps with uWSGI on my host. I set everything up based on manuals provided by my Host & uWsgi and it works. However i would like to customized everything a bit further, so that i can understand everything a bit better.
As far as i understand my uWSGI service uwsgi.ini currently works in an emperor mode and provides vassals for my two different app named baron_app.ini and prince_app.ini to handle my different apps.
Question
I noticed that the err.log is a kind of confusing for debugging with multiple apps.
for instance...
announcing my loyalty to the Emperor...
Sat May 2 21:37:58 2020 - [emperor] vassal baron_app.ini is now loyal....
[pid: 26852|app: 0|req: 2/2].....
Question: Is there a way to give my vassals a name so that it will printed in the Log ? Or a way to tell uWSGI to set some kind of process & app log relation (Emperor - Vassals - Worker etc.) in the Log?
For instance i could imagine something like this, could be easier when it comes to find errors.
#baron_app: announcing my loyalty to the Emperor...
#emperor: Sat May 2 21:37:58 2020 - [emperor] vassal baron_app.ini is now loyal....
#prince_app: [pid: 26852|app: 0|req: 2/2].....
i tried something like procname-prefix and vassal_name but it seems not to work - maybe because i donĀ“t know where to put it, in the uwsgi.ini or vassals*.ini?
my current settings...
...< etc < services.d < uwsgi.ini**
[program:uwsgi]
command=uwsgi --master -- %(ENV_HOME)s/uwsgi/apps-enabled
autostart=true
autorestart=true
stderr_logfile = ~/uwsgi/err.log
stdout_logfile = ~/uwsgi/out.log
stopsignal=INT
vacuum = 1
...< uwsgi < apps-enabled < baron_app.ini**
[uwsgi]
base = /home/kiowa/baron_app/baron_app
chdir = /home/kiowa/baron_app/
static_files = /home/kiowa/baron_app/
http = :8080
master = true
wsgi-file = %(base)/wsgi.py
touch-reload = %(wsgi-file)
static-map = /static=%(static_files)/static_storage/production_static
enable-threads = true
single-interpreter = true
app = wsgi
virtualenv = /home/kiowa/.local/env_baron
plugin = python
uid = kiowa
gid = kiowa
...< uwsgi < apps-enabled < baron_app.ini**
[uwsgi]
base = /home/kiowa/prince_app/baron_app
chdir = /home/kiowa/prince_app/
static_files = /home/kiowa/prince_app/
http = :8000
master = true
wsgi-file = %(base)/wsgi.py
touch-reload = %(wsgi-file)
static-map = /static=%(static_files)/static_storage/production_static
enable-threads = true
single-interpreter = true
app = wsgi
virtualenv = /home/kiowa/.local/prince_app
plugin = python
uid = kiowa
gid = kiowa
Ok i was able to seperate my vassals log files by putting this into my vassals.ini
; set app / error Log - check
logger = file:%(var_logs)/vassal_baron/baron_app.log
; disable default req log and set request Log and - check
req-logger = file:%(var_logs)/vassal_baron/baron_request.log
disable-logging = true

Touch command kill uWSGI service

I'm getting a little issue with uWSGI and my Django application on production server. I have a FreeBSD jail which has only one Django application. When I made code improvements, I do a touch on settings file in order to take into account modifications.
However, touch kills my uWSGI service each time. So I need to start uWSGI manually else I get a 502 Bad Gateway issue with my browser.
Environment:
Django version : 1.11.20
uWSGI version : 2.0.15
Python version : 3.6.2
uWSGI.ini file:
This is my uwsgi.ini file :
[uwsgi]
pythonpath=/usr/local/www/app/src/web
virtualenv = /usr/local/www/app/venv
module=main.wsgi:application
env = DJANGO_SETTINGS_MODULE=main.settings.prod
env = no_proxy=*.toto.fr
env = LANG=en_US.UTF-8
master=true
processes=2
vaccum=true
chmod-socket=660
chown-socket=www:www
socket=/tmp/uwsgi.sock
socket-timeout = 60
post-buffering = 8192
max-requests = 5000
buffer-size = 32768
offload-threads = 1
uid=www
gid=www
logdate=true
log-maxsize = 20000000
manage-script-name=true
touch-reload = /usr/local/www/app/src/web/main/settings/prod.py
Issue:
When I make a deployment, once it's done, I do :
touch /usr/local/www/app/src/web/main/settings/prod.py
Then I have my uWSGI service out.
This is the last log I have :
Thank you very much !

uWSGI error every time I use the requests module in Django app on Docker

I'm running a Django app with uWSGI in Docker with docker-compose. I get the same error every time I:
Send a POST request with AJAX
In handling said request in my view, I use python's requests module, i.e. r = requests.get(some_url)
uWSGI says the following:
!!! uWSGI process 13 got Segmentation Fault !!!
DAMN ! worker 1 (pid: 13) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 24)
spawned 4 offload threads for uWSGI worker 1
The console in the browser says net::ERR_EMPTY_RESPONSE
I've tried using the requests module in different places, and wherever I put it I get the same Segmentation Fault error. I'm also able to run everything fine outside of docker with no errors, so I've narrowed it down to: docker + requests module = errror.
Is there something that could be blocking the requests sent with the requests module from within the docker container? Thanks in advance for your help.
Here's my uwsgi.ini file:
[uwsgi]
chdir = %d
module = my_project.wsgi:application
master = true
processes = 2
http = 0.0.0.0:8000
vacuum = true
pidfile = /tmp/my_project.pid
daemonize = %d/my_project.log
check-static = %d
static-expires = /* 7776000
offload-threads = %k
uid = 1000
gid = 1000
# there is no /etc/mime.types on the docker Arch Linux image
mime-file = %d/mime.types
Dockerfile:
FROM alpine:3.8
ENV PYTHONUNBUFFERED 1
RUN mkdir /my_project
WORKDIR /my_project
RUN apk add build-base python3-dev py3-pip python3
# deps for python cryptography
RUN apk add libffi-dev musl-dev openssl-dev
# dep for uwsgi
RUN apk add linux-headers
ADD requirements.txt /my_project/
RUN pip3 install -r requirements.txt
ADD . /my_project/
ENTRYPOINT ./start.sh
docker-compose.yml:
version: '3'
services:
web:
build: .
entrypoint: ./start.sh
volumes:
- .:/my_project
ports:
- "8000:8000"
environment:
- DEBUG_LEVEL=INFO
network_mode: "host"
start.sh:
#!/bin/sh
echo '' > logfile.log
uwsgi --ini uwsgi.ini
tail -f logfile.log
Solution: Change base image to Ubuntu 16.04 and everything works fine now.

"[Errno 12] Cannot allocate memory" on Geoip2() in Django with uWSGI

The following code runs successfully in manage.py shell:
from django.contrib.gis.geoip2 import GeoIP2
g = GeoIP2()
If I start the server manually with manage.py runserver
and put the code in my Django view, it also runs fine.
My Django application is hosted with uWSGI and Nginx as reverse proxy.
Both uWSGI and Nginx run with www-data user.
Here is the exception that I get when running on uWSGI:
...
File "/home/myuser/Env/virtenv/myproject/index/views.py" in index
28. g = GeoIP2()
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/django/contrib/gis/geoip2/base.py" in __init__
95. self._city = geoip2.database.Reader(city_db, mode=cache)
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/geoip2/database.py" in __init__
82. self._db_reader = maxminddb.open_database(filename, mode)
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/maxminddb/__init__.py" in open_database
37. return maxminddb.reader.Reader(database, mode)
File "/home/myuser/Env/virtenv/local/lib/python2.7/site-packages/maxminddb/reader.py" in __init__
52. db_file.fileno(), 0, access=mmap.ACCESS_READ)
Exception Type: error at /
Exception Value: [Errno 12] Cannot allocate memory
Local variables are:
Variable Value
database '/home/myuser/Env/virtenv/myproject/geoip/GeoLite2-City.mmdb'
db_file <closed file '/home/myuser/Env/virtenv/myproject/geoip/GeoLite2-City.mmdb', mode 'rb' at 0x7f8c5cf5d390>
mode 0
self <maxminddb.reader.Reader object at 0x7f8c5cf5f550>
I use Virtualbox and my guest OS is Ubuntu 16.04. I have 4GB swap file there.
If I free up some RAM the problem persists.
This shouldn't be a memory issue on OS level, though,
as I can create the GeoIP2 object in the shell and also if I run the server manually.
Next thing I checked is that the geoip directory in my project is owned by www-data and has 775.
Both files inside (GeoLite2-City.mmdb and GeoLite2-Country.mmdb) are also owned by www-data and have 774.
/etc/systemd/system/uwsgi.service contains:
[Unit]
Description=uWSGI Emperor service
[Service]
ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown www-data:www-data /run/uwsgi'
ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target
/etc/uwsgi/sites/my_site.ini contains:
[uwsgi]
project = myproject
base = /home/myuser
home = %(base)/Env/virtenv/%(project)
binary-path = /usr/local/bin/uwsgi
chdir = %(home)
chmod-socket = 660
chown-socket = www-data:www-data
#emperor = true
#enable-threads = true
gid = www-data
limit-as = 1024
logto = /tmp/uwsgi.log
master = true
module = myproject.wsgi:application
pidfile = /opt/logs/uwsgi/master.pid
# number of cores on machine
processes = 2
python-path = %(home)
py-autoreload = 2
socket = /run/uwsgi/%(project).sock
uid = www-data
vacuum = true
virtualenv = %(base)/Env/virtenv
vhost = true
workers = 4
env = AWS_KEY=***************
env = AWS_SECRET=***************
env = DJANGO_SETTINGS_MODULE=myproject.settings.local
env = GMAIL_PASS=***************
env = PSQL_PASS=***************
env = SECRET_KEY=*********************************************
I am wondering where the limitation of uWSGI is?
limit-as = 512 in uWSGI's ini file was the culprit. Doubling it to 1024 mb resolves the problem.
From the official uWSGI 2 documentation here:
limit-as
Argument: number
Limit process address space (vsz) (in megabytes).
Limits the address space usage of each uWSGI (worker) process using
POSIX/UNIX setrlimit(). For example, limit-as 256 will disallow uWSGI
processes to grow over 256MB of address space. Address space is the
virtual memory a process has access to. It does not correspond to
physical memory. Read and understand this page before enabling this
option: http://en.wikipedia.org/wiki/Virtual_memory

start: Job failed to start UWSGI,

Trying to configure ubuntu+nginx+uwsgi+django
upstart script /etc/init/uwsgi.conf
description "uWSGI application server in Emperor mode"
start on runlevel [2345]
stop on runlevel [!2345]
setuid voxa
setgid www-data
exec /usr/local/bin/uwsgi --emperor /etc/uwsgi/sites
uwsgi configuration
[uwsgi]
project = project
base = /home/user
chdir = %(base)/%(project)
home = home/user/Env/project_env
module = %(project).wsgi:application
master = true
processes = 5
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
socket = %(base)/%(project)/%(project).sock
chmod-socket = 664
vacuum = true
But after running the command sudo service uwsgi start I get an error
start: Job failed to start
What should I check to handle it?
UPD:
with virtualenv enabled app successfully runs with uwsgi command
uwsgi --http :8000 --module project.wsgi
uWSGI doesn't have permissions to create socket file in specified directory. To solve that, you can run emperor as root and drop privileges in vassal, after creating socket. Simply add to your vassal config:
username = voxa
gropuname = www-data
And remove setuid and setgid from your upstart config file.
If you're worried that someone will abuse that and use other user/group, you can use emperor tyrant mode, by adding --emperor-tyrant to uwsgi start line in upstart config. That will disallow change of username and groupname to other than owner of vassal config file.