i am started uWSGI via supervisord in Emperor Mode to deploy multiple Django Apps in near future. So far i only deployed one App for testing purposes.
I would like to seperate the emperor logs from my vassals. So far the loggers work. Except that i can not apply log-maxsize to my vassals logger - thats also applies to my vassals req-logger.
[uwsgi]
[program:uwsgi]
command=uwsgi
--master
--emperor %(ENV_HOME)s/etc/uwsgi/vassals
--logto %(ENV_HOME)s/logs/uwsgi/emperor/emperor.log
--log-maxsize 20
--enable-threads
--log-master
<...autostart etc...>
[garden_app] - vassal
<...>
; ---------
; # Logger #
; ---------
; diable default req log and set request Log
req-logger = file:%(var_logs)/vassal_garden/garden_request.log
disable-logging = true
log-4xx = true
log-5xx = true
log-maxsize = 20
; set app / error Log
logger = file:%(var_logs)/vassal_garden/garden_app.log
log-maxsize = 20
<...>
As you can see i set the log-maxsize very low to see the effects immediately.
First of all - all logs working separately.
However, while my emperor creates new files (emperor.log.122568) after reaching the log-maxsize my Vassal files still growing above the log-maxsize or in other words nothing happens they don´t create garden_app.log.56513.
So my Question is: How to set log-maxsize for my vassals loggers? Applies log-maxsize only to logto?
I also tried logto or logto2 on my vassal but then my emperor says "unloyal bad behaving vassal" or "Permission denied".
My Solution after looking long into. Now i get separate logs and rotates.
change the options from logger to logto . logger will do the log job, but it does not rotate, for whatever reason. Also do not use file:.
Make sure to make supervisorctl reread and supervisorctl update after changes on your uwsgi.ini
uwsgi.ini
[program:uwsgi]
command=uwsgi
--master
--emperor %(ENV_HOME)s/etc/uwsgi/vassals
--logto %(ENV_HOME)s/logs/uwsgi/emperor/emperor.log
--log-maxsize 350000
--enable-threads
<...autostart etc...>
[garden_app] - vassal
<...>
; ---------
; # Logger #
; ---------
; default req logger and set request Log and - currently disabled
req-logger = file:%(var_logs)/vassal_garden/garden_request.log
disable-logging = true
log-4xx = true
log-5xx = true
log-maxsize = 350000
; set app / error Log - check
logto = %(var_logs)/vassal_garden/garden_app.log
log-maxsize = 350000
log-backupname = %(var_logs)/vassal_garden/garden_app.old.log
<...>
I have a Flask app running on Heroku with uwsgi server in which each user connects to his own database. I have implemented the solution reported here for a very similar situation. In particular, I have implemented the connection registry as follows:
class DBSessionRegistry():
_registry = {}
def get(self, URI, **kwargs):
if URI not in self._registry:
current_app.logger.info(f'INFO - CREATING A NEW CONNECTION')
try:
engine = create_engine(URI,
echo=False,
pool_size=5,
max_overflow=5)
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
a_session = Session()
self._registry[URI] = a_session
except ArgumentError:
raise Exception('Error')
current_app.logger.info(f'SESSION ID: {id(self._registry[URI])}')
current_app.logger.info(f'REGISTRY ID: {id(self._registry)}')
current_app.logger.info(f'REGISTRY SIZE: {len(self._registry.keys())}')
current_app.logger.info(f'APP ID: {id(current_app)}')
return self._registry[URI]
In my create_app() I assign a registry to the app:
app.DBregistry = DBSessionRegistry()
and whenever I need to talk to the DB I call:
current_app.DBregistry.get(URI)
where the URI is dependent on the user. This works nicely if I use uwsgi with one single process. With more processes,
[uwsgi]
processes = 4
threads = 1
sometimes it gets stuck on some requests, returning a 503 error code. I have found that the problem appears when the requests are handled by different processes in uwsgi. This is an excerpt of the log, which I commented to illustrate the issue:
# ... EVERYTHING OK UP TO HERE.
# ALL PREVIOUS REQUESTS HANDLED BY PROCESS pid = 12
INFO in utils: SESSION ID: 139860361716304
INFO in utils: REGISTRY ID: 139860484608480
INFO in utils: REGISTRY SIZE: 1
INFO in utils: APP ID: 139860526857584
# NOTE THE pid IN THE NEXT LINE...
[pid: 12|app: 0|req: 1/1] POST /manager/_save_task =>
generated 154 bytes in 3457 msecs (HTTP/1.1 200) 4 headers in 601
bytes (1 switches on core 0)
# PREVIOUS REQUEST WAS MANAGED BY PROCESS pid = 12
# THE NEXT REQUEST IS FROM THE SAME USER AND TO THE SAME URL.
# SO THERE IS NO NEED FOR CREATING A NEW CONNECTION, BUT INSTEAD...
INFO - CREATING A NEW CONNECTION
# TO THIS POINT, I DON'T UNDERSTAND WHY IT CREATED A NEW CONNECTION.
# THE SESSION ID CHANGES, AS IT IS A NEW SESSION
INFO in utils: SESSION ID: 139860363793168 # <<--- CHANGED
INFO in utils: REGISTRY ID: 139860484608480
INFO in utils: REGISTRY SIZE: 1
# THE APP AND THE REGISTRY ARE UNIQUE
INFO in utils: APP ID: 139860526857584
# uwsgi GIVES UP...
*** HARAKIRI ON WORKER 4 (pid: 11, try: 1) ***
# THE FAILED REQUEST WAS MANAGED BY PROCESS pid = 11
# I ASSUME THIS IS WHY IT CREATED A NEW CONNECTION
HARAKIRI: -- syscall> 7 0x7fff4290c6d8 0x1 0xffffffff 0x4000 0x0 0x0
0x7fff4290c6b8 0x7f33d6e3cbc4
HARAKIRI: -- wchan> poll_schedule_timeout
HARAKIRI !!! worker 4 status !!!
HARAKIRI [core 0] - POST /manager/_save_task since 1587660997
HARAKIRI !!! end of worker 4 status !!!
heroku[router]: at=error code=H13 desc="Connection closed without
response" method=POST path="/manager/_save_task"
DAMN ! worker 4 (pid: 11) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 4 (new pid: 14)
# FROM HERE ON, NOTHINGS WORKS ANYMORE
This behavior is consistent over several attempts: when the pid changes, the request fails. Even with a pool_size = 1 in the create_engine function the issue persists. No issue instead is uwsgi is used with one process.
I am pretty sure it is my fault, there is something I don't know or I don't understand about how uwsgi and/or sqlalchemy work. Could you please help me?
Thanks
What is hapeening is that you are trying to share memory between processes.
There are some exaplanations in these posts.
(is it possible to share memory between uwsgi processes running flask app?).
(https://stackoverflow.com/a/45383617/11542053)
You can use an extra layer to store your sessions outsite of the app.
For that, you can use uWsgi's SharedArea(https://uwsgi-docs.readthedocs.io/en/latest/SharedArea.html) which is very low level or you can user other approaches like uWsgi's caching(https://uwsgi-docs.readthedocs.io/en/latest/Caching.html)
hope it helps.
I'm getting a little issue with uWSGI and my Django application on production server. I have a FreeBSD jail which has only one Django application. When I made code improvements, I do a touch on settings file in order to take into account modifications.
However, touch kills my uWSGI service each time. So I need to start uWSGI manually else I get a 502 Bad Gateway issue with my browser.
Environment:
Django version : 1.11.20
uWSGI version : 2.0.15
Python version : 3.6.2
uWSGI.ini file:
This is my uwsgi.ini file :
[uwsgi]
pythonpath=/usr/local/www/app/src/web
virtualenv = /usr/local/www/app/venv
module=main.wsgi:application
env = DJANGO_SETTINGS_MODULE=main.settings.prod
env = no_proxy=*.toto.fr
env = LANG=en_US.UTF-8
master=true
processes=2
vaccum=true
chmod-socket=660
chown-socket=www:www
socket=/tmp/uwsgi.sock
socket-timeout = 60
post-buffering = 8192
max-requests = 5000
buffer-size = 32768
offload-threads = 1
uid=www
gid=www
logdate=true
log-maxsize = 20000000
manage-script-name=true
touch-reload = /usr/local/www/app/src/web/main/settings/prod.py
Issue:
When I make a deployment, once it's done, I do :
touch /usr/local/www/app/src/web/main/settings/prod.py
Then I have my uWSGI service out.
This is the last log I have :
Thank you very much !
trying to run
$ rails c RAILS_ENV=development
1 warning and 1 error are raised which I do not understand
# warning :
config.eager_load is set to nil. Please update your config/environments/*.rb files accordingly:
* development - set it to false
* test - set it to false (unless you use a tool that preloads your test environment)
* production - set it to true
# error
/config/initializers/devise.rb:13:in `+': no implicit conversion of nil into String (TypeError)
However , the config.eager_load is set to false in the development environment
config/environment/development.rb
Rails.application.configure do
config.cache_classes = false
config.eager_load = false
…/…
And looking at the config/initializers/devise.rb ( line 13) I have
config/initializers/devise.rb
Devise.setup do |config|
…/…
(13) config.mailer_sender = 'no-reply#' + Rails.application.secrets.domain_name
…/…
which lead to the config/secrets.yml file
config/secrets.yml
development:
domain_name: example.com
it's quite understandable , as running rails c ( whithout RAILS_ENV) , I get
$ rails c
development environment (Rails 4.2.3)
irb: warn: can't alias context from irb_context.
irb(main):001:0> Rails.application.secrets.domain_name
=> "example.com"
this warning is also cryptic :
irb: warn: can't alias context from irb_context
could not find any info on Google search... but at least it runs in development ....
why this warning and error using RAILS_ENV ? any enlightenment welcome
too bad .. I should HAVE READ the latest 4.2 doc !!!
so I should NOT be using RAILS_ENV at all !!
$ rails console staging
Loading staging environment (Rails 4.2.3)
irb(main):001:0> exit
$ rails console development
Loading development environment (Rails 4.2.3)
irb: warn: can't alias context from irb_context.
irb(main):001:0> exit
Based on the following instructions:
https://gist.github.com/dergachev/3866825#vagrant-setup
Ubuntu Linaro
uname -a
Linux ken-desktop 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
cat /proc/version
Linux version 3.11.0-18-generic (buildd#toyol) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014
Virtualbox-4.2
VBoxManage --version
4.2.16_Ubuntur86992
Vagrant 1.5 vagrant_1.5.0_x86_64.deb
In a cookbooks folder I cloned the following chef cookbooks:
git clone git://github.com/opscode-cookbooks/vim.git
git clone git://github.com/opscode-cookbooks/git.git
git clone git://github.com/opscode-cookbooks/apt.git
git clone git://github.com/tiokksar/chef-oh-my-zsh-solo.git
git clone git://github.com/opscode-cookbooks/openssl.git
git clone git://github.com/getaroom/chef-couchbase.git
I also installed this:
https://github.com/dotless-de/vagrant-vbguest
vagrant plugin install vagrant-vbguest
I try to make a nice Vagrantfile creating a precise64 VM with usb automatically mounted.
But each time I try to add an usbfilter on my virtualbox VM, I end up with that message:
% vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'hashicorp/precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'hashicorp/precise64' is up to date...
==> default: Setting the name of the VM: smartofficeVM_default_1395303674511_42792
==> default: The cookbook path '/home/ken/smartofficeVM/databags' doesn't exist. Ignoring...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 3000 => 3000 (adapter 1)
default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Error: Connection refused. Retrying...
default: Error: Connection refused. Retrying...
default: Error: Connection refused. Retrying...
default: Error: Connection refused. Retrying...
default: Error: Connection refused. Retrying...
The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'poweroff' state. Please verify everything is configured
properly and try again.
If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.
my configuration file is the following:
% cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# All Vagrant configuration is done here. The most common configuration
# options are documented and commented below. For a complete reference,
# please see the online documentation at vagrantup.com.
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "hashicorp/precise64"
# The url from where the 'config.vm.box' box will be fetched if it
# doesn't already exist on the user's system.
# config.vm.box_url = "http://domain.com/path/to/above.box"
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
config.vm.network "public_network"
# If true, then any SSH connections made will enable agent forwarding.
# Default value: false
# config.ssh.forward_agent = true
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
# config.vm.synced_folder "../data", "/vagrant_data"
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
# config.vm.provider "virtualbox" do |vb|
# # Don't boot with headless mode
# vb.gui = true
#
# # Use VBoxManage to customize the VM. For example to change memory:
# vb.customize ["modifyvm", :id, "--memory", "1024"]
# end
#
# View the documentation for the provider you're using for more
# information on available options.
# Enable provisioning with Puppet stand alone. Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
# You will need to create the manifests directory and a manifest in
# the file hashicorp/precise32.pp in the manifests_path directory.
#
# An example Puppet manifest to provision the message of the day:
#
# # group { "puppet":
# # ensure => "present",
# # }
# #
# # File { owner => 0, group => 0, mode => 0644 }
# #
# # file { '/etc/motd':
# # content => "Welcome to your Vagrant-built virtual machine!
# # Managed by Puppet.\n"
# # }
#
# config.vm.provision "puppet" do |puppet|
# puppet.manifests_path = "manifests"
# puppet.manifest_file = "site.pp"
# end
# Enable provisioning with chef solo, specifying a cookbooks path, roles
# path, and data_bags path (all relative to this Vagrantfile), and adding
# some recipes and/or roles.
#
config.vm.provision "chef_solo" do |chef|
chef.cookbooks_path = "cookbooks"
#chef.roles_path = "../my-recipes/roles"
chef.data_bags_path = "databags"
#chef.add_role "web"
chef.add_recipe "apt"
chef.add_recipe "zsh"
chef.add_recipe "chef-oh-my-zsh-solo"
chef.add_recipe "vim"
chef.add_recipe "git"
chef.add_recipe "openssl"
chef.add_recipe "couchbase::server"
# setup users (from data_bags/users/*.json)
# chef.add_recipe "users::sysadmins" # creates users and sysadmin group
# chef.add_recipe "users"
# chef.add_recipe "users::sysadmin_sudo" # adds %sysadmin group to sudoers
# homesick_agent and its dependencies
# chef.add_recipe "root_ssh_agent::ppid" # maintains agent during 'sudo su root'
# chef.add_recipe "ssh_known_hosts"
# populates /etc/ssh/ssh_known_hosts from data_bags/ssh_known_hosts/*.json
# You may also specify custom JSON attributes:
#chef.json = { :users => "admin" }
chef.json = {
"couchbase" => {
"server"=> {
"password" => "123"
}
}
}
chef.log_level = :debug
end
# Enable provisioning with chef server, specifying the chef server URL,
# and the path to the validation key (relative to this Vagrantfile).
#
# The Opscode Platform uses HTTPS. Substitute your organization for
# ORGNAME in the URL and validation key.
#
# If you have your own Chef Server, use the appropriate URL, which may be
# HTTP instead of HTTPS depending on your configuration. Also change the
# validation key to validation.pem.
#
# config.vm.provision "chef_client" do |chef|
# chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
# chef.validation_key_path = "ORGNAME-validator.pem"
# end
#
# If you're using the Opscode platform, your validator client is
# ORGNAME-validator, replacing ORGNAME with your organization name.
#
# If you have your own Chef Server, the default validation client name is
# chef-validator, unless you changed the configuration.
#
# chef.validation_client_name = "ORGNAME-validator"
end
On detail is: If I remove the following lines, it's starting properly (but no usb available)
vb.customize ["modifyvm", :id, "--usb", "on"]
vb.customize ["modifyvm", :id, "--usbehci", "on"]
EDIT
Logs from Vlogs file
cat VBox.log
VirtualBox VM 4.2.16_Ubuntu r86992 linux.amd64 (Sep 21 2013 11:46:57) release log
00:00:00.033561 Log opened 2014-03-20T08:21:15.686771000Z
00:00:00.033570 OS Product: Linux
00:00:00.033572 OS Release: 3.11.0-18-generic
00:00:00.033575 OS Version: #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014
00:00:00.033610 DMI Product Name:
00:00:00.033624 DMI Product Version:
00:00:00.033756 Host RAM: 3882MB total, 3328MB available
00:00:00.033763 Executable: /usr/lib/virtualbox/VBoxHeadless
00:00:00.033765 Process ID: 10288
00:00:00.033767 Package type: LINUX_64BITS_GENERIC (OSE)
00:00:00.039722 Installed Extension Packs:
00:00:00.039747 VNC (Version: 4.2.16 r86992; VRDE Module: VBoxVNC)
00:00:00.046777 SUP: Loaded VMMR0.r0 (/usr/lib/virtualbox/VMMR0.r0) at 0xffffffffa0518020 - ModuleInit at ffffffffa052e0f0 and ModuleTerm at ffffffffa052e390
00:00:00.046820 SUP: VMMR0EntryEx located at ffffffffa052f510, VMMR0EntryFast at ffffffffa052f240 and VMMR0EntryInt at ffffffffa052f230
00:00:00.049809 OS type: 'Ubuntu_64'
00:00:00.073143 File system of '/home/ken/VirtualBox VMs/smartofficeVM_default_1395303674511_42792/Snapshots' (snapshots) is unknown
00:00:00.073166 File system of '/home/ken/VirtualBox VMs/smartofficeVM_default_1395303674511_42792/box-disk1.vmdk' is ext4
00:00:00.091096 VMSetError: /build/buildd/virtualbox-4.2.16-dfsg/src/VBox/Main/src-client/ConsoleImpl2.cpp(2300) int Console::configConstructorInner(PVM, util::AutoWriteLock*); rc=VERR_NOT_FOUND
00:00:00.091111 VMSetError: Implementation of the USB 2.0 controller not found!
00:00:00.091113 Because the USB 2.0 controller state is part of the saved VM state, the VM cannot be started. To fix this problem, either install the 'Oracle VM VirtualBox Extension Pack' or disable USB 2.0 support in the VM settings
00:00:00.217513 ERROR [COM]: aRC=NS_ERROR_FAILURE (0x80004005) aIID={db7ab4ca-2a3f-4183-9243-c1208da92392} aComponent={Console} aText={Implementation of the USB 2.0 controller not found!
00:00:00.217535 Because the USB 2.0 controller state is part of the saved VM state, the VM cannot be started. To fix this problem, either install the 'Oracle VM VirtualBox Extension Pack' or disable USB 2.0 support in the VM settings (VERR_NOT_FOUND)}, preserve=false
00:00:00.224473 Power up failed (vrc=VERR_NOT_FOUND, rc=NS_ERROR_FAILURE (0X80004005))
VAGRANT= debug vragant up log
http://pastebin.com/2GMhmy9T
Anybody has some expertise on the topic?
Thank you very much.
SOLUTION: I though it was already installed... when reading : 00:00:00.039722 Installed Extension Packs: 00:00:00.039747 VNC (Version: 4.2.16 r86992; VRDE Module: VBoxVNC) But in fact I have to install the extension guest pack on the Host too. It's a bit confusing. thank you very much. You can add a proper answer, I ll validate it.
The following line in VBox logs:
00:00:00.217535 Because the USB 2.0 controller state is part of the saved VM state, the VM cannot be started. To fix this problem, either install the 'Oracle VM VirtualBox Extension Pack' or disable USB 2.0 support in the VM settings (VERR_NOT_FOUND)}, preserve=false
Highlights that you have to install the VirtualBox Extension Pack in order to fix the issue.
Download and install VirtualBox extension pack from there (according to your VirtualBox version). It may solve your problem.