I use python 2.7.3 and daemon runner in my script. In a run(loop) method i want to sleep for the some time, but not with the such code:
while True:
time.sleep(10)
I want wait on a some synchronizing primitive, for example multiprocessing.Event. There is my code:
# -*- coding: utf-8 -*-
import logging
from daemon import runner
import signal
import multiprocessing
import spyder_cfg
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', datefmt='%m-%d %H:%M', filename=spyder_cfg.log_file)
class Daemon(object):
def __init__(self, pidfile_path):
self.stdin_path = '/dev/null'
self.stdout_path = '/dev/tty'
self.stderr_path = '/dev/tty'
self.pidfile_path = None
self.pidfile_timeout = 5
self.pidfile_path = pidfile_path
def setup_daemon_context(self, daemon_context):
self.daemon_context = daemon_context
def run(self):
logging.info('Spyder service has started')
logging.debug('event from the run() = {}'.format(self.daemon_context.stop_event))
while not self.daemon_context.stop_event.wait(10):
try:
logging.info('Spyder is working...')
except BaseException as exc:
logging.exception(exc)
logging.info('Spyder service has been stopped')
def handle_exit(self, signum, frame):
try:
logging.info('Spyder stopping...')
self.daemon_context.stop_event.set()
except BaseException as exc:
logging.exception(exc)
if __name__ == '__main__':
app = Daemon(spyder_cfg.pid_file)
d = runner.DaemonRunner(app)
d.daemon_context.working_directory = spyder_cfg.work_dir
d.daemon_context.files_preserve = [h.stream for h in logging.root.handlers]
d.daemon_context.signal_map = {signal.SIGUSR1: app.handle_exit}
d.daemon_context.stop_event = multiprocessing.Event()
app.setup_daemon_context(d.daemon_context)
logging.debug('event from the main = {}'.format(d.daemon_context.stop_event))
d.do_action()
It is my log file records:
06-04 11:32 root DEBUG event from the main = <multiprocessing.synchronize.Event object at 0x7f0ef0930d50>
06-04 11:32 root INFO Spyder service has started
06-04 11:32 root DEBUG event from the run() = <multiprocessing.synchronize.Event object at 0x7f0ef0930d50>
06-04 11:32 root INFO Spyder is working...
06-04 11:32 root INFO Spyder stopping...
There is not 'Spyder service has been stopped' print in the log, my program hang on the set() call. While debugging i see that it hang when Event.set() call, the set method hang on semaphore while all waiting entities wake up. There is no reason if Event will be global object or threading.Event. I see this one answer, but its not good for me. Is there an alternative for wait with the timeout wait with the same behavior as multiprocessing.Event?
I do print stack from the signal handler and i think there is deadlock, because signal handler use same stack with the my main process and when i call Event.set(), method wait() higher on the stack...
def handle_exit(self, signum, frame):
try:
logging.debug('Signal handler:{}'.format(traceback.print_stack()))
except BaseException as exc:
logging.exception(exc)
d.do_action()
File ".../venv/work/local/lib/python2.7/site-packages/daemon/runner.py", line 189, in do_action
func(self)
File ".../venv/work/local/lib/python2.7/site-packages/daemon/runner.py", line 134, in _start
self.app.run()
File ".../venv/work/skelet/src/spyder.py", line 32, in run
while not self.daemon_context.stop_event.wait(10):
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 337, in wait
self._cond.wait(timeout)
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 246, in wait
self._wait_semaphore.acquire(True, timeout)
File ".../venv/work/skelet/src/spyder.py", line 41, in handle_exit
logging.debug('Signal handler:{}'.format(traceback.print_stack()))
that is why this fix solve the problem:
def handle_exit(self, signum, frame):
t = Timer(1, self.handle_exit2)
t.start()
def handle_exit2(self):
self.daemon_context.stop_event.set()
Related
Background:
I have an app that scrapes through our IT infrastructure resources (vCenter, storage, and backup entities) to pull things into a central inventory for quick reference. Each collection spins up in its own thread, and I've taken measures to implement a producer/consumer setup to better scale for our resources. What I've noticed is that when I have collections running from multiple types (Ex: vCenter and storage), the web interface chugs. My thought is because I've got a ton of threads running from multiple sources and the GIL is causing everything to get queued up under one main thread. So, I thought that I could have the main producer/consumer model run as processes instead of threads since they are fairly independent of each other.
What's wrong:
When I made the code switch from spinning up threads to processes, the worker process tries to load up the models that it should, but it fails because the sub process is separate, and the applications aren't loaded up. It throws django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
What I have found is since this spawned process doesn't inherit anything from the Django main process, it's trying to access the models without having anything started. What I need is a method to spin these up but still have access to the Django ORM
What I need:
I need a way to spin up a process that can still interact with Django since it will be doing most of the heavy lifting. My thought process is that if I can spin off the collections into their own process, it will keep everything snappy and won't interfere with the webserver speed.
To clarify, these processes are not getting spawned from a view, and are only communicating with each other via a multiprocessing Queue. I'm not sending data back and forth between the spawned processes, but they are querying the database and writing data to it.
From what I found, the only thing that's remotely close to this is Celery, but in my brief research, that seems to be a bit more than I want to involve. What it seems like I need to do is to have each of these spawned processes installed as an app in the Django settings, but that doesn't seem right to me.
Code samples:
Stacktrace:
File "C:\Program Files\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Program Files\Python310\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "C:\Users\jfort\PycharmProjects\VmInventory\VMwareInventory\Processes\Vcenter.py", line 5, in <module>
from VMwareInventory.VMwareRest.VMwareRest import RESTVCenter
File "C:\Users\jfort\PycharmProjects\VmInventory\VMwareInventory\VMwareRest\VMwareRest.py", line 19, in <module>
from VMwareInventory.models import *
File "C:\Users\jfort\PycharmProjects\VmInventory\VMwareInventory\models\__init__.py", line 2, in <module>
from .Base.cost import Cost
File "C:\Users\jfort\PycharmProjects\VmInventory\VMwareInventory\models\Base\cost.py", line 2, in <module>
from .base import BaseModel
File "C:\Users\jfort\PycharmProjects\VmInventory\VMwareInventory\models\Base\base.py", line 4, in <module>
class BaseModel(models.Model):
File "C:\Program Files\Python310\lib\site-packages\django\db\models\base.py", line 127, in __new__
app_config = apps.get_containing_app_config(module)
File "C:\Program Files\Python310\lib\site-packages\django\apps\registry.py", line 260, in get_containing_app_config
self.check_apps_ready()
File "C:\Program Files\Python310\lib\site-packages\django\apps\registry.py", line 138, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
apps.py:
from django.apps import AppConfig
from VMwareInventory.settings_local import environment
from VMwareInventory.threading.initial_settings import set_default_database_items
class VmwareinventoryConfig(AppConfig):
name = 'VMwareInventory'
def ready(self):
set_default_database_items()
if environment == "prod":
from .threading.scraping import TimerScrape
TimerScrape()
threading\scraping.py (Where the TimerScrape() function lives):
# Python imports:
from threading import Thread, enumerate
from multiprocessing import Queue
from datetime import datetime, timezone
from time import sleep
# from queue import Queue
# Local imports:
from VMwareInventory.models import Vcenter, StorageSystem, BackupSystem, Setting, Platform, Application, Function, \
Region, Site, Environment, TagsReport
from VMwareInventory.reports.tags_report import TagGenericReport
from VMwareInventory.reports.missing_tags_report import MissingTagsReport
from VMwareInventory.Processes.Backup import BackupWorker
from VMwareInventory.Processes.Storage import StorageWorker
from VMwareInventory.Processes.Vcenter import VcenterWorker
# Django imports:
from django.db import connection
class TimerScrape(Thread):
def __init__(self):
Thread.__init__(self)
self.name = "timerscrape_thread"
self.vcenter_queue = Queue()
self.vcenter_list = []
self.storage_queue = Queue()
self.storage_list = []
self.backup_queue = Queue()
self.backup_list = []
self.vcenter_worker = 0
self.storage_worker = 0
self.backup_worker = 0
self.daemon = True
self.start()
def run(self):
sleep(60)
while True:
vcenters = Vcenter.objects.all()
netapps = StorageSystem.objects.all()
rubriks = BackupSystem.objects.all()
vcenter_threads = Setting.objects.get(name='vcenter_scraping_threads')
storage_threads = Setting.objects.get(name='storage_scraping_threads')
backup_threads = Setting.objects.get(name='backup_scraping_threads')
wait_hours = int(Setting.objects.get(name='scrape_wait_time').value)
connection.close()
wait_seconds = wait_hours * 3600
current_time = datetime.now(timezone.utc)
# get list of current threading and their names
threads = enumerate()
# print(threads)
thread_list = []
worker_list = []
for thread in threads:
if thread.name == "vCenter_worker_thread":
worker_list.append(thread)
elif thread.name == "storage_worker_thread":
worker_list.append(thread)
elif thread.name == "backup_worker_thread":
worker_list.append(thread)
else:
thread_list.append(thread.name)
self.vcenter_worker = 0
self.storage_worker = 0
self.backup_worker = 0
for thread in worker_list:
if thread.name == "vCenter_worker_thread":
self.vcenter_worker += 1
elif thread.name == "storage_worker_thread":
self.storage_worker += 1
elif thread.name == "backup_worker_thread":
self.backup_worker += 1
while self.vcenter_worker < int(vcenter_threads.value):
VcenterWorker(self.vcenter_queue, self.vcenter_list)
self.vcenter_worker += 1
while self.storage_worker < int(storage_threads.value):
StorageWorker(self.storage_queue, self.storage_list)
self.storage_worker += 1
while self.backup_worker < int(backup_threads.value):
BackupWorker(self.backup_queue, self.backup_list)
self.backup_worker += 1
Processes\Vcenter.py
# Python imports:
from multiprocessing import Process
# Local imports:
from VMwareInventory.VMwareRest.VMwareRest import RESTVCenter
class VcenterWorker(Process):
def __init__(self, queue, vcenter_list):
Process.__init__(self)
self.queue = queue
self.list = vcenter_list
self.name = "vCenter_worker_process"
self.start()
def run(self):
while True:
vcenter = self.queue.get()
self.list.remove(vcenter)
self.vcscrape(vcenter.name, vcenter.user, vcenter.password)
self.queue.task_done()
#staticmethod
def vcscrape(name, user, pwd):
vc_scrape = RESTVCenter(name, user, pwd)
vc_scrape.join()
return
OK, after posting this directly on the Django forums, I was given some guidance. I basically had to break these off as "standalone" Django processes in order to allow them access into the Django ORM.
I had to setup a settings, and run django.setup() in each of the processes and shift any models imports into code that would run after that execution.
Following a tutorial I have written a management command to start tornado and it looks like:
import signal
import time
import tornado.httpserver
import tornado.ioloop
from django.core.management.base import BaseCommand, CommandError
from privatemessages.tornadoapp import application
class Command(BaseCommand):
args = '[port_number]'
help = 'Starts the Tornado application for message handling.'
def sig_handler(self, sig, frame):
"""Catch signal and init callback"""
tornado.ioloop.IOLoop.instance().add_callback(self.shutdown)
def shutdown(self):
"""Stop server and add callback to stop i/o loop"""
self.http_server.stop()
io_loop = tornado.ioloop.IOLoop.instance()
io_loop.add_timeout(time.time() + 2, io_loop.stop)
def handle(self, *args, **options):
if len(args) == 1:
try:
port = int(args[0])
except ValueError:
raise CommandError('Invalid port number specified')
else:
port = 8888
self.http_server = tornado.httpserver.HTTPServer(application)
self.http_server.listen(port, address="127.0.0.1")
# Init signals handler
signal.signal(signal.SIGTERM, self.sig_handler)
# This will also catch KeyboardInterrupt exception
signal.signal(signal.SIGINT, self.sig_handler)
print "start"
tornado.ioloop.IOLoop.instance().start()
print "end"
Here I when I run this management command I get tornado tornado.access:403 GET /2/ (127.0.0.1) 1.92ms error
For test purpose I have printed "start" and "end". I guess when this command is executed successfully "end" should be printed.
Here only "start" is printed not "end". I guess there is something error on tornado.ioloop.IOLoop.instance().start() but I dont know what it is.
Can anyone guide me what is wrong in here ?
You forgot to put self.http_server.start() before starting the ioloop.
...
self.http_server = tornado.httpserver.HTTPServer(application)
self.http_server.listen(port, address="127.0.0.1")
self.http_server.start()
...
Update
Along with the httpserver start missing, are you using this library?
Your management command is exactly this one
Their tutorial is in russian and honestly, I don't read russian.
Update2:
What is see in their code, is that the url /(?P<thread_id>\d+)/ is a websocket handler:
application = tornado.web.Application([
(r"/", MainHandler),
(r'/(?P<thread_id>\d+)/', MessagesHandler),
])
...
class MessagesHandler(tornado.websocket.WebSocketHandler):
...
But the error you posted seems like something tried to access it in http via GET.
Honestly, without a debugger and the same environment, I can't figure out the issue.
The following two lines of code hangs forever:
import urllib2
urllib2.urlopen('https://www.5giay.vn/', timeout=5)
This is with python2.7, and I have no http_proxy or any other env variables set. Any other website works fine. I can also wget the site without any issue. What could be the issue?
If you run
import urllib2
url = 'https://www.5giay.vn/'
urllib2.urlopen(url, timeout=1.0)
wait for a few seconds, and then use C-c to interrupt the program, you'll see
File "/usr/lib/python2.7/ssl.py", line 260, in read
return self._sslobj.read(len)
KeyboardInterrupt
This shows that the program is hanging on self._sslobj.read(len).
SSL timeouts raise socket.timeout.
You can control the delay before socket.timeout is raised by calling
socket.setdefaulttimeout(1.0).
For example,
import urllib2
import socket
socket.setdefaulttimeout(1.0)
url = 'https://www.5giay.vn/'
try:
urllib2.urlopen(url, timeout=1.0)
except IOError as err:
print('timeout')
% time script.py
timeout
real 0m3.629s
user 0m0.020s
sys 0m0.024s
Note that the requests module succeeds here although urllib2 did not:
import requests
r = requests.get('https://www.5giay.vn/')
How to enforce a timeout on the entire function call:
socket.setdefaulttimeout only affects how long Python waits before an exception is raised if the server has not issued a response.
Neither it nor urlopen(..., timeout=...) enforce a time limit on the entire function call.
To do that, you could use eventlets, as shown here.
If you don't want to install eventlets, you could use multiprocessing from the standard library; though this solution will not scale as well as an asynchronous solution such as the one eventlets provides.
import urllib2
import socket
import multiprocessing as mp
def timeout(t, cmd, *args, **kwds):
pool = mp.Pool(processes=1)
result = pool.apply_async(cmd, args=args, kwds=kwds)
try:
retval = result.get(timeout=t)
except mp.TimeoutError as err:
pool.terminate()
pool.join()
raise
else:
return retval
def open(url):
response = urllib2.urlopen(url)
print(response)
url = 'https://www.5giay.vn/'
try:
timeout(5, open, url)
except mp.TimeoutError as err:
print('timeout')
Running this will either succeed or timeout in about 5 seconds of wall clock time.
I have a pika consumer
when i run it and sends SIGHUP signal, It gives me an exception
Consumertest.py
import signal
import traceback
import pika
from time import sleep
received_signal = False
def signal_handler(signal, frame):
global received_signal
received_signal = True
exit(1)
def sighup_handler(signal, frame):
print "sighup received"
signal.signal(signal.SIGHUP, sighup_handler)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
mq_server = "localhost"
mq_exchange = "my_exchange"
my_queue = "test_queue"
try:
mq_connection = pika.BlockingConnection(pika.ConnectionParameters(mq_server))
except:
exit(1)
mq_channel = mq_connection.channel()
mq_channel.exchange_declare(exchange=mq_exchange, durable=True)
mq_channel.queue_declare(queue=my_queue, durable=True)
mq_channel.queue_bind(my_queue, mq_exchange)
def callback(ch, method, properties, body):
try:
sleep(10)
ch.basic_reject(delivery_tag=method.delivery_tag)
except Exception as e:
traceback.print_exc() # region Message consumption
try:
print ' [*] Waiting for messages. To exit press CTRL+C'
mq_channel.basic_consume(callback, queue=my_queue)
mq_channel.start_consuming()
except Exception as e:
traceback.print_exc()
while True:
pass
exception:
[*] Waiting for messages. To exit press CTRL+C
sighup received
Traceback (most recent call last):
File "/run/media/bluto/04D0CF8ED0CF8500/Email_Projects/new_email_workers/testsigHup.py", line 47, in <module>
mq_channel.start_consuming()
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 814, in start_consuming
self.connection.process_data_events()
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 168, in process_data_events
if self._handle_read():
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 271, in _handle_read
if self._read_poller.ready():
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 54, in ready
events = self.poller.poll(self.poll_timeout)
error: (4, 'Interrupted system call')
I am new to tornado web server. When I start the tornado web server using python main_tornado.py It is working. Please see the below code.
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
When I stop the server using CTRL+C it gave the following error.
^CTraceback (most recent call last):
File "main_tornado.py", line 19, in <module>
tornado.ioloop.IOLoop.instance().start()
File "/home/nyros/Desktop/NewWeb/venv/lib/python3.2/site-packages/tornado/ioloop.py", line 301, in start
event_pairs = self._impl.poll(poll_timeout)
KeyboardInterrupt
Please solve my problem. Thanks..
You can stop Tornado main loop with tornado.ioloop.IOLoop.instance().stop(). To have this method called after passing signal with Ctrl+C you can periodically check global flag to test if main loop should end and register handler for SIGINT signal which will change value of this flag:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import signal
import logging
import tornado.ioloop
import tornado.web
import tornado.options
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
class MyApplication(tornado.web.Application):
is_closing = False
def signal_handler(self, signum, frame):
logging.info('exiting...')
self.is_closing = True
def try_exit(self):
if self.is_closing:
# clean up here
tornado.ioloop.IOLoop.instance().stop()
logging.info('exit success')
application = MyApplication([
(r"/", MainHandler),
])
if __name__ == "__main__":
tornado.options.parse_command_line()
signal.signal(signal.SIGINT, application.signal_handler)
application.listen(8888)
tornado.ioloop.PeriodicCallback(application.try_exit, 100).start()
tornado.ioloop.IOLoop.instance().start()
Output:
$ python test.py
[I 181209 22:13:43 web:2162] 200 GET / (127.0.0.1) 0.92ms
^C[I 181209 22:13:45 test:21] exiting...
[I 181209 22:13:45 test:28] exit success
UPDATE
I've just saw in question Tornado long polling requests this simple solution:
try:
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
Obviously, this is a less safe way.
UPDATE
Edited the code to remove use of global.
You can simply stop the Tornado ioloop from a signal handler. It should be safe thanks to add_callback_from_signal() method, the event loop will exit nicely, finishing any eventually concurrently running task.
import tornado.ioloop
import tornado.web
import signal
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
def sig_exit(signum, frame):
tornado.ioloop.IOLoop.instance().add_callback_from_signal(do_stop)
def do_stop(signum, frame):
tornado.ioloop.IOLoop.instance().stop()
if __name__ == "__main__":
application.listen(8888)
signal.signal(signal.SIGINT, sig_exit)
tornado.ioloop.IOLoop.instance().start()
The code is OK. The CTRL+C generates KeyboardInterrupt. To stop the server you can use CTRL+Pause Break(on windows) instead of CTRL+C. On linux CTRL+C also generates the KeyboardInterrupt also. If you will use CTRL+Z program will stop but port gets busy.
I'd say the cleanest, safest and most portable solution would be to put all closing and clean-up calls in a finally block instead of relying on KeyboardInterrupt exception:
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
# .instance() is deprecated in Tornado 5
loop = tornado.ioloop.IOLoop.current()
if __name__ == "__main__":
try:
print("Starting server")
application.listen(8888)
loop.start()
except KeyboardInterrupt:
pass
finally:
loop.stop() # might be redundant, the loop has already stopped
loop.close(True) # needed to close all open sockets
print("Server shut down, exiting...")