Correct code as expected:
from multiprocessing import Pool
import signal
import time
import os
def consumer(i):
while True:
# print os.getpid()
pass
def handler(signum, frame):
print 'Here you go'
signal.signal(signal.SIGTERM, handler)
p = Pool(5)
p.map_async(consumer, [1 for i in range(5)])
while True:
pass
p.terminate()
# p.close()
p.join()
==================================================
I have found the problem, when I use map function, the main func is blocked, and signal handler will be called only when map function is funished.
So, use "map_async" function is much more better to fix this problem.
Here is what I found:
A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes.
==================================================
I wrote a program like the following, and I expect to exit/(like the program print string) in the program when I type "kill pid" in the terminal, but it not work. Is there any other strategy that block the SIGTERM get in the main func.
from multiprocessing import Pool
import signal
import time
import os
def consumer(i):
while True:
# print os.getpid()
pass
def handler(signum, frame):
print 'Here you go'
signal.signal(signal.SIGTERM, handler)
p = Pool(5)
p.map(consumer, [1 for i in range(5)])
p.terminate()
# p.close()
p.join()
You need to use the map_async method as the map one blocks until results are not ready. Therefore, in your example the call to terminate is never reached.
Related
I created two simple scripts:
script.py:
import time
import sys
import signal
try:
print('i am running')
time.sleep(10)
print('i am done')
except KeyboardInterrupt:
print("you don't like me??")
and test.py:
import subprocess
import signal
from threading import Thread
import time
import os
p = subprocess.Popen('python script.py', shell=True)
t = Thread(target=p.wait)
t.start()
print('sleeping')
time.sleep(2)
print('interrupt')
p.send_signal(signal.SIGINT)
#p.send_signal(signal.SIGTERM)
t.join()
print('process finished')
If I run test.py (on ubuntu) the expected result would be:
sleeping
i am running
interrupt
you don't like me??
process finished
instead the SIGINT seems to be ignored:
sleeping
i am running
interrupt
i am done
process finished
SIGTERM terminates the process as anticipated. However no KeyboardInterrupt is raised.
Even if I add the following lines to script.py
def signal_handler(signal, frame):
print('You pressed Ctrl+C!')
signal.signal(signal.SIGINT, signal_handler)
no SIGINT seems to be received.
However, when I press C+CTRL myself a SIGINT is received. But that's not an option for me since the SIGINT must be time triggered.
Does anybody have a clue why this happens?
Cheers,
Thomas
(I've removed the use of threading in my examples because it doesn't add anything to the example other than more lines of code)
This is to do with how signals are handled in process groups, you might find this other SO answer answer helpful.
import subprocess
import signal
import time
import os
p = subprocess.Popen('python script.py', shell=True, preexec_fn=os.setsid)
print('sleeping')
time.sleep(2)
os.killpg(os.getpgid(p.pid), signal.SIGINT)
print('interrupt')
p.wait()
print('process finished')
This yields the expected result:
andy#batman[14:58:04]:~/so$ python test.py
sleeping
i am running
interrupt
you don't like me??
process finished
Signals are handled by process groups, so sending one from a process within the process group doesn't work like you think.
Interestingly, if you don't use shell=True (which you shouldn't use if you can avoid it), it works just fine.
import subprocess
import signal
import time
import os
p = subprocess.Popen(['python', 'script.py'])
print('sleeping')
time.sleep(2)
p.send_signal(signal.SIGINT)
print('interrupt')
p.wait()
print('process finished')
So if I'm honest this answer is a little bit crap because I can show you two things which ostensibly work, but not really explain why.
I am just wondering why a regular list that is passed into a process as an argument (and modified in the process) doesn't remain as a pass by reference as if I had passed it into a function normally?
The following is some example code:
from multiprocessing import Process
def appendThings(x):
x.append(1)
x.append(2)
x.append(3)
x = []
p = Process(target=appendThings, args=(x))
p.start()
p.join()
print(x)
I expected to see:
[1,2,3]
but instead got:
[]
General insight into multiprocessing is welcome too as I am currently learning :)
Two things you should note:
You should not terminate the process by p.terminate(). That would just terminate the process before it can run its course.
The following is some example code:
from multiprocessing import Process
def appendThings(x):
x.append(1)
x.append(2)
x.append(3)
print ("I reached here!")
x = []
p = Process(target=appendThings, args=(x,))
p.start()
p.terminate()
p.join()
print(x)
It will output only the following:
[]
Process don't share memory by themselves.
So if you do the following:
def appendThings(x):
x.append(1)
x.append(2)
x.append(3)
print (x)
It will print:
[1,2,3]
So in multiprocessing you should use Manager to share the objects between processes. A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value and Array. You can read more about Manager() from the Python Official Doco.
Finally you should modify your code as follows to make it do what you intended:
from multiprocessing import Process, Manager
def appendThings(x):
x.append(1)
x.append(2)
x.append(3)
x = Manager().list()
#x =list()
p = Process(target=appendThings, args=(x,))
p.start()
p.join()
print(x)
So the output will be:
[1, 2, 3]
Multiprocessing can not pass python objects directly. The arguments sent to the Process are a copy.
You have at least a couple of options to return data:
Shared ctypes
multiprocessing.Manager()
A working example using the manager:
from multiprocessing import Process, Manager
def append_things(x):
x.append(1)
x.append(2)
x.append(3)
if __name__ == '__main__':
x = Manager().list([])
p = Process(target=append_things, args=(x,))
p.start()
p.join()
print(x)
I posted a question about how to catch a "sudo shutdown -r 2" event in Python. I was sent to this thread: Run code in python script on shutdown signal .
I'm running a Raspberry Pi v2 with Jessy.
I have read about
signal
and have tried to follow the ideas in the above thread, but so far I have not been successful. Here is my code:
import time
import signal
import sys
def CloseAll(Code, Frame):
f = open('/mnt/usbdrive/output/TestSignal.txt','a')
f.write('Signal Code:' + Code)
f.write('Signal Frame:' + Frame)
f.write('\r\n')
f.close()
sys.exit(0)
signal.signal(signal.SIGTERM,CloseAll)
print('Program is running')
try:
while True:
#get readings from sensors every 15 seconds
time.sleep(15)
f = open('/mnt/usbdrive/output/TestSignal.txt','a')
f.write('Hello ')
f.write('\r\n')
f.close()
except KeyboardInterrupt:
f = open('/mnt/usbdrive/output/TestSignal.txt','a')
f.write('Done')
f.write('\r\n')
f.close()
The program runs in a "screen" session/window and reacts as expected to a CNTL-C. However, when I exit the screen session, leaving the program running, and enter "sudo shutdown -r 2", the Pi reboots as expected after 2 minutes, but the TestSignal.txt file does not show that the signal.SIGTERM event was processed.
What am I doing wrong? Or better yet, how can I trap the shutdown event, usually initiated by a cron job, and close my Python program running in a screen session gracefully?
When you do not try to await such an event, but in a parallel session send SIGTERMto that process (e.g. by calling kill -15 $PID on the process id $PID of the python script running) , you should see an instructive error message ;-)
Also the comment about the mount point should be of interest after you repaired the python errors (TypeError: cannot concatenate 'str' and 'int' objects).
Try something like:
import time
import signal
import sys
LOG_PATH = '/mnt/usbdrive/output/TestSignal.txt'
def CloseAll(Code, Frame):
f = open(LOG_PATH, 'a')
f.write('Signal Code:' + str(Code) + ' ')
f.write('Signal Frame:' + str(Frame))
f.write('\r\n')
f.close()
sys.exit(0)
signal.signal(signal.SIGTERM, CloseAll)
print('Program is running')
try:
while True:
# get readings from sensors every 15 seconds
time.sleep(15)
f = open(LOG_PATH, 'a')
f.write('Hello ')
f.write('\r\n')
f.close()
except KeyboardInterrupt:
f = open(LOG_PATH, 'a')
f.write('Done')
f.write('\r\n')
f.close()
as a starting point. If this works somehow on your system why not rewrite some portions like:
# ... 8< - - -
def close_all(signum, frame):
with open(LOG_PATH, 'a') as f:
f.write('Signal Code:%d Signal Frame:%s\r\n' % (signum, frame))
sys.exit(0)
signal.signal(signal.SIGTERM, close_all)
# 8< - - - ...
Edit: To further isolate the error and adapt more to production like mode, one might rewrite the code like this (given that syslog is running on the machine, which it should, but I never worked on devices of that kind):
#! /usr/bin/env python
import datetime as dt
import time
import signal
import sys
import syslog
LOG_PATH = 'foobarbaz.log' # '/mnt/usbdrive/output/TestSignal.txt'
def close_all(signum, frame):
"""Log to system log. Do not spend too much time after receipt of TERM."""
syslog.syslog(syslog.LOG_CRIT, 'Signal Number:%d {%s}' % (signum, frame))
sys.exit(0)
# register handler for SIGTERM(15) signal
signal.signal(signal.SIGTERM, close_all)
def get_sensor_readings_every(seconds):
"""Mock for sensor readings every seconds seconds."""
time.sleep(seconds)
return dt.datetime.now()
def main():
"""Main loop - maybe check usage patterns for file resources."""
syslog.syslog(syslog.LOG_USER, 'Program %s is running' % (__file__,))
try:
with open(LOG_PATH, 'a') as f:
while True:
f.write('Hello at %s\r\n' % (
get_sensor_readings_every(15),))
except KeyboardInterrupt:
with open(LOG_PATH, 'a') as f:
f.write('Done at %s\r\n' % (dt.datetime.now(),))
if __name__ == '__main__':
sys.exit(main())
Points to note:
the log file for the actual measurements is separate from the logging channel for operational alerts
the log file handle is safeguarded in context managing blocks and in usual operation is just kept open
for alerting the syslog channel is used.
as a sample for the message routing the syslog.LOG_USER on my system (OS X) gives me in all terminals a message, whilst the syslog.LOG_ERR priority message in signal handler only targets the system log.
should be more to the point during shutdown hassle (not opening a file, etc.)
The last point (5.) is important in case all processes receive a SIGTERM during shutdown, i.e. all want to do something (slowing things down), maybe screenalso does not accept any buffered input anymore (or does not flush), note stdout is block buffered not line buffered.
The decoupling of the output channels, should also ease the eventual disappearance of the mount point of the measurement log file.
I am writing some code that makes use of the multiprocessing module. However, since I am a newbie, what often happens is that some error pops up, putting a halt to the main application.
However, that applications' children still remain running, and I get a long, long list of running pythonw processes in my task manager list.
After an error occurs, what can I do to make sure all the child processes are killed as well?
There are two pieces to this puzzle.
How can I detect and kill all the child processes?
How can I make a best effort to ensure my code from part 1 is run whenever one process dies?
For part 1, you can use multiprocessing.active_children() to get a list of all the active children and kill them with Process.terminate(). Note the use of Process.terminate() comes with the usual warnings.
from multiprocessing import Process
import multiprocessing
def f(name):
print 'hello', name
while True: pass
if __name__ == '__main__':
for i in xrange(5):
p = Process(target=f, args=('bob',))
p.start()
# At user input, terminate all processes.
raw_input("Press Enter to terminate: ")
for p in multiprocessing.active_children():
p.terminate()
One solution to part 2 is to use sys.excepthook, as described in this answer. Here is a combined example.
from multiprocessing import Process
import multiprocessing
import sys
from time import sleep
def f(name):
print 'hello', name
while True: pass
def myexcepthook(exctype, value, traceback):
for p in multiprocessing.active_children():
p.terminate()
if __name__ == '__main__':
for i in xrange(5):
p = Process(target=f, args=('bob',))
p.start()
sys.excepthook = myexcepthook
# Sleep for a bit and then force an exception by doing something stupid.
sleep(1)
1 / 0
I'm writing a python daemon using the daemon module. I want it to respond to SIGALRM, SIGHUP, SIGUSR1 etc to perform daemon control functions.
I find that the signal handler gets called OK, however the daemon terminates when I expect it to continue running. How do I get the handler to run without terminating the daemon?
I've tried registering handlers both with signal.signal and with context.signal_map. The behaviour is the same in both cases.
Proto handler looks like this:
def myhandler(signum, frame):
logger.info("Received Signal: %s at frame: %s" % (signum, frame))
Signal registration looks like this
context.signal_map = {
signal.SIGHUP: myhandler,
signal.SIGUSR1: myhandler,
signal.SIGUSR2: myhandler,
}
This piece of code works for me:
import signal
import daemon
import time
def do_main_program():
while True:
with open("/tmp/current_time.txt", "w") as f:
f.write("The time is now " + time.ctime())
time.sleep(5)
def reload_program_config(signum, frame):
with open("/tmp/reload.txt", "w") as f:
f.write("The time is now " + time.ctime())
return None
def run():
context = daemon.DaemonContext()
context.signal_map = {
signal.SIGHUP: 'terminate',
signal.SIGUSR1: reload_program_config,
}
with context:
do_main_program()
if __name__ == "__main__":
run()