I am writing some code that makes use of the multiprocessing module. However, since I am a newbie, what often happens is that some error pops up, putting a halt to the main application.
However, that applications' children still remain running, and I get a long, long list of running pythonw processes in my task manager list.
After an error occurs, what can I do to make sure all the child processes are killed as well?
There are two pieces to this puzzle.
How can I detect and kill all the child processes?
How can I make a best effort to ensure my code from part 1 is run whenever one process dies?
For part 1, you can use multiprocessing.active_children() to get a list of all the active children and kill them with Process.terminate(). Note the use of Process.terminate() comes with the usual warnings.
from multiprocessing import Process
import multiprocessing
def f(name):
print 'hello', name
while True: pass
if __name__ == '__main__':
for i in xrange(5):
p = Process(target=f, args=('bob',))
p.start()
# At user input, terminate all processes.
raw_input("Press Enter to terminate: ")
for p in multiprocessing.active_children():
p.terminate()
One solution to part 2 is to use sys.excepthook, as described in this answer. Here is a combined example.
from multiprocessing import Process
import multiprocessing
import sys
from time import sleep
def f(name):
print 'hello', name
while True: pass
def myexcepthook(exctype, value, traceback):
for p in multiprocessing.active_children():
p.terminate()
if __name__ == '__main__':
for i in xrange(5):
p = Process(target=f, args=('bob',))
p.start()
sys.excepthook = myexcepthook
# Sleep for a bit and then force an exception by doing something stupid.
sleep(1)
1 / 0
Related
I created two processes, one process that spawn multi threads is response for writing data to Queue, the other is reading data from Queue. It always deadblock in high frequent, fewer not. Especially when you add sleep in run method in write module(comment in codes). Let me put my codes below:
environments: python2.7
main.py
from multiprocessing import Process,Queue
from write import write
from read import read
if __name__ == "__main__":
record_queue = Queue()
table_queue = Queue()
pw = Process(target=write,args=[record_queue, table_queue])
pr = Process(target=read,args=[record_queue, table_queue])
pw.start()
pr.start()
pw.join()
pr.join()
write.py
from concurrent.futures import ThreadPoolExecutor, as_completed
def write(record_queue, table_queue):
thread_num = 3
pool = ThreadPoolExecutor(thread_num)
futures = [pool.submit(run, record_queue, table_queue) for _ in range (thread_num)]
results = [r.result() for r in as_completed(futures)]
def run(record_queue, table_queue):
while True:
if table_queue.empty():
break
table = table_queue.get()
# adding this code below reduce deadlock opportunity.
#import time
#import random
#time.sleep(random.randint(1, 3))
process_with_table(record_queue, table_queue, table)
def process_with_table(record_queue, table_queue, table):
#for short
for item in [x for x in range(1000)]:
record_queue.put(item)
read.py
from concurrent.futures import ThreadPoolExecutor, as_completed
import threading
import Queue
def read(record_queue, table_queue):
count = 0
while True:
item = record_queue.get()
count += 1
print ("item: ", item)
if count == 4:
break
I googled it and there are same questions on SO, but i cant see the similarity compared with my code, so can anyone help my codes, thanks...
I seem to find a solution, change run method in write module to :
def run(record_queue, table_queue):
while True:
try:
if table_queue.empty():
break
table = table_queue.get(timeout=3)
process_with_table(record_queue, table_queue, table)
except multiprocessing.queues.Empty:
import time
time.sleep(0.1)
and never see deadlock or blocking on get method.
I created two simple scripts:
script.py:
import time
import sys
import signal
try:
print('i am running')
time.sleep(10)
print('i am done')
except KeyboardInterrupt:
print("you don't like me??")
and test.py:
import subprocess
import signal
from threading import Thread
import time
import os
p = subprocess.Popen('python script.py', shell=True)
t = Thread(target=p.wait)
t.start()
print('sleeping')
time.sleep(2)
print('interrupt')
p.send_signal(signal.SIGINT)
#p.send_signal(signal.SIGTERM)
t.join()
print('process finished')
If I run test.py (on ubuntu) the expected result would be:
sleeping
i am running
interrupt
you don't like me??
process finished
instead the SIGINT seems to be ignored:
sleeping
i am running
interrupt
i am done
process finished
SIGTERM terminates the process as anticipated. However no KeyboardInterrupt is raised.
Even if I add the following lines to script.py
def signal_handler(signal, frame):
print('You pressed Ctrl+C!')
signal.signal(signal.SIGINT, signal_handler)
no SIGINT seems to be received.
However, when I press C+CTRL myself a SIGINT is received. But that's not an option for me since the SIGINT must be time triggered.
Does anybody have a clue why this happens?
Cheers,
Thomas
(I've removed the use of threading in my examples because it doesn't add anything to the example other than more lines of code)
This is to do with how signals are handled in process groups, you might find this other SO answer answer helpful.
import subprocess
import signal
import time
import os
p = subprocess.Popen('python script.py', shell=True, preexec_fn=os.setsid)
print('sleeping')
time.sleep(2)
os.killpg(os.getpgid(p.pid), signal.SIGINT)
print('interrupt')
p.wait()
print('process finished')
This yields the expected result:
andy#batman[14:58:04]:~/so$ python test.py
sleeping
i am running
interrupt
you don't like me??
process finished
Signals are handled by process groups, so sending one from a process within the process group doesn't work like you think.
Interestingly, if you don't use shell=True (which you shouldn't use if you can avoid it), it works just fine.
import subprocess
import signal
import time
import os
p = subprocess.Popen(['python', 'script.py'])
print('sleeping')
time.sleep(2)
p.send_signal(signal.SIGINT)
print('interrupt')
p.wait()
print('process finished')
So if I'm honest this answer is a little bit crap because I can show you two things which ostensibly work, but not really explain why.
Correct code as expected:
from multiprocessing import Pool
import signal
import time
import os
def consumer(i):
while True:
# print os.getpid()
pass
def handler(signum, frame):
print 'Here you go'
signal.signal(signal.SIGTERM, handler)
p = Pool(5)
p.map_async(consumer, [1 for i in range(5)])
while True:
pass
p.terminate()
# p.close()
p.join()
==================================================
I have found the problem, when I use map function, the main func is blocked, and signal handler will be called only when map function is funished.
So, use "map_async" function is much more better to fix this problem.
Here is what I found:
A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes.
==================================================
I wrote a program like the following, and I expect to exit/(like the program print string) in the program when I type "kill pid" in the terminal, but it not work. Is there any other strategy that block the SIGTERM get in the main func.
from multiprocessing import Pool
import signal
import time
import os
def consumer(i):
while True:
# print os.getpid()
pass
def handler(signum, frame):
print 'Here you go'
signal.signal(signal.SIGTERM, handler)
p = Pool(5)
p.map(consumer, [1 for i in range(5)])
p.terminate()
# p.close()
p.join()
You need to use the map_async method as the map one blocks until results are not ready. Therefore, in your example the call to terminate is never reached.
I'm trying to run a pygame program using pythonw to avoid having the console window show up. This causes a weird issue related to print statements.
Basically, the program will just exit after a few seconds with no error message. The more printing I do, the faster it happens.
If I run it in idle or at the command prompt (or in linux) the program works fine. This problem only happens when launched with pythonw (right-click, Open With, pythonw).
I'm using python 2.7.11 on Windows XP 32-bit. pygame 1.9.1release.
Is there a workaround for this? Why does the program simply terminate with no error?
import pygame
from pygame.locals import *
succeeded, failed = pygame.init()
display_surface = pygame.display.set_mode((320, 240))
clock = pygame.time.Clock()
terminate = False
while terminate is False:
for event in pygame.event.get():
if event.type == QUIT:
terminate = True
area = display_surface.fill((0,100,0))
pygame.display.flip()
elapsed = clock.tick(20)
print str(elapsed)*20
pygame.quit()
You don't need to remove print statements. Save them for later debugging. ;-)
Two steps to solve this problem:
Firstly, keep all the code in py file - don't change it to pyw now; Say it is actualCode.py
Then, create a new file runAs.pyw with the following lines in it
# In runAs.pyw file, we will first send stdout to StringIO so that it is not printed
import sys # access to stdout
import StringIO # StringIO implements a file like class without the need of disc
sys.stdout = StringIO.StringIO() # sends stdout to StringIO (not printed anymore)
import actualCode # or whatever the name of your file is, see further details below
Note that, just importing actualCode runs the file, so, in actualCode.py you should not enclose the code which is executed, in what I call is it main running file condition. For example,
# In actualCode.py file
....
....
....
if __name__ == '__main__': # Don't use this condition; it evaluates to false when imported
... # These lines won't be executed when this file is imported,
... # So, keep these lines outside
# Note: The file in your question, as it is, is fine
I am looking something like this.
If i have a method in a class, now that i accept 2 command line arguments (start, stop), so when i run python script.py start, it will start automatically, now after sometime if user enters script.py stop it should end gracefully, can it be done.
This is what i have tried to do
from multiprocessing import Process, Queue
from time import sleep
import sys
def start(name):
print('hello', name)
print "Sleeping for 60 sec"
# sleep (60)
if __name__ == '__main__':
p = Process(target=start, args=('bob',))
p.start()
Thanks