Python interpreter hangs - experimenting with thread and queues - python-2.7

Question

Following successfully hangs on exit
import threading
import Queue as queue
import time
import sys
class WorkItem(threading.Thread):
def __init__(self):
self.P1 = 20
self.P2 = 40
threading.Thread.__init__(self)
def run(self):
print "P1 = %d" % (self.P1)
print "P2 = %d" % (self.P2)
class WorkQueue(object):
def __init__(self,queueLimit = 5):
self.WorkQueue = queue.Queue(queueLimit)
self.dispatcherThread = threading.Thread(target=self.DequeueWorker)
self.dispatcherThread.start()
self.QueueStopEvent = threading.Event()
self.QueueStopEvent.clear()
def DequeueWorker(self):
print "DequeueWorker Enter .."
while not self.QueueStopEvent.isSet():
workItem = self.WorkQueue.get(True)
workItem.start()
def DispatchToQueue(self,workItem):
self.WorkQueue.put(workItem,True)
def Stop(self):
self.QueueStopEvent.set()
self.queue = None
def main():
q = WorkQueue()
for i in range(1,20):
t = WorkItem()
q.DispatchToQueue(t)
time.sleep(10)
q.Stop()
if __name__ == "__main__":
main()
I can see the DequeueWorker is the one still running and pending and trying to understand why since I do signal the Stop event. I was expecting the loop would exit.
>>> $w
=> Frame id=0, function=DequeueWorker
Frame id=1, function=run
Frame id=2, function=__bootstrap_inner
Frame id=3, function=__bootstrap
Help appreciated !!

Answer 1

You're calling get with block set to True, which means it will block until an item is actually available in the queue. In your code, once the work queue is exhausted, the next get will indefinitely block since it is waiting for an additional work item that will never come (and not letting the next iteration of the while loop execute, so the status of QueueStopEvent doesn't get checked anymore). Try modifying your DequeueWorker method to this:
def DequeueWorker(self):
print "DequeueWorker Enter .."
while not self.QueueStopEvent.isSet():
try:
workItem = self.WorkQueue.get(True, timeout=3)
workItem.start()
except queue.Empty: continue
Now when get is called after the work queue is exhausted, it will timeout (after 3 seconds in this case, I chose that arbitrarily) and raise the queue.Empty exception. In this case, we're simply going to let the loop continue to the next iteration where the loop will break itself when QueueStopEvent eventually gets set.
Other options would be to invoke get with block set to False or to use the get_nowait method inside that try/except block:
workItem = self.WorkQueue.get(False)
workItem = self.WorkQueue.get_nowait()
Although that creates a very tight while loop when the queue is empty.