I'm running a multiprocessing script that is supposed to launch 2.000.000 jobs of about 0.01 seconds. Each job put the result in a queue imported from Queue because the queue from Multiprocessing module couldn't handle more than 517 results in it.
My programm freeze before getting the results from the queue. Here is the core of my multiprocess function:
while argslist != []:
p = mp.Process(target=function, args=(result_queue, argslist.pop(),))
jobs.append(p)
p.start()
for p in jobs:
p.join()
print 'over'
res = [result_queue.get() for p in jobs]
print 'got it'
output: "over" but never "got it"
when I replace
result_queue.get()
by
result_queue.get_nowait()
I got the raise Empty error saying my queue is Empty...
but if I do the queue.get() just after the queue.put() in my inner function, then it works, showing me that the queue is well filed by my function..
queue.Queue is not shared between processes, so it won't work with that, you must use multiprocessing.Queue.
To avoid a deadlock you should not join your processes before getting the results from the queue. A multiprocessing.Queue is effectively limited by its underlying pipes' buffer, so if that fills up no more items can be flushed to the pipe and queue.put() will block until a consumer calls queue.get(), but if the consumer is joining a blocked process, then you have a deadlock.
You can avoid all of this by using a multiprocessing.Pool and its map() instead.
Thank you mata, I switched back to the multiprocessing.Queue() but I don't want to use a pool because I want to keep track of how many jobs did run. I finally added an if statement to regularly empty my queue.
def multiprocess(function, argslist, ncpu):
total = len(argslist)
done = 0
result_queue = mp.Queue(0)
jobs = []
res = []
while argslist != []:
if len(mp.active_children()) < ncpu:
p = mp.Process(target=function, args=(result_queue, argslist.pop(),))
jobs.append(p)
p.start()
done += 1
print "\r",float(done)/total*100,"%", #here is to keep track
# here comes my emptying step
if len(jobs) == 500:
tmp = [result_queue.get() for p in jobs]
for r in tmp:
res.append(r)
result_queue = mp.Queue(0)
jobs = []
tmp = [result_queue.get() for p in jobs]
for r in tmp:
res.append(r)
return res
Then comes to my mind this question:
Is 500 jobs the limit because of python or because of my machine or my system?
Will this threshold be buggy if my multiprocessing function is used in other conditions?
Related
I'm trying to implement a crawler that visits some URL, collects new relative URLs from it and builds a report. I'm trying to do it concurrently using Crystal fibers and channels, like the following:
urls = [...] # of String
visited_urls = []
pool_size.times do
spawn do
loop do
url = urls.shift?
break if url.nil?
channel.send(url) if some_condition
end
end
end
# TODO: here the problem!
loop do
url = channel.receive?
break if url.nil? || channel.closed?
visited_urls << url
end
puts visited_urls.inspect
But here I have a problem - infinite second loop (it calls channel.receive? till the last item in the channel and than waits for a new message that never arrives). Issue exists because I never know how much items actually in the channel, so I can't do like proposed in the Concurency section of the Crystal lang Guides.
So maybe there are some good practices how to work with the channel when we don't know how much items it will store and we need to receive? Thanks!
A common solution to this is to have a kill value. Either as part of the main data flow like this:
results = Channel(String|Symbol).new(POOL_SIZE * 2)
POOL_SIZE.times do
spawn do
while has_work?
results.send "some work result"
end
results.send :done
end
end
done_workers = 0
loop do
message = results.receive
if message == :done
done_workers += 1
break if done_workers == POOL_SIZE
elsif message.is_a? String
puts "Got: #{message}"
end
end
Or via a secondary channel to signal the event:
results = Channel(String).new(POOL_SIZE * 2)
done = Channel(Nil).new(POOL_SIZE)
POOL_SIZE.times do
spawn do
while has_work?
results.send "some work result"
end
done.send nil
end
end
done_workers = 0
loop do
select
when message = results.receive
puts "Got: #{message}"
when done.receive
done_workers += 1
break if done_workers == POOL_SIZE
end
end
Sorry for the long code, I have tried to make it as simple as possible and yet reproducible.
In short, this python script starts four processes that randomly distribute numbers into lists. Then, the result is added to a multiprocessing.Queue().
import random
import multiprocessing
import numpy
import sys
def work(subarray, queue):
result = [numpy.array([], dtype=numpy.uint64) for i in range (0, 4)]
for element in numpy.nditer(subarray):
index = random.randint(0, 3)
result[index] = numpy.append(result[index], element)
queue.put(result)
print "after the queue.put"
jobs = []
queue = multiprocessing.Queue()
subarray = numpy.array_split(numpy.arange(1, 10001, dtype=numpy.uint64), 4)
for i in range(0, 4):
process = multiprocessing.Process(target=work, args=(subarray[i], queue))
jobs.append(process)
process.start()
for j in jobs:
j.join()
print "the end"
All processes ran the print "after the queue.put" line. However, it doesn't get to the print "the end" line. Weird enough, if I change the arange from 10001 to 1001, it gets to the end. What is happening?
most of the child processes are blocking on put call.
multiprocessing queue put
block if necessary until a free slot is available.
this can be avoided by adding a call to queue.get() before join.
Also, in multiprocessing code please isolate the parent process by having:
if __name__ == '__main__':
# main code here
Compulsory usage of if name==“main” in windows while using multiprocessing
I will expand my comment into a short answer. As I also do not understand the weird behavior it is merely a workaround.
A first observation is that the code runs to the end if the queue.put line is commented out, so it must be a problem related to the queue. The results are actually added to the queue so the problem must be in the interplay between the queue and join.
The following code works as expected
import random
import multiprocessing
import numpy
import sys
import time
def work(subarray, queue):
result = [numpy.array([], dtype=numpy.uint64) for i in range (4)]
for element in numpy.nditer(subarray):
index = random.randint(0, 3)
result[index] = numpy.append(result[index], element)
queue.put(result)
print("after the queue.put")
jobs = []
queue = multiprocessing.Queue()
subarray = numpy.array_split(numpy.arange(1, 15001, dtype=numpy.uint64), 4)
for i in range(4):
process = multiprocessing.Process(target=work, args=(subarray[i], queue))
jobs.append(process)
process.start()
res = []
while len(res)<4:
res.append(queue.get())
print("the end")
This is the reason:
Joining processes that use queues
Bear in mind that a process that has put items in a queue will wait before terminating until all the buffered items are fed by the “feeder” thread to the underlying pipe. (The child process can call the cancel_join_thread() method of the queue to avoid this behaviour.)
This means that whenever you use a queue you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined. Otherwise you cannot be sure that processes which have put items on the queue will terminate. Remember also that non-daemonic processes will be joined automatically.
In our Django project, there is a view which creates multiple objects (from 5 to even 100). The problem is that creating phase takes a very long time.
Don't know why is that so but I suppose that it could be because on n objects, there are n database lookups and commits.
For example 24 objects takes 67 seconds.
I want to speed up this process.
There are two things I think may be worth to consider:
To create these objects in one query so only one commit is executed.
Create a ThreadPool and create these objects parallel.
This is a part of the view which causes problems (We use Postgres on localhost so connection is not a problem)
#require_POST
#login_required
def heu_import(request):
...
...
product = Product.objects.create(user=request.user,name=name,manufacturer=manufacturer,category=category)
product.groups.add(*groups)
occurences = []
counter = len(urls_xpaths)
print 'Occurences creating'
start = datetime.now()
eur_currency = Currency.objects.get(shortcut='eur')
for url_xpath in urls_xpaths:
counter-=1
print counter
url = url_xpath[1]
xpath = url_xpath[0]
occ = Occurence.objects.create(product=product,url=url,xpath=xpath,active=True if xpath else False,currency=eur_currency)
occurences.append(occ)
print 'End'
print datetime.now()-start
...
return render(request,'main_app/dashboard/new-product.html',context)
Output:
Occurences creating
24
.
.
.
0
End
0:01:07.727000
EDIT:
I tried to put the for loop into the with transaction.atomic(): block but it seems to help only a bit (47 seconds instead of 67).
EDIT2:
I'm not sure but it seems that SQL queries are not a problem:
Please use bulk_create for inserting multiple objects.
occurences = []
for url_xpath in urls_xpaths:
counter-=1
print counter
url = url_xpath[1]
xpath = url_xpath[0]
occurances.append(Occurence(product=product,url=url,xpath=xpath,active=True if xpath else False,currency=eur_currency))
Occurence.objects.bulk_create(occurences)
I've got a computation, consisting of 3 "map" steps, and the last step depends on results of the first two. I am performing this task using dask.distributed running on several PCs.
Dependency graph looks like following.
map(func1, list1) -> res_list1-\
| -> create_list_3(res_list1, res_list2)-> list3 -> map(func3, list3)
map(func2, list2) -> res_list2-/
If we imagine that these computations are independent, then it is straightforward to call map function 3 times.
from distributed import Executor, progress
def process(jobid):
e = Executor('{address}:{port}'.format(address=config('SERVER_ADDR'),
port=config('SERVER_PORT')))
futures = []
futures.append(e.map(func1, list1))
futures.append(e.map(func2, list2))
futures.append(e.map(func3, list3))
return futures
if __name__ == '__main__':
jobid = 'blah-blah-blah'
r = process(jobid)
progress(r)
However, list3 is constructed from results of func1 and func2, and its creation is not easily mappable (list1, list2, res_list1 and res_list2 are stored in the Postgresql database and creation of list3 is a JOIN query, taking some time).
I've tried to add call to submit to the list of futures, however, that did not work as expected:
def process(jobid):
e = Executor('{address}:{port}'.format(address=config('SERVER_ADDR'),
port=config('SERVER_PORT')))
futures = []
futures.append(e.map(func1, list1))
futures.append(e.map(func2, list2))
futures.append(e.submit(create_list_3))
futures.append(e.map(func3, list3))
return futures
In this case one dask-worker has received the task to execute create_list_3, but others have simultaneously received tasks to call func3, that have erred, because list3 did not exist.
Obvious thing - I'm missing synchronization. Workers must stop and wait till creation of list3 is finished.
Documentation to dask describes custom task graphs, that can provide a synchronization.
However, examples in the documentation do not include map functions, only simple calculations, like calls add and inc.
Is it possible to use map and custom dask graph in my case, or should I implement sync with some other means, that are not included in dask?
If you want to link dependencies between tasks then you should pass the outputs from previous tasks into the inputs of another.
futures1 = e.map(func1, list1)
futures2 = e.map(func2, list2)
futures3 = e.map(func3, futures1, futures2)
For any invocation of func3 Dask will handle waiting until the inputs are ready and will send the appropriate results to that function from wherever they get computed.
However it looks like you want to handle data transfer and synchronization through some other custom means. If this is so then perhaps it would be useful to pass along some token to the call to func3.
futures1 = e.map(func1, list1)
futures2 = e.map(func2, list2)
def do_nothing(*args):
return None
token1 = e.submit(do_nothing, futures1)
token2 = e.submit(do_nothing, futures2)
list3 = e.submit(create_list_3)
def func3(arg, tokens=None):
...
futures3 = e.map(func3, list3, tokens=[token1, token2])
This is a bit of a hack, but would force all func3 functions to wait until they were able to get the token results from the previous map calls.
However I recommend trying to do something like the first option. This will allow dask to be much smarter about when it runs and can release resources. Barriers like token1/2 result in sub-optimal scheduling.
I am newbe here and also in Python. I have a question about dictionaries and multiprocessing. I want to run this part of the code on the second core of my Raspberry Pi (on first is running GUI application). So, I created a dictionary (keys(20) + array with the length of 256 for each of this key - script below is just short example). I initialized this dictionary in a separate script and put all values inside this dictionary on zero. Script table1.py (this dictionary should be available from both cores)
diction = {}
diction['FrameID']= [0]*10
diction['Flag']= ["Z"]*10
In the second script (should run on the second core), I read the values that I get from the serial port and put/set them in this dictionary (parsing + conversion) according to the appropriate place. Since I get much information through a serial port, the information is changing all the time. Script Readmode.py
from multiprocessing import Process
import time
import serial
import table1
def replace_all(text, dic):
for i, j in dic.iteritems():
text = text.replace(i, j)
return text
def hexTobin(hex_num):
scale = 16 ## equals to hexadecimal
num_of_bits = len(hex_num)*4
bin_num = bin(int(hex_num, scale))[2:].zfill(num_of_bits) #hex to binary
return bin_num
def readSerial():
port = "/dev/ttyAMA0"
baudrate = 115200
ser = serial.Serial(port, baudrate, bytesize=8, parity=serial.PARITY_NONE, stopbits=1, xonxoff=False, rtscts=False)
line = []
for x in xrange(1):
ser.write(":AAFF:AA\r\n:F1\r\n") # new
while True:
for c in ser.read():
line.append(c)
a=''.join(line[-2:])
if a == '\r\n':
b=''.join(line)
print("what I get:" + b)
c=b[b.rfind(":"):len(b)] #string between last ":" start delimiter and stop delimiter
reps = {':':'', '\r\n':''} #throw away start and stop delimiter
txt = replace_all(c, reps)
print("hex num: " + txt)
bina_num=hexTobin(txt) # convert hex to bin
print("bin num: " + bina_num)
ssbit = bina_num[:3] # first three bits
print("select first three bits: " + ssbit)
abit=int(ssbit,2) # binary to integer
if abit == 5:
table1.diction['FrameID'][0]=abit
if abit == 7:
table1.diction['FrameID'][5]=abit
print("int num: ",abit)
print(table1.diction)
line = []
break
ser.close()
p1=Process(target=readSerial)
p1.start()
During that time I want to read information in this dictionary and use them in another process. But When I try to read that values there are all zero.
My question is how to create a dictionary that will be available for both process and can be updated based on data get from serial port?
Thank you for your answer in advance.
In Python, when you start two different scripts, even if they import some common modules, they share nothing (except the code). If two scripts both import table1, then they both have their own instance of the table1 module and therefore their own instance of the table1.diction variable.
If you think about it, it has to be like that. Otherwise all Python scripts would share the same sys.stdout, for example. So, more or less nothing is shared between two different scripts executing at the same time.
The multiprocessing module lets you create more than one process from the same single script. So you'll need to merge your GUI and your reading function into the same script. But then you can do what you want. Your code will look something like this:
import multiprocessing
# create shared dictionary sd
p1=Process(target=readSerial, args = (sd,)
p1.start()
# create the GUI behaviour you want
But hang on a minute. This won't work either. Because when the Process is created it starts a new instance of the Python interpreter and creates all its own instances again, just like starting a new script. So even now, by default, nothing in readSerial will be shared with the GUI process.
But fortunately the multiprocessing module provides some explicit techniques to share data between the processes. There is more than one way to do that, but here is one that works:
import multiprocessing
import time
def readSerial(d):
d["test"] = []
for i in range(100):
l = d["test"]
l.append(i)
d["test"] = l
time.sleep(1)
def doGUI(d):
while True:
print(d)
time.sleep(.5)
if __name__ == '__main__':
with multiprocessing.Manager() as manager:
sd = manager.dict()
p = multiprocessing.Process(target=readSerial, args=(sd,))
p.start()
doGUI(sd)
You'll notice that the append to the list in the readSerial function is a bit odd. That is because the dictionary object we are working with here is not a normal dictionary. It's actually a dictionary proxy that conceals a pipe used to send the data between the two processes. When I append to the list inside the dictionary the proxy needs to be notified (by assigning to the dictionary) so it knows to send the updated data across the pipe. That is why we need the assignment (rather than simply directly mutating the list inside the dictionary). For more on this look at the docs.