I am trying to use multiprocessing in python for running 10-20 input files which are in xml format together using parallel process which can be achieved by multiprocessing. But when i try to run the code
service = Process(name='my_service', target=function_name, args=(run_xml, result_dest,))
service.start()
I get pickling error when i tried to run the code and can any one suggest me the possible solution?
f(self, obj) # Call unbound method with explicit self
File "D:\Program Files\Aon\PathWise 2.0\apps\py_vendors\Python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle : it's not found as __main__.new
Related
I have multiple python scripts, each with print statements and prompts for input. I run these scripts from a single python script as below.
os.system('python script1.py ' + sys.argv[1])
os.system('python script2.py ' + sys.argv[1]).....
The run is completed successfully, however, when I run all the scripts from a single file, I no longer see any print statements or prompts for input on the run console. Have researched and attempted many different ways to get this to work w/o success. Help would be much appreciated. Thanks.
If I understand correctly you want to run multiple python scripts synchronously, i.e. one after another.
You could use a bash script instead of python, but to answer your question of starting them from python...
Checkout out the subprocess module: https://docs.python.org/3.4/library/subprocess.html
In particular the call method, it accepts a stdin and stdout which you can pass sys.stdin and sys.stdout to.
import sys
import subprocess
subprocess.call(['python', 'script1.py', sys.argv[1]], stdin=sys.stdin, stdout=sys.stdout)
subprocess.call(['python', 'script2.py', sys.argv[1]], stdin=sys.stdin, stdout=sys.stdout)
^
This will work in python 2.7 and 3, another way of doing this is by importing your file (module) and calling the methods in it. The difference here is that you're no longer running the code in a separate process.
subroutine.py
def run_subroutine():
name = input('Enter a name: ')
print(name)
master.py
import subroutine
subroutine.run_subroutine()
Here's the context:
I am using python 2.7.5. And I would like to run UNIX commands as well as maven commands in a python script.
I was successful to do so, using os.system("cmd"), but I need to work on the result of the given command. After reading the doc and some threads in here, I decided to use the subprocess module to redirect the output to the stdout using PIPE. Unexpectedly, I am getting an OSError as shown in the attached image. Your help will be much appreciated.
In addition to the given sample in the attached image, I have tried:
p = os.popen("java -version")
result = subprocess.check_output(p, shell=True)
subprocess.call("ls /usr", shell=True)
p.s. Using shell=True is strongly discouraged (doc), since it can be dangerous when coupled with unsanitized input.
Also, I took a look at the given script in the error message /usr/lib64/python2.7/subprocess.py, line 711 adn 1327 but didn't learn more than what is mentionned in the error message: raise child_exception
Subprocess Terminal Output
You aren't using subprocess.check_output correctly. You're trying to pass a pipe file object (the return value of os.popen) to check_output but it's expecting a command argument or argument vector.
Also, the subprocess.call function won't capture the executed command's output, so you would only use that if you want the output of ls /usr (or whatever) to be seen by the user running the script interactively. (Which is pretty much the same as os.system.)
Try this instead (showing with and without the shell):
import subprocess
out1a = subprocess.check_output(['java', '-version'], stderr=subprocess.STDOUT)
print(out1a)
out1b = subprocess.check_output('java -version', stderr=subprocess.STDOUT, shell=True)
print(out1b)
out2a = subprocess.check_output(['ls', '/usr'])
print(out2a)
out2b = subprocess.check_output('ls /usr', shell=True)
print(out2b)
# Cannot capture output this way, but it will be visible to user
subprocess.call('ls /usr', shell=True)
Note that in the case of the java -version command, the version info gets printed to the command's standard error output so you must redirect that in order to capture it as the returned value of check_output (hence the stderr=subprocess.STDOUT).
I am trying to simulate multiple Modelica FMUs in parallel using python/pyfmi and multiprocessing. However I am not able to return any pyfmi FMI objects from the subprocesses once the FMUs are initialized. It seems that pyfmi FMI objects (e.g. pyfmi.fmi.FMUModelCS2 or pyfmi.fmi.FMUState2) are not pickable. I also tried dill to pickle, which doesn't work for me eather. With dill the objects are picklable though, meaning no error, but somehow corrupted if I try to reload them afterwards. Does anyone have an idea of how to solve this issue? Thanks!
The problem is that pyfmi.fmiFMUModelCS2 is a Cython class dependent on external libraries which makes it unpickable. So it is not possible unfortunately.
If you want to use multiprocessing the only way forward that I see is that you first create the processes and then load the FMUs into the separate processes. In this way you do not need to pickle the classes.
I faced a similar problem when I created EstimationPy. I ended up creating a wrapper for running parallel simulation of the same FMU using multiple processes.
I suggest you to look at the implementation here
https://github.com/lbl-srg/EstimationPy/blob/master/estimationpy/fmu_utils/fmu_pool.py
And to the example http://lbl-srg.github.io/EstimationPy/modules/examples/first_order.html#run-multiple-simulations
The pathos module allows multiprocessing with a similar interface as the multiprocessing but relies on dill instead of pickle for serialisation.
The Pool method works for parallel execution of model.simulate, provided that results are handled in memory:
n_core = 2
n_simulation = 10
# ====
import pyfmi
model = pyfmi.load_fmu(path_fmu)
def worker(*args):
model.reset()
print "================> %d" % args[0]
return model.simulate(options=dict(result_handling="memory"))["y"]
from pathos.multiprocessing import Pool
pool = Pool(n_core)
out = pool.map(worker, range(n_simulation))
pool.close()
pool.join()
Note in the above snippet that it is necessary to handle results in memory : options=dict(result_handling="memory").
The default is to use temporary files which works for when the amount of simulations is small.
However, the longer the queue, the higher the chance to get something like
Exception in thread Thread-27:
Traceback (most recent call last):
File "/home/USER/anaconda2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/USER/anaconda2/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/USER/anaconda2/lib/python2.7/site-packages/multiprocess/pool.py", line 389, in _handle_results
task = get()
File "/home/USER/anaconda2/lib/python2.7/site-packages/dill/dill.py", line 260, in loads
return load(file)
File "/home/USER/anaconda2/lib/python2.7/site-packages/dill/dill.py", line 250, in load
obj = pik.load()
File "/home/USER/anaconda2/lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/home/USER/anaconda2/lib/python2.7/pickle.py", line 1139, in load_reduce
value = func(*args)
TypeError: __init__() takes exactly 2 arguments (1 given)
which I fail to grasp.
on my Raspberry Pi, I encounter a strange behaviour regarding the use of the PiCamera module.
The following code runs smoothly when either started from IDLE (F5) or from the command prompt ($python test.py)
import picamera
if __name__ == "__main__":
camera=picamera.PiCamera()
camera.close()
But when I put the camera object into a class the code will run only when started from IDLE (F5):
import picamera
class VF:
def __init__(self):
self.camera = picamera.PiCamera()
def __del__(self):
self.camera.close()
if __name__ == "__main__":
myvf = VF()
When I start the above code from the command prompt, I get the following error message:
mmal: mmal_vc_component_enable: failed to enable component: ENOSPC
Traceback (most recent call last): File "test.py", line 14, in
myvf = VF()
File "test.py", line 6, in init
self.camera = picamera.PiCamera()
File "/usr/lib/python2.7/dist-packages/picamera/camera.py", line
379, in init
camera_num, self.STEREO_MODES[stereo_mode], stereo_decimate)
File "/usr/lib/python2.7/dist-packages/picamera/camera.py", line
505, in _init_camera
prefix="Camera component couldn't be enabled")
File "/usr/lib/python2.7/dist-packages/picamera/exc.py", line 133,
in mmal_check
raise PiCameraMMALError(status, prefix)
picamera.exc.PiCameraMMALError: Camera component couldn't be enabled:
Out of resources (other than memory)
The camera module is working correct, I just stripped the code down to the least possible size. Does anybody know this problem, or a similar problem, and can probably provide a solution? The Python Version is 2.7 and the Raspberry Rasbiab-System is completely up to date.
Thanks in advance.
I struggled with this one for hours, and kept getting the "out of resources" error. I finally figured out that in my take-the-picture function, I needed to make sure I did it like this:
camera = PiCamera()
(...camera settings here...)
camera.capture(myfileName)
camera.close()
If I didn't do the close(), I'd get that error every time.
So make sure that camera.close() is getting called right after the 'snap'. It solved the problem for me.
Found out, that the camera-module is not properly shut down when the destructor is not explicitly called (had LED turned off, so didn't see this).
IDLE handles a running camera by somehow resetting it before the script starts, but not the python interpreter.
So everything is ok now when the destructor is called before the script ends.
My goal it to process a large CSV file using Celery that is uploaded through a Django form. When the file's size is less than SETTINGS.FILE_UPLOAD_MAX_MEMORY_SIZE, I can pass the form's cleaned_data variable to a celery task and read the file with:
#task
def taskFunction(cleaned_data):
for line in csv.reader(cleaned_data['upload_file']):
MyModel.objects.create(field=line[0])
However, when the file's size is greater than the above setting, I get the following error:
expected string or Unicode object, NoneType found
Where the stack trace shows the error occurring during pickle:
return dumper(obj, protocol=pickle_protocol)
It appears that when the uploaded file is read from a temporary file, pickle fails.
The simple solution to this problem is to increase the FILE_UPLOAD_MAX_MEMORY_SIZE. However, I am curious if there is a better way to manage this issue?
Save it to a temp file and pass the file name to celery instead. Delete after processing.