Is it possible to create constraints in parallel? - pyomo

Question

I have a relatively simple Pyomo model, where I wish to create the constraints in parallel. I have created the following functions:
def constraintA():
constraintA = pyo.ConstraintList()
for y, h in model.YH:
constraintA.add(model.varA[y, h] <= data[y, h])
return constraintA
def constraintB():
constraintB = pyo.ConstraintList()
for y, h in model.YH:
constraintB.add(model.varB[y, h] >= other_data[y, h])
return constraintB
def constraintC():
constraintC = pyo.ConstraintList()
for y, h in model.YH:
constraintC.add(model.varC[y, h] <= data[y,h] * 0.5)
return constraintC
Note that the model, data and other_data are considered 'global' and therefore can be used within these constraint functions. Now, what I would like to do is create attributes in the pyomo model with the respective constraints, but then in parallel.
So I start with:
constraints = [constraintA, constraintB, constraintC]
with ProcessPoolExecutor() as pool:
for work_item in enumerate(constraints):
fs = [pool.submit(c) for c in constraints]
However, I get "finished raised AttributeError". Does this mean that despite Python only has to read and copy it, it is locking the object and therefore cannot access the attribute and therefore is unable to create the constraints?
In order to resolve this issue, I was wondering whether I need to explicitly copy the model and the data and other_data to the constraint functions themselves. Or is there another way to achieve this, without "passing variables around"?

Answer 1

For all intents and purposes, Pyomo model construction is a serial exercise.
This is primarily due to the CPython GIL (Global Interpreter Lock), which effectively serializes everything in Python (multithreading in CPython is actually serial, and relies on things like I/O waits to provide "parallelism"). In theory, you can construct model fragments in distributed processes (either through forked processes or something like mpi4py). The problem is that the process of serializing the model fragments and collecting them back into a single process is actually more costly than just generating the model serially in the single process.
There are some Pyomo-based ideas that may enable parallel model construction in the future: one is possibly supporting distributed model construction as well as LP / NL file creation, then merging the LP / NL file fragments instead of bringing back the original modeling objects, and the other would rely on a compiled C extension that would accept a Constraint "template" and construct objects on the C side (where we can avoid the GIL). Both of these are very experimental ideas at the moment.
There is also always the possibility that CPython will remove the GIL (in which case Pyomo will almost certainly immediately add multithreading to constraint construction). There have been many efforts over the years to remove the GIL from CPython, and maybe one will finally succeed and be adopted by CPython.
The last thing to point to is #AirSquid's comment that model construction is usually not the bottleneck. If it is, then some time looking at how you are defining your model / constraints would be warranted (Pyomo is very forgiving in what it will allow you to do, but not all approaches are equally performant).