How to wrap multithreaded C++ library using Python C/API? - c++

Question

This is a somewhat long question, but I hope I can express it clearly.
I am trying to wrap a C++ library using Python/C API. The main library, say, mylib, has its own object system (it is something like an interpreter for another language ) and uniquely identifies each object in its environment by an Id. It creates multiple threads in its init() function and does different things on different threads (say creating objects on one thread and interpreting commands in another thread).
Now I am trying to wrap it in two levels:
I created a Dummy class with the Id of an object in mylib. The Dummy constructor actually calls a function in mylib to create a new object and store its Id. Other methods in Dummy class similarly call equivalent functions in mylib. This does not use Python/C API.
I created mylibmodule.cpp, which uses the Python/C API to provide the functions that will be called from the Python interpreter.
I call the init() function of mylib in PyMODINIT_FUNC init_mylib().
I code functions like :
static PyObject * py_new_Dummy(PyObject* self, PyObject *args){
// ... process arguments
return reinterpret_cast<PyObject*>(new Dummy);
}
Note that the Dummy constructor does call functions in mylib that are executed on threads created by using pthreads.
I compile this into _mylib.so and I have a mylib.py:
import _mylib
class MyClass(obj):
def __init__(self, *args)
self.__ptr = _mylib.py_new_Dummy()
Now to the actual problem: I can import mylib in the Python interpreter, but as soon as I try:
a = MyClass(some_args)
I get a segmentation fault. A gdb backtrace shows
Program received signal SIGSEGV, Segmentation fault.
__pthread_mutex_lock (mutex=0x0) at pthread_mutex_lock.c:50
Even funnier is that if I disable spawning multiple threads in the mylib code (still linked with pthreads), I can create MyClass instances, but I get a segmentation violation at exit from the Python interpreter.
The "Thin Ice" section in the Python documentation (http://docs.python.org/extending/) did not enlighten me. I am wondering if I should use PyGILState_Ensure and PyGILState_Release around all Python C/API calls in mylibmodule.cpp. Or should it be Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS?
Can anybody help? Is there any definitive documentation on how exactly Python plays with pthreads?

Answer 1

From your description it doesn't really sound like a threading issue at all: you claim you define the Dummy class without using the Python API, but that would mean Dummy instances are not PyObjects, so the reinterpret_cast will do the wrong thing. You can't create PyObjects by just instantiating a C++ class; you need to play along with Python's object system and create a proper PyType struct and a PyObject struct and properly initialize both. You also need to make sure your refcounts are correct.
Once you have that sorted, the main thing to remember about threads is that any call that touches Python objects or that uses any of the Python API (except the functions to grab the GIL) must have the GIL acquired. If any of the threads in your C++ library try to call back to Python code or touch Python objects, the access needs to be wrapped in PyGILState_Ensure/PyGILState_Release.

Answer 2

Thank you Thomas for pointing out the red herring. The problem was in the initialization of the threads in the C++ side.
And yes, it did not need any GIL manipulation as the none of the additional C++ threads were accessing Python C/API.