Python3 C++ Module - Exception - GIL not held - c++

Question

I have created a minimal reproduction of an issue I am having, trying to create a C++ based Python3 module. The build environment is CMake, Visual Studio Pro 2019, WinSDK 10.0.18362, Python 3.9.4. When executing:
python3 -c "import pymod"
I will get an exception in release mode with a NULL access. In debug Python, I get additional information.
My CMakeLists.txt
cmake_minimum_required(VERSION 3.15)
project(pymod_minimal_repo_example)
find_package(Python3 COMPONENTS Interpreter Development)
add_library(pymod SHARED)
target_sources(pymod PRIVATE pymodmodule.cpp)
set_target_properties(pymod PROPERTIES SUFFIX ".pyd")
target_compile_options(pymod PRIVATE /Zi)
target_link_options(pymod PRIVATE /DEBUG:FULL)
target_link_libraries(pymod PRIVATE ${Python3_LIBRARIES})
target_include_directories(pymod PRIVATE ${Python3_INCLUDE_DIRS})
Following this python reference: https://docs.python.org/3.9/extending/extending.html#a-simple-example I created the following:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
static struct PyModuleDef pymodmodule = {
PyModuleDef_HEAD_INIT, // m_base
"pymod", // m_name
NULL, // m_doc
-1, // m_size - submod not support must be static struct
NULL, // m_methods - no functions present
NULL, // m_slots - must be NULL
NULL, // m_traverse - not needed
NULL, // m_clear - not needed
NULL // m_free - not needed
};
PyMODINIT_FUNC PyInit_pymod(void) {
return PyModule_Create(&pymodmodule);
}
It is a functionally void module which does nothing, but should still run. Note that having m_methods defined, produces the same failure.
When the failure occurs, the following is output to the console:
E:\projects\pymod\build\Debug>python -c "import pymod"
Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown
WinDbg jit debugger then catches the issue. A partial call stack shows my PyInit_pymod is called and when creating the python module, it cascades to a failure:
0:000> k
# Child-SP RetAddr Call Site
00 000000dc`abfed4a8 00007fff`71d51385 KERNELBASE!wil::details::DebugBreak+0x2
01 000000dc`abfed4b0 00007fff`71d511a8 python39_d!fatal_error_exit+0x15 [D:\a\1\s\Python\pylifecycle.c # 2201]
02 000000dc`abfed4e0 00007fff`71d4ea98 python39_d!fatal_error+0x1b8 [D:\a\1\s\Python\pylifecycle.c # 2286]
03 000000dc`abfed540 00007fff`71cbe730 python39_d!_Py_FatalErrorFunc+0x38 [D:\a\1\s\Python\pylifecycle.c # 2302]
04 000000dc`abfed580 00007fff`71aa8044 python39_d!_Py_FatalError_TstateNULL+0x10 [D:\a\1\s\Python\ceval.c # 251]
05 000000dc`abfed5b0 00007fff`71aa6cf2 python39_d!_PyInterpreterState_GET+0x34 [D:\a\1\s\Include\internal\pycore_pystate.h # 105]
06 000000dc`abfed5f0 00007fff`d0be13e7 python39_d!PyModule_Create2+0x12 [D:\a\1\s\Objects\moduleobject.c # 168]
>>>> 07 000000dc`abfed620 00007fff`751165dc pymod!PyInit_pymod+0x27 [E:\projects\pymod\pymodmodule.cpp # 19]
08 000000dc`abfed660 00007fff`751167ee python39!_PyImport_LoadDynamicModuleWithSpec+0x104 [C:\A\34\s\Python\importdl.c # 165]
09 000000dc`abfed6d0 00007fff`75116749 python39!_imp_create_dynamic_impl+0x86 [C:\A\34\s\Python\import.c # 2299]
0a 000000dc`abfed700 00007fff`750cb94b python39!_imp_create_dynamic+0x39 [C:\A\34\s\Python\clinic\import.c.h # 330]
0b 000000dc`abfed730 00007fff`750ad500 python39!cfunction_vectorcall_FASTCALL+0x9b [C:\A\34\s\Objects\methodobject.c # 426]
0c 000000dc`abfed7a0 00007fff`750ad2ef python39!PyVectorcall_Call+0x5c [C:\A\34\s\Objects\call.c # 248]
0d 000000dc`abfed800 00007fff`750ad418 python39!_PyObject_Call+0x4f [C:\A\34\s\Objects\call.c # 287]
0e (Inline Function) --------`-------- python39!PyObject_Call+0xc [C:\A\34\s\Objects\call.c # 293]
0f 000000dc`abfed830 00007fff`7508c65f python39!do_call_core+0xb8 [C:\A\34\s\Python\ceval.c # 5092]
10 000000dc`abfed880 00007fff`75083963 python39!_PyEval_EvalFrameDefault+0x5d6f [C:\A\34\s\Python\ceval.c # 3581]
11 (Inline Function) --------`-------- python39!_PyEval_EvalFrame+0x13 [C:\A\34\s\Include\internal\pycore_ceval.h # 40]
12 000000dc`abfedbb0 00007fff`750855a7 python39!_PyEval_EvalCode+0x2b3 [C:\A\34\s\Python\ceval.c # 4327]
13 000000dc`abfedc80 00007fff`7508823d python39!_PyFunction_Vectorcall+0x257 [C:\A\34\s\Objects\call.c # 396]
14 000000dc`abfedd80 00007fff`7508812f python39!_PyEval_EvalFrameDefault+0x194d [C:\A\34\s\Python\ceval.c # 3487]
15 000000dc`abfee0b0 00007fff`750888e5 python39!_PyEval_EvalFrameDefault+0x183f [C:\A\34\s\Python\ceval.c # 3504]
16 000000dc`abfee3e0 00007fff`750888e5 python39!_PyEval_EvalFrameDefault+0x1ff5 [C:\A\34\s\Python\ceval.c # 3518]
17 000000dc`abfee710 00007fff`750888e5 python39!_PyEval_EvalFrameDefault+0x1ff5 [C:\A\34\s\Python\ceval.c # 3518]
18 000000dc`abfeea40 00007fff`750854c4 python39!_PyEval_EvalFrameDefault+0x1ff5 [C:\A\34\s\Python\ceval.c # 3518]
I cannot find any information on the console message, nor the int3. I have tried full purge and reinstall/update of Python.
Can anyone offer some direction to help, or know the cause?
Edit: Modified that PyModule_pymod function to display Pre and Post msg to console. Debug build exceptions, Release appears not to:
PyMODINIT_FUNC PyInit_pymod(void) {
printf("Pre-PyModule_Create\n");
PyObject *obj = PyModule_Create(&pymodmodule);
printf("Post-PyModule_Create\n");
return obj;
}
Release:
E:\projects\pymod\build\Release>python -c "import pymod"
Pre-PyModule_Create
Post-PyModule_Create
Debug:
E:\projects\pymod\build\Debug>python -c "import pymod"
Pre-PyModule_Create
Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown

Answer 1

I ran into the same issue as you, and while debugging TLS structures, I noticed that Visual Studio had loaded python310.dll instead of python310_d.dll.
To make a long story short, when you define the _DEBUG macro, Python.h will use python310_d.lib, so you need to run the debug version of Python:
python_d -c "import pymod"
I was confused further by the fact that I did not have this issue while using pybind11. As it turned out, pybind11 undefines the _DEBUG macro while including Python.h, so it is always using python310.lib instead of python310_d.lib, even for debugging builds.
As I do not see an option to use the python_d interpreter in Visual Studio, I ended up using the release library for debugging builds using the following code:
#if defined(_MSC_VER) && defined(_DEBUG)
#undef _DEBUG
#include <Python.h>
#define _DEBUG 1
#else
#include <Python.h>
#endif
As a sidenote, when you link against python310_d.dll, you will have to name your debug module pymod_d.pyd instead of pymod.pyd.