Valgrind error and memory leaks with Python/C API - c++

Question

I'm actually developping a game in C++ and trying to do the AI with a script langage. To do so, i choosed Python2 with Python/C api.
My AI is actually working but there is a big problem : when I run valgrind on my program, there is a lot of error and memory leaks. So, I would know if this happened because of my code or by the API ?
Here is a summary of my class AI :
IA::IA()
{
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyRun_SimpleString("import sys");
pName = PyBytes_FromString((char*)"Test");
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pFunc = PyDict_GetItemString(pDict, "push_f");
}
IA::~IA()
{
Py_DECREF(pValue);
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
void IA::LaunchIA(float x, float y, float z)
{
PyObject *toSend;
toSend = Py_BuildValue("(OOO)", TlistMob, TlistPlayer, pDPosIA);
pResult = PyObject_CallObject(pFunc, toSend);
PyErr_Print();
printf("return = %f\n", (float)PyInt_AsLong(pResult));
}
My (very) simple Python code :
def push_f(MobList, PlayerList, pos):
return 0
And the valgrind error (x1000) :
==11602== Memcheck, a memory error detector
==11602== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==11602== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==11602== Command: ./a.out
==11602==
==11602== Invalid read of size 4
==11602== at 0x4FCE173: PyObject_Free (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F02FC2: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4FBDE9A: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F85BAD: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F872FF: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F88559: PyImport_ImportModuleLevel (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EFF697: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F4B1E2: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x5021446: PyEval_CallObjectWithKeywords (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EF45C5: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x502201B: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EF0B88: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== Address 0x693c020 is 2,560 bytes inside a block of size 2,731 free'd
==11602== at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11602== by 0x4F81D28: PyMarshal_ReadLastObjectFromFile (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F85A22: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F872FF: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F88559: PyImport_ImportModuleLevel (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EFF697: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F4B1E2: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x5021446: PyEval_CallObjectWithKeywords (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EF45C5: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x502201B: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EF0B88: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F851B3: PyImport_ExecCodeModuleEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== Block was alloc'd at
==11602== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11602== by 0x4F81CDF: PyMarshal_ReadLastObjectFromFile (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F85A22: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F872FF: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F88559: PyImport_ImportModuleLevel (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EFF697: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F4B1E2: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x5021446: PyEval_CallObjectWithKeywords (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EF45C5: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x502201B: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4EF0B88: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== by 0x4F851B3: PyImport_ExecCodeModuleEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==
==11602==
==11602== HEAP SUMMARY:
==11602== in use at exit: 491,741 bytes in 204 blocks
==11602== total heap usage: 3,301 allocs, 3,097 frees, 3,567,424 bytes allocated
==11602==
==11602== LEAK SUMMARY:
==11602== definitely lost: 0 bytes in 0 blocks
==11602== indirectly lost: 0 bytes in 0 blocks
==11602== possibly lost: 1,072 bytes in 2 blocks
==11602== still reachable: 490,669 bytes in 202 blocks
==11602== suppressed: 0 bytes in 0 blocks
==11602== Rerun with --leak-check=full to see details of leaked memory
==11602==
==11602== For counts of detected and suppressed errors, rerun with: -v
==11602== Use --track-origins=yes to see where uninitialised values come from
==11602== ERROR SUMMARY: 497 errors from 25 contexts (suppressed: 0 from 0)
In my main, you need to know that I'm creating only one IA object.
Am I doing something wrong ? Or is it just the API ?
(This is not a duplicate because I run valgrind on my C++ executable and not Python, my c++ is running the script)
Thanks in advance !!!

Answer 1

I initially suggested this as a duplicate. I don't think that's the case any more, but it still gives useful advice on removing false positives from the Valgrind output for Python programs.
Beyond that there are a number of specific issues with your program:
No error checking - the Python C API typically uses a NULL return value to indicate an error. There are clearly various ways of writing error checking code, but I'd be tempted to go for something like
IA::IA() :
pModule(NULL), pDict(NULL), pFunc(NULL), pName(NULL) // initially null initialize everything
{
setenv("PYTHONPATH",".",1);
Py_Initialize();
// I don't think you actually use this line, so maybe remove it
if (PyRun_SimpleString("import sys") == -1) goto error;
pName = PyBytes_FromString((char*)"Test");
if (!pName) goto error;
pModule = PyImport_Import(pName);
if (!pModule) goto error;
pDict = PyModule_GetDict(pModule);
if (!pDict) goto error;
pFunc = PyDict_GetItemString(pDict, "push_f");
if (!pFunc) goto error;
return; // completed OK
error:
Py_XDECREF(pName); // XDECREF is OK with NULL...
Py_XDECREF(pModule);
Py_XDECREF(pDict);
Py_XDECREF(pFunc);
PyErr_Print();
throw std::runtime_error(""); // ??? - possibly get and use the error string
// see https://stackoverflow.com/a/1418703/4657412
}
I'm aware people are suspicious of goto but in this case it's a reasonably clean way of jumping to an error handling block. You can structure it differently if you like.
The destructor doesn't decref pFunc, which looks like a memory leak.
IA::LaunchIA similarly is lacking working error checking.
IA::LaunchIA never decrefs toSend, pResult, TlistMob, TlistPlayer, pDPosIA. Some of these are related to the incompleteness of the code you show, but if they aren't decrefed then you're leaking memory.
You call PyErr_Print() without checking for an error. The documentation says:
Call this function only when the error indicator is set. (Otherwise it will cause a fatal error!)
I suspect that this might be your biggest problem.
Those are all the issues I can see. Without a minimal complete example it's impossible to actually check what you're seeing. Given you're using C++ you might be well advised to use/make a decent, object oriented wrapper for PyObject* to avoid having to worry about reference counting yourself - Boost Python has one.