stack smashing calling python function importing tensorflow c++ - c++

I am new to tensorflow as well as including python code in c++, therefore I would apprechiate any tips/comments on the following weird behaviour:
I have a c++ class pythoninterface with headerfile pythoninterface.h:
#include <string>
#include <iostream>
class pythoninterface{
private:
const char* file;
const char* funct;
const char* filepath;
public:
pythoninterface();
~pythoninterface();
void CallFunction();
};
The sourcefile pythoninterface.cpp:
#include <Python.h>
#include <string>
#include <sstream>
#include <vector>
#include "pythoninterface.h"
pythoninterface::pythoninterface(){
file = "TensorflowIncludePy";
funct = "myTestFunction";
filepath = "/path/To/TensorflowIncludePy.py";
}
void pythoninterface::CallFunction(){
PyObject *pName, *pModule, *pDict, *pFunc, *pValue, *presult;
// Initialize the Python Interpreter
Py_Initialize();
//Set in path where to find the custom python module other than the path where Python's system modules/packages are found.
std::stringstream changepath;
changepath << "import sys; sys.path.insert(0, '" << filepath << "')";
const std::string tmp = changepath.str();
filepath = tmp.c_str();
PyRun_SimpleString (this->filepath);
// Build the name object
pName = PyString_FromString(this->file);
// Load the module object
pModule = PyImport_Import(pName);
if(pModule != NULL) {
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, this->funct);
if (PyCallable_Check(pFunc))
{
pValue=Py_BuildValue("()");
printf("pValue is empty!\n");
PyErr_Print();
presult=PyObject_CallObject(pFunc,pValue);
PyErr_Print();
} else
{
PyErr_Print();
}
printf("Result is %d!\n",PyInt_AsLong(presult));
Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
}
else{
std::cout << "Python retuned null pointer, no file!" << std::endl;
}
// Finish the Python Interpreter
Py_Finalize();
}
And the Python File from which the function should be included (TensorflowIncludePy.py):
def myTestFunction():
print 'I am a function without an input!'
gettingStartedTF()
return 42
def gettingStartedTF():
import tensorflow as tf #At this point the error occurs!
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
return 42
Finally in my main function I only create a pythoninterface object p and call the function p.CallFunction(). The communication between the c++ and python code works all right, but when (at runtime) the line import tensorflow as tf is reached, I get a *** stack smashing detected *** error message and the program finishes. Can anyone guess what the problem might be or had a similar issue before?
I know there is a c++ tensorflow API, but I feel more comfortable with using tensorflow in python so I thought this might be the perfect solution for me (apparently it is not...:P)

Related

Python Coverage for C++ PyImport

Situation:
I'm attempting to get coverage reports on all python code in my current project. I've utilized Coverage.py with great success for the most part. Currently I'm using it like this taking advantage of the sitecustomize.py process. For everything that's being started from the command line, and it works amazing.
Issue:
I can't get python modules run from C++ via PyImport_Import() type statements to actually trace and output coverage data.
Example:
[test.cpp]
#include <stdio.h>
#include <iostream>
#include <Python.h>
int main()
{
Py_Initialize();
PyObject* sysPath = PySys_GetObject("path");
PyList_Append(sysPath, PyString_FromString("."));
// Load the module
PyObject *pName = PyString_FromString("test_mod");
PyObject *pModule = PyImport_Import(pName);
if (pModule != NULL) {
std::cout << "Python module found\n";
// Load all module level attributes as a dictionary
PyObject *pDict = PyModule_GetDict(pModule);
PyObject *pFunc = PyObject_GetAttrString(pModule, "getInteger");
if(pFunc)
{
if(PyCallable_Check(pFunc))
{
PyObject *pValue = PyObject_CallObject(pFunc, NULL);
std::cout << PyLong_AsLong(pValue) << std::endl;
}
else
{
printf("ERROR: function getInteger()\n");
}
}
else
{
printf("ERROR: pFunc is NULL\n");
}
}
else
std::cout << "Python Module not found\n";
return 0;
}
[test_mod.py]
#!/bin/python
def getInteger():
print('Python function getInteger() called')
c = 100*50/30
return c
print('Randomness')
Output:
If I manually run test_mod.py it outputs as expected. However, if I run the compiled test.cpp binary, it doesn't output anything for coverage data. I know sitecustomize.py is still being hit, as I added some debugging to ensure I wasn't going insane. I can also see in the coverage debug log that it does indeed want to trace the module..
[cov.log]
New process: executable: /usr/bin/python
New process: cmd: ???
New process: parent pid: 69073
-- config ----------------------------------------------------
_include: None
_omit: None
attempted_config_files: /tmp/.coveragerc
branch: True
concurrency: thread
multiprocessing
config_files: /tmp/.coveragerc
cover_pylib: False
data_file: /tmp/python_data/.coverage
debug: process
trace
sys
config
callers
dataop
dataio
disable_warnings: -none-
exclude_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER)
extra_css: None
fail_under: 0.0
html_dir: htmlcov
html_title: Coverage report
ignore_errors: False
note: None
New Section 1 Page 2note: None
parallel: True
partial_always_list: while (True|1|False|0):
if (True|1|False|0):
partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
paths: {'source': ['/tmp/python_source', '/opt/test']}
plugin_options: {}
plugins: -none-
precision: 0
report_include: None
report_omit: None
run_include: None
run_omit: None
show_missing: False
skip_covered: False
source: /opt/test/
timid: False
xml_output: coverage.xml
xml_package_depth: 99
-- sys -------------------------------------------------------
version: 4.5.4
coverage: /usr/lib64/python2.7/site-packages/coverage/__init__.pyc
cover_paths: /usr/lib64/python2.7/site-packages/coverage
pylib_paths: /usr/lib64/python2.7
tracer: PyTracer
plugins.file_tracers: -none-
plugins.configurers: -none-
config_files: /tmp/.coveragerc
configs_read: /tmp/.coveragerc
data_path: /tmp/python_data/.coverage
python: 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
platform: Linux-3.10.0-1062.el7.x86_64-x86_64-with-redhat-7.7-Maipo
implementation: CPython
executable: /usr/bin/python
cwd: /opt/test
path: /usr/lib64/python27.zip
/usr/lib64/python2.7
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/lib-tk
/usr/lib64/python2.7/lib-old
/usr/lib64/python2.7/lib-dynload
/usr/lib64/python2.7/site-packages
environment: COVERAGE_DEBUG = process,trace,sys,config,callers,dataop,dataio
COVERAGE_DEBUG_FILE = /tmp/cov.log
COVERAGE_PROCESS_START = /tmp/.coveragerc
command_line: ???
source_match: /opt/test
source_pkgs_match: -none-
include_match: -none-
omit_match: -none-
cover_match: -none-
pylib_match: -none-
-- end -------------------------------------------------------
<module> : /usr/lib64/python2.7/site.py #556
New Section 1 Page 3<module> : /usr/lib64/python2.7/site.py #556
main : /usr/lib64/python2.7/site.py #539
addsitepackages : /usr/lib64/python2.7/site.py #317
addsitedir : /usr/lib64/python2.7/site.py #190
addpackage : /usr/lib64/python2.7/site.py #152
<module> : <string> #1
process_startup : /usr/lib64/python2.7/site-packages/coverage/control.py #1289
start : /usr/lib64/python2.7/site-packages/coverage/control.py #690
_init : /usr/lib64/python2.7/site-packages/coverage/control.py #362
_write_startup_debug : /usr/lib64/python2.7/site-packages/coverage/control.py #382
write_formatted_info : /usr/lib64/python2.7/site-packages/coverage/debug.py #120
Not tracing '/usr/lib64/python2.7/threading.py': falls outside the --source trees
<module> : /usr/lib64/python2.7/site.py #556
main : /usr/lib64/python2.7/site.py #539
addsitepackages : /usr/lib64/python2.7/site.py #317
addsitedir : /usr/lib64/python2.7/site.py #190
addpackage : /usr/lib64/python2.7/site.py #152
<module> : <string> #1
process_startup : /usr/lib64/python2.7/site-packages/coverage/control.py #1289
start : /usr/lib64/python2.7/site-packages/coverage/control.py #701
start : /usr/lib64/python2.7/site-packages/coverage/collector.py #318
settrace : /usr/lib64/python2.7/threading.py #99
_trace : /usr/lib64/python2.7/site-packages/coverage/pytracer.py #111
_should_trace : /usr/lib64/python2.7/site-packages/coverage/control.py #593
[... Not tracing a bunch of common python code ...]
Tracing './test_mod.py'
<module> : ./test_mod.py #3
_trace : /usr/lib64/python2.7/site-packages/coverage/pytracer.py #111
_should_trace : /usr/lib64/python2.7/site-packages/coverage/control.py #593
I reproduced the issue using your code and you only forgot to call Py_Finalize(). As a result, the report is never generated whereas the data were collected.
It works with the following piece of code:
#include <stdio.h>
#include <iostream>
#include <Python.h>
int main()
{
Py_Initialize();
PyEval_InitThreads();
PyObject* sysPath = PySys_GetObject("path");
PyList_Append(sysPath, PyString_FromString("."));
// Load the module
PyObject *pName = PyString_FromString("test_mod");
PyObject *pModule = PyImport_Import(pName);
if (pModule != NULL) {
std::cout << "Python module found\n";
// Load all module level attributes as a dictionary
PyObject *pDict = PyModule_GetDict(pModule);
PyObject *pFunc = PyObject_GetAttrString(pModule, "getInteger");
if(pFunc)
{
if(PyCallable_Check(pFunc))
{
PyObject *pValue = PyObject_CallObject(pFunc, NULL);
std::cout << PyLong_AsLong(pValue) << std::endl;
}
else
{
printf("ERROR: function getInteger()\n");
}
}
else
{
printf("ERROR: pFunc is NULL\n");
}
}
else
std::cout << "Python Module not found\n";
Py_Finalize();
return 0;
PyObject *PySys_GetObject(char *name) returns a borrowed reference. Is not it the case that the reference count should be incremented? What about:
// ...
PyObject* sysPath = PySys_GetObject("path");
Py_INCREF(sysPath);
PyList_Append(sysPath, PyString_FromString("."));
Py_DECREF(sysPath);
// sysPath = NULL;
// ...
I'm only just starting with the Python-C API myself, but my understanding is that importing modules doesn't actually add them to your main module. You need to do that separately. I'm not sure if this will help with your issue, but my approach that's worked (minus the error checking) has been as follows:
// Initialize main module
PyObject* mainModule = PyImport_AddModule("__main__");;
// Initialize module to be added
PyObject* moduleNamePyObject= PyUnicode_DecodeFSDefault("moduleName");
PyImport_Import(moduleNamePyObject);
// Add module to main module
PyObject_SetAttrString(mainModulePtr, "moduleName", modulePyObject);
Normally, when importing a module, Python tries to find the module file next to the importing module (the module that contains the import statement). Python then tries the directories in “sys.path”. The current working directory is usually not considered. In our case, the import is performed via the API, so there is no importing module in whose directory Python could search for “test_mod.py”. The plug-in is also not on “sys.path”. One way of enabling Python to find the plug-in is to add the current working directory to the module search path by doing the equivalent of “sys.path.append(‘.’)” via the API.
Py_Initialize();
PyObject* sysPath = PySys_GetObject((char*)"path");
PyObject* programName = PyString_FromString(<DIRECTORY>.c_str());
PyList_Append(sysPath, programName);
Py_DECREF(programName);
If you are using python3 ,
Change PyString_FromString to PyUnicode_FromString.
Sources :
https://realmike.org/blog/2012/07/08/embedding-python-tutorial-part-1/
Python Embedding: PyImport_Import not from the current directory

Embedded Python does not work pointing to Python35.zip with NumPy - how to fix?

Okay here's the basic example from the Python website for a simple runpy.exe to run Python scripts below. It works fine using Visual Studio 2015 on x64 Windows after referencing the Python includes and linking to python35.lib for basic functions (the docs don't mention pyvenv.cfg must be in the EXE directory). However, calling a script that imports NumPy leads to this error ImportError: No module named 'numpy' Failed to load "eig" only when using embedded python35.zip, so how does one include NumPy in an embedded Python EXE? I.e. I want to also "embed" NumPy (as a .zip, directory, .dll, or .pyd etc.). I've tried adding the NumPy includes and also linking to npymath.lib but I get the same import error. I've also dug through some Cython wrapper code but haven't found a solution. Here is the Python embedded sample code:
#include <Python.h>
#include <iostream>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict, *pFunc;
PyObject *pArgs, *pValue;
int i;
if (argc < 3) {
fprintf(stderr, "Usage: runpy pythonfile funcname [args]\n");
return 1;
}
Py_SetPath(L"python35.zip"); //this is in the current directory
Py_Initialize();
pName = PyUnicode_DecodeFSDefault(argv[1]);
/* Error checking of pName left out */
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != NULL) {
pFunc = PyObject_GetAttrString(pModule, argv[2]);
/* pFunc is a new reference */
if (pFunc && PyCallable_Check(pFunc)) {
pArgs = PyTuple_New(argc - 3);
for (i = 0; i < argc - 3; ++i) {
pValue = PyLong_FromLong(atoi(argv[i + 3]));
if (!pValue) {
Py_DECREF(pArgs);
Py_DECREF(pModule);
fprintf(stderr, "Cannot convert argument\n");
return 1;
}
/* pValue reference stolen here: */
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pValue != NULL) {
printf("Result of call: %ld\n", PyLong_AsLong(pValue));
Py_DECREF(pValue);
}
else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr, "Call failed\n");
return 1;
}
}
else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
}
Py_XDECREF(pFunc);
Py_DECREF(pModule);
}
else {
PyErr_Print();
fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
return 1;
}
Py_Finalize();
return 0;
}
Embed file is here: https://www.python.org/ftp/python/3.5.2/python-3.5.2-embed-amd64.zip, python35.zip inside the archive. Here is the simple test script (runpy eig eig 10 to test - note if you don't embed Python35.zip and have NumPy / SciPy installed it WILL run):
eig.py
import numpy as np
from scipy import linalg
def eig(a):
c = np.random.rand(a,a)*100
c = np.corrcoef(c)
print('You are taking the eigsh of a ', a, '^2 matrix')
e, f = linalg.eig(c)
return print('Eigvals are: ',np.diag(f))
Anyone know how to fix this issue? Much appreciated.
Update: Here's the compiled version x64 Python 3.5 Windows NumPy SciPy and Pandas with Intel MKL included: https://www.dropbox.com/sh/2smbgen2i9ilf2e/AADI8A3pCAFU-EqNLTbOiUwJa?dl=0
This does not work because numpy is not in the zipfile python35.zip.
The runpy-program sets the path to python35.zip: It is thus the only path in the Pythonpath for this programs exception...
You have to add the parent-folder of your local numpy-folder also to the Pythonpath to make it working.

C's Python API: Calling python function 2X causes Access violation writing location

I'm trying to write some C code that calls some python code to do some data analysis, and I came upon a weird issue. If I call initialize python, call a function, finalize python, and then repeat the same 1 time, I get an access violation writing location error the second time I try to call the function. The following simple code:
#include "stdafx.h"
#include <iostream>
#include "Python.h"
int main()
{
for (int testInc = 0; testInc < 2; testInc++)
{
std::cout << testInc + 1 << std::endl;
PyObject *pName, *pModule, *pFunc, *pValue;
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\"PathToCode\")");
pName = PyUnicode_DecodeFSDefault("MyModuleName");
pModule = PyImport_Import(pName);
pFunc = PyObject_GetAttrString(pModule, "MyFunctionName");
pValue = PyObject_CallObject(pFunc, NULL);
printf("Result: %s\n", PyBytes_AS_STRING(PyUnicode_AsEncodedString(pValue, "ASCII", "strict")));
Py_DECREF(pName);
Py_DECREF(pModule);
Py_DECREF(pFunc);
Py_DECREF(pValue);
Py_Finalize();
}
return 0;
}
(checks on Py_Object*'s being == NULL omitted for brevity, but they all pass). With the python code being:
def myFunctionName():
import numpy
return "Hi!"
consistently throws the error "Unhandled exception at 0x00007FFE1EBC199C (multiarray.cp35-win_amd64.pyd) in TestApplication.exe: 0xC0000005: Access violation writing location 0x000000000000000A." on the second pass through the for loop, and I'm struggling to figure out why. If I place the initialize and finalize commands outside of the for loop, then this works fine, but my understanding of these commands leads me to believe that this code should be functional. Also, if I omit the "import" command in the python script, my C code also runs fine then, leading me to believe that something weird is happening with the import. Am I misunderstanding something?

Call Python function from C/C++

I am trying to call a simple python function which is defined in ("ctest.py") as
def square(a)
return a**2
the following ("pytest.c") (in same directory) is the C code that I am trying to use to call the function. The issue that I am experiencing is that when the C program tries to load the python module NULL is returned.
#include <Python.h>
int main(int argc, char* argv[])
{
printf("Calling Python Function\n");
Py_Initialize(); // Initialize the Python interpreter.
PyObject *pName, *pModule, *pDict, *pFunc, *pArgs, *pValue; // Create some Python objects that will later be assigned values.
// Convert the file name to a Python string.
pName = PyString_FromString("ctest.py");
if (pName==NULL)
printf("file not found");
else
printf("%s\n", PyString_AsString(pName));
// Import the file as a Python module.
pModule = PyImport_Import(pName); //PROBLEM LINE
if(pModule==NULL)
printf("no Module\n");
// Create a dictionary for the contents of the module.
pDict = PyModule_GetDict(pModule);
printf("After Dictionary retrieval\n");
// Get the add method from the dictionary.
pFunc = PyDict_GetItemString(pDict, "square");
printf("after function retrieval\n");
// Convert 2 to a Python integer.
pValue = PyInt_FromLong(2);
// Call the function with the arguments.
PyObject* pResult = PyObject_CallObject(pFunc, pValue);
// Print a message if calling the method failed.
if(pResult == NULL)
printf("Calling the add method failed.\n");
// Convert the result to a long from a Python object.
long result = PyInt_AsLong(pResult);
// Destroy the Python interpreter.
Py_Finalize();
// Print the result.
printf("The result is %d.\n", result);
return 0;
}
The C Code is built with:
gcc -o pytest -lpythhon2.7 -I/usr/include/python2.7 pytest.c
It looks like you are running into a naming/path issue.
You might have a look at this answer:
Why does PyImport_Import fail to load a module from the current directory?

Python embedded in CPP: how to get data back to CPP

While working on a C++ project, I was looking for a third party library for something that is not my core business. I found a really good library, doing exactly what's needed, but it is written in Python. I decided to experiment with embedding Python code in C++, using the Boost.Python library.
The C++ code looks something like this:
#include <string>
#include <iostream>
#include <boost/python.hpp>
using namespace boost::python;
int main(int, char **)
{
Py_Initialize();
try
{
object module((handle<>(borrowed(PyImport_AddModule("__main__")))));
object name_space = module.attr("__dict__");
object ignored = exec("from myModule import MyFunc\n"
"MyFunc(\"some_arg\")\n",
name_space);
std::string res = extract<std::string>(name_space["result"]);
}
catch (error_already_set)
{
PyErr_Print();
}
Py_Finalize();
return 0;
}
A (very) simplified version of the Python code looks like this:
import thirdparty
def MyFunc(some_arg):
result = thirdparty.go()
print result
Now the problem is this:
'MyFunc' executes fine, i can see the print of 'result'.
What i cannot do is read 'result' back from the C++ code. The extract command never finds 'result' in any namespace.
I tried defining 'result' as a global, i even tried returning a tuple, but i cannot get it to work.
First of all, change your function to return the value. printing it will complicate things since you want to get the value back. Suppose your MyModule.py looks like this:
import thirdparty
def MyFunc(some_arg):
result = thirdparty.go()
return result
Now, to do what you want, you have to go beyond basic embedding, as the documentation says. Here is the full code to run your function:
#include <Python.h>
int
main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pFunc;
PyObject *pArgs, *pArg, *pResult;
int i;
Py_Initialize();
pName = PyString_FromString("MyModule.py");
/* Error checking of pName left out as exercise */
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != NULL) {
pFunc = PyObject_GetAttrString(pModule, "MyFunc");
/* pFunc is a new reference */
if (pFunc) {
pArgs = PyTuple_New(0);
pArg = PyString_FromString("some parameter")
/* pArg reference stolen here: */
PyTuple_SetItem(pArgs, 0, pArg);
pResult = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pResult != NULL) {
printf("Result of call: %s\n", PyString_AsString(pResult));
Py_DECREF(pResult);
}
else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr,"Call failed\n");
return 1;
}
}
else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find function");
}
Py_XDECREF(pFunc);
Py_DECREF(pModule);
}
else {
PyErr_Print();
fprintf(stderr, "Failed to load module");
return 1;
}
Py_Finalize();
return 0;
}
Based on ΤΖΩΤΖΙΟΥ, Josh and Nosklo's answers i finally got it work using boost.python:
Python:
import thirdparty
def MyFunc(some_arg):
result = thirdparty.go()
return result
C++:
#include <string>
#include <iostream>
#include <boost/python.hpp>
using namespace boost::python;
int main(int, char **)
{
Py_Initialize();
try
{
object module = import("__main__");
object name_space = module.attr("__dict__");
exec_file("MyModule.py", name_space, name_space);
object MyFunc = name_space["MyFunc"];
object result = MyFunc("some_args");
// result is a dictionary
std::string val = extract<std::string>(result["val"]);
}
catch (error_already_set)
{
PyErr_Print();
}
Py_Finalize();
return 0;
}
Some important points:
I changed 'exec' to 'exec_file' out of
convenience, it also works with
plain 'exec'.
The main reason it failed is that i
did not pass a "local" name_sapce to
'exec' or 'exec_file' - this is now
fixed by passing name_space twice.
If the python function returns
unicode strings, they are not
convertible to 'std::string', so i
had to suffix all python strings
with '.encode('ASCII', 'ignore')'.
I think what you need is either PyObject_CallObject(<py function>, <args>), which returns the return value of the function you call as a PyObject, or PyRun_String(<expression>, Py_eval_input, <globals>, <locals>) which evaluates a single expression and returns its result.
You should be able to return the result from MyFunc, which would then end up in the variable you are currently calling "ignored". This eliminates the need to access it in any other way.