I am trying to extend Python 3.6 using C++ on windows, following the official documentation and guide. The Anaconda distribution is used with numpy updated to the latest version. In the project python.h and arrayobject.h has been added to the C++ directories, and python36.lib linked, as advised.
For the code, a simple example which is supposed to create an array with the elements 0,1,2,3, when calling the func1 method:
#include <python.h>
#include <arrayobject.h>
static PyObject* fun(PyObject* self, PyObject* args)
{
PyObject *X;
int x[4] = {0,1,2,3};
int dims[1];
dims[0] = 1;
X = PyArray_SimpleNew(1, dims, NPY_INT64, x);
return X;
}
static PyMethodDef MyMethod[] = {
{ "func1", fun, METH_VARARGS,nullptr },
{ nullptr, nullptr, 0, nullptr }
};
static struct PyModuleDef MyModule = {
PyModuleDef_HEAD_INIT,
"MyModule",
NULL,
-1,
MyMethod
};
PyMODINIT_FUNC PyInit_MyModule(void)
{
(void)PyModule_Create(&MyModule);
import_array();
}
The code builds fine. When I take the resulting .pyd file and import it in Spyder, the kernel crashes. Specifically, the import_array(); command seems to cause the crash, as without it the kernel doesn't crash. However, as the documentation notes, the method crashes then. What is the fix?
It was solved by changing the order:
PyMODINIT_FUNC PyInit_MyModule(void)
{
import_array();
return PyModule_Create(&MyModule);
}
All the documentation seems to concern Python 2.X only, which used different initialization. In python 3, the code needs to modified. (There is a small error also in the creation of the array, but I let it be).
Related
Creating an application in C++, I integrated CPython to facilitate the development of certain top-level logics.
The application has a plugin subsystem, which can loaded/unloaded plugins at runtime, this implies to add and remove Python definitions at runtime.
I found that I can add functions with PyModule_AddFunctions, and similarly, I can add constants, objects, etc...
But I found no equivalent PyModule_RemoveFunction.
How to remove a Python function from a module using C++?
Note: Ideally, I would like to avoid solutions like:
Removing the full module and reloading everything
Crafting Python code, that when executed would remove the function.
Let see an example:
DISCLAIM: I removed most of the error checks for simplicity.
#define PY_SSIZE_T_CLEAN
#include <Python.h> //before std includes
// Just an example function to add/remove from Python
static PyObject* log(PyObject* , PyObject* args)
{
//Do something
Py_INCREF(Py_None);
return Py_None;
}
// Setup a "AppModule" module
PyMODINIT_FUNC initAppModule()
{
static PyModuleDef AppModuleInfo{
PyModuleDef_HEAD_INIT,
"AppModule", // name of module
nullptr, // module documentation, may be NULL
-1, // size of per-interpreter state of the module, or -1 if the module keeps state in global variables.
nullptr
};
const auto AppModule = PyModule_Create(&AppModuleInfo);
PyModule_AddFunctions(AppModule, AppModuleMethods);
return AppModule;
}
// Adding a function when a plugin is loaded
void PluginAddingAFunction()
{
static PyMethodDef AppModuleMethods[]{
{"log", log, METH_VARARGS,
"Log a message in the standard output."},
{NULL, NULL, 0, NULL} // Sentinel
};
PyObject *modName = PyUnicode_FromString("AppModule");
PyObject *mod = PyImport_Import(modName);
PyModule_AddFunctions(mod, AppModuleMethods);
}
// Removing the function when the plugin is unloaded
void PluginRemoveAFunction()
{
PyObject *modName = PyUnicode_FromString("AppModule");
PyObject *mod = PyImport_Import(modName);
// How to do this?
//PyModule_RemoveFunctions(mod, "log");
}
int main(int argn, const char* argv[])
{
Py_SetProgramName(argv[0]);
PyImport_AppendInittab("AppModule", &initAppModule);
Py_Initialize();
PyObject *pmodule = PyImport_ImportModule("AppModule");
PluginAddingAFunction(); // <<===== This is done at any time, when loading a plugin
PyRun_SimpleString("import AppModule\n"
"AppModule.log('Hi World')\n"); // <== Example code
PluginRemoveAFunction(); // <<===== This is done when unloading a plugin
Py_FinalizeEx();
}
You can use PyObject_DelAttr C-apis.
int PyObject_DelAttr(PyObject *o, PyObject *attr_name)
Delete attribute named attr_name, for object o. Returns -1 on failure. This is the equivalent of the Python statement del o.attr_name
Reference.
So you could do something like this to remove log function
void PluginRemoveAFunction()
{
PyObject *modName = PyUnicode_FromString("AppModule");
PyObject *mod = PyImport_Import(modName);
PyObject *funcName = PyUnicode_FromString("log");
PyObject_DelAttr(mod, funcName);
}
This question already has answers here:
Integrate Python And C++
(11 answers)
Closed 3 years ago.
I'm looking for a way to run C++ code alongside python in real time. I have a neural network program which takes in inputs, and outputs a certain integer. The process of running the neural network is the main "meat" of my program and uses up the most resources so I'd like to code it in c++ to make the overall system more efficient.
What I'm looking to do now if run a python main program that passes in python variables to an external CPP file. The CPP file does its math magic and spits put an integer in Python. Is this possible? Is it advisable?
I have heard of Cython before but would much rather code this by scratch to ensure it is optimized for my specific task.
Kindest reguards!
I did something like you want, but with C++ as start point:
C++ code creates python-like module with C functions.
After that C++ code runs target python module, that invokes functions from python-like module.
As result, python module is running and is invoking C++ functions.
Here the sample of C++ code:
// Sample of C++ function, which python module invokes
static PyObject* OutDtmf(PyObject* self, PyObject* args)
{
PyObject* a1;
if (PyArg_UnpackTuple(args, "OutDtmf", 1, 1, &a1))
{
// Check the result
PyObject* astr = PyUnicode_AsUTF8String(a1);
const char* ustr = PyBytes_AsString(astr);
OutDtmf(ustr);
}
Py_RETURN_NONE;
}
// Pack of definitions
// --------------------
static PyMethodDef WarSysMethods[] = {
{ "Finish", FinishScript, METH_VARARGS, NULL },
{ "KpsoSetControl", KpsoSetControl, METH_VARARGS, NULL },
{ "KpsoSetLine", KpsoSetLine, METH_VARARGS, NULL },
{ "OutDtmf", OutDtmf, METH_VARARGS, NULL },
{ "PsoSetLine", PsoSetLine, METH_VARARGS, NULL},
{ NULL, NULL, 0 , nullptr }
};
static struct PyModuleDef WarSysModuleDef = {
PyModuleDef_HEAD_INIT,
"WarSys",
NULL,
-1,
WarSysMethods };
PyMODINIT_FUNC PyInit_WarSys(void)
{
PyObject *module;
module = PyModule_Create(&WarSysModuleDef);
return module;
}
// Start point for creation of python-like module and loading target python module
void StartScript(bool serverMode, const char* testModuleName)
{
// Initialization Python -> C++
PyImport_AppendInittab("WarSys", PyInit_WarSys);
// Initialization C++ -> Python
Py_Initialize();
PyObject* pDict; // borrowed
TestModule = PyImport_ImportModule(testModuleName);
if (!TestModule)
{
PyErr_Print();
return;
}
pDict = PyModule_GetDict(TestModule);
// Read function objects
FuncInit = PyDict_GetItemString(pDict, "Init");
....................
// Invokes python function in module (f.e. Init)
PyObject_CallObject(command.Function, command.Arguments)
in python code use:
import WarSys
and invokes functions WarSys.Finish(False) or other.
I am using Boost Python, I generate a large vector of integers in C++, and I would like to access this vector in Python without copying it.
In C++ I have:
BOOST_PYTHON_MODULE(myModule)
{
class_<vector<int>>("vectorInt").def(vector_indexing_suite<vector<int>>());
def("ReturnVectorPtr", ReturnVectorPtr, return_value_policy<manage_new_object>());
}
vector<int>* ReturnVectorPtr()
{
return new vector<int>();
}
Then in python I have:
import myModule
myModule.ReturnVectorPtr()
This causes Python to crash, although I'm not even storing the return value. Any ideas on what my mistake is?
Edit:
The following code works for getting the data in the vector from C++ to python, but leaks memory. Are the vectors being copied and then not disposed?
In C++:
BOOST_PYTHON_MODULE(myModule)
{
class_<vector<int>>("vectorInt").def(vector_indexing_suite<vector<int>>());
def("ModifyVectorInPlace", ModifyVectorInPlace);
}
void ModifyVectorInPlace(vector<int>& data)
{
// Modify data...
return;
}
Then in python I have:
import myModule
vectorInt = myModule.vectorInt()
myModule.ModifyVectorInPlace(vectorInt)
What is going on?
Edit 2:
I tried the "Raw C++ Pointers" example from here, exactly as written:
https://wiki.python.org/moin/boost.python/PointersAndSmartPointers
It crashes too. It seems that I can't get a pointer to anything passed into Python for some reason...
Edit 3:
The crash appears to be a segfault from invoke.hpp, in this function:
template <class RC, class F BOOST_PP_ENUM_TRAILING_PARAMS_Z(1, N, class AC)>
inline PyObject* invoke(invoke_tag_<false,false>, RC const& rc, F& f BOOST_PP_ENUM_TRAILING_BINARY_PARAMS_Z(1, N, AC, & ac) )
{
return rc(f( BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () BOOST_PP_INTERCEPT) ));
}
It turns out this was a bug in the interaction between Mingw-w64 and Python. I performed the procedure described here and the problem was solved:
http://ascend4.org/Setting_up_a_MinGW-w64_build_environment#Setup_Python_for_compilation_of_extensions
For performance reasons I want to port parts of my python program to C++ and I therefore try to write a simple extension for my program. The C++ part will build a dictionary, which then needs to be delivered to the Python program.
One way I found seems to be to build my dict-like object in C++, e.g. a boost::unordered_map, and then translate it to Python using the Py_BuildValue[1] method, which is able to produce Python dicts. But this method which includes converting the container into a string representation and back seems a bit too much 'around the corner' to be the most performant solution!?
So my question is: What is the most performant way to build a Python dictionary in C++? I saw that boost has a Python library which supports mapping containers between C++ and Python, but I didn't find the exact thing I need in the documentation so far. If there is such way I would prefer to directly build a Python dict in C++, so that no copying etc. is needed. But if the most performant way to do this is another one, I'm good with that too.
Here is the (simplified) C++-code I compile into a .dll/.pyd:
#include <iostream>
#include <string>
#include <Python.h>
#include "boost/unordered_map.hpp"
#include "boost/foreach.hpp"
extern "C"{
typedef boost::unordered_map<std::string, int> hashmap;
static PyObject*
_rint(PyObject* self, PyObject* args)
{
hashmap my_hashmap; // DO I NEED THIS?
my_hashmap["a"] = 1; // CAN I RATHER INSERT TO PYTHON DICT DIRECTLY??
BOOST_FOREACH(hashmap::value_type i, my_hashmap) {
// INSERT ELEMENT TO PYTHON DICT
}
// return PYTHON DICT
}
static PyMethodDef TestMethods[] = {
{"rint", _rint, METH_VARARGS, ""},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC
inittest(void)
{
Py_InitModule("test", TestMethods);
}
} // extern "C"
This I want to use in Python like:
import test
new_dict = test.rint()
The dictionary will map strings to integers. Thanks for any help!
Use the CPython API directly yes:
PyObject *d = PyDict_New()
for (...) {
PyDict_SetItem(d, key, val);
}
return d;
Or write a python object that emulate a dict, by overriding __setitem__ and __getitem__. In both method, use your original hashmap. At the end, no copy will happen!
I have written some C++ code that generates a std::vector.
I also have a python script that manipulates some data that, for now, I am declaring like this (below).
import numpy
x = numpy.random.randn(1000)
y = numpy.random.randn(1000)
I can run the script fine. From my C++ code:
using namespace boost::python;
try{
Py_Initialize();
object main = import("__main__");
object global(main.attr("__dict__"));
object result = exec_file("scatterPlot.py", global, global);
Py_Finalize();
}
catch(error_already_set){
PyErr_Print();
}
return;
I have no idea how to get my C++ data to python. I've around quite a bit, but there doesn't seem to be anything definitive.
I have in my C++
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
This seems to work, but as I understand, it only provides a class "PyVec" for my python script but not the data I need. Am I wrong?
I've also seen some other people use boost::shared_ptr in a python mailing list.
I also found this example but found it confusing.
I can think of a few approaches
Pass something to the boost::python::exec_file method
Using the boost_indexing_suite_ext
Uinsg boost::shared_ptr
Which approach is easiest to get going? No approach seems clear to me.
Here are some more links I've looked at:
from the boost website
from the python website
another mailing list thread
UPDATE:
This works for passing an int to my python code like below
int main(){
int five_squared=0;
int a =3;
try {
Py_Initialize();
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
main_namespace["var"]=a;
object ignored = exec("result = 5 ** var", main_namespace);
five_squared = extract<int>(main_namespace["result"]);
} catch( error_already_set ) {
PyErr_Print();
}
std::cout << five_squared << std::endl;
return 0;
}
But I want to pass a vector, when I try to do that in a similar fashion as above I get this error
TypeError: No to_python (by-value)
converter found for C++ type:
std::vector >
So, obviously I need to tell python how to deal with std::vector. I think this code could help with that.
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
But since std::vector is pretty common, there must be a defined way to do this... right?
The following code works for me (Python 2.6, Boost 1.39). This is almost the same as your code, except without the BOOST_PYTHON_MODULE line itself (but with the class_ definition for the vector). BOOST_PYTHON_MODULE only needs to be used when creating extension modules.
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
using namespace boost::python;
using namespace std;
int main()
{
vector<double> vec;
vec.push_back(1.2);
vec.push_back(3.4);
try {
Py_Initialize();
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
object main_module = import("__main__");
object globals = main_module.attr("__dict__");
globals["var"]=vec;
object ignored = exec("result = sum(var)", globals, globals);
double result = extract<double>(globals["result"]);
std::cout << result << std::endl;
} catch( error_already_set ) {
PyErr_Print();
}
return 0;
}
I'm not sure if I understand correctly. After exporting your class "PyVec" which can hold std::vector<double>, you can export any c++ function taking vector as input or return type. So of course you can populate your vector within c++ and access this data in Python with the interfaced type "PyVec".