I'm wondering if boost.python allows C++ functionality to be exposed to python after the module has loaded. For example it would be nice if something like this might work:
#include <boost/python.hpp>
int a;
void expose_var() {
boost::python::scope().attr( "a" ) = a;
}
BOOST_PYTHON_MODULE( mod )
{
boost::python::def( "expose_var", expose_var, "Expose an attribute." );
}
Then in python:
import mod
mod.expose_var()
mod.a = 2
With similar code I'm getting an error when I call the equivalent of expose_var():
AttributeError: 'NoneType' object has no attribute '__dict__'
I want to do this as I'm exposing a C++ library that is heavily templated and I don't wish to expose every possible combination of template parameters by default. I'd like to let the python client ask for specific combinations to be exposed at runtime.
Related
I'm using boost::python to do a hybrid C++/python application: the C++ app calls a collection of python scripts, which in turn use the C++ program's functions, classes, etc., exposed as python objects. (Python 2.x.)
BOOST_PYTHON_MODULE(MyModule) exposes the C++ to python as expected.
My initialization code:
Py_Initialize();
initMyModule(); // import MyModule
namespace bpl = boost::python;
Now I want my C++ code to get at MyModule, too. In python, you just write globals()['MyModule']. But this (and things like it) don't work in C++:
bpl::object globals = bpl::eval("globals()");
This fails at run-time with
File "<string>", line 1, in <module>; NameError: name 'globals' is not defined
As an aside, I see many examples of setting up __main__ like this:
bpl::object m = bpl::import("__main__");
bpl::dict g = m.attr("__dict__"); // like locals(), but not globals()
This doesn't fail, and gives locals, but according to the Py_Initialize docs, __main__ is already set up. And it doesn't let you see globals, where you'd find your imported module.
You don't need the explicit bpl::import("__main__");.
Here are the globals:
bpl::dict globals()
{
bpl::handle<> mainH(bpl::borrowed(PyImport_GetModuleDict()));
return bpl::extract<bpl::dict>(bpl::object(mainH));
}
Since everything is managed by smart pointers, returning and manipulating bpl::dict directly works fine.
bpl::object myMod = globals()["MyModule"];
globals()["myNewGlobal"] = 88;
So I'm working on a little project in which I'm using Python as an embedded scripting engine. So far I've not had much trouble with it using boost.python, but there's something I'd like to do with it if it's possible.
Basically, Python can be used to extend my C++ classes by adding functions and even data values to the class. I'd like to be able to have these persist in the C++ side, so one python function can add data members to a class, and then later the same instance passed to a different function will still have them. The goal here being to write a generic core engine in C++, and let users extend it in Python in any way they need without ever having to touch the C++.
So what I thought would work was that I would store a boost::python::object in the C++ class as a value self, and when calling the python from the C++, I'd send that python object through boost::python::ptr(), so that modifications on the python side would persist back to the C++ class. Unfortunately when I try this, I get the following error:
TypeError: No to_python (by-value) converter found for C++ type: boost::python::api::object
Is there any way of passing an object directly to a python function like that, or any other way I can go about this to achieve my desired result?
Thanks in advance for any help. :)
Got this fantastic solution from the c++sig mailing list.
Implement a std::map<std::string, boost::python::object> in the C++ class, then overload __getattr__() and __setattr__() to read from and write to that std::map. Then just send it to the python with boost::python::ptr() as usual, no need to keep an object around on the C++ side or send one to the python. It works perfectly.
Edit: I also found I had to override the __setattr__() function in a special way as it was breaking things I added with add_property(). Those things worked fine when getting them, since python checks a class's attributes before calling __getattr__(), but there's no such check with __setattr__(). It just calls it directly. So I had to make some changes to turn this into a full solution. Here's the full implementation of the solution:
First create a global variable:
boost::python::object PyMyModule_global;
Create a class as follows (with whatever other information you want to add to it):
class MyClass
{
public:
//Python checks the class attributes before it calls __getattr__ so we don't have to do anything special here.
boost::python::object Py_GetAttr(std::string str)
{
if(dict.find(str) == dict.end())
{
PyErr_SetString(PyExc_AttributeError, JFormat::format("MyClass instance has no attribute '{0}'", str).c_str());
throw boost::python::error_already_set();
}
return dict[str];
}
//However, with __setattr__, python doesn't do anything with the class attributes first, it just calls __setattr__.
//Which means anything that's been defined as a class attribute won't be modified here - including things set with
//add_property(), def_readwrite(), etc.
void Py_SetAttr(std::string str, boost::python::object val)
{
try
{
//First we check to see if the class has an attribute by this name.
boost::python::object obj = PyMyModule_global["MyClass"].attr(str.c_str());
//If so, we call the old cached __setattr__ function.
PyMyModule_global["MyClass"].attr("__setattr_old__")(ptr(this), str, val);
}
catch(boost::python::error_already_set &e)
{
//If it threw an exception, that means that there is no such attribute.
//Put it on the persistent dict.
PyErr_Clear();
dict[str] = val;
}
}
private:
std::map<std::string, boost::python::object> dict;
};
Then define the python module as follows, adding whatever other defs and properties you want:
BOOST_PYTHON_MODULE(MyModule)
{
boost::python::class_<MyClass>("MyClass", boost::python::no_init)
.def("__getattr__", &MyClass::Py_GetAttr)
.def("__setattr_new__", &MyClass::Py_SetAttr);
}
Then initialize python:
void PyInit()
{
//Initialize module
PyImport_AppendInittab( "MyModule", &initMyModule );
//Initialize Python
Py_Initialize();
//Grab __main__ and its globals
boost::python::object main = boost::python::import("__main__");
boost::python::object global = main.attr("__dict__");
//Import the module and grab its globals
boost::python::object PyMyModule = boost::python::import("MyModule");
global["MyModule"] = PyMyModule;
PyMyModule_global = PyMyModule.attr("__dict__");
//Overload MyClass's setattr, so that it will work with already defined attributes while persisting new ones
PyMyModule_global["MyClass"].attr("__setattr_old__") = PyMyModule_global["MyClass"].attr("__setattr__");
PyMyModule_global["MyClass"].attr("__setattr__") = PyMyModule_global["MyClass"].attr("__setattr_new__");
}
Once you've done all of this, you'll be able to persist changes to the instance made in python over to the C++. Anything that's defined in C++ as an attribute will be handled properly, and anything that's not will be appended to dict instead of the class's __dict__.
My app have some events, each event can have some actions. These actions is implemented in C++. I want to expose those core functions to python and use python to write the action. The advantage is I can modify actions without recompile. For example:
CppClass o;
// --- this is a action----
o.f1();
o.f2();
// ------------------------
use python to script the action:
def action1(o):
o.f1()
o.f2()
In c++, use interpreter to run this script, find action1 and call it with a PyObject which convert from c++ object. Actually, I have not expose f1() & f2() to python, I just use python to regroup the definition of action, all function is running by c++ binary code. Notice that I have not to give a definition of f1() & f2() in python.
The problem is: how I expose global functions? such as:
def action2():
gf1()
gf2()
boost::python can expose function, but it is different, it need compile a DLL file and the main() is belong to python script. Of course I can make global functions to be a class static member, but I just want to know. Notice that I HAVE TO give a definition of gf1() & gf2() in python.
Jython can do this easily: just import Xxx in python code and call Xxx.gf1() is ok. But in cython, how I define gf1() in python? This is a kind of extension, but extension requires Xxx be compiled ahead. It seems only way is make gf() into a class?
Solved. boost::python's doc is really poor...
For example: expose function
void ff(int x, int y)
{
std::cout<<x<<" "<<y<<std::endl;
}
to python:
import hello
def foo(x, y)
hello.ff(x, y)
you need to expose it as a module:
BOOST_PYTHON_MODULE(hello)
{
boost::python::def("ff", ff, boost::python::arg("x"), boost::python::arg("y"));
}
But this still is not a 'global function', so expose it to python's main scope:
BOOST_PYTHON_MODULE(__main__)
{
boost::python::def("ff", ff, boost::python::arg("x"), boost::python::arg("y"));
}
then you can write:
def foo(x, y)
ff(x, y)
You might also want to have a look at Cython.
Cython's main function is to translate (a subset of) Python code to C
or C++ code to build native code Python extensions. As a consequence,
it allows interfacing C/C++ code with a very terse and Python-ish
syntax.
Cython's user guide provides a
good example of how to call a simple C++ class from Python:
http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#declaring-a-c-class-interface
In addition to creating extensions, Cython can also generate C/C++
code that embeds the Python interpreter, so you do not need to build
and ship an external DLL. See details at: http://wiki.cython.org/EmbeddingCython
ffpython is c++ lib that wrap python 2.x api. see code repo
call python function: ffpython.call("fftest", "test_stl", a1, a2, a3);
reg c++ class:
ffpython.reg_class<foo_t, PYCTOR(int)>("foo_t")
.reg(&foo_t::get_value, "get_value")
.reg(&foo_t::set_value, "set_value")
.reg(&foo_t::test_stl, "test_stl")
.reg_property(&foo_t::m_value, "m_value");
You can add a property to a class using a getter and a setter (in a simplistic case):
class<X>("X")
.add_property("foo", &X::get_foo, &X::set_foo);
So then you can use it from python like this:
>>> x = mymodule.X()
>>> x.foo = 'aaa'
>>> x.foo
'aaa'
But how to add a property to a module itself (not a class)?
There is
scope().attr("globalAttr") = ??? something ???
and
def("globalAttr", ??? something ???);
I can add global functions and objects of my class using the above two ways, but can't seem to add properties the same way as in classes.
__getattr__ and __setattr__ aren't called on modules, so you can't do this in ordinary Python without hacks (like storing a class in the module dictionary). Given that, it's very unlikely there's an elegant way to do it in Boost Python either.
boost.python/HowTo on Python Wiki has an example of exposing C++ object as a module attribute inside BOOST_PYTHON_MODULE:
namespace bp = boost::python;
BOOST_PYTHON_MODULE(example)
{
bp::scope().attr("my_attr") = bp::object(bp::ptr(&my_cpp_object));
}
To set the attribute outside of BOOST_PYTHON_MODULE use
bp::import("example").attr("my_attr") = bp::object(bp::ptr(&my_cpp_object));
Now you can do in python something like
from example import my_attr
Of course, you need to register class of my_cpp_object in advance (e.g. you can do this inside the same BOOST_PYTHON_MODULE call) and ensure C++ object lifetime exceeds that of the python module. You can use any bp::object instead of wrapping C++ one.
Note that BOOST_PYTHON_MODULE swallows exceptions, so if you make a mistake, you don't receive any error indication and BOOST_PYTHON_MODULE-generated function will just immediately return. To ease debugging of this case you can catch exceptions inside BOOST_PYTHON_MODULE or temporary add some logging statement as a last line of BOOST_PYTHON_MODULE to see that it is reached:
BOOST_PYTHON_MODULE(example)
{
bp::scope().attr("my_attr2") = new int(1); // this will compile
std::cout << "init finished\n"; // OOPS, this line will not be reached
}
I have implemented a class in C++. I want to use it with Python.
Please suggest step by step method and elaborate each step.
Somthing like this...
class Test{
private:
int n;
public:
Test(int k){
n=k;
}
void setInt(int k){
n = k;
}
int getInt(){
return n;
}
};
Now, in Python
>>> T1 = Test(12)
>>> T1.getInt()
12
>>> T1.setInt(32)
>>> T1.getInt()
32
Please suggest.How can I do this ?
NOTE: I would like to know manual way to do that. I don't want any third party library dependency.
Look into Boost.Python. It's a library to write python modules with C++.
Also look into SWIG which can also handle modules for other scripting languages. I've used it in the past to write modules for my class and use them within python. Works great.
You can do it manually by using the Python/C API, writing the interface yourself. It's pretty lowlevel, but you will gain a lot of additional knowledge of how Python works behind the scene (And you will need it when you use SWIG anyway).
ctypes is good. It is really easy to use, and it comes standard with Python. Unfortunately it can only talk to shared libraries (Unix) or DLLs (Windows) that have a C-style interface, which means you can't directly interface to a C++ object. But you could use a handle system where a handle refers to a particular object.
>>> getInt(h)
12
I think that is simple, easy to understand, and doesn't require extra libraries.
I would suggest you try SWIG or sip (KDE/PyQt).
SWIG link : http://www.swig.org/
SIP link: http://freshmeat.net/projects/python-sip/
These can be used to wrap C++ classes and provide a Pythonic interface to them.
This is a very old post, and probably the OP already found a solution to his problem, but since no one has yet provided a valid example, I will give it a try and maybe help someone like me with the same problem in hand.
Since the OP did not specify, I will restrict my answer to Windows, although a Linux solution would be straightforward. I will try to provide a minimal working example that reproduces the code in question.
Step 1) Start with the C++ code (a single file)
class Test{
private:
int n;
public:
Test(int k){
n=k;
}
void setInt(int k){
n = k;
}
int getInt(){
return n;
}
};
extern "C"
{
// include below each method you want to make visible outside
__declspec(dllexport) Test* init(int k) {return new Test(k);}
__declspec(dllexport) void setInt(Test *self, int k) {self->setInt(k);}
__declspec(dllexport) int getInt(Test *self) {return self->getInt();}
// Note: the '__declspec(dllexport)' is only necessary in Windows
}
Step 2) Compile the DLL shared library. E.g. from your working directory:
g++ -shared mytest.cpp -o libtest.dll
Step 3) Create the Python file with the wrapper:
# mytest.py
import ctypes
import platform
# From Python 3.8 onwards, there is a reported bug in CDLL.__init__()
mode = dict(winmode=0) if platform.python_version() >= '3.8' else dict()
lib = ctypes.CDLL('./libtest.dll', **mode)
class Test(object):
def __init__(self, val: int):
# Declare input and output types for each method you intend to use
lib.init.argtypes = [ctypes.c_int]
lib.init.restype = ctypes.c_void_p
lib.setInt.argtypes = [ctypes.c_void_p, ctypes.c_int]
lib.setInt.restype = ctypes.c_void_p
lib.getInt.argtypes = [ctypes.c_void_p]
lib.getInt.restype = ctypes.c_int
self.obj = lib.init(val)
def setInt(self, n):
lib.setInt(self.obj, n)
def getInt(self):
return lib.getInt(self.obj)
T1 = Test(12)
print(T1.getInt())
T1.setInt(32)
print(T1.getInt())
Step 4) Run the code
Below are my results when I run the script with 4 different Python versions:
PS D:\Desktop\mytest> py -3.7 .\mytest.py
12
32
PS D:\Desktop\mytest> py -3.8 .\mytest.py
12
32
PS D:\Desktop\mytest> py -3.9 .\mytest.py
12
32
PS D:\Desktop\mytest> py -3.10 .\mytest.py
Traceback (most recent call last):
File "D:\Desktop\mytest\mytest.py", line 7, in <module>
lib = ctypes.CDLL('./libtest.dll', **mode)
File "D:\Python\3102\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'D:\Desktop\mytest\libtest.dll' (or one of its dependencies). Try using the full path with constructor syntax.
In the last case, I have tried with the full path and still same error.
I'm running on Windows 10 64bit and compiling with g++ 8.1.0 from MinGW64
Comments:
My answer is based in the solutions provided here, here, and here
A common problem arises when the Python and compiler (g++) architectures does not match (i.e. 32 or 64 bit)