Can someone suggest a way to manipulate c++ objects with Cython, when
the c++ instance of one class is expected to feed the constructor of another wrapped
class as described below?
Please look at the note on the pyx file for the class PySession, which
takes a python PyConfigParams object as argument and then needs to
extract values from it in order to construct a c++ ConfigParams
object. the ConfigParams object is then used to feed the constructor
of Session.
It would be ideal to have a procedure which would allow me to "inject"
the ConfigParams c++ object wrapped by the PyConfigParams object
directly into the constructor of Session, without having to dismantle
it first and then building a new c++ object to feed the constructor.
This works, of course. However, it is a cumbersome, sort of brutal way to implement this solution, not to mention unreliable.
I am aware of PyCapsule, however it may require to touch the c++ headers, which is something I can not do.
Related to this, but a different question is: what if I need a wrapped
class (let's say here PySession) to simulate the behavior of the C++
api by returning a ConfigParams instance? would I need to do the
reverse and dismantle the c++ object to build a Python PyConfigParams
which would then be returned to the Python user in the Python world?
Any suggestions are very welcome!
Thank you!
Let's suppose I have two c++ classes named ConfigParams and Session.
An instance of ConfigParams is used to feed the constructor of the
Session class:
C++ classes
ConfigParams class
// ConfigParams.h
#include <iostream>
using namespace std;
class ConfigParams
{
int parameter1;
public:
ConfigParams(int par1) { this->parameter1 = par1;}
int getPar1() { return this->parameter1; }
};
Session class
// Session.h
#include <iostream>
using namespace std;
#include "configparams.h"
class Session
{
int sessionX;
public:
Session(ConfigParams parameters) { this->sessionX = parameters.getPar1(); }
void doSomething();
};
void Session::doSomething()
{
cout << "Session parameters set as: " << endl;
cout << "X = " << this->sessionX << endl;
}
Cython pyx and pxd files for the above classes:
PyConfigParams
# configparams.pxd
cdef extern from "configparams.h":
cppclass ConfigParams:
ConfigParams(int par1)
int getPar1()
# configparams.pyx
cdef class PyConfigParams:
cdef ConfigParams* thisptr
def __cinit__(self, i):
self.thisptr = new ConfigParams(<int> i)
def getPar1(self):
return self.thisptr.getPar1()
PySession class
# session.pxd
from configparams cimport *
cdef extern from "session.h":
cdef cppclass Session:
Session(ConfigParams parameters)
void doSomething()
# session.pyx
cdef class PySession:
cdef Session* thisptr
def __cinit__(self, pars):
# Note that here I have to extract the values
# from the pars (python PyConfigParams object)
# in order to build a c++ ConfigParams object
# which feeds the c ++ constructor of Session.
cdef ConfigParams* cpppargsptr = new ConfigParams(<int> pars.getPar1())
self.thisptr = new Session(cpppargsptr[0])
def doSomething(self):
self.thisptr.doSomething()
Solution:
Forward declare PyConfigParams in the configparams.pxd module (so it can be invoked from the session.pyx module)
# configparams.pxd
cdef extern from "configparams.h":
cppclass ConfigParams:
ConfigParams(int par1)
int getPar1()
cdef class PyConfigParams:
cdef ConfigParams* thisptr
Import PyConfigParams in the session.pyx module, and cast the argument for the constuctor, this will grant access to the PyConfigParams pointer to the c++ object, which will need to be dereferenced.
# session.pyx
from configparams cimport PyConfigParams
from cython.operator cimport dereference as deref
cdef class PySession:
cdef Session* thisptr
def __cinit__(self, PyConfigParams pars):
self.thisptr = new Session(deref(pars.thisptr))
def doSomething(self):
self.thisptr.doSomething()
Related
I wrote a wrapper that allows me to use python functions in C++, the api looks like this:
auto module = python.getModule("pycode"); // calls PyUnicode_DecodeFSDefault
CppPyFunction<int(int, int)> multiply(module, "multiply");
std::cout << "multiply(a, b): " << multiply(a, b); // calls PyTuple_New, PyTuple_SetItem, PyObject_CallObject
However, I dont really understand how to use python classes directly
I can use classes in a somewhat hacky way:
// python code
class Person:
def __init__(self, name):
self.name = name;
print("Person: __init__" + str(name))
def printName(self, string):
print(self.name)
def makePerson(name):
return Person(index);
def printName(person)
person.printName()
Then the C++ code would be
auto personModule = python.getModule("Person");
CppPyFunction<PyObject * (std::string)> makePerson(module, "makePerson");
CppPyFunction<void(PyObject*)> printName(module, "printName");
auto alice = makePerson("Alice");
auto bob = makePerson("Bob");
printName(alice);
printName(bob);
But this is abit of a hack because it is not natural to create the functions def makePerson(name): and def printName(person).
Question1: Is there a way to call the Person constructor and methods directly from C++ without having to create a "factory method" in the python code?
Question2: Is there a way to construct a template "class" representation in C++ with nice syntax, eg
typedef ?? CppPyClass<???> ??? Person;
Person a("Alice");
Person b("Bob);
a.printName();
b.printName();
I'm working on a c++ application that uses pybind11 to embed python and I've run into a bit of a problem when trying to call an embedded function from a class method.
to start with here are my bindings:
#ifdef _DEBUG
#undef _DEBUG
#include <python.h>
#define _DEBUG
#else
#include <python.h>
#endif
#include <embed.h>
namespace py = pybind11;
using namespace py::literals;
void DebugInfo(std::string string_)
{
String LogMessage_(string_.c_str());
LOGINFO(LogMessage_);
}
PYBIND11_EMBEDDED_MODULE(Test, m) {
m.def("DebugInfo", &DebugInfo, "Posts message to DEBUGINFO");
}
I could then have a .py file with:
import Test
test.DebugInfo("I'm a lumberjack and that's OK")
and it will print just fine to debug
The trouble starts when I try and call it from within a class method.
import Test
class PyTest(object):
def __init__(self):
test.DebugInfo("I'm a lumberjack and that's OK")
test = PyTest()
when this runs it throws an exception against cast.h specifically against line 1985 which is part of this function:
template <return_value_policy policy>
class unpacking_collector {
public:
template <typename... Ts>
explicit unpacking_collector(Ts &&...values) {
// Tuples aren't (easily) resizable so a list is needed for collection,
// but the actual function call strictly requires a tuple.
auto args_list = list();
int _[] = { 0, (process(args_list, std::forward<Ts>(values)), 0)... };
ignore_unused(_);
m_args = std::move(args_list);
}
const tuple &args() const & { return m_args; }
const dict &kwargs() const & { return m_kwargs; }
tuple args() && { return std::move(m_args); }
dict kwargs() && { return std::move(m_kwargs); }
/// Call a Python function and pass the collected arguments
object call(PyObject *ptr) const {
PyObject *result = PyObject_Call(ptr, m_args.ptr(), m_kwargs.ptr());
if (!result)
throw error_already_set(); //EXCEPTION THROWS HERE!
return reinterpret_steal<object>(result);
}
And because it's probably relevant here's how I'm calling the whole thing from my main application
//Start the Python Interpreter
py::scoped_interpreter guard{};
//Python variables
py::object thing_;
std::string test_py = Import_File("test.py");
auto locals = py::dict();
py::exec(test_py, py::globals(), locals);
thing_ = locals["test"].cast<py::object>();
thing_.attr("start")();
and the contents of test.py
import Test
class PyTest(object):
def __init__(self, message = "Test Object initialized"):
self.message = message
iterstr = str(self.iter)
message = self.message + iterstr
self.iter = 0
Test.DebugInfo(message)
def start(self):
self.message = "Starting Python Object"
self.iter = self.iter + 1
iterstr = str(self.iter)
message = self.message + iterstr
Test.DebugInfo(message)
def update(self):
self.message = "Python Object Update Cycle:"
self.iter = self.iter + 1
iterstr = str(self.iter)
message = self.message + iterstr
Test.DebugInfo(message)
test = PyTest()
I'm not sure if I've run into a limitation of pybind11, a bug in it, or if I've just screwed the whole thing up.
Any insight would be greatly appreciated.
This was also filed as an issue with pybind11 here: https://github.com/pybind/pybind11/issues/1452
I came across both this SO and the issue, but I figured this out. Copying it here for anyone who stumbles across this first in the future
Basically, you don't actually want a blank py::dict for locals; it will cause all sorts of problems. If you look embedding sample code from the docs or the tests, the locals value always copies the global scope.
See:
* https://pybind11.readthedocs.io/en/stable/advanced/embedding.html
* https://github.com/pybind/pybind11/blob/master/tests/test_embed/test_interpreter.cpp#L57
Your options are to copy the global scope, or, in this case, simply don't pass in a locals
py::scoped_interpreter guard{};
auto globals = py::globals();
py::exec(test_py, globals);
thing_ = globals["Object"].cast<py::object>();
thing_.attr("start")();
It looks like that in the case of top-level code (not inside any module), the globals variable holds the values at this scope.
So after some experimentation I discovered that the problem is caused by pybind not being able to detect an imported module outside of the scope of the function.
import foo
def bar():
foo.func()
will always cause an error. However,
def bar():
import foo
foo.func()
will function as intended.
I'm trying to use Python embedded in C++ with Boost::python.
My embedded script are supposed to use decorator to register their methods like following:
class Test:
def __init__(self, object_id):
self.object_id = object_id
#decorator('my_id_1')
def func1(self):
print("{}.func1".format(self.object_id))
decorator is declared on the C++ side, defining the method the __init__ and __call__. Everything works has expected, until the call of the method, which lead to SIGSEGV or SIGARBT.
Here is an example of what I would like to do in Python:
#CPP side
item = {}
class decorator:
def __init__(self, _id):
self._id = _id
def __call__(self, func):
item[self._id] = func #saved as PyObject* in CPP
print func
return func
#Script side
class Test(CppBase):
def __init__(self, object_id):
CppBase.__init__(self)
self.object_id = object_id
#decorator('my_id_1')
def func1(self):
print("{}.func1".format(self.object_id))
#decorator('my_id_2')
def func2(self):
print("{}.func2".format(self.object_id))
#CPP side
obj1 = Test("obj1")
item['my_id_1'](obj1) #The crash append here
To do the call, I'm using the following function: boost::python::call<void>(my_PyObject_func, boost::ref(my_obj_instance))
I won't put my all C++ code because I'm actually updating a working project made from the old Python C API, and the whole API is quite huge. However, if you think I forgot some significant part of it, just tell me, and I will post those parts. Furthermore, I removed a lot of simple check such as being sure that the global Python var contain my object, no python error happened or the hash contain the requested id, to make the code lighter.
Here are my C++ Object definition
class CppBase: public boost::python::object {
public:
CppBase();
void open(std::string, boost::python::tuple arg);
void setRoutes(const std::hash<std::string, const Route*>&);
inline std::hash<std::string, const Route*>*const routes() const { return route; }
private:
std::string name;
QHash<std::string, const Decorator*> *route;
};
class Decorator {
public:
Decorator(std::string identifier);
PyObject* call(PyObject* func);
void invoke(boost::python::object&, boost::python::tuple) const;
static void commit(CppBase&);
private:
PyObject* method;
std::string identifier;
static std::hash<std::string, const Decorator*> routes;
};
Here is how I register my Python module
BOOST_PYTHON_MODULE(app)
{
boost::python::class_<CppBase, boost::noncopyable>("CppApp") //I tried to remove 'noncopyable', nothing change
;
boost::python::class_<Decorator, boost::noncopyable>("decorator", boost::python::init<std::string>())
.def("__repr__", &Decorator::repr)
.def("__call__", &Decorator::call)
;
}
Here is the implementation of CppBase::open that I think is the only one important to show in my class definition.
...
void CppBase::open(std::string id, boost::python::tuple arg /* unused so far */){
boost::python::call<void>(route->value(id), boost::ref(*this))
}
...
Here is the Python script sample, running with this example:
class MyScriptSubClass(CppApp):
def __init__(self, object_id):
CppBase.__init__(self)
self.object_id = object_id
#decorator('my_id_1')
def func1(self):
print("{}.func1".format(self.object_id))
Here is how I try to make everything work
//... Creating Python context, Executing the Script file...
boost::python::object cls(main_module.attr("MyScriptSubClass")); //Getting the classDefinition
CppBase core = boost::python::extract<CppBase>(cls()); //Instanciating the object with the previous catched definition
Decorator::commit(core); //Save all the decorator intercepted until now into the object
core.open('my_id_1'); //Calling the function matching with this id
I hope I made everything clear.
In advance, thank you.
The observer pattern appears frequently in my C++ project, which I now want to expose to the Python interpreter via Cython bindings. I tried to construct a minimal example illustrating the situation. A Spectacle accepts any object derived from the abstract base class Observer, such as an Onlooker. When we call Spectacle::event(), each registered observer is notified.
This is the content of the file ObserverPattern.h:
class Spectacle {
private:
std::vector<Observer*> observers;
public:
Spectacle() {};
virtual ~Spectacle() {};
virtual void registerObserver(Observer* observer) {
this->observers.push_back(observer);
}
virtual void event() {
std::cout << "event triggered" << std::endl;
for (Observer* observer : this->observers) {
observer->onEvent();
}
}
};
class Observer {
public:
Observer() {};
virtual ~Observer() {};
virtual void onEvent() = 0;
};
class Onlooker : public Observer {
public:
Onlooker() {};
virtual ~Onlooker() {};
virtual void onEvent() {
std::cout << "event observed" << std::endl;
}
};
And this is the content of my .pyx file, containing the bindings:
cdef extern from "ObserverPattern.h":
cdef cppclass _Spectacle "Spectacle":
_Spectacle() except +
void registerObserver(_Observer* observer)
void event()
cdef extern from "ObserverPattern.h":
cdef cppclass _Observer "Observer":
_Observer() except +
void onEvent()
cdef extern from "ObserverPattern.h":
cdef cppclass _Onlooker "Onlooker":
_Onlooker() except +
void onEvent()
cdef class Spectacle:
cdef _Spectacle _this
def event(self):
self._this.event()
def registerObserver(self, Observer observer):
self._this.registerObserver(observer._this)
cdef class Observer:
cdef _Observer* _this # must be a pointer because _Observer has pure virtual method
cdef class Onlooker(Observer):
pass # what should be the class body?
This does compile, but segfaults when event() is called and the observers are notified:
>>> spec = CythonMinimal.Spectacle()
>>> look = CythonMinimal.Onlooker()
>>> spec.registerObserver(look)
>>> spec.event()
event triggered
Segmentation fault: 11
What is the problem here and how could a fix look like?
Your problem is essentially "implement a C++ interface in Python".
The only portable way to do this is to write an actual C++ class
that will call back into Python.
Cython has undocumented experimental_cpp_class_def option that allows to
create C++ classes using Cython syntax. It's not pretty (IMO), but it works
for many scenarios.
Here is how you could implement Observer that delegates to the provided
Python callable:
from cpython.ref cimport PyObject, Py_INCREF, Py_DECREF
cdef cppclass ObserverImpl(_Observer):
PyObject* callback
__init__(object callback): # constructor. "this" argument is implicit.
Py_INCREF(callback)
this.callback = <PyObject*>callback
__dealloc__(): # destructor
Py_DECREF(<object>this.callback)
void onEvent():
(<object>this.callback)() # exceptions will be ignored
And that's how you could use it:
def registerObserver(self, callback not None): # user passes any Python callable
self._this.registerObserver(new ObserverImpl(callback))
C++ objects, just like C structures, can't hold Cython-managed object
references. That's why you have to use PyObject* field and manage reference
counting yourself. Inside methods you can, of course, cast to and use any Cython feature.
Another tricky moment is exception propagation. onEvent() method, being defined in C++, can't propagate Python exceptions. Cython will simply ignore exceptions it can't propagate. If you want to do better, catch them yourself and store somewhere for later inspection or rethrow as C++ exception. (I think it's impossible to throw C++ exceptions in Cython syntax, but you can call an external throwing helper function.)
If your observer has more than one method, then callback would be a Python class and instead of calling it directly you would call its methods, like (<object>this.callback).onEvent().
Obviously, ObserverImpl can also be coded directly in C++. Py_INCREF, Py_DECREF and PyObject_Call/PyObject_CallMethod are the only Python APIs necessary.
I'm porting a Python extension module written in C++ from Boost.Python to SWIG.
The C++ code defines an abstract class X with a static factory method
class X {
public:
static X* create(const char* descr);
...
};
The factory method returns a pointer to an instance of some derived class.
With Boost.Python you can wrap the C++ class X in a Python class X that has an
__init__(self, descr)
method that calls X::create. In fact, it is done as follows:
namespace bp = boost::python;
bp::class_<X>("X", boost::no_init)
.def("__init__", bp::make_constructor(&X::create))
...
Is there a way of doing the same thing with SWIG?
As suggested, it is better to use __new__ for controlling how a class is created. In SWIG, you should create an interface (.i) file that looks as follows:
%extend X {
static X * __new__(const char *desc) { return create(desc); }
};