I'm working on a c++ application that uses pybind11 to embed python and I've run into a bit of a problem when trying to call an embedded function from a class method.
to start with here are my bindings:
#ifdef _DEBUG
#undef _DEBUG
#include <python.h>
#define _DEBUG
#else
#include <python.h>
#endif
#include <embed.h>
namespace py = pybind11;
using namespace py::literals;
void DebugInfo(std::string string_)
{
String LogMessage_(string_.c_str());
LOGINFO(LogMessage_);
}
PYBIND11_EMBEDDED_MODULE(Test, m) {
m.def("DebugInfo", &DebugInfo, "Posts message to DEBUGINFO");
}
I could then have a .py file with:
import Test
test.DebugInfo("I'm a lumberjack and that's OK")
and it will print just fine to debug
The trouble starts when I try and call it from within a class method.
import Test
class PyTest(object):
def __init__(self):
test.DebugInfo("I'm a lumberjack and that's OK")
test = PyTest()
when this runs it throws an exception against cast.h specifically against line 1985 which is part of this function:
template <return_value_policy policy>
class unpacking_collector {
public:
template <typename... Ts>
explicit unpacking_collector(Ts &&...values) {
// Tuples aren't (easily) resizable so a list is needed for collection,
// but the actual function call strictly requires a tuple.
auto args_list = list();
int _[] = { 0, (process(args_list, std::forward<Ts>(values)), 0)... };
ignore_unused(_);
m_args = std::move(args_list);
}
const tuple &args() const & { return m_args; }
const dict &kwargs() const & { return m_kwargs; }
tuple args() && { return std::move(m_args); }
dict kwargs() && { return std::move(m_kwargs); }
/// Call a Python function and pass the collected arguments
object call(PyObject *ptr) const {
PyObject *result = PyObject_Call(ptr, m_args.ptr(), m_kwargs.ptr());
if (!result)
throw error_already_set(); //EXCEPTION THROWS HERE!
return reinterpret_steal<object>(result);
}
And because it's probably relevant here's how I'm calling the whole thing from my main application
//Start the Python Interpreter
py::scoped_interpreter guard{};
//Python variables
py::object thing_;
std::string test_py = Import_File("test.py");
auto locals = py::dict();
py::exec(test_py, py::globals(), locals);
thing_ = locals["test"].cast<py::object>();
thing_.attr("start")();
and the contents of test.py
import Test
class PyTest(object):
def __init__(self, message = "Test Object initialized"):
self.message = message
iterstr = str(self.iter)
message = self.message + iterstr
self.iter = 0
Test.DebugInfo(message)
def start(self):
self.message = "Starting Python Object"
self.iter = self.iter + 1
iterstr = str(self.iter)
message = self.message + iterstr
Test.DebugInfo(message)
def update(self):
self.message = "Python Object Update Cycle:"
self.iter = self.iter + 1
iterstr = str(self.iter)
message = self.message + iterstr
Test.DebugInfo(message)
test = PyTest()
I'm not sure if I've run into a limitation of pybind11, a bug in it, or if I've just screwed the whole thing up.
Any insight would be greatly appreciated.
This was also filed as an issue with pybind11 here: https://github.com/pybind/pybind11/issues/1452
I came across both this SO and the issue, but I figured this out. Copying it here for anyone who stumbles across this first in the future
Basically, you don't actually want a blank py::dict for locals; it will cause all sorts of problems. If you look embedding sample code from the docs or the tests, the locals value always copies the global scope.
See:
* https://pybind11.readthedocs.io/en/stable/advanced/embedding.html
* https://github.com/pybind/pybind11/blob/master/tests/test_embed/test_interpreter.cpp#L57
Your options are to copy the global scope, or, in this case, simply don't pass in a locals
py::scoped_interpreter guard{};
auto globals = py::globals();
py::exec(test_py, globals);
thing_ = globals["Object"].cast<py::object>();
thing_.attr("start")();
It looks like that in the case of top-level code (not inside any module), the globals variable holds the values at this scope.
So after some experimentation I discovered that the problem is caused by pybind not being able to detect an imported module outside of the scope of the function.
import foo
def bar():
foo.func()
will always cause an error. However,
def bar():
import foo
foo.func()
will function as intended.
Related
I've embedded Python (3.6) code into my C++ application successfully. I use the Python/C API to call methods from this library. An overview of the python module is as follows:
class MyClass(object):
def __init__(args: str) -> None:
...
return
def callable_method(*args) -> Tuple:
...
return some_tuple
#staticmethod()
def create_from_cli_args(argv: List[str]) -> 'MyClass':
...
return MyClass(parsed_args)
The static method is a new addition to my code to move exception logic from the __init__ function to another function and yield an object. The C++ code to construct an object before was as follows:
PyObject *module_name = PyUnicode_FromString("module_path.my_class");
// Load the module object
PyObject *module = PyImport_Import(module_name);
if(module == nullptr) return;
Py_DECREF(module_name);
// Get callable list from module
PyObject *dict = PyModule_GetDict(module);
if(dict == nullptr) return;
Py_DECREF(module);
// Builds the name of a callable class
PyObject *my_class = PyDict_GetItemString(dict, "MyClass");
if(python_class == nullptr) return;
// Check if python class is callable
if(!PyCallable_Check(my_class)) return;
// Populate args for class instantiation
PyObject *py_args = PyTuple_New(1);
PyTuple_SetItem(py_args, 0, PyUnicode_FromString("Initialize_Configs"));
// Construct object
PyObject *my_class_obj = PyObject_CallObject(my_class, py_args);
if(my_class_obj == nullptr) return;
The above code snippet works, however, with the addition of the static method I'm trying to use to create an object of my_class, I'm unable to figure out how to call a static method of a class. I've tried using PyObject *my_class = PyDict_GetItemString(dict, "MyClass.create_my_class"); as well as PyObject_CallMethod(my_class, "create_my_class", kargs, kwargs) but that doesn't work either. I'd appreciate any help.
As the person in the comments suggested, I was incorrectly calling the static method. The solution is:
...
if(!PyCallable_Check(my_class)) return;
// A static method can be called the same as a normal method in python.
PyObject *my_class_object = PyObject_CallMethod(my_class, "create_from_cli_args", "s", "--cli_args");
if(!my_class_object) return;
I'm trying to use Python embedded in C++ with Boost::python.
My embedded script are supposed to use decorator to register their methods like following:
class Test:
def __init__(self, object_id):
self.object_id = object_id
#decorator('my_id_1')
def func1(self):
print("{}.func1".format(self.object_id))
decorator is declared on the C++ side, defining the method the __init__ and __call__. Everything works has expected, until the call of the method, which lead to SIGSEGV or SIGARBT.
Here is an example of what I would like to do in Python:
#CPP side
item = {}
class decorator:
def __init__(self, _id):
self._id = _id
def __call__(self, func):
item[self._id] = func #saved as PyObject* in CPP
print func
return func
#Script side
class Test(CppBase):
def __init__(self, object_id):
CppBase.__init__(self)
self.object_id = object_id
#decorator('my_id_1')
def func1(self):
print("{}.func1".format(self.object_id))
#decorator('my_id_2')
def func2(self):
print("{}.func2".format(self.object_id))
#CPP side
obj1 = Test("obj1")
item['my_id_1'](obj1) #The crash append here
To do the call, I'm using the following function: boost::python::call<void>(my_PyObject_func, boost::ref(my_obj_instance))
I won't put my all C++ code because I'm actually updating a working project made from the old Python C API, and the whole API is quite huge. However, if you think I forgot some significant part of it, just tell me, and I will post those parts. Furthermore, I removed a lot of simple check such as being sure that the global Python var contain my object, no python error happened or the hash contain the requested id, to make the code lighter.
Here are my C++ Object definition
class CppBase: public boost::python::object {
public:
CppBase();
void open(std::string, boost::python::tuple arg);
void setRoutes(const std::hash<std::string, const Route*>&);
inline std::hash<std::string, const Route*>*const routes() const { return route; }
private:
std::string name;
QHash<std::string, const Decorator*> *route;
};
class Decorator {
public:
Decorator(std::string identifier);
PyObject* call(PyObject* func);
void invoke(boost::python::object&, boost::python::tuple) const;
static void commit(CppBase&);
private:
PyObject* method;
std::string identifier;
static std::hash<std::string, const Decorator*> routes;
};
Here is how I register my Python module
BOOST_PYTHON_MODULE(app)
{
boost::python::class_<CppBase, boost::noncopyable>("CppApp") //I tried to remove 'noncopyable', nothing change
;
boost::python::class_<Decorator, boost::noncopyable>("decorator", boost::python::init<std::string>())
.def("__repr__", &Decorator::repr)
.def("__call__", &Decorator::call)
;
}
Here is the implementation of CppBase::open that I think is the only one important to show in my class definition.
...
void CppBase::open(std::string id, boost::python::tuple arg /* unused so far */){
boost::python::call<void>(route->value(id), boost::ref(*this))
}
...
Here is the Python script sample, running with this example:
class MyScriptSubClass(CppApp):
def __init__(self, object_id):
CppBase.__init__(self)
self.object_id = object_id
#decorator('my_id_1')
def func1(self):
print("{}.func1".format(self.object_id))
Here is how I try to make everything work
//... Creating Python context, Executing the Script file...
boost::python::object cls(main_module.attr("MyScriptSubClass")); //Getting the classDefinition
CppBase core = boost::python::extract<CppBase>(cls()); //Instanciating the object with the previous catched definition
Decorator::commit(core); //Save all the decorator intercepted until now into the object
core.open('my_id_1'); //Calling the function matching with this id
I hope I made everything clear.
In advance, thank you.
I have a groovy script defined, which includes an inner class
foo.groovy:
// does something
Bar {
int foobar() {
return 1
}
}
// Does something else
Why I have the class defined in the script is a longer story, but it's necessary, unless I want to make a big redesign of the project structure (And the customer doesn't want to pay for this time)
Anyway, I have a GroovyTestCase, where I want to call this class from.
I want to do something like this:
class Test extends GroovyTestCase {
void testSomething() {
Bar bar = new Bar()
assertTrue(1, bar.foobar())
}
}
Is it possible to reference a class defined in a groovy script? If so, how?
foo.groovy
println "foooo 01"
class Bar {
int foobar() {
return 1
}
}
println "foooo 02"
Test.groovy (you can make it as a class)
def shell = new GroovyShell()
//parse script
def script = shell.parse(new File("/11/foo.groovy"))
//get class loader used for script parsing
def cl = shell.getClassLoader()
//get list of parsed/loaded classes
println cl.getLoadedClasses() // [class Bar, class foo]
def bar = cl.loadClass('Bar').newInstance()
println bar.foobar() // 1
but beware! if your foo.groovy script loaded some other classes that you want to access in testcase, you have to do it through the same shell classloader, otherwise there could be strange errors like could not assign String to String...
We are trying to embed several Python procedures in our C++ code using and it fails with an error
TypeError: No to_python (by_value) converter found for C++ type: boost::python::detail::kwds_proxy
We honestly studied all the examples we managed to find over the network (this and this), but still we have no any clear solution for passing the ****kwargs** variable from C++ to Python. This failure seems to be very rare.
Python function we are trying to call recieves a string value and a dictionary:
from ipalib import api
user_kw = dict(givenname=u'Test', sn=u'User')
api.Command.user_add.(u'Pythonist', **user_kw)
This is its C++ implementation:
//Importing modules
bp::import("__main__");
ipalib = bp::import("ipalib");
ipalib_namespace = ipalib.attr("__dict__");
api = ipalib.attr("api");
... //Starting Python environment
//Construct args
std::string name = "Pythonist";
bp::dict new_user;
new_user["givenname"] = "Test";
new_user["sn"] = "User";
//Calling the function
bp::object user_add_wrapper = api.attr("Command").attr("user_add");
user_add_wrapper(name, new_user);
And on the last line Boost throws an exception. What are we doing wrong? Thank you.
user_add_wrapper(name, new_user) tries to pass new_user as a dictionary to user_add_wrapper(), rather than passing the unpacked contents of new_user. The new_user dictionary needs to be unpacked. Additionally, calling the python::object with an unpacked dictionary requires the first argument to be an unpacked boost::python::tuple. To account for these requirements, invoke user_add_wrapper() as follows:
user_add_wrapper(*bp::make_tuple(name), **new_user);
This behavior is part of the seamless interoperability provided by Boost.Python, but it is rather obscure and I only recall noticing it mentioned indirectly in the change log rather than the tutorial or reference.
Below is a complete minimal example. Given example.py:
def user_add(name, givenname, sn):
print locals()
The following program invokes user_add() by unpacking a dictionary:
#include <boost/python.hpp>
int main()
{
Py_Initialize(); // Start interpreter.
// Create the __main__ module.
namespace python = boost::python;
python::object main = python::import("__main__");
python::object main_namespace = main.attr("__dict__");
try
{
python::object example = python::import("example");
python::object example_namespace = example.attr("__dict__");
// Construct args.
std::string name = "Pythonist";
python::dict new_user;
new_user["givenname"] = "Test";
new_user["sn"] = "User";
// Invoke user_add by unpacking the new_user dictionary.
example_namespace["user_add"](*python::make_tuple(name), **new_user);
}
catch (const python::error_already_set&)
{
PyErr_Print();
}
}
And produces the following output:
{'givenname': 'Test', 'sn': 'User', 'name': 'Pythonist'}
I have C++ classes in following format (copying just important parts):
class my_stringimpl {
public:
static sample_string* create(const char* str, int len) {
my_stringimpl* sample = static_cast<my_stringimpl*>(malloc(sizeof(my_stringimpl) + len*sizeof(char)));
char* data_ptr = reinterpret_cast<char*>(sample+1);
memset(data_ptr, 0, len);
memcpy(data_ptr, str, len);
return new (sample) my_stringimpl(len);
}
private:
int m_length;
};
class my_string {
public:
my_string(const char* str, int len)
: m_impl(my_stringimpl::create(str, len)) { }
~my_string() {
delete m_impl;
}
private:
my_stringimpl* m_impl;
};
For this my_string class I am adding pretty printer. I added the following defs in a python script (which I am including in my .gdbinit file) - just func defs copied here:
def string_to_char(ptr, length=None):
error_message = ''
if length is None:
length = int(0)
error_message = 'null string'
else:
length = int(length)
string = ''.join([chr((ptr + i).dereference()) for i in xrange(length)])
return string + error_message
class StringPrinter(object):
def __init__(self, val):
self.val = val
class StringImplPrinter(StringPrinter):
def get_length(self):
return self.val['m_length']
def get_data_address(self):
return self.val.address + 1
def to_string(self):
return string_to_char(self.get_data_address(), self.get_length())
class MyStringPrinter(StringPrinter):
def stringimpl_ptr(self):
return self.val['m_impl']
def get_length(self):
if not self.stringimpl_ptr():
return 0
return StringImplPrinter(self.stringimpl_ptr().dereference()).get_length()
def to_string(self):
if not self.stringimpl_ptr():
return '(null)'
return StringImplPrinter(self.stringimpl_ptr().dereference()).to_string()
But, on usage I am getting the below error -
Python Exception <class 'gdb.error'> Cannot convert value to int.:
If I try to change the value in 'ptr' to int and then do the arthimetic before casting back to char (like in def above), it gives below error:
Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'cast':
Can anybody tell what is that I am doing wrong? I am really struck here. :(. In nutshell, I am trying to achieve the following c/c++ expr equivalent,
*(char*){hex_address}
in python. How can I do it?
It's better to post full stack traces or at least to indicate exactly which lines throws the exception. Python provides this...
Your string_to_char function can be replaced by Value.string or Value.lazy_string, which were designed for exactly this use. I imagine the error is coming from there; in which case this ought to remove it.
Also, your printers should implement "hint" methods that return "string".