I've embedded Python (3.6) code into my C++ application successfully. I use the Python/C API to call methods from this library. An overview of the python module is as follows:
class MyClass(object):
def __init__(args: str) -> None:
...
return
def callable_method(*args) -> Tuple:
...
return some_tuple
#staticmethod()
def create_from_cli_args(argv: List[str]) -> 'MyClass':
...
return MyClass(parsed_args)
The static method is a new addition to my code to move exception logic from the __init__ function to another function and yield an object. The C++ code to construct an object before was as follows:
PyObject *module_name = PyUnicode_FromString("module_path.my_class");
// Load the module object
PyObject *module = PyImport_Import(module_name);
if(module == nullptr) return;
Py_DECREF(module_name);
// Get callable list from module
PyObject *dict = PyModule_GetDict(module);
if(dict == nullptr) return;
Py_DECREF(module);
// Builds the name of a callable class
PyObject *my_class = PyDict_GetItemString(dict, "MyClass");
if(python_class == nullptr) return;
// Check if python class is callable
if(!PyCallable_Check(my_class)) return;
// Populate args for class instantiation
PyObject *py_args = PyTuple_New(1);
PyTuple_SetItem(py_args, 0, PyUnicode_FromString("Initialize_Configs"));
// Construct object
PyObject *my_class_obj = PyObject_CallObject(my_class, py_args);
if(my_class_obj == nullptr) return;
The above code snippet works, however, with the addition of the static method I'm trying to use to create an object of my_class, I'm unable to figure out how to call a static method of a class. I've tried using PyObject *my_class = PyDict_GetItemString(dict, "MyClass.create_my_class"); as well as PyObject_CallMethod(my_class, "create_my_class", kargs, kwargs) but that doesn't work either. I'd appreciate any help.
As the person in the comments suggested, I was incorrectly calling the static method. The solution is:
...
if(!PyCallable_Check(my_class)) return;
// A static method can be called the same as a normal method in python.
PyObject *my_class_object = PyObject_CallMethod(my_class, "create_from_cli_args", "s", "--cli_args");
if(!my_class_object) return;
Related
i have wrapped a simple c++ interface / class using pybind11
py::class_<IBaseObject, SmartPtr<IBaseObject>>(m, "BaseObject")
.def(py::init([]()
{
return BaseObject_Create();
}))
)
IBaseObject is interface, SmartPtr is custom holder type and BaseObject_Create is factory function which returns IBaseObject*.
Instatiating class from python works fine, however i also want to instantiate python wrapper class from C++ while passing IBaseObject* as parameter. Is this possible?
If you just want to instantiate your class in your C++ binding code, you can use pybinds Python C++ Interface:
https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html#calling-python-functions
Some ways to do this:
// Option 1: Use your existing class handle.
py::class_<IBaseObject, ...> py_cls(...);
py::object py_obj = py_cls();
// - Cast it if you need to.
auto* obj = py_obj.cast<IBaseObject*>();
// - or cast to `SmartPtr<IBaseObject>()` if its shareable.
// Option 2: Re-import elsewhere in your code.
py::object py_cls = py::module::import("my_pybind_module").attr("IBaseObject");
py::object py_obj = py_cls();
// - etc...
// Option 3: Use `eval` (blech)
py::module m = py::module::import("my_pybind_module");
py::object py_obj = py::eval("IBaseObject()", /* globals */ m.attr("__dict__"));
I'm working on a c++ application that uses pybind11 to embed python and I've run into a bit of a problem when trying to call an embedded function from a class method.
to start with here are my bindings:
#ifdef _DEBUG
#undef _DEBUG
#include <python.h>
#define _DEBUG
#else
#include <python.h>
#endif
#include <embed.h>
namespace py = pybind11;
using namespace py::literals;
void DebugInfo(std::string string_)
{
String LogMessage_(string_.c_str());
LOGINFO(LogMessage_);
}
PYBIND11_EMBEDDED_MODULE(Test, m) {
m.def("DebugInfo", &DebugInfo, "Posts message to DEBUGINFO");
}
I could then have a .py file with:
import Test
test.DebugInfo("I'm a lumberjack and that's OK")
and it will print just fine to debug
The trouble starts when I try and call it from within a class method.
import Test
class PyTest(object):
def __init__(self):
test.DebugInfo("I'm a lumberjack and that's OK")
test = PyTest()
when this runs it throws an exception against cast.h specifically against line 1985 which is part of this function:
template <return_value_policy policy>
class unpacking_collector {
public:
template <typename... Ts>
explicit unpacking_collector(Ts &&...values) {
// Tuples aren't (easily) resizable so a list is needed for collection,
// but the actual function call strictly requires a tuple.
auto args_list = list();
int _[] = { 0, (process(args_list, std::forward<Ts>(values)), 0)... };
ignore_unused(_);
m_args = std::move(args_list);
}
const tuple &args() const & { return m_args; }
const dict &kwargs() const & { return m_kwargs; }
tuple args() && { return std::move(m_args); }
dict kwargs() && { return std::move(m_kwargs); }
/// Call a Python function and pass the collected arguments
object call(PyObject *ptr) const {
PyObject *result = PyObject_Call(ptr, m_args.ptr(), m_kwargs.ptr());
if (!result)
throw error_already_set(); //EXCEPTION THROWS HERE!
return reinterpret_steal<object>(result);
}
And because it's probably relevant here's how I'm calling the whole thing from my main application
//Start the Python Interpreter
py::scoped_interpreter guard{};
//Python variables
py::object thing_;
std::string test_py = Import_File("test.py");
auto locals = py::dict();
py::exec(test_py, py::globals(), locals);
thing_ = locals["test"].cast<py::object>();
thing_.attr("start")();
and the contents of test.py
import Test
class PyTest(object):
def __init__(self, message = "Test Object initialized"):
self.message = message
iterstr = str(self.iter)
message = self.message + iterstr
self.iter = 0
Test.DebugInfo(message)
def start(self):
self.message = "Starting Python Object"
self.iter = self.iter + 1
iterstr = str(self.iter)
message = self.message + iterstr
Test.DebugInfo(message)
def update(self):
self.message = "Python Object Update Cycle:"
self.iter = self.iter + 1
iterstr = str(self.iter)
message = self.message + iterstr
Test.DebugInfo(message)
test = PyTest()
I'm not sure if I've run into a limitation of pybind11, a bug in it, or if I've just screwed the whole thing up.
Any insight would be greatly appreciated.
This was also filed as an issue with pybind11 here: https://github.com/pybind/pybind11/issues/1452
I came across both this SO and the issue, but I figured this out. Copying it here for anyone who stumbles across this first in the future
Basically, you don't actually want a blank py::dict for locals; it will cause all sorts of problems. If you look embedding sample code from the docs or the tests, the locals value always copies the global scope.
See:
* https://pybind11.readthedocs.io/en/stable/advanced/embedding.html
* https://github.com/pybind/pybind11/blob/master/tests/test_embed/test_interpreter.cpp#L57
Your options are to copy the global scope, or, in this case, simply don't pass in a locals
py::scoped_interpreter guard{};
auto globals = py::globals();
py::exec(test_py, globals);
thing_ = globals["Object"].cast<py::object>();
thing_.attr("start")();
It looks like that in the case of top-level code (not inside any module), the globals variable holds the values at this scope.
So after some experimentation I discovered that the problem is caused by pybind not being able to detect an imported module outside of the scope of the function.
import foo
def bar():
foo.func()
will always cause an error. However,
def bar():
import foo
foo.func()
will function as intended.
I am trying to pass back a string from a Java method called from C++. I am not able to find out what JNI function should I call to access the method and be returned a jstring value.
My code follows:
C++ part
main() {
jclass cls;
jmethodID mid;
jstring rv;
/** ... omitted code ... */
cls = env->FindClass("ClassifierWrapper");
mid = env->GetMethodID(cls, "getString","()Ljava/lang/String");
rv = env->CallStatic<TYPE>Method(cls, mid, 0);
const char *strReturn = env->GetStringUTFChars(env, rv, 0);
env->ReleaseStringUTFChars(rv, strReturn);
}
Java Code
public class ClassifierWrapper {
public String getString() { return "TEST";}
}
The Method Signature (from "javap -s Class")
public java.lang.String getString();
Signature: ()Ljava/lang/String;
You should have
cls = env->FindClass("ClassifierWrapper");
Then you need to invoke the constructor to get a new object:
jmethodID classifierConstructor = env->GetMethodID(cls,"<init>", "()V");
if (classifierConstructor == NULL) {
return NULL; /* exception thrown */
}
jobject classifierObj = env->NewObject( cls, classifierConstructor);
You are getting static method (even though the method name is wrong). But you need to get the instance method since getString() is not static.
jmethodID getStringMethod = env->GetMethodID(cls, "getString", "()Ljava/lang/String;");
Now invoke the method:
rv = env->CallObjectMethod(classifierObj, getStringMethod, 0);
const char *strReturn = env->GetStringUTFChars(env, rv, 0);
The complete working solution is as below:
Java Side
public class ClassifierWrapper {
public ClassifierWrapper(){}
public String getString() { return "TEST";}
}
Native Side
jclass cls;
jmethodID mid;
jstring rv;
cls = jniEnv->FindClass("ClassifierWrapper"); //plase also consider your package name as package\name\classname
jmethodID classifierConstructor = jniEnv->GetMethodID(cls,"<init>", "()V");
if (classifierConstructor == NULL) {
return NULL; /* exception thrown */
}
jobject classifierObj = jniEnv->NewObject( cls, classifierConstructor);
jmethodID getStringMethod = jniEnv->GetMethodID(cls, "getString", "()Ljava/lang/String;");
rv = (jstring)(jniEnv->CallObjectMethod(classifierObj, getStringMethod));
const char *strReturn = jniEnv->GetStringUTFChars( rv, 0);
jniEnv->ReleaseStringUTFChars(rv, strReturn);
The signature ()Ljava/lang/String is wrong, due that a class name into JVM must terminate with ;, then in this case signature must be ()Ljava/lang/String;
The first problem is that ClassifierWrapper.getString() is not static. You will need to make it static or instantiate ClassifierWrapper.
The second problem is that you are using GetMethodId instead of GetStaticMethodId.
To invoke a method that returns an Object (such as a String) you would call CallStaticObjectMethod(). That will return a jobject local reference to the String that the method returned. You can safely cast the jobject to a jstring (see http://java.sun.com/docs/books/jni/html/types.html) and use GetStringUTFChars to retrieve the characters and GetStringUTFLength to get the number of characters.
JNI is very tricky. You need to check the error code for everything (use ExceptionCheck() when there is no error code). If you don't check for errors it will fail silently in most cases and usually not at the point where the actual bug is.
You also need to understand the difference between local and global references (and what methods generate new references) in order to not leak memory and run into the reference limit. For instance, FindClass returns a local reference to a class object, but GetMethodId returns a MethodID.
Good luck
In my implementation for python integration into a c++ application, I am adding support for nodes that might or might not be valid. Internally these are stored as weak pointers, so I was thinking of having an isValid() method that users can use before calling the exposed methods. If they call an exposed method on an invalid node it would throw an exception.
However, I was wondering if it's possible to be a bit more pythonic than that. Is it possible to internally check whether the pointer is valid before calling the exposed method, and if it isn't making the python object None?
An example of what I want is here:
>>> my_valid_node = corelibrary.getNode("valid_node")
>>> my_valid_node.printName()
valid_node
Now however, something somewhere else in the system might invalidate the node, but from python's point of view, I want the node to become None.
>>> my_valid_node.printName()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'printName'
Can anyone think of a way to do this?
There is no clean way to make a reference to an object become a reference to None when an external event occurs. Nevertheless, when working towards a Pythonic interface, one could:
Implement the __nonzero__ method to allow the object to be evaluated in a boolean context.
Throw a Python exception when the weak_ptr fails to lock. One simple solution, would be to access a member attribute on a default constructed boost::python::object, as it references None.
Note that attribute lookup customization points, such as __getattr__, will not be sufficient enough as the object pointed to by weak_ptr may expire between attribute access and dispatch to the C++ member functions.
Here is a complete minimal example based on the above details. In this example, spam and spam_factory, a factory that instantiates spam objects managed by shared_ptr, are considered legacy types. A spam_proxy auxiliary class that references spam via weak_ptr and additionally auxiliary functions help adapt the legacy types to Python.
#include <string>
#include <boost/make_shared.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/smart_ptr/weak_ptr.hpp>
#include <boost/python.hpp>
/// Assume legacy APIs.
// Mockup class containing data.
class spam
{
public:
explicit spam(const char* name)
: name_(name)
{}
std::string name() { return name_; }
private:
std::string name_;
};
// Factory for creating and destroying the mockup class.
class spam_factory
{
public:
boost::shared_ptr<spam> create(const char* name)
{
instance_ = boost::make_shared<spam>(name);
return instance_;
}
void destroy()
{
instance_.reset();
}
private:
boost::shared_ptr<spam> instance_;
};
/// Auxiliary classes and functions to help obtain Pythonic semantics.
// Helper function used to cause a Python AttributeError exception to
// be thrown on None.
void throw_none_has_no_attribute(const char* attr)
{
// Attempt to extract the specified attribute on a None object.
namespace python = boost::python;
python::object none;
python::extract<python::object>(none.attr(attr))();
}
// Mockup proxy that has weak-ownership.
class spam_proxy
{
public:
explicit spam_proxy(const boost::shared_ptr<spam>& impl)
: impl_(impl)
{}
std::string name() const { return lock("name")->name(); }
bool is_valid() const { return !impl_.expired(); }
boost::shared_ptr<spam> lock(const char* attr) const
{
// Attempt to obtain a shared pointer from the weak pointer.
boost::shared_ptr<spam> impl = impl_.lock();
// If the objects lifetime has ended, then throw.
if (!impl) throw_none_has_no_attribute(attr);
return impl;
}
private:
boost::weak_ptr<spam> impl_;
};
// Use a factory to create a spam instance, but wrap it in the proxy.
spam_proxy spam_factory_create(
spam_factory& self,
const char* name)
{
return spam_proxy(self.create(name));
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose the proxy class as if it was the actual class.
python::class_<spam_proxy>("Spam", python::no_init)
.def("__nonzero__", &spam_proxy::is_valid)
.add_property("name", &spam_proxy::name)
;
python::class_<spam_factory>("SpamFactory")
.def("create", &spam_factory_create) // expose auxiliary method
.def("destroy", &spam_factory::destroy)
;
}
Interactive usage:
>>> import example
>>> factory = example.SpamFactory()
>>> spam = factory.create("test")
>>> assert(spam.name == "test")
>>> assert(bool(spam) == True)
>>> if spam:
... assert(bool(spam) == True)
... factory.destroy() # Maybe occurring from a C++ thread.
... assert(bool(spam) == False) # Confusing semantics.
... assert(spam.name == "test") # Confusing exception.
...
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
AttributeError: 'NoneType' object has no attribute 'name'
>>> assert(spam is not None) # Confusing type.
One could argue that while the interface is Pythonic, the object's semantics are not. With weak_ptr semantics not being too common in Python, one generally does not expect the object referenced by a local variable to be destructed. If weak_ptr semantics are necessary, then consider introducing a means to allow the user to obtain shared ownership within a specific context via the context manager protocol. For example, the following pattern allows an object's validity to be checked once, then be guaranteed within a limited scope:
>>> with spam: # Attempt to acquire shared ownership.
... if spam: # Verify ownership was obtained.
... spam.name # Valid within the context's scope.
... factory.destroy() # spam is still valid.
... spam.name # Still valid.
... # spam destroyed once context's scope is exited.
Here is a complete extension of the previous example, where in the spam_proxy implements the context manager protocol:
#include <string>
#include <boost/make_shared.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/smart_ptr/weak_ptr.hpp>
#include <boost/python.hpp>
/// Assume legacy APIs.
// Mockup class containing data.
class spam
{
public:
explicit spam(const char* name)
: name_(name)
{}
std::string name() { return name_; }
private:
std::string name_;
};
// Factory for creating and destroying the mockup class.
class spam_factory
{
public:
boost::shared_ptr<spam> create(const char* name)
{
instance_ = boost::make_shared<spam>(name);
return instance_;
}
void destroy()
{
instance_.reset();
}
private:
boost::shared_ptr<spam> instance_;
};
/// Auxiliary classes and functions to help obtain Pythonic semantics.
// Helper function used to cause a Python AttributeError exception to
// be thrown on None.
void throw_none_has_no_attribute(const char* attr)
{
// Attempt to extract the specified attribute on a None object.
namespace python = boost::python;
python::object none;
python::extract<python::object>(none.attr(attr))();
}
// Mockup proxy that has weak-ownership and optional shared ownership.
class spam_proxy
{
public:
explicit spam_proxy(const boost::shared_ptr<spam>& impl)
: shared_impl_(),
impl_(impl)
{}
std::string name() const { return lock("name")->name(); }
bool is_valid() const { return !impl_.expired(); }
boost::shared_ptr<spam> lock(const char* attr) const
{
// If shared ownership exists, return it.
if (shared_impl_) return shared_impl_;
// Attempt to obtain a shared pointer from the weak pointer.
boost::shared_ptr<spam> impl = impl_.lock();
// If the objects lifetime has ended, then throw.
if (!impl) throw_none_has_no_attribute(attr);
return impl;
}
void enter()
{
// Upon entering the runtime context, guarantee the lifetime of the
// object remains until the runtime context exits if the object is
// alive during this call.
shared_impl_ = impl_.lock();
}
bool exit(boost::python::object type,
boost::python::object value,
boost::python::object traceback)
{
shared_impl_.reset();
return false; // Do not suppress the exception.
}
private:
boost::shared_ptr<spam> shared_impl_;
boost::weak_ptr<spam> impl_;
};
// Use a factory to create a spam instance, but wrap it in the proxy.
spam_proxy spam_factory_create(
spam_factory& self,
const char* name)
{
return spam_proxy(self.create(name));
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose the proxy class as if it was the actual class.
python::class_<spam_proxy>("Spam", python::no_init)
.def("__nonzero__", &spam_proxy::is_valid)
// Support context manager protocol.
.def("__enter__", &spam_proxy::enter)
.def("__exit__", &spam_proxy::exit)
.add_property("name", &spam_proxy::name)
;
python::class_<spam_factory>("SpamFactory")
.def("create", &spam_factory_create) // expose auxiliary method
.def("destroy", &spam_factory::destroy)
;
}
Interactive usage:
>>> import example
>>> factory = example.SpamFactory()
>>> spam = factory.create("test")
>>> with spam:
... assert(bool(spam) == True)
... if spam:
... assert(spam.name == "test")
... factory.destroy()
... assert(bool(spam) == True)
... assert(spam.name == "test")
...
>>> assert(bool(spam) == False)
The exact pattern may not be the most Pythonic, but it provides a clean way to guarantee the object's lifetime within a limited scope.
We are trying to embed several Python procedures in our C++ code using and it fails with an error
TypeError: No to_python (by_value) converter found for C++ type: boost::python::detail::kwds_proxy
We honestly studied all the examples we managed to find over the network (this and this), but still we have no any clear solution for passing the ****kwargs** variable from C++ to Python. This failure seems to be very rare.
Python function we are trying to call recieves a string value and a dictionary:
from ipalib import api
user_kw = dict(givenname=u'Test', sn=u'User')
api.Command.user_add.(u'Pythonist', **user_kw)
This is its C++ implementation:
//Importing modules
bp::import("__main__");
ipalib = bp::import("ipalib");
ipalib_namespace = ipalib.attr("__dict__");
api = ipalib.attr("api");
... //Starting Python environment
//Construct args
std::string name = "Pythonist";
bp::dict new_user;
new_user["givenname"] = "Test";
new_user["sn"] = "User";
//Calling the function
bp::object user_add_wrapper = api.attr("Command").attr("user_add");
user_add_wrapper(name, new_user);
And on the last line Boost throws an exception. What are we doing wrong? Thank you.
user_add_wrapper(name, new_user) tries to pass new_user as a dictionary to user_add_wrapper(), rather than passing the unpacked contents of new_user. The new_user dictionary needs to be unpacked. Additionally, calling the python::object with an unpacked dictionary requires the first argument to be an unpacked boost::python::tuple. To account for these requirements, invoke user_add_wrapper() as follows:
user_add_wrapper(*bp::make_tuple(name), **new_user);
This behavior is part of the seamless interoperability provided by Boost.Python, but it is rather obscure and I only recall noticing it mentioned indirectly in the change log rather than the tutorial or reference.
Below is a complete minimal example. Given example.py:
def user_add(name, givenname, sn):
print locals()
The following program invokes user_add() by unpacking a dictionary:
#include <boost/python.hpp>
int main()
{
Py_Initialize(); // Start interpreter.
// Create the __main__ module.
namespace python = boost::python;
python::object main = python::import("__main__");
python::object main_namespace = main.attr("__dict__");
try
{
python::object example = python::import("example");
python::object example_namespace = example.attr("__dict__");
// Construct args.
std::string name = "Pythonist";
python::dict new_user;
new_user["givenname"] = "Test";
new_user["sn"] = "User";
// Invoke user_add by unpacking the new_user dictionary.
example_namespace["user_add"](*python::make_tuple(name), **new_user);
}
catch (const python::error_already_set&)
{
PyErr_Print();
}
}
And produces the following output:
{'givenname': 'Test', 'sn': 'User', 'name': 'Pythonist'}