I have an array of objects in Python
[obj1, obj2, obj3]
and I want to pass them to off to a C++ function to perform some computation. I'm using SWIG to write my interface. The class type of the passed object is already defined in C++.
What's the best way to do this?
It depends on if your function is already written and cannot be changed, in which case you may need to check Swig docs to see if there is already a typemap from PyList to std::vector (I think there is). If not, taking PyObject* as the argument to the function and using the Python C API for manipulating lists should work fine. I haven't had any problems with it so far. For self-documentation, I recommend typedef'ing PyObject* to some kind of expected type, like "PythonList" so that the parameters have some meaning.
This may also be useful:
How to expose std::vector<int> as a Python list using SWIG?
Related
I wish to pass a Python numpy array to this C++ function:
int data(std::span<int> x) { return x.size(); }
Is it possible to write a custom type caster for pybind11 that supports this signature?
pybind already has good support for vector<int>, but I would like to add support for a zero-copy version of vector, i.e. pass by reference.
It looks like there are some clues on how to do this here.
There's two questions here, so I'll answer them each individually:
It is certainly possible to write a custom type caster for std::span -- my own implementation based off the pybind11 type caster for abseil's span can be found here. I haven't gotten around to submitting a patch upstream yet, but someone else is also interested in this (see github discussion).
Most of the pybind11 STL type casters make copies of their sequences, so it's likely that any future std::span casters (like mine) are also going to make copies. That doesn't mean it's impossible -- the abseil span implementation does have a specialization where they would convert a span to a python buffer if it was a basic type.
It's easy to do something like that in Python, but implementing it in C++ seems to be more challenging.
I actually have some solution to this, but I'd like to see if you can see any better solution.
Here's what I want to do.
I have a list of values of different types (string, integer, can be also instance of some class etc.). Now here's the first problem - in C++ (unlike in Python) all values in vector/array have to be of the same type.
The solution I can see is that I can use std::any like this: vector<std::any> list.
I also have an array/vector of functions (or pointers to functions) with different parameter types and returned values - one function can accept string and integer and return a char and other can accept a char and return an int. Here's another problem: in C++ you can have an array/vector of functions only if they have the same parameters and returned values (as far as I know) because in your declaration of the vector you need to define the parameter types and the returned value.
The other problem is that I need to retrieve the information about the parameters and the returned value for each function. In other words, having those functions, I need to know that this function accepts 2 strings and 1 integer and returns a char for example. In Python I can use inspect.signature function to retrieve information about type annotations of a function. In C++, I don't know if there is a way to do this.
The solution I can see here is to use std::any again (although I will use another solution, I will explain why later).
The solution I can see to this problem is that I won't retrieve that information but instead the user of the class which accepts this vector of functions will simply have to specify what are the parameter types and returned value for each function. In other words, the solution I can see is that I won't be retrieving the information about parameter types programmatically.
The other problem I have is that later I need to call one of those functions with some parameters. In Python I do this like this:
arguments = [1, 'str', some_object] // here I prepare a list of arguments (they are of different types)
func(**arguments)
In C++ I can do unpacking as well, but not if the parameters are of different types.
The solution I can see here is as follows. Those functions in the vector will all accepts only argument which is vector<std::any> args which will simply contain all of the arguments. Later when I want to call the function, I will simply construct a vector with std::any values and pass it as an argument. This would also solve the previous problem of not being able to store vector of functions with different parameters.
Can you see better solutions?
You might wonder what I need all of this is for. I do some program synthesis stuff and I need to programmatically construct programs from existing functions. I'm writing a library and I want the user of my library to be able to specify those base functions out of which I construct programs. In order to do what I want, I need to know what are the parameters and returned values of those functions and I need to call them later.
I believe what you are looking for is std::apply. You can use std::tuple instead of std::vector to store a list of values of different types -- as long as the types are known at compile-time. Then std::apply(f, t) in C++ is basically the same as f(*t) in Python.
I have a list of values of different types (string, integer, can be also instance of some class etc.).
A type which is a union of subtypes is called a sum type or tagged union. C++ has the template std::variant for that.
Now here's the first problem - in C++ (unlike in Python) all values in vector/array have to be of the same type.
Of course, so use cleverly C++ containers. You might want some std::map or std::vector of your particular instance of std::variant.
I also have an array/vector of functions
You probably want some std::vector of std::function-s and code with C++ lambda expressions
You should read a good C++ programming book
I'm writing a library and I want the user of my library to be able to specify those base functions out of which I construct programs.
You could get inspiration from SWIG and consider generating some C++ code in your library. So write (in Python or C++) your C++ metaprogram (generating some C++ code, like ANTLR does) which generates the user code, and your user would adapt his build automation tool for such a need (like users of GNU bison do).
You might also consider embedding Guile (or Lua) in your application.
PS. You might be interested by other programming languages like Ocaml, Go, Scheme (with Guile, and read SICP), Common Lisp (with SBCL), or Rust.
To be able to pass vectors of a custom type by reference between Python and C++, my project uses PYBIND11_MAKE_OPAQUE and pybind11::bind_vector<> on vectors of my type. However, I also need to work with a vector of vector of my custom type. Below is an example.
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <vector>
class Example {};
PYBIND11_MAKE_OPAQUE(std::vector<Example>);
PYBIND11_MAKE_OPAQUE(std::vector<std::vector<Example>>);
PYBIND11_MODULE(ExModule, m)
{
pybind11::class_<Example>(m, "Example")
.def(pybind11::init<>());
pybind11::bind_vector<std::vector<Example>>(m, "ExampleVector");
pybind11::bind_vector<std::vector<std::vector<Example>>>(m, "Example2DVector");
}
If I create a 2D vector of my type in Python, then try to access it, I get an error. Below is the example Python code.
from ExModule import Example, ExampleVector, Example2DVector
# a is a 10x10 vector of Examples
a = Example2DVector([ExampleVector([Example() for i in range(10)]) for i in range(10)])
b = a[4]
Error message:
TypeError: Unable to convert function return value to a Python type! The signature was
(self: ExModule.Example2DVector, arg0: int) -> std::vector<Example, std::allocator<Example> >
This seems to be because the return type on the index operation on the 2D vector type is an opaque type. What I think should be happening is the return should be constructed into a ExampleVector. I can't do that from Python, it must be done in the module wrapper. Is this a bug or missing feature? If not, how do I overload the index operator on the Example2DVector class?
The example works, but my code did not. This was due to the fact that in my code the 1D vector class was bound to a Python type in a different pybind11 module. So there was no mapped Python type for the 1D vector C++ type when it defined the __getitem__ for the 2D vector class. However, I think that if I imported the module containing the 1D vector Python type bind later that it should work, but it does not. That might be a bug.
EDIT:
This behavior is not a bug (I figured, Jakob seems like a pretty smart guy). As discussed in the manual on binding STL containers, there is a section on "module local" bindings. By default type bindings are local to the module they are defined in to avoid multiple different bindings of the same type.
However, our project contains a "datatypes" module, and many modules that use those types. In this case we do not want datatypes defined in the "datatypes" module to be module local. Otherwise we end up with the given problem of return values not being converted into the correct Python type.
We can turn off the default module local binding in the binding definition. Using the question's example, we can turn off the module local binding for ExampleVector and accesses to an Example2DVector (defined in another module) will no longer fail.
pybind11::bind_vector<std::vector<Example>>(m, "ExampleVector", pybind11::module_local(false));
Quote form docs:
This macro must be specified at the top level (and outside of any namespaces), since it instantiates a partial template overload. If your binding code consists of multiple compilation units, it must be present in every file (typically via a common header) preceding any usage of std::vector. Opaque types must also have a corresponding class_ declaration to associate them with a name in Python, and to define a set of available operations
#ktb, It's not a bug, please see https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html?highlight=compilation%20unit#making-opaque-types
I'm wrapping some C++ code with SWIG to Python, e.g. member functions like
Leek::update(const std::vector<double>& v);
In Python, I would like to use this function with both sequences and/or Numpy arrays. I understand that I could write my own %typemap(in) that checks and converts those types.
However, SWIG already offers a typemap for std::vectors to sequences, is there a way to 'extend' the existing typemap with a typemap for Numpy arrays, such that it checks and converts both types?
Until now, I've only been able to overwrite the built-in typemap, which is undesirable.
I've begun running into slight issues with certain functions that I've bound to lua using luabind. These functions return or take std:string objects as arguments, and the data that c++ eventually gets is garbage, apparently because lua doesn't recognize the datatype. I'm fine with manually binding std:: classes over as I need them, but I have to wonder if that's already been done for the c standard library anywhere. I can't seem to find any sort of bindings library; does anyone know if such a project exists?
Luabind already has automatic conversions between Lua strings and std::string (as well as char*). So you should be able to use them as parameters and/or return values just fine. If you are unable to do so, then something is going wrong, either on your side or on Luabind's side.
Luabind also has an adapter to convert a return type that conforms to an STL-style container into a Lua iterator function. Thus, you can use it in a for-loop like this:
for object in CppFuncThatReturnsStdVector() do
--Do something with "object"
end
Other than these, I am not aware of any particular effort to make a Luabind library that directly exposes the standard C++ library to Lua. It probably wouldn't be a good idea anyway; Lua tables cover most of the needs you might have for STL-style containers. And most of the other stuff are things Lua can handle with its own standard library.