To be able to pass vectors of a custom type by reference between Python and C++, my project uses PYBIND11_MAKE_OPAQUE and pybind11::bind_vector<> on vectors of my type. However, I also need to work with a vector of vector of my custom type. Below is an example.
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <vector>
class Example {};
PYBIND11_MAKE_OPAQUE(std::vector<Example>);
PYBIND11_MAKE_OPAQUE(std::vector<std::vector<Example>>);
PYBIND11_MODULE(ExModule, m)
{
pybind11::class_<Example>(m, "Example")
.def(pybind11::init<>());
pybind11::bind_vector<std::vector<Example>>(m, "ExampleVector");
pybind11::bind_vector<std::vector<std::vector<Example>>>(m, "Example2DVector");
}
If I create a 2D vector of my type in Python, then try to access it, I get an error. Below is the example Python code.
from ExModule import Example, ExampleVector, Example2DVector
# a is a 10x10 vector of Examples
a = Example2DVector([ExampleVector([Example() for i in range(10)]) for i in range(10)])
b = a[4]
Error message:
TypeError: Unable to convert function return value to a Python type! The signature was
(self: ExModule.Example2DVector, arg0: int) -> std::vector<Example, std::allocator<Example> >
This seems to be because the return type on the index operation on the 2D vector type is an opaque type. What I think should be happening is the return should be constructed into a ExampleVector. I can't do that from Python, it must be done in the module wrapper. Is this a bug or missing feature? If not, how do I overload the index operator on the Example2DVector class?
The example works, but my code did not. This was due to the fact that in my code the 1D vector class was bound to a Python type in a different pybind11 module. So there was no mapped Python type for the 1D vector C++ type when it defined the __getitem__ for the 2D vector class. However, I think that if I imported the module containing the 1D vector Python type bind later that it should work, but it does not. That might be a bug.
EDIT:
This behavior is not a bug (I figured, Jakob seems like a pretty smart guy). As discussed in the manual on binding STL containers, there is a section on "module local" bindings. By default type bindings are local to the module they are defined in to avoid multiple different bindings of the same type.
However, our project contains a "datatypes" module, and many modules that use those types. In this case we do not want datatypes defined in the "datatypes" module to be module local. Otherwise we end up with the given problem of return values not being converted into the correct Python type.
We can turn off the default module local binding in the binding definition. Using the question's example, we can turn off the module local binding for ExampleVector and accesses to an Example2DVector (defined in another module) will no longer fail.
pybind11::bind_vector<std::vector<Example>>(m, "ExampleVector", pybind11::module_local(false));
Quote form docs:
This macro must be specified at the top level (and outside of any namespaces), since it instantiates a partial template overload. If your binding code consists of multiple compilation units, it must be present in every file (typically via a common header) preceding any usage of std::vector. Opaque types must also have a corresponding class_ declaration to associate them with a name in Python, and to define a set of available operations
#ktb, It's not a bug, please see https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html?highlight=compilation%20unit#making-opaque-types
Related
I wish to pass a Python numpy array to this C++ function:
int data(std::span<int> x) { return x.size(); }
Is it possible to write a custom type caster for pybind11 that supports this signature?
pybind already has good support for vector<int>, but I would like to add support for a zero-copy version of vector, i.e. pass by reference.
It looks like there are some clues on how to do this here.
There's two questions here, so I'll answer them each individually:
It is certainly possible to write a custom type caster for std::span -- my own implementation based off the pybind11 type caster for abseil's span can be found here. I haven't gotten around to submitting a patch upstream yet, but someone else is also interested in this (see github discussion).
Most of the pybind11 STL type casters make copies of their sequences, so it's likely that any future std::span casters (like mine) are also going to make copies. That doesn't mean it's impossible -- the abseil span implementation does have a specialization where they would convert a span to a python buffer if it was a basic type.
It's easy to do something like that in Python, but implementing it in C++ seems to be more challenging.
I actually have some solution to this, but I'd like to see if you can see any better solution.
Here's what I want to do.
I have a list of values of different types (string, integer, can be also instance of some class etc.). Now here's the first problem - in C++ (unlike in Python) all values in vector/array have to be of the same type.
The solution I can see is that I can use std::any like this: vector<std::any> list.
I also have an array/vector of functions (or pointers to functions) with different parameter types and returned values - one function can accept string and integer and return a char and other can accept a char and return an int. Here's another problem: in C++ you can have an array/vector of functions only if they have the same parameters and returned values (as far as I know) because in your declaration of the vector you need to define the parameter types and the returned value.
The other problem is that I need to retrieve the information about the parameters and the returned value for each function. In other words, having those functions, I need to know that this function accepts 2 strings and 1 integer and returns a char for example. In Python I can use inspect.signature function to retrieve information about type annotations of a function. In C++, I don't know if there is a way to do this.
The solution I can see here is to use std::any again (although I will use another solution, I will explain why later).
The solution I can see to this problem is that I won't retrieve that information but instead the user of the class which accepts this vector of functions will simply have to specify what are the parameter types and returned value for each function. In other words, the solution I can see is that I won't be retrieving the information about parameter types programmatically.
The other problem I have is that later I need to call one of those functions with some parameters. In Python I do this like this:
arguments = [1, 'str', some_object] // here I prepare a list of arguments (they are of different types)
func(**arguments)
In C++ I can do unpacking as well, but not if the parameters are of different types.
The solution I can see here is as follows. Those functions in the vector will all accepts only argument which is vector<std::any> args which will simply contain all of the arguments. Later when I want to call the function, I will simply construct a vector with std::any values and pass it as an argument. This would also solve the previous problem of not being able to store vector of functions with different parameters.
Can you see better solutions?
You might wonder what I need all of this is for. I do some program synthesis stuff and I need to programmatically construct programs from existing functions. I'm writing a library and I want the user of my library to be able to specify those base functions out of which I construct programs. In order to do what I want, I need to know what are the parameters and returned values of those functions and I need to call them later.
I believe what you are looking for is std::apply. You can use std::tuple instead of std::vector to store a list of values of different types -- as long as the types are known at compile-time. Then std::apply(f, t) in C++ is basically the same as f(*t) in Python.
I have a list of values of different types (string, integer, can be also instance of some class etc.).
A type which is a union of subtypes is called a sum type or tagged union. C++ has the template std::variant for that.
Now here's the first problem - in C++ (unlike in Python) all values in vector/array have to be of the same type.
Of course, so use cleverly C++ containers. You might want some std::map or std::vector of your particular instance of std::variant.
I also have an array/vector of functions
You probably want some std::vector of std::function-s and code with C++ lambda expressions
You should read a good C++ programming book
I'm writing a library and I want the user of my library to be able to specify those base functions out of which I construct programs.
You could get inspiration from SWIG and consider generating some C++ code in your library. So write (in Python or C++) your C++ metaprogram (generating some C++ code, like ANTLR does) which generates the user code, and your user would adapt his build automation tool for such a need (like users of GNU bison do).
You might also consider embedding Guile (or Lua) in your application.
PS. You might be interested by other programming languages like Ocaml, Go, Scheme (with Guile, and read SICP), Common Lisp (with SBCL), or Rust.
I have a class that is constructed with a path to a text file. It parses the text file and stores a lot of data in various vectors and maps as its members. I'd like to share the class as a dll with users of different versions of MSVS (something that's new to me).
My original implementation when it was just for me returned the STL containers directly. After reading, my understanding is that this is dangerous because different compilers or different versions of the same compiler can easily implement the containers differently. One solution I saw was to explicitly instantiate any templates you were using and export them as well. I also had strings so I'd need to instantiate and export that since a std::string is actually an alias for a more complex template. However, even if I went that route it appears there's nothing I can do about exporting maps.
What I've done now is that instead of giving the user access to the containers, I have accessor functions that take an index (or a key for the maps, or a key and index for a vector of maps I've got) and fetch the value. All my parameters and return values are primitive types, including const char* for the strings.
Am I understanding the problem correctly, and is this a reasonable approach to it? Do I need to worry about the integral primitives in c++ not being strictly defined in the standard? I suppose I could use the std-defined integral types as well. One issue is that the user won't be able to iterate over the containers or check size. I could provide the size as a member(all the vectors are the same size), and then I guess it'd just be up to the user to provide their own vector and fill it if they want the other vector functionality.
I have an old C library I try to modify. There is a line like:
memcpy(&m_agents[idx].params, params, sizeof(dtCrowdAgentParams));
And ofcourse C stule pod like values and structs get copied. But I tried to add to params.some_struct.some_inner_map_field std::map and it does not get initialised on memcpy.
So I wonder how to copy my map keeping it alive? (if possible not extending methods of params and params.some_struct)
I would need to see the full definition of the struct to be sure, but this should work:
m_agents[idx].params = *params;
For POD types, it will do the same thing as memcpy. For non-POD types, it will call their assignment operator.
Replacing it with
m_agents[idx].params = *params;
will correct that instance. However, it's reasonable to suppose that the C library you are working with is assuming POD throughout and so you will need to check the whole codebase to ensure it's not doing anything that will break things elsewhere.
Are your maps mutable within the library or can you treat them as constant while the library is handling the data? If you can treat them as constant consider passing a handle, or pointer, to you map into the library instead of including the map itself in the parameter struct.
I have an array of objects in Python
[obj1, obj2, obj3]
and I want to pass them to off to a C++ function to perform some computation. I'm using SWIG to write my interface. The class type of the passed object is already defined in C++.
What's the best way to do this?
It depends on if your function is already written and cannot be changed, in which case you may need to check Swig docs to see if there is already a typemap from PyList to std::vector (I think there is). If not, taking PyObject* as the argument to the function and using the Python C API for manipulating lists should work fine. I haven't had any problems with it so far. For self-documentation, I recommend typedef'ing PyObject* to some kind of expected type, like "PythonList" so that the parameters have some meaning.
This may also be useful:
How to expose std::vector<int> as a Python list using SWIG?