I have a C++ function which takes a vector< pair< pair<string,string>, string > > which I wish to call from python.
I wrote a wrapper as below:
bool create_cache_wrapper(boost::python::tuple tup_)
{
vector< pair< pair<string,string>, string > > rbv;
for(int i=0;i<py::len(tup_);++i)
{
std::string start,end,bucket;
bucket = py::extract<std::string>(tup_[i][1]);
start = py::extract<std::string>(tup_[i][0][0]);
end = py::extract<std::string>(tup_[i][0][1]);
rbv.push_back(make_pair(make_pair(start,end),bucket));
}
return create_cache(rbv);
}
I have registered this function with python as below:
BOOST_PYTHON_MODULE(my_lib)
{
using namespace boost::python;
def("create_cache", create_cache_wrapper);
}
This works fine but the issue is, it accepts only a tuple but not a list. How do I write the wrapper such that it works any sequence container in python? Is there a better way to do this without having to write a wrapper? I want the python api to be simple like
create_cache( ( (('',''),'b1'), ) )
create_cache( [ (('',''),'b1') ] )
This could be accomplished by making the wrapper take an object instead:
bool create_cache_wrapper(boost::python::object tup_)
{
// ...
}
py::len() and the indexing operators should work on objects, and they should raise a Python error if the type in question does not support len() or indexing. This should allow the caller to provide any indexable type, even user-defined ones!
I normally use boost python iterator wrappers to achieve what you described - it works with any type of iterable. I adapted your code below so you have an idea on how to do it yourself:
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
using namespace boost::python;
bool create_cache_wrapper(object iterable)
{
stl_input_iterator<object> end;
for(std_input_iterator<object> i(iterable); i!=end; ++i)
{
std::string start,end,bucket;
bucket = py::extract<std::string>((*i)[1]);
start = py::extract<std::string>((*i)[0][0]);
end = py::extract<std::string>((*i)[0][1]);
rbv.push_back(make_pair(make_pair(start,end),bucket));
}
return create_cache(rbv);
}
You can go all the way using stl_input_iterator, even for the your pairs inside the first iterable.
Related
Currently I'm trying to convert a py::dict into its counterpart C++s std::map. Trying to use the
automatic conversion like this fails :
#include <pybind11/stl.h>
namespace py = pybind11;
using namespace py::literals;
...
py::dict py_kwargs = py::dict("number1"_a = 5, "number2"_a = 42);
auto cpp_kwargs = py_kwargs.cast<std::map<int, int>>();
with an exception stating :
Unable to cast Python instance of type <class 'dict'> to C++ type 'std::map<int,int,std::less<int>,std::allocator<std::pair<int const ,int> > >'
What am I missing here?
Also on a side note, how should I go about the situation , where the python dictionary has different key types, for example like :
py::dict py_kwargs = py::dict("name"_a = "World", "number"_a = 42);
How should I go about the conversion in such cases?
OK, I found the issue, before that I ended up doing the conversion like this :
map<std::string, int> convert_dict_to_map(py::dict dictionary)
{
map<std::string, int> result;
for (std::pair<py::handle, py::handle> item : dictionary)
{
auto key = item.first.cast<std::string>();
auto value = item.second.cast<int>();
//cout << key << " : " << value;
result[key] = value;
}
return result;
}
and then, looked carefully at the :
auto cppmap = kwargs.cast<map<int, int>>();
and finally noticed my issue.
it should have been :
auto cppmap = kwargs.cast<map<std::string, int>>();
I made a mistake, when I changed my example dict and later on reverted back the changes but forgot to change the signature!
Anyway, it seems the first solution may be a better choice, as it allows the developer to work with Python's dynamic nature better.
That is, a Python dictionary may very well, include different pairs (such as string:string, string:int, int:float, etc all in the same dictionary object). Therefore, making sure the items are validly reconstructed in c++ can be done much better using the first crude method!
I have a header file like:
#include <vector>
inline std::vector<uint8_t>& vec() {
static std::vector<uint8_t> v { 'a', 'b', 'c', 'd' };
return v;
}
inline const std::vector<uint8_t>& cvec() {
return vec();
}
I can wrap it in SWIG using std_vector.i and pyabc.i but that is quite inefficient (there's a jump between C++ and Python code for every access) and given that these are literally just a bunch of bytes I ought to be able to wrap them with Python's memoryview interface.
How can I expose my std::vector<uint8_t> as a Python memoryview?
Exposing it as a memoryview requires creating a Py_buffer first. In Python 3.3+ there is a convenient helper function, PyMemoryView_FromMemory that does a lot of the work for us. In earlier versions though we'll need to take a few extra steps, so our basic out typemap looks like:
%typemap(out) std::vector<uint8_t>&, const std::vector<uint8_t>& {
Py_buffer *buf=(Py_buffer*)malloc(sizeof *buf);
const bool ro = info<$1_type>::is_readonly();
if (PyBuffer_FillInfo(buf, NULL, &((*$1)[0]), (*$1).size(), ro, PyBUF_ND)) {
// error, handle
}
$result = PyMemoryView_FromBuffer(buf);
}
Here we're basically allocating some memory for the Py_buffer. This just contains the details of the buffer internally for Python. The memory we allocate will be owned by the memoryview object once it's created. Unfortunately since it's going to be released with a call to free() we need to allocate it with malloc(), even though it's C++ code.
Besides the Py_buffer and an optional Py_Object PyBuffer_FillInfo takes a void* (the buffer itself), the size of the buffer, a boolean indicating if it's writeable and a flag. In this case our flag simply indicates that we have provided C-style contiguous memory for the buffer.
For deciding if it is readonly or not we used SWIG's built in $n_type variable and a helper (which could be a C++11 type trait if we wanted).
To complete our SWIG interface we need to provide that helper and include the header file, so the whole thing becomes:
%module test
%{
#include "test.hh"
namespace {
template <typename T>
struct info {
static bool is_readonly() {
return false;
}
};
template <typename T>
struct info<const T&> {
static bool is_readonly() {
return true;
}
};
}
%}
%typemap(out) std::vector<uint8_t>&, const std::vector<uint8_t>& {
Py_buffer *buf=(Py_buffer*)malloc(sizeof *buf);
const bool ro = info<$1_type>::is_readonly();
if (PyBuffer_FillInfo(buf, NULL, &((*$1)[0]), (*$1).size(), ro, PyBUF_ND)) {
// error, handle
}
$result = PyMemoryView_FromBuffer(buf);
}
%include "test.hh"
We can then test it with:
import test
print test.vec()
print len(test.vec())
print test.vec()[0]
print test.vec().readonly
test.vec()[0]='z'
print test.vec()[0]
print "This should fail:"
test.cvec()[0] = 0
Which worked as expected, tested using Python 2.7.
Compared to just wrapping it using std_vector.i this approach does have some drawbacks. The biggest being that we can't resize the vector, or convert it back to a vector later trivially. We could work around that, at least partially by creating a SWIG proxy for the vector like normal and using the second parameter of PyBuffer_FillInfo to store it internally. (This would also be needed if we had to manage the ownership of the vector for instance).
I haven't used the STL much before, but I started to on this huffman compression project. Everything seems to work except the "for_each" function, which wont except a function argument. Since I don't normally use xcode (I usually use eclipse cdt) I'm not sure if the problem is with my code or with xcode.
This is the Huff.h file
class Huff {
private:
typedef pair<char, int> c_pair;
vector<Node> nodes;
vector<Code> code;
string content;
void copy_to(c_pair c);
public:
Huff(string);
~Huff();
string compress();
bool set_content();
string get_content();
string get_compress();
};
And this is the part of the Huff.cpp file that will not work.
//---Compress---
void Huff::copy_to(c_pair c){
Node n(c.second, c.first, NULL, NULL);
nodes.push_back(n);
}
string Huff::compress(){
map<char, int> freq;
for(int i = 0; i < content.length(); i++)
freq[content[i]]++;
for_each(freq.begin(), freq.end(), copy_to); //I've also tried this->copy_to
return "110";
}
for_each(freq.begin(), freq.end(), copy_to); //I've also tried this->copy_to
copy_to is a member function which cannot be passed to std::for_each.
What you need is a callable entity which doesn't need implicit this : such an entity can be either functor or free function, and in C++11, lambda also.
The lambda solution would be pretty simple if you can use it:
for_each(freq.begin(),
freq.end(),
[this](c_pair & c) { this->copy_to(c); } );
Learn about lambda expression here:
What is a lambda expression in C++11?
As pointed out, you can't use a member function that way with for_each.
The C++03 alternative is to use mem_fun and bind1st to build a function object:
std::for_each(freq.begin(), freq.end(),
std::bind1st(std::mem_fun(&Huff::copy_to), this));
Or using Boost.Bind:
std::for_each(freq.begin(), freq.end(),
boost::bind(&Huff::copy_to, this, _1));
I'm trying to do something which should be very simple, but I'm not having much luck figuring out how from the existing documentation.
For a python 2 project I am trying to return a list gettext-translated string as a unicode instances to python. The return value for gettext() is a UTF-8 encoded char*, which should be pretty simple to convert to a python unicode instrance using PyUnicode_FromString. I have a feeling this is trivial to do, but I can't seem to figure out how.
Basd on comments from Ignacio Vazquez-Abrams and Thomas K I did get this working for a single string; for that case you can bypass all the boost.python infrastructure. Here is an example:
PyObject* PyMyFunc() {
const char* txt = BaseClass::MyFunc();
return PyUnicode_FromString(txt);
}
which is exposed with the usual def statement:
class_<MyCclass>("MyClass")
.def("MyFunc", &MyClass::PyMyFunc);
Unfortuantely this does not work when you want to return a list of unicode instances. This is my naive implementation:
boost::python::list PyMyFunc() {
std::vector<std::string> raw_strings = BaseClass::MyFunc();
std::vector<std::string>::const_iterator i;
boost::python::list result;
for (i=raw_strings.begin(); i!=raw_strings.end(); i++)
result.append(PyUnicode_FromString(i->c_str()));
return result;
}
but this does not compile: boost::python::list does seem to handle PyObject values.
With some help from the C++-SIG mailinglist I have this working now. There are two extra steps needed:
use boost::python::handle<> to create a C++ wrapper around the PyObject* which takes care of reference handling
use boost::python::object to create a C++ wrapper around the handle, which allows using a PyObject* instance as a (reasonably) normal C++ class instance, and thus something boost::python::list can handle.
With that knowledge the working code looks like this:
boost::python::list PyMyFunc() {
std::vector<std::string> raw_strings = BaseClass::MyFunc();
std::vector<std::string>::const_iterator i;
boost::python::list result;
for (i=raw_strings.begin(); i!=raw_strings.end(); i++)
result.append(
boost::python::object(
boost::python::handle<>(
PyUnicode_FromString(i->c_str()))));
return result;
}
I have written some C++ code that generates a std::vector.
I also have a python script that manipulates some data that, for now, I am declaring like this (below).
import numpy
x = numpy.random.randn(1000)
y = numpy.random.randn(1000)
I can run the script fine. From my C++ code:
using namespace boost::python;
try{
Py_Initialize();
object main = import("__main__");
object global(main.attr("__dict__"));
object result = exec_file("scatterPlot.py", global, global);
Py_Finalize();
}
catch(error_already_set){
PyErr_Print();
}
return;
I have no idea how to get my C++ data to python. I've around quite a bit, but there doesn't seem to be anything definitive.
I have in my C++
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
This seems to work, but as I understand, it only provides a class "PyVec" for my python script but not the data I need. Am I wrong?
I've also seen some other people use boost::shared_ptr in a python mailing list.
I also found this example but found it confusing.
I can think of a few approaches
Pass something to the boost::python::exec_file method
Using the boost_indexing_suite_ext
Uinsg boost::shared_ptr
Which approach is easiest to get going? No approach seems clear to me.
Here are some more links I've looked at:
from the boost website
from the python website
another mailing list thread
UPDATE:
This works for passing an int to my python code like below
int main(){
int five_squared=0;
int a =3;
try {
Py_Initialize();
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
main_namespace["var"]=a;
object ignored = exec("result = 5 ** var", main_namespace);
five_squared = extract<int>(main_namespace["result"]);
} catch( error_already_set ) {
PyErr_Print();
}
std::cout << five_squared << std::endl;
return 0;
}
But I want to pass a vector, when I try to do that in a similar fashion as above I get this error
TypeError: No to_python (by-value)
converter found for C++ type:
std::vector >
So, obviously I need to tell python how to deal with std::vector. I think this code could help with that.
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
But since std::vector is pretty common, there must be a defined way to do this... right?
The following code works for me (Python 2.6, Boost 1.39). This is almost the same as your code, except without the BOOST_PYTHON_MODULE line itself (but with the class_ definition for the vector). BOOST_PYTHON_MODULE only needs to be used when creating extension modules.
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
using namespace boost::python;
using namespace std;
int main()
{
vector<double> vec;
vec.push_back(1.2);
vec.push_back(3.4);
try {
Py_Initialize();
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
object main_module = import("__main__");
object globals = main_module.attr("__dict__");
globals["var"]=vec;
object ignored = exec("result = sum(var)", globals, globals);
double result = extract<double>(globals["result"]);
std::cout << result << std::endl;
} catch( error_already_set ) {
PyErr_Print();
}
return 0;
}
I'm not sure if I understand correctly. After exporting your class "PyVec" which can hold std::vector<double>, you can export any c++ function taking vector as input or return type. So of course you can populate your vector within c++ and access this data in Python with the interfaced type "PyVec".