I'm trying to instantiate a class written in C++ from python. For some reason, I'm getting a syntax error when invoking the "print" method, which takes no argument and should just print an int:
IronPython 2.7.5b2 (2.7.5.0) on .NET 4.0.30319.18444 (32-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> clr.AddReferenceToFileAndPath('c:\\users\\pletzer\\documents\\visual studio
>>> \\Projects\\AlexTest\\Debug\\AlexTest.dll')
>>> import at
>>> a = at.AlexTest(2)
>>> a.print()
File "<stdin>", line 1
a.print()
^
SyntaxError: syntax error
Thanks in advance for any suggestion. The C++ class is
// AlexTest.h
#include <iostream>
#pragma once
using namespace System;
namespace at {
public ref class AlexTest
{
public:
AlexTest(int i) {
mi = i;
}
void print() {
std::cout << "mi = i\n";
}
private:
int mi;
};
}
Changing the name of the method from "print" to "display" fixes the issue.
Also, can use (raw string r'...')
clr.AddReferenceToFileAndPath(r'c:\users\pletzer\documents\visual studio\Projects\AlexTest\Debug\AlexTest.dll')
to avoid having to type double back slashes
Related
I am trying to call a python function from a C++ code which contains main() function using Pybind11. But I found very few references are available. Most of existing documents talk about the reversed direction, i.e. calling C++ from Python.
Is there any complete example showing how to do that? The only reference I found is: https://github.com/pybind/pybind11/issues/30
But it has very little information.
The answer to your question really has two parts: one about calling a Python function from C++, the other about embedding the interpreter.
Calling a function in pybind11 is simply a matter of getting that function into a pybind11::object variable, on which you can invoke operator() to attempt to call the object. (It doesn't have to be a function, but just something callable: for example, it could also be an object with a __call__ method). For example, to call math.sqrt(2) from C++ code you'd use:
auto math = py::module::import("math");
auto resultobj = math.attr("sqrt")(2);
double result = resultobj.cast<double>();
or you could condense it all to just:
double result = py::module::import("math").attr("sqrt")(2).cast<double>();
The second part of the question involves how to do this from a C++ executable. When building an executable (i.e. when your C++ code contains main()) you have to embed the Python interpreter in your binary before you can do anything with Python (like calling a Python function).
Embedded support is a new feature added in the current pybind11 master branch (which will become the 2.2 release). Here's a basic example that starts an embedded Python interpreter and calls a Python function (math.sqrt):
#include <pybind11/embed.h>
#include <iostream>
namespace py = pybind11;
int main() {
py::scoped_interpreter python;
auto math = py::module::import("math");
double root_two = math.attr("sqrt")(2.0).cast<double>();
std::cout << "The square root of 2 is: " << root_two << "\n";
}
Outputs:
The square root of 2 is: 1.41421
More examples and documentation of calling functions and embedding are available at http://pybind11.readthedocs.io/en/master/advanced/pycpp/object.html and http://pybind11.readthedocs.io/en/master/advanced/embedding.html, respectively.
Jasons answer is pretty much on point, but I want to add a slightly more complex (and clean) example calling a python method with a numpy input.
I want to showcase two points:
We can cast a py::object to a py::function using py::reinterpret_borrow<py::function>
We can input a std::vector that automatically gets converted to a numpy.array
Note that the user is responsible for making sure that the PyModule.attr is actually a python function. Also note that the type conversion works for a wide variety of c++ types (see here for details).
In this example I want to use the method scipy.optimize.minimize with a starting point x0 that is provided from the c++ interface.
#include <iostream>
#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/embed.h> // python interpreter
#include <pybind11/stl.h> // type conversion
namespace py = pybind11;
int main() {
std::cout << "Starting pybind" << std::endl;
py::scoped_interpreter guard{}; // start interpreter, dies when out of scope
py::function min_rosen =
py::reinterpret_borrow<py::function>( // cast from 'object' to 'function - use `borrow` (copy) or `steal` (move)
py::module::import("py_src.exec_numpy").attr("min_rosen") // import method "min_rosen" from python "module"
);
py::object result = min_rosen(std::vector<double>{1,2,3,4,5}); // automatic conversion from `std::vector` to `numpy.array`, imported in `pybind11/stl.h`
bool success = result.attr("success").cast<bool>();
int num_iters = result.attr("nit").cast<int>();
double obj_value = result.attr("fun").cast<double>();
}
with the python script py_src/exec_numpy.py
import numpy as np
from scipy.optimize import minimize, rosen, rosen_der
def min_rosen(x0):
res = minimize(rosen, x0)
return res
Hope this helps someone!
project structure
CMakeLists.txt
calc.py
main.cpp
main.cpp
#include <pybind11/embed.h>
#include <iostream>
namespace py = pybind11;
using namespace py::literals;
int main() {
py::scoped_interpreter guard{};
// append source dir to sys.path, and python interpreter would find your custom python file
py::module_ sys = py::module_::import("sys");
py::list path = sys.attr("path");
path.attr("append")("..");
// import custom python class and call it
py::module_ tokenize = py::module_::import("calc");
py::type customTokenizerClass = tokenize.attr("CustomTokenizer");
py::object customTokenizer = customTokenizerClass("/Users/Caleb/Desktop/codes/ptms/bert-base");
py::object res = customTokenizer.attr("custom_tokenize")("good luck");
// show the result
py::list input_ids = res.attr("input_ids");
py::list token_type_ids = res.attr("token_type_ids");
py::list attention_mask = res.attr("attention_mask");
py::list offsets = res.attr("offset_mapping");
std::string message = "input ids is {},\noffsets is {}"_s.format(input_ids, offsets);
std::cout << message << std::endl;
}
calc.py
from transformers import BertTokenizerFast
class CustomTokenizer(object):
def __init__(self, vocab_dir):
self._tokenizer = BertTokenizerFast.from_pretrained(vocab_dir)
def custom_tokenize(self, text):
return self._tokenizer(text, return_offsets_mapping=True)
def build_tokenizer(vocab_dir: str) -> BertTokenizerFast:
tokenizer = BertTokenizerFast.from_pretrained(vocab_dir)
return tokenizer
def tokenize_text(tokenizer: BertTokenizerFast, text: str) -> dict:
res = tokenizer(text, return_offsets_mapping=True)
return dict(res)
CMakeLists.txt
cmake_minimum_required(VERSION 3.4)
project(example)
set(CMAKE_CXX_STANDARD 11)
# set pybind11 dir
set(pybind11_DIR /Users/Caleb/Softwares/pybind11)
find_package(pybind11 REQUIRED)
# set custom python interpreter(under macos)
link_libraries(/Users/Caleb/miniforge3/envs/py38/lib/libpython3.8.dylib)
add_executable(example main.cpp)
target_link_libraries(example PRIVATE pybind11::embed)
I've written part of a class in C++ and I want to be able to use it in conjunction with a Python GUI, so I'm using Boost.Python to try and make it easy. The issue I'm running into is that in following their guide (http://www.boost.org/doc/libs/1_55_0/libs/python/doc/tutorial/doc/html/python/exposing.html), I keep getting the following exception whenever I run bjam:
PacketWarrior/pcap_ext.cc:21:5: error: too few template arguments for class template 'class_'
Obviously it's complaining at me for omitting what they claim are optional arguments to the 'class_' template function, but I can't figure out why. I'm assuming it's a compiler issue but I don't know how to fix it. I'm running OS X 10.9 and using darwin for the default toolset, but GCC throws the same error. My Boost version is 1_55_0 if that helps at all.
Class header file (header guards omitted):
#include <queue>
#include "pcap.h"
#include "Packet.h"
class PacketEngine {
public:
PacketEngine();
~PacketEngine();
const char** getAvailableDevices(char *error_buf);
bool selectDevice(const char* dev);
Packet getNextPacket();
private:
char *selected_device;
char **devices;
int num_devices;
std::queue<Packet> packet_queue;
};
The cc file containing the references to Boost.Python and my class:
#include <boost/python/module.hpp>
#include <boost/python/def.hpp>
#include "PacketEngine.h"
BOOST_PYTHON_MODULE(pcap_ext) {
using namespace boost::python;
class_<PacketEngine>("PacketEngine")
.def("getAvailableDevices", &PacketEngine::getAvailableDevices);
}
And my bjam file (irrelevant parts and comments omitted):
use-project boost : ../../../Downloads/boost_1_55_0 ;
project
: requirements <library>/boost/python//boost_python
<implicit-dependency>/boost//headers
: usage-requirements <implicit-dependency>/boost//headers
;
python-extension pcap_ext : PacketWarrior/pcap_ext.cc ;
install convenient_copy
: pcap_ext
: <install-dependencies>on <install-type>SHARED_LIB <install-type>PYTHON_EXTENSION
<location>.
;
local rule run-test ( test-name : sources + )
{
import testing ;
testing.make-test run-pyd : $(sources) : : $(test-name) ;
}
run-test pcap : pcap_ext pcap.py ;
Any ideas as to how to circumvent this exception are greatly appreciated! I looked into the obvious route of just adding the optional parameters but I don't think they're relevant to my project. The class_ definition can be found here:
http://www.boost.org/doc/libs/1_37_0/libs/python/doc/v2/class.html
In short, include either:
boost/python.hpp: The Boost.Python convenient header file.
boost/python/class.hpp: The header that defines boost::python::class_.
The current included header files are declaring class_ with no default template arguments from def_visitor.hpp.
Also, trying to directly expose PacketEngine::getAvailableDevices() will likely present a problem:
It accepts a char* argument, but strings are immutable in Python.
There are no types that automatically convert to/from a const char** in Boost.Python.
It may be reasonable for a Python user to expect PacketEngine.getAvailableDevices() to return an iterable type containing Python strs, or throw an exception on error. This can be accomplished in a non-intrusive manner by writing a helper or auxiliary function that delegates to original function, but is exposed to Python as PacketEngine.getAvailableDevices().
Here is a complete example based on the original code:
#include <exception> // std::runtime_error
#include <boost/python.hpp>
namespace {
const char* devices_str[] = {
"device A",
"device B",
"device C",
NULL
};
} // namespace
class PacketEngine
{
public:
PacketEngine() : devices(devices_str) {}
const char** getAvailableDevices(char *error_buf)
{
// Mockup example to force an error on second call.
static bool do_error = false;
if (do_error)
{
strcpy(error_buf, "engine not responding");
}
do_error = true;
return devices;
}
private:
const char **devices;
};
/// #brief Auxiliary function for PacketEngine::getAvailableDevices that
/// provides a more Pythonic API. The original function accepts a
/// char* and returns a const char**. Both of these types are
/// difficult to use within Boost.Python, as strings are immutable
/// in Python, and Boost.Python is focused to providing
/// interoperability to C++, so the const char** type has no direct
/// support.
boost::python::list PacketEngine_getAvailableDevices(PacketEngine& self)
{
// Get device list and error from PacketEngine.
char error_buffer[256] = { 0 };
const char** devices = self.getAvailableDevices(error_buffer);
// On error, throw an exception. Boost.Python will catch it and
// convert it to a Python's exceptions.RuntimeError.
if (error_buffer[0])
{
throw std::runtime_error(error_buffer);
}
// Convert the c-string array to a list of Python strings.
namespace python = boost::python;
python::list device_list;
for (unsigned int i = 0; devices[i]; ++i)
{
const char* device = devices[i];
device_list.append(python::str(device, strlen(device)));
}
return device_list;
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<PacketEngine>("PacketEngine")
.def("getAvailableDevices", &PacketEngine_getAvailableDevices);
}
Interactive usage:
>>> import example
>>> engine = example.PacketEngine()
>>> for device in engine.getAvailableDevices():
... print device
...
device A
device B
device C
>>> devices = engine.getAvailableDevices()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: engine not responding
I am trying to use SWIG in an effort to call member functions of a C++ object from Python. Currently I have a small example class with a getter and setter to modify a member variable of the C++ class. Here is the C++ header file:
#ifndef _square_
#define _square_
#include <iostream>
class Square
{
private:
double x;
double y;
const char *name;
public:
void setName(const char*);
const char* getName();
Square() {
name = "construct value";
};
};
#endif
Here is the .cpp implementation file:
#include <iostream>
using namespace std;
#include "Square.h"
const char* Square::getName()
{
return name;
}
void Square::setName(const char* name)
{
this->name = name;
return;
}
And the Square.i file for SWIG:
%module Square
%{
#include "Square.h"
%}
%include "Square.h"
SWIG seems to generate the Square_wrap.cxx file without issue, and the resulting object files seem to link fine:
$ swig -python -c++ Square.i
$ g++ -c -fpic Square.cxx Square_wrap.cxx -I/usr/include/python2.7
$ g++ -shared Square.o Square_wrap.o -o _Square.so
Now for some example Python to test the results:
$ cat test2.py
#!/usr/bin/python
import Square
s = Square.Square()
print s.getName()
s.setName("newnametest")
print s.getName()
If I run this through the Python interpreter everything works fine:
$ python test2.py
construct value
newnametest
But if I interactively enter in the test lines via Python's CLI, things do not work:
$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Square
>>>
>>> s = Square.Square()
>>>
>>> print s.getName()
construct value
>>> s.setName("newnametest")
>>> print s.getName()
>>> s.getName()
'<stdin>'
>>> s.setName('newnametest')
>>> s.getName()
''
>>> s.setName("newnametest")
>>> s.getName()
''
Does Python handle a Python script file differently under the hood in comparison to the CLI, or am I somehow abusing the Python interface generated by SWIG? Any tips on how to debug or understand the issue under the hood would be much appreciated.
As far as I see, you are just storing the reference on the cpp file (this->name = name). It would be good to copy it, because there are high chances the string doesn't last enough and is just discarded after the function returns (and garbage collected slightly after that). This would explain why in the script it works (there is no GCollection nor anything else happens between the two calls).
Try making a copy with strdup or using std::string.
I'm using SWIT to convert a vc project to python.
I found when a struct has a member which type is like "typedef char TEXT[16]" cannot be converted correctly.
for example:
typedef char TEXT[16];
struct MYSTRUCT
{
TEXT TradingDay;
};
The wrapper cpp cannot compile all right.
"error C2075: 'Target of operator new()' : array initialization needs curly braces"
BUT,if typedef is not an array , like this:
typedef int NUMBER;
struct MYSTRUCT2
{
NUMBER Money;
};
there will be all right.
what should I do?
thx!
P.S:
i file:
%module MyDataAPI
%include "typemaps.i"
%header %{
#include "../References/MyDataAPI.h"
%}
namespace MyDataAPI
{
struct MYSTRUCT
{
TEXT TradingDay;
};
struct MYSTRUCT2
{
NUMBER Money;
};
}
Make sure your typedef statements are processed by SWIG. %header only adds code to the generated file, that data is not processed by SWIG. %inline both adds the code directly to the generated file and processes it with SWIG. Here's my .i file:
%module x
%inline %{
typedef char TEXT[16];
typedef int NUMBER;
namespace MyDataAPI
{
struct MYSTRUCT
{
TEXT TradingDay;
};
struct MYSTRUCT2
{
NUMBER Money;
};
}
%}
And use:
T:\>py
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import x
>>> a=x.MYSTRUCT()
>>> a.TradingDay
''
>>> a.TradingDay='ABCDEFGHIJKLMNOPQ' # Note this is too long, 17 chars...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: in method 'MYSTRUCT_TradingDay_set', argument 2 of type 'char [16]'
>>> a.TradingDay='ABCDEFGHIJKLMNOP'
>>> a.TradingDay
'ABCDEFGHIJKLMNOP'
>>> b=x.MYSTRUCT2()
>>> b.Money
0
>>> b.Money=100
>>> b.Money
100
I'm using SWIG 2.0 to create a Python wrapper for a C++ library. One method has an argument of type "const std::map&". SWIG happily generates a wrapper for it, but I can't figure out how to invoke the method. If I pass, for example, {"a":"b"} for that argument, I get a "NotImplementedError: Wrong number or type of arguments for overloaded function" error.
I looked at the generated .cxx file in the hope it would clarify, but it didn't. Here's the code that processes that argument:
res4 = SWIG_ConvertPtr(obj3, &argp4, SWIGTYPE_p_std__mapT_std__string_std__string_t, 0 | 0);
if (!SWIG_IsOK(res4)) {
SWIG_exception_fail(SWIG_ArgError(res4), "in method '" "new_Context" "', argument " "4"" of type '" "std::map< std::string,std::string > const &""'");
}
It clearly knows that argument exists, and that it's supposed to be something that gets converted to a map. But I can't figure out what it actually wants me to pass for it.
When you're using a C++ template (e.g. a std::map<string, string>) you need to create an alias for it in your .i file so you can use it in python:
namespace std {
%template(map_string_string) map<string, string>;
}
Now let's say you want to wrap a function that looks like this:
void foo(const std::map<string, string> &arg);
On the python side, you need to pass a map_string_string to foo, not a python dict. It turns out that you can easily convert a python dict to a map though by doing this:
map_string_string({ 'a' : 'b' })
so if you want to call foo, you need to do this:
foo(map_string_string({ 'a' : 'b' }))
Here's full example code that works.
// test.i
%module test
%include "std_string.i"
%include "std_map.i"
namespace std {
%template(map_string_string) map<string, string>;
}
void foo(const std::map<std::string, std::string> &val);
%{
#include <iostream>
#include <string>
#include <map>
using namespace std;
void
foo(const map<string, string> &val)
{
map<string, string>::const_iterator i = val.begin();
map<string, string>::const_iterator end = val.end();
while (i != end) {
cout << i->first << " : " << i->second << endl;
++i;
}
}
%}
And the python test code:
#run_test.py
import test
x = test.map_string_string({ 'a' : 'b', 'c' : 'd' })
test.foo(x)
And my command line:
% swig -python -c++ test.i
% g++ -fPIC -shared -I/usr/include/python2.7 -o _test.so test_wrap.cxx
% python run_test.py
a : b
c : d