I have written some C++ code that generates a std::vector.
I also have a python script that manipulates some data that, for now, I am declaring like this (below).
import numpy
x = numpy.random.randn(1000)
y = numpy.random.randn(1000)
I can run the script fine. From my C++ code:
using namespace boost::python;
try{
Py_Initialize();
object main = import("__main__");
object global(main.attr("__dict__"));
object result = exec_file("scatterPlot.py", global, global);
Py_Finalize();
}
catch(error_already_set){
PyErr_Print();
}
return;
I have no idea how to get my C++ data to python. I've around quite a bit, but there doesn't seem to be anything definitive.
I have in my C++
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
This seems to work, but as I understand, it only provides a class "PyVec" for my python script but not the data I need. Am I wrong?
I've also seen some other people use boost::shared_ptr in a python mailing list.
I also found this example but found it confusing.
I can think of a few approaches
Pass something to the boost::python::exec_file method
Using the boost_indexing_suite_ext
Uinsg boost::shared_ptr
Which approach is easiest to get going? No approach seems clear to me.
Here are some more links I've looked at:
from the boost website
from the python website
another mailing list thread
UPDATE:
This works for passing an int to my python code like below
int main(){
int five_squared=0;
int a =3;
try {
Py_Initialize();
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
main_namespace["var"]=a;
object ignored = exec("result = 5 ** var", main_namespace);
five_squared = extract<int>(main_namespace["result"]);
} catch( error_already_set ) {
PyErr_Print();
}
std::cout << five_squared << std::endl;
return 0;
}
But I want to pass a vector, when I try to do that in a similar fashion as above I get this error
TypeError: No to_python (by-value)
converter found for C++ type:
std::vector >
So, obviously I need to tell python how to deal with std::vector. I think this code could help with that.
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
But since std::vector is pretty common, there must be a defined way to do this... right?
The following code works for me (Python 2.6, Boost 1.39). This is almost the same as your code, except without the BOOST_PYTHON_MODULE line itself (but with the class_ definition for the vector). BOOST_PYTHON_MODULE only needs to be used when creating extension modules.
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
using namespace boost::python;
using namespace std;
int main()
{
vector<double> vec;
vec.push_back(1.2);
vec.push_back(3.4);
try {
Py_Initialize();
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
object main_module = import("__main__");
object globals = main_module.attr("__dict__");
globals["var"]=vec;
object ignored = exec("result = sum(var)", globals, globals);
double result = extract<double>(globals["result"]);
std::cout << result << std::endl;
} catch( error_already_set ) {
PyErr_Print();
}
return 0;
}
I'm not sure if I understand correctly. After exporting your class "PyVec" which can hold std::vector<double>, you can export any c++ function taking vector as input or return type. So of course you can populate your vector within c++ and access this data in Python with the interfaced type "PyVec".
Related
I'm writing a program with C++ and boost::python, and it seems very strange to me that boost::python::exec returns anything at all. For example, in the docs here, it says:
Effects
Execute Python source code from code in the context specified by the dictionaries globals and locals.
Returns
An instance of object which holds the result of executing the code.
And yet the documentation for python 3's exec function says that:
The return value is None.
So what's the point of returning anything if the function always returns none? Why not just make it a void function, or even better yet, have it return a python error if something goes wrong? Or maybe, I'm just misunderstanding the documentation, and there is something useful in there after all. Which is why I'm asking this question.
As I was trying to figure this out, I tried this sample program:
#include <boost\python.hpp>
#include <iostream>
int main()
{
using namespace boost::python;
Py_Initialize();
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
while (true)
{
try
{
std::cout << ">>> ";
std::string comm;
std::getline(std::cin, comm);
if (comm == "exit")
break;
object bar = exec(comm.c_str(), main_namespace);
if (bar.is_none())
std::cout << "None\n";
}
catch (error_already_set const &)
{
PyErr_Print();
}
}
}
And it seems like exec never returned an object that isn't None.
So is there ever, under any circumstance a reason to keep around the return value of a boost::python::exec call, or should I just always throw it away?
The concept of a void function in Python is one with no return value. If you attempt to assign the result of a void function, the result will always be None.
Boost::Python seems to follow this in their implementation of py::exec, although this is no surprise as even the CPython PyRun_String function that py::exec calls returns a PyObject that is always None.
So to answer your question, yes you can ignore the return value.
I have a C++ function which takes a vector< pair< pair<string,string>, string > > which I wish to call from python.
I wrote a wrapper as below:
bool create_cache_wrapper(boost::python::tuple tup_)
{
vector< pair< pair<string,string>, string > > rbv;
for(int i=0;i<py::len(tup_);++i)
{
std::string start,end,bucket;
bucket = py::extract<std::string>(tup_[i][1]);
start = py::extract<std::string>(tup_[i][0][0]);
end = py::extract<std::string>(tup_[i][0][1]);
rbv.push_back(make_pair(make_pair(start,end),bucket));
}
return create_cache(rbv);
}
I have registered this function with python as below:
BOOST_PYTHON_MODULE(my_lib)
{
using namespace boost::python;
def("create_cache", create_cache_wrapper);
}
This works fine but the issue is, it accepts only a tuple but not a list. How do I write the wrapper such that it works any sequence container in python? Is there a better way to do this without having to write a wrapper? I want the python api to be simple like
create_cache( ( (('',''),'b1'), ) )
create_cache( [ (('',''),'b1') ] )
This could be accomplished by making the wrapper take an object instead:
bool create_cache_wrapper(boost::python::object tup_)
{
// ...
}
py::len() and the indexing operators should work on objects, and they should raise a Python error if the type in question does not support len() or indexing. This should allow the caller to provide any indexable type, even user-defined ones!
I normally use boost python iterator wrappers to achieve what you described - it works with any type of iterable. I adapted your code below so you have an idea on how to do it yourself:
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
using namespace boost::python;
bool create_cache_wrapper(object iterable)
{
stl_input_iterator<object> end;
for(std_input_iterator<object> i(iterable); i!=end; ++i)
{
std::string start,end,bucket;
bucket = py::extract<std::string>((*i)[1]);
start = py::extract<std::string>((*i)[0][0]);
end = py::extract<std::string>((*i)[0][1]);
rbv.push_back(make_pair(make_pair(start,end),bucket));
}
return create_cache(rbv);
}
You can go all the way using stl_input_iterator, even for the your pairs inside the first iterable.
I am embedding python in my C++ application using boost python. I am a C++ programmer, with very limited knowledge of Python.
I have a C++ class, PyExpression. Each instance of this class has a string expStr, which is a short user-entered (at runtime) python program, that is executed by calling boost::python::exec. Briefly, I have this set up as:
//import main and its globals
bp::object main = bp::import("__main__");
bp::object main_namespace = main.attr("__dict__");
where main and main_namespace are members of the C++ class PyExpression.
void PyExpression::Run()
{
bp::object pyrun = exec(expStr,main_namespace);
}
The problem here is that different C++ instances of PyExpression modify the same global python namespace, main_namespace, and I want each PyExpression instance to have its own "global" namespace.
If I pass in boost::python::dict class_dict instead of main_namespace above, it works at a basic level. But if PyExpression::expStr imports a module, e.g. import sys, then I get an ImportError. Also, using class_dict, I can no longer call globals(), locals(), vars(), as they all become undefined.
I have also tried exposing PyExpression as a python module. Briefly,
BOOST_PYTHON_MODULE(PyExpModule)
{
bp::class_<PyExpression>("PyExpression", bp::no_init)
//a couple .def functions
}
int pyImport = PyImport_AppendInittab( "PyExpModule", &initPyExpModule );
bp::object thisExpModule = bp::object( (bp::handle<>(PyImport_ImportModule("PyExpModule"))) );
bp::object PyExp_namespace = thisExpModule.attr("__dict__");
Unfortunately, using PyExp_namespace, again I get the ImportError when the string to be executed imports a python module, and again, the namespace is shared between all instances of PyExpression.
In short, I want to be able to use a namespace object/dictionary, that is preferably a class member of PyExpression, have only that instance of PyExpression have access to the namespace, and the namespace to act like a global namespace such that other modules can be imported, and the `globals(), locals(), vars() are all defined.
If anyone can point me to a sketch of working code, I would very much appreciate it. I can't find relevant material on this problem.
Before providing a solution, I want to provide some clarification on Python behavior.
Boost.Python's object is essentially a higher-level handle of a smart pointer. Thus, multiple object instances may point to the same Python object.
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
The above code imports a module named __main__. In Python, modules are essentially singletons due to the import behavior. Therefore, although the C++ main_module may be a member of the C++ PyExpression class, they all point to the same Python __main__ module, as it is a singleton. This results in main_namespace pointing to the same namespace.
Much of Python is built around dictionaries. For example, with an example module:
class Foo:
def __init__(self):
self.x = 42;
def bar(self):
pass
There are 3 dictionaries of interests:
example.__dict__ is the example module's namespace.
>>> example.__dict__.keys()
['__builtins__', '__file__', '__package__', '__name__', 'Foo', '__doc__']
example.Foo.__dict__ is a dictionary that describes the Foo class. Additionally, it would contain the equivalent of C++'s static member variables and functions.
>>> example.Foo.__dict__.keys()
['__module__', 'bar', '__doc__', '__init__']
example.Foo().__dict__ is a dictionary containing instance-specific variables. This would contain the equivalent of C++'s non-static member variables.
>>> example.Foo().__dict__.keys()
['x']
The Python exec statement takes two optional arguments:
The first argument specifies the dictionary that will be used for globals(). If the second argument is omitted, then it is also used for locals().
The second argument specifies the dictionary that will be used for locals(). Variable changes occurring within exec are applied to locals().
To get the desired behavior, example.Foo().__dict__ needs to be used as locals(). Unfortunately, this becomes slightly more complicated because of the following two factors:
Although import is a Python keyword, the CPython implementation is dependent on __builtins__.__import__. Thus, there needs to be a guarantee that the __builtin__ module is assessable as __builtins__ within the namespace passed to exec.
If a C++ class called Foo is exposed as a Python class through Boost.Python, then there is no easy way to access the Python Foo instance from within the C++ Foo instance.
To account for these behaviors, the C++ code will need to:
Get a handle to the Python object's __dict__.
Inject the __builtin__ module into the Python object's __dict__.
Extract the C++ object from the Python object.
Pass the Python object's __dict__ to the C++ object.
Here is an example solution that only sets variables on the instance for which code is being evaluated:
#include <boost/python.hpp>
class PyExpression
{
public:
void run(boost::python::object dict) const
{
exec(exp_.c_str(), dict);
}
std::string exp_;
};
void PyExpression_run(boost::python::object self)
{
// Get a handle to the Python object's __dict__.
namespace python = boost::python;
python::object self_dict = self.attr("__dict__");
// Inject the __builtin__ module into the Python object's __dict__.
self_dict["__builtins__"] = python::import("__builtin__");
// Extract the C++ object from the Python object.
PyExpression& py_expression = boost::python::extract<PyExpression&>(self);
// Pass the Python object's `__dict__` to the C++ object.
py_expression.run(self_dict);
}
BOOST_PYTHON_MODULE(PyExpModule)
{
namespace python = boost::python;
python::class_<PyExpression>("PyExpression")
.def("run", &PyExpression_run)
.add_property("exp", &PyExpression::exp_, &PyExpression::exp_)
;
}
// Helper function to check if an object has an attribute.
bool hasattr(const boost::python::object& obj,
const std::string& name)
{
return PyObject_HasAttrString(obj.ptr(), name.c_str());
}
int main()
{
PyImport_AppendInittab("PyExpModule", &initPyExpModule);
Py_Initialize();
namespace python = boost::python;
try
{
// python: import PyExpModule
python::object py_exp_module = python::import("PyExpModule");
// python: exp1 = PyExpModule.PyExpression()
// python: exp1.exp = "import time; x = time.localtime().tm_year"
python::object exp1 = py_exp_module.attr("PyExpression")();
exp1.attr("exp") =
"import time;"
"x = time.localtime().tm_year"
;
// python: exp2 = PyExpModule.PyExpression()
// python: exp2.exp = "import time; x = time.localtime().tm_mon"
python::object exp2 = py_exp_module.attr("PyExpression")();
exp2.attr("exp") =
"import time;"
"x = time.localtime().tm_mon"
;
// Verify neither exp1 nor exp2 has an x variable.
assert(!hasattr(exp1, "x"));
assert(!hasattr(exp2, "x"));
// python: exp1.run()
// python: exp2.run()
exp1.attr("run")();
exp2.attr("run")();
// Verify exp1 and exp2 contain an x variable.
assert(hasattr(exp1, "x"));
assert(hasattr(exp2, "x"));
// python: print exp1.x
// python: print exp2.x
std::cout << python::extract<int>(exp1.attr("x"))
<< "\n" << python::extract<int>(exp2.attr("x"))
<< std::endl;
}
catch (python::error_already_set&)
{
PyErr_Print();
}
}
And the output:
[twsansbury#localhost]$ ./a.out
2013
5
Due to how libraries are loaded from imports, it may require providing arguments to the linker that will cause all symbols, not only used ones, to the dynamic symbol table. For example, when compiling the above example with gcc, using -rdynamic was required. Otherwise, import time will fail due to an undefined PyExc_IOError symbol.
Python does not provide a 100% reliable isolation mechanism for this kind of task. That said, the essential tool you are looking for is the Python C-API Py_NewInterpreter, which is documented here. You will have to call it upon the creation of your PyExpression object, to create a new (semi)-isolated environment (N.B.: the destructor should call Py_EndInterpreter).
This is untested, but I'd guess something liket this would do the job:
PyThreadState* current_interpreter = Py_NewInterpreter();
bp::object pyrun = exec(expStr);
Py_EndInterpreter(current_interpreter);
You may wrap that into an object. If you wish to do so, you must manage the "thread" state as explained in this other stackoverflow thread.
I'm trying to refactor parts of my project, particularly the Python/C++ interface.
The standard boost::python python initialization was working before:
boost::python::object main_module = boost::python::import("__main__");
boost::python::object globals(main_module.attr("__dict__"));
//...
However, after factoring that into a class of its own, I'm getting
TypeError: No to_python (by-value) converter found for C++ type: boost::python::api::proxy<boost::python::api::attribute_policies>
When an instantiating a PyInterface object, as below:
namespace py = boost::python;
class PyInterface
{
private:
py::object
main_module,
global,
tmp;
//...
public:
PyInterface();
//...
};
PyInterface::PyInterface()
{
std::cout << "Initializing..." << std::endl;
Py_Initialize();
std::cout << "Accessing main module..." << std::endl;
main_module = py::import("__main__");
std::cout << "Retrieve global namespace..." << std::endl;
global(main_module.attr("__dict__"));
//...
}
//in test.cpp
int main()
{
PyInterface python;
//...
}
Running gives the following output:
Initializing...
Accessing main module...
Retrieving global namespace...
TypeError: No to_python (by-value) converter found for C++ type: boost::python::api::proxy<boost::python::api::attribute_policies>
The only thing I can think is that it has something to do with declaring "globals" before using it. In which case, is there another way that I can do this?
Ah! Fixed it.
Changing the call to globals in the constructor from
globals(main_method.attr("__dict__"));
to using the assignment operator instead:
globals = main_method.attr("__dict__");
Looking back, that seems perfectly obvious, but at least I know I wasn't the only one stumped judging by the lack of anyone derping me.
I want to embed python in my C++ application. I'm using Boost library - great tool. But i have one problem.
If python function throws an exception, i want to catch it and print error in my application or get some detailed information like line number in python script that caused error.
How can i do it? I can't find any functions to get detailed exception information in Python API or Boost.
try {
module=import("MyModule"); //this line will throw excetion if MyModule contains an error
} catch ( error_already_set const & ) {
//Here i can said that i have error, but i cant determine what caused an error
std::cout << "error!" << std::endl;
}
PyErr_Print() just prints error text to stderr and clears error so it can't be solution
Well, I found out how to do it.
Without boost (only error message, because code to extract info from traceback is too heavy to post it here):
PyObject *ptype, *pvalue, *ptraceback;
PyErr_Fetch(&ptype, &pvalue, &ptraceback);
//pvalue contains error message
//ptraceback contains stack snapshot and many other information
//(see python traceback structure)
//Get error message
char *pStrErrorMessage = PyString_AsString(pvalue);
And BOOST version
try{
//some code that throws an error
}catch(error_already_set &){
PyObject *ptype, *pvalue, *ptraceback;
PyErr_Fetch(&ptype, &pvalue, &ptraceback);
handle<> hType(ptype);
object extype(hType);
handle<> hTraceback(ptraceback);
object traceback(hTraceback);
//Extract error message
string strErrorMessage = extract<string>(pvalue);
//Extract line number (top entry of call stack)
// if you want to extract another levels of call stack
// also process traceback.attr("tb_next") recurently
long lineno = extract<long> (traceback.attr("tb_lineno"));
string filename = extract<string>(traceback.attr("tb_frame").attr("f_code").attr("co_filename"));
string funcname = extract<string>(traceback.attr("tb_frame").attr("f_code").attr("co_name"));
... //cleanup here
This is the most robust method I've been able to come up so far:
try {
...
}
catch (bp::error_already_set) {
if (PyErr_Occurred()) {
msg = handle_pyerror();
}
py_exception = true;
bp::handle_exception();
PyErr_Clear();
}
if (py_exception)
....
// decode a Python exception into a string
std::string handle_pyerror()
{
using namespace boost::python;
using namespace boost;
PyObject *exc,*val,*tb;
object formatted_list, formatted;
PyErr_Fetch(&exc,&val,&tb);
handle<> hexc(exc),hval(allow_null(val)),htb(allow_null(tb));
object traceback(import("traceback"));
if (!tb) {
object format_exception_only(traceback.attr("format_exception_only"));
formatted_list = format_exception_only(hexc,hval);
} else {
object format_exception(traceback.attr("format_exception"));
formatted_list = format_exception(hexc,hval,htb);
}
formatted = str("\n").join(formatted_list);
return extract<std::string>(formatted);
}
In the Python C API, PyObject_Str returns a new reference to a Python string object with the string form of the Python object you're passing as the argument -- just like str(o) in Python code. Note that the exception object does not have "information like line number" -- that's in the traceback object (you can use PyErr_Fetch to get both the exception object and the traceback object). Don't know what (if anything) Boost provides to make these specific C API functions easier to use, but, worst case, you could always resort to these functions as they are offered in the C API itself.
This thread has been very useful for me, but I had problems with the Python C API when I tried to extract the error message itself with no traceback. I found plenty of ways to do that in Python, but I couldn't find any way to do this in C++. I finally came up with the following version, which uses the C API as little as possible and instead relies much more on boost python.
PyErr_Print();
using namespace boost::python;
exec("import traceback, sys", mainNamespace_);
auto pyErr = eval("str(sys.last_value)", mainNamespace_);
auto pyStackTrace = eval("'\\n'.join(traceback.format_exception(sys.last_type, sys.last_value, sys.last_traceback))", mainNamespace_);
stackTraceString_ = extract<std::string>(pyStackTrace);
errorSummary_ = extract<std::string>(pyErr);
The reason this works is because PyErr_Print() also sets the value for sys.last_value, sys.last_type, and sys.last_traceback. Those are set to the same values as sys.exc_info would give, so this is functionally similar to the following python code:
import traceback
import sys
try:
raise RuntimeError("This is a test")
except:
err_type = sys.exc_info()[0]
value = sys.exc_info()[1]
tb = sys.exc_info()[2]
stack_trace = "\n".join(traceback.format_exception(err_type, value, tb))
error_summary = str(value)
print(stack_trace)
print(error_summary)
I hope someone finds this useful!
Here's some code based on some of the other answers and comments, nicely formatted with modern C++ and comments. Minimally tested but it seems to work.
#include <string>
#include <boost/python.hpp>
#include <Python.h>
// Return the current Python error and backtrace as a string, or throw
// an exception if there was none.
std::string python_error_string() {
using namespace boost::python;
PyObject* ptype = nullptr;
PyObject* pvalue = nullptr;
PyObject* ptraceback = nullptr;
// Fetch the exception information. If there was no error ptype will be set
// to null. The other two values might set to null anyway.
PyErr_Fetch(&ptype, &pvalue, &ptraceback);
if (ptype == nullptr) {
throw std::runtime_error("A Python error was detected but when we called "
"PyErr_Fetch() it returned null indicating that "
"there was no error.");
}
// Sometimes pvalue is not an instance of ptype. This converts it. It's
// done lazily for performance reasons.
PyErr_NormalizeException(&ptype, &pvalue, &ptraceback);
if (ptraceback != nullptr) {
PyException_SetTraceback(pvalue, ptraceback);
}
// Get Boost handles to the Python objects so we get an easier API.
handle<> htype(ptype);
handle<> hvalue(allow_null(pvalue));
handle<> htraceback(allow_null(ptraceback));
// Import the `traceback` module and use it to format the exception.
object traceback = import("traceback");
object format_exception = traceback.attr("format_exception");
object formatted_list = format_exception(htype, hvalue, htraceback);
object formatted = str("\n").join(formatted_list);
return extract<std::string>(formatted);
}
Btw I was curious why everyone is using handle<> instead of handle. Apparently it disables template argument deduction. Not sure why you'd want that here but it isn't the same anyway, and the Boost docs say to use handle<> too so I guess there is a good reason.