I am using Boost Python, I generate a large vector of integers in C++, and I would like to access this vector in Python without copying it.
In C++ I have:
BOOST_PYTHON_MODULE(myModule)
{
class_<vector<int>>("vectorInt").def(vector_indexing_suite<vector<int>>());
def("ReturnVectorPtr", ReturnVectorPtr, return_value_policy<manage_new_object>());
}
vector<int>* ReturnVectorPtr()
{
return new vector<int>();
}
Then in python I have:
import myModule
myModule.ReturnVectorPtr()
This causes Python to crash, although I'm not even storing the return value. Any ideas on what my mistake is?
Edit:
The following code works for getting the data in the vector from C++ to python, but leaks memory. Are the vectors being copied and then not disposed?
In C++:
BOOST_PYTHON_MODULE(myModule)
{
class_<vector<int>>("vectorInt").def(vector_indexing_suite<vector<int>>());
def("ModifyVectorInPlace", ModifyVectorInPlace);
}
void ModifyVectorInPlace(vector<int>& data)
{
// Modify data...
return;
}
Then in python I have:
import myModule
vectorInt = myModule.vectorInt()
myModule.ModifyVectorInPlace(vectorInt)
What is going on?
Edit 2:
I tried the "Raw C++ Pointers" example from here, exactly as written:
https://wiki.python.org/moin/boost.python/PointersAndSmartPointers
It crashes too. It seems that I can't get a pointer to anything passed into Python for some reason...
Edit 3:
The crash appears to be a segfault from invoke.hpp, in this function:
template <class RC, class F BOOST_PP_ENUM_TRAILING_PARAMS_Z(1, N, class AC)>
inline PyObject* invoke(invoke_tag_<false,false>, RC const& rc, F& f BOOST_PP_ENUM_TRAILING_BINARY_PARAMS_Z(1, N, AC, & ac) )
{
return rc(f( BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () BOOST_PP_INTERCEPT) ));
}
It turns out this was a bug in the interaction between Mingw-w64 and Python. I performed the procedure described here and the problem was solved:
http://ascend4.org/Setting_up_a_MinGW-w64_build_environment#Setup_Python_for_compilation_of_extensions
Related
In linux, with gdb debugger I can print every object if the program is compiled with -g option. Is it possible to write a function like
template<typename T> ostream& to_stream(ostream& out, const T& obj);
that serialize my object with the name and type?
Naturally with help of gdb call or system call or other.
If you don't have gdb, you can't do much.
magic_get works in some limited cases.
For the rest you mostly have to manually specify the field names, possibly through a macro to make it more convenient, but still manual.
visual c++ - Printing values of all fields in a C++ structure - Stack Overflow
If you have gdb (as in, you can guarantee the program will be run in gdb) however, (as specified in the question),
use a technique in
c - Sending commands to gdb from the debugged program - Stack Overflow
we have (you need the Python code copied from that post run in gdb)
void gdb_run([[maybe_unused]] char const* str) {}
template<class T> void gdb_pretty_print([[maybe_unused]] T const& arg){
gdb_run("up\nprint arg");
}
struct Example{
int a;
char b;
};
int main(){
gdb_pretty_print(Example{1, '0'});
}
If you want to get the result to pass to C++ you can do something like
template<class T> void gdb_pretty_print([[maybe_unused]] T const& arg){
char const* volatile result=nullptr;
gdb_run("up\n"
"python gdb.set_convenience_variable('result', gdb.execute('print arg', to_string=True))\n"
"set result=$result");
__builtin_printf("result = %s\n", result);
}
Actually I'm not sure if volatile will do it, alternatively compile the particular with -O0.
Alternatively you may try to set break point within gdb_pretty_print itself, but I tried it's pretty hard.
class GdbPrettyPrint(gdb.Breakpoint):
def __init__(self, function_name):
super().__init__(function_name, internal=1)
self.silent = True
def stop(self):
gdb.execute("print arg")
return False
def set_breaks():
for b in gdb.rbreak("^gdb_pretty_print<"):
GdbPrettyPrint(b.location.split(":")[-1])
b.delete()
You have to run the set_breaks() function unfortunately. For some reason rbreak alone doesn't work for me.
Example program
template<class T> void gdb_pretty_print(T const& arg){
(void) arg;
}
struct Example{
int a;
char b;
};
int main(){
gdb_pretty_print(Example{1, '0'});
}
Yes, you can do it with python Pretty Printing API. To start with pretty printing, you can look at this question and Tom Tromey's blog: http://tromey.com/blog/?p=524.
I am trying to extend Python 3.6 using C++ on windows, following the official documentation and guide. The Anaconda distribution is used with numpy updated to the latest version. In the project python.h and arrayobject.h has been added to the C++ directories, and python36.lib linked, as advised.
For the code, a simple example which is supposed to create an array with the elements 0,1,2,3, when calling the func1 method:
#include <python.h>
#include <arrayobject.h>
static PyObject* fun(PyObject* self, PyObject* args)
{
PyObject *X;
int x[4] = {0,1,2,3};
int dims[1];
dims[0] = 1;
X = PyArray_SimpleNew(1, dims, NPY_INT64, x);
return X;
}
static PyMethodDef MyMethod[] = {
{ "func1", fun, METH_VARARGS,nullptr },
{ nullptr, nullptr, 0, nullptr }
};
static struct PyModuleDef MyModule = {
PyModuleDef_HEAD_INIT,
"MyModule",
NULL,
-1,
MyMethod
};
PyMODINIT_FUNC PyInit_MyModule(void)
{
(void)PyModule_Create(&MyModule);
import_array();
}
The code builds fine. When I take the resulting .pyd file and import it in Spyder, the kernel crashes. Specifically, the import_array(); command seems to cause the crash, as without it the kernel doesn't crash. However, as the documentation notes, the method crashes then. What is the fix?
It was solved by changing the order:
PyMODINIT_FUNC PyInit_MyModule(void)
{
import_array();
return PyModule_Create(&MyModule);
}
All the documentation seems to concern Python 2.X only, which used different initialization. In python 3, the code needs to modified. (There is a small error also in the creation of the array, but I let it be).
I have a small Python module written in C++. This module compiles well and can be imported to Python without any errors. These are some snippets of the code:
//...multiple includes
typedef variant<sqlite3 **, ... other types> VariantData; //!ONE
class ORM{
//... private fields
public:
VariantData Connect(int idx){
if(idx == 0) {
sqlite3 * conn;
sqlite3_open("reestr.db", &conn);
return &conn; //!TWO
// how to return a pointer to conn ???
// am I doing it right ???
}
}
//... other methods
// Python stuff
BOOST_PYTHON_MODULE(corm){
to_python_converter<VariantData, variant_to_object>(); //!THREE
implicitly_convertible<sqlite3 **, VariantData>(); //!THREE
//other unimportant code
So, this code compiles without errors. But,please, pay attention to four lines of code, marked by ONE, TWO and THREE, since they are causing errors. In my interpretation at line ONE I define a variant datatype that can take different values with sqlite3 ** among them. At lines THREE I define Python stuff responsible for converting variables and passing them between C++ and Python. However, I get into trouble when I try to use this module in Python. I do it like:
>>> import corm
>>> c = corm.ORM(...initialization parameters...)
>>> r = c.Connect() # !ERROR
... No to_python (by-value) converter found for C++ type: sqlite *
I really wonder why Python is talking about sqlite * type here, while the method itself does return &conn; and so returns a variable of type sqlite3 ** (as I believe). So, in a nutshell, my question is how to return a pointer of type sqlite3 ** from Connect method.
This is a solution. I'm not sure whether it is good, but at least it works:
typedef variant<void *, ... other types> VariantData;
class ORM{
//... private fields
public:
VariantData Connect(int idx){
if(idx == 0) {
sqlite3 * conn;
sqlite3_open("reestr.db", &conn);
void * ref = ::operator new(sizeof(conn));
ref = conn;
return ref;
}
...
}
....
};
struct variant_to_object {
static PyObject* convert(VariantData const& v){
return boost::python::incref(
boost::python::object(v).ptr());
}
};
BOOST_PYTHON_MODULE(corm){
to_python_converter<VariantData, variant_to_object>();
implicitly_convertible<void *, VariantData>();
But of course, I should take care of ref pointer and delete it, when it is no longer used.
I have a header file like:
#include <vector>
inline std::vector<uint8_t>& vec() {
static std::vector<uint8_t> v { 'a', 'b', 'c', 'd' };
return v;
}
inline const std::vector<uint8_t>& cvec() {
return vec();
}
I can wrap it in SWIG using std_vector.i and pyabc.i but that is quite inefficient (there's a jump between C++ and Python code for every access) and given that these are literally just a bunch of bytes I ought to be able to wrap them with Python's memoryview interface.
How can I expose my std::vector<uint8_t> as a Python memoryview?
Exposing it as a memoryview requires creating a Py_buffer first. In Python 3.3+ there is a convenient helper function, PyMemoryView_FromMemory that does a lot of the work for us. In earlier versions though we'll need to take a few extra steps, so our basic out typemap looks like:
%typemap(out) std::vector<uint8_t>&, const std::vector<uint8_t>& {
Py_buffer *buf=(Py_buffer*)malloc(sizeof *buf);
const bool ro = info<$1_type>::is_readonly();
if (PyBuffer_FillInfo(buf, NULL, &((*$1)[0]), (*$1).size(), ro, PyBUF_ND)) {
// error, handle
}
$result = PyMemoryView_FromBuffer(buf);
}
Here we're basically allocating some memory for the Py_buffer. This just contains the details of the buffer internally for Python. The memory we allocate will be owned by the memoryview object once it's created. Unfortunately since it's going to be released with a call to free() we need to allocate it with malloc(), even though it's C++ code.
Besides the Py_buffer and an optional Py_Object PyBuffer_FillInfo takes a void* (the buffer itself), the size of the buffer, a boolean indicating if it's writeable and a flag. In this case our flag simply indicates that we have provided C-style contiguous memory for the buffer.
For deciding if it is readonly or not we used SWIG's built in $n_type variable and a helper (which could be a C++11 type trait if we wanted).
To complete our SWIG interface we need to provide that helper and include the header file, so the whole thing becomes:
%module test
%{
#include "test.hh"
namespace {
template <typename T>
struct info {
static bool is_readonly() {
return false;
}
};
template <typename T>
struct info<const T&> {
static bool is_readonly() {
return true;
}
};
}
%}
%typemap(out) std::vector<uint8_t>&, const std::vector<uint8_t>& {
Py_buffer *buf=(Py_buffer*)malloc(sizeof *buf);
const bool ro = info<$1_type>::is_readonly();
if (PyBuffer_FillInfo(buf, NULL, &((*$1)[0]), (*$1).size(), ro, PyBUF_ND)) {
// error, handle
}
$result = PyMemoryView_FromBuffer(buf);
}
%include "test.hh"
We can then test it with:
import test
print test.vec()
print len(test.vec())
print test.vec()[0]
print test.vec().readonly
test.vec()[0]='z'
print test.vec()[0]
print "This should fail:"
test.cvec()[0] = 0
Which worked as expected, tested using Python 2.7.
Compared to just wrapping it using std_vector.i this approach does have some drawbacks. The biggest being that we can't resize the vector, or convert it back to a vector later trivially. We could work around that, at least partially by creating a SWIG proxy for the vector like normal and using the second parameter of PyBuffer_FillInfo to store it internally. (This would also be needed if we had to manage the ownership of the vector for instance).
I have written some C++ code that generates a std::vector.
I also have a python script that manipulates some data that, for now, I am declaring like this (below).
import numpy
x = numpy.random.randn(1000)
y = numpy.random.randn(1000)
I can run the script fine. From my C++ code:
using namespace boost::python;
try{
Py_Initialize();
object main = import("__main__");
object global(main.attr("__dict__"));
object result = exec_file("scatterPlot.py", global, global);
Py_Finalize();
}
catch(error_already_set){
PyErr_Print();
}
return;
I have no idea how to get my C++ data to python. I've around quite a bit, but there doesn't seem to be anything definitive.
I have in my C++
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
This seems to work, but as I understand, it only provides a class "PyVec" for my python script but not the data I need. Am I wrong?
I've also seen some other people use boost::shared_ptr in a python mailing list.
I also found this example but found it confusing.
I can think of a few approaches
Pass something to the boost::python::exec_file method
Using the boost_indexing_suite_ext
Uinsg boost::shared_ptr
Which approach is easiest to get going? No approach seems clear to me.
Here are some more links I've looked at:
from the boost website
from the python website
another mailing list thread
UPDATE:
This works for passing an int to my python code like below
int main(){
int five_squared=0;
int a =3;
try {
Py_Initialize();
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
main_namespace["var"]=a;
object ignored = exec("result = 5 ** var", main_namespace);
five_squared = extract<int>(main_namespace["result"]);
} catch( error_already_set ) {
PyErr_Print();
}
std::cout << five_squared << std::endl;
return 0;
}
But I want to pass a vector, when I try to do that in a similar fashion as above I get this error
TypeError: No to_python (by-value)
converter found for C++ type:
std::vector >
So, obviously I need to tell python how to deal with std::vector. I think this code could help with that.
BOOST_PYTHON_MODULE(vector_indexing_suite_ext){
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
}
But since std::vector is pretty common, there must be a defined way to do this... right?
The following code works for me (Python 2.6, Boost 1.39). This is almost the same as your code, except without the BOOST_PYTHON_MODULE line itself (but with the class_ definition for the vector). BOOST_PYTHON_MODULE only needs to be used when creating extension modules.
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
using namespace boost::python;
using namespace std;
int main()
{
vector<double> vec;
vec.push_back(1.2);
vec.push_back(3.4);
try {
Py_Initialize();
boost::python::class_<std::vector<double> >("PyVec")
.def(boost::python::vector_indexing_suite<std::vector<double> >());
object main_module = import("__main__");
object globals = main_module.attr("__dict__");
globals["var"]=vec;
object ignored = exec("result = sum(var)", globals, globals);
double result = extract<double>(globals["result"]);
std::cout << result << std::endl;
} catch( error_already_set ) {
PyErr_Print();
}
return 0;
}
I'm not sure if I understand correctly. After exporting your class "PyVec" which can hold std::vector<double>, you can export any c++ function taking vector as input or return type. So of course you can populate your vector within c++ and access this data in Python with the interfaced type "PyVec".