Been breaking my head trying to pass vectors by reference to a C++ template method but all I get back is an empty list, apparently the parameter is being passed by value.
I am using Cython 0.18 & Python 2.7
Any ideas?
C++ side
class VectByRef
{
public:
VectByRef::VectByRef();
template<typename T>
void GetVector(T& var);
}
template<typename T>
void VectByRef::GetVector(T& var)
{
var.push_back(1);
var.push_back(2);
var.push_back(3);
}
Cython side
cdef extern from "VectByRef.h":
cdef cppclass VectByRef:
VectByRef() except
vector[cython.int] GetVector(vector[cython.int])
def getVector(self):
cdef vector[cython.int] resultVectInt
self._vectByRef.GetVector(<vector[cython.int]> resultVectInt)
print(resultVectInt) # The result is an empty list []
I'm not sure this actually answers the question, but I can't comment yet to get clarification.
When I tried to compile this I got an error:
error: no matching function for call to ‘VectByRef::GetVector(std::vector<int>)’
The error was resolved by changing the call on the Cython side to
.GetVector(<vector[cython.int]&> resultVectInt)
removing the type qualification altogether also worked:
.GetVector(resultVectInt)
Both of these gave the desired result as well: [1, 2, 3]
This was also using Python 2.7, with both Cython 0.17 & 0.19. So I'm guessing either something else is going on, or there's a specific bug with 0.18.
Related
Currently, I try to get the latest version of facebook's flint to get compiled.
The code fails with the D compiler version 2.081.1 on a similar construct as
import std.stdio;
void main()
{
long[] foo = [];
foo.clear();
}
Here is the short link to the example: https://run.dlang.io/is/ZSsPNS
with the messages:
onlineapp.d(5): Error: template object.clear cannot deduce function
from argument types !()(long[]), candidates are:
/dlang/dmd/linux/bin64/../../src/druntime/import/object.d(2855):
object.clear(T : Value[Key], Value, Key)(T aa)
/dlang/dmd/linux/bin64/../../src/druntime/import/object.d(2860):
object.clear(T : Value[Key], Value, Key)(T* aa)
How can this ambiguity be solved?
Any help is very much appreciated!
There is no ambiguity there - clear is a function that operates on associative arrays, not dynamic arrays, which is what long[] is.
Now, that of course doesn't mean the code works. :p
The issue is clear was renamed to destroy in 2.066, and removed in 2.070. Then, in 2.071, the current function was introduced, which clears AAs of their contents.
So, in summary: replace clear with destroy, and things should work.
I am using Boost Python, I generate a large vector of integers in C++, and I would like to access this vector in Python without copying it.
In C++ I have:
BOOST_PYTHON_MODULE(myModule)
{
class_<vector<int>>("vectorInt").def(vector_indexing_suite<vector<int>>());
def("ReturnVectorPtr", ReturnVectorPtr, return_value_policy<manage_new_object>());
}
vector<int>* ReturnVectorPtr()
{
return new vector<int>();
}
Then in python I have:
import myModule
myModule.ReturnVectorPtr()
This causes Python to crash, although I'm not even storing the return value. Any ideas on what my mistake is?
Edit:
The following code works for getting the data in the vector from C++ to python, but leaks memory. Are the vectors being copied and then not disposed?
In C++:
BOOST_PYTHON_MODULE(myModule)
{
class_<vector<int>>("vectorInt").def(vector_indexing_suite<vector<int>>());
def("ModifyVectorInPlace", ModifyVectorInPlace);
}
void ModifyVectorInPlace(vector<int>& data)
{
// Modify data...
return;
}
Then in python I have:
import myModule
vectorInt = myModule.vectorInt()
myModule.ModifyVectorInPlace(vectorInt)
What is going on?
Edit 2:
I tried the "Raw C++ Pointers" example from here, exactly as written:
https://wiki.python.org/moin/boost.python/PointersAndSmartPointers
It crashes too. It seems that I can't get a pointer to anything passed into Python for some reason...
Edit 3:
The crash appears to be a segfault from invoke.hpp, in this function:
template <class RC, class F BOOST_PP_ENUM_TRAILING_PARAMS_Z(1, N, class AC)>
inline PyObject* invoke(invoke_tag_<false,false>, RC const& rc, F& f BOOST_PP_ENUM_TRAILING_BINARY_PARAMS_Z(1, N, AC, & ac) )
{
return rc(f( BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () BOOST_PP_INTERCEPT) ));
}
It turns out this was a bug in the interaction between Mingw-w64 and Python. I performed the procedure described here and the problem was solved:
http://ascend4.org/Setting_up_a_MinGW-w64_build_environment#Setup_Python_for_compilation_of_extensions
As I'm still new to this, I'm facing some problems, here's my C++ code:
#include <python.h>
#define DLLEXPORT extern "C" __declspec(dllexport)
DLLEXPORT PyObject *Add(PyObject *pSelf, PyObject *pArgs)
{
int s,d;
if(!PyArg_ParseTuple(pArgs, "ii" , &s, &d))
{
PyErr_SetString(PyExc_TypeError,
"Add() invalid parameter");
return NULL;
}
return Py_BuildValue("i", s + d);
}
And the Python code:
import ctypes
MyDll = ctypes.cdll.LoadLibrary(r"PyToCppTest.dll")
jj = MyDll.Add(1,2)
I get an error when I run the above Python code:
OSError: exception: access violation reading 0x000000000000000A
I want to pass the data, without converting it, from Python to C++, then convert it inside C++.
Use either an extension or ctypes; you're not supposed to call your extension through ctypes. The point of extensions is to be able to create modules that look native to people using them from Python. ctypes serves to call C code that was written completely oblivious of Python.
There are a few things that are wrong with your code. First and foremost, the proper include is:
#include <Python.h>
Note the capital P. You're probably on Windows, but this wouldn't work on Linux without the capital P.
Also, I don't see the point of the *pSelf pointer in your function declaration, you should get rid of it:
PyObject *Add(PyObject *pArgs)
Now, your main problem is this:
MyDll.Add(1,2)
...does not call MyDll.Add with a tuple. It calls it with two integer arguments, 1 and 2. If you want to pass a tuple, you'd do:
MyDll.Add((1,2))
However, Python's ctypes won't know what to do with this (it normally accepts integer arguments), so you'll need to tell it that Add actually wants a tuple, and returns a Python object:
import ctypes
MyDll = ctypes.cdll.LoadLibrary("PyToCppTest.dll")
MyCFunc = ctypes.PYFUNCTYPE(
ctypes.py_object, # return val: a python object
ctypes.py_object # argument 1: a tuple
)
MyFunc = MyCFunc(('Add', MyDll))
jj = MyFunc((1,2))
I have the following class:
#include <array>
template<unsigned short D>
class Point {
private:
std::array<float, D> coordinates;
public:
Point() { for(int i=D-1; i>=0; --i) coordinates[i] = 0.0; }
Point(const Point& rhs) = default;
Point& operator=(const Point& rhs) = default;
~Point() = default;
float& get_ref(const unsigned short dimension)
{ return coordinates[dimension-1]; }
};
I'm trying to wrap it with:
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(fernpy) {
using namespace boost::python;
class_< Point<2> >("point")
.def("__call__", &Point<2>::get_ref, return_internal_reference<>());
}
I'm using gcc-4.7 to compile for boost 1.48, python-2.7 on Fedora 17. All the code is a file called testpy.cpp. I'm using these commands to compile:
g++ -std=c++11 -g -fPIC -I/usr/include/python2.7 -c testpy.cpp
g++ -shared -g -lpython2.7 -lboost_python -o libfern.so testpy.o
The compiler returns a bunch of boost internal errors, too many to post here. This excerpt seems to be the core of it. There are a bunch of "required from"s before it and "note"s after.
/usr/include/boost/python/object/make_instance.hpp:27:9: error: no matching function for call to ‘assertion_failed(mpl_::failed************ boost::mpl::or_<boost::is_class<float>, boost::is_union<float>, mpl_::bool_<false>, mpl_::bool_<false>, mpl_::bool_<false> >::************)’
It works just fine if I return a plain float from get_ref and remove the return_internal_reference<>() argument from the .def line of the wrapper. It's weird because I'm doing the same thing with another, more complicated class template and it works just fine there too. I've been Googling and banging my head against this for almost a full day now. Anybody have any idea what the heck is going on?
UPDATE:
I ended up using the "getitem" and "setitem" special methods from python, a la this link. The link shows how to define a nifty struct template with static wrappers for access functions, so you don't have to mess with the interface to your original C++ class.
From a python point-of-view, floats are an immutable-type. As such, python does not allow changing the value.
For example, the following occurs in python:
coordinates = [ 5, 10, 15 ]
x = cooardinates[ 2 ] # Bind x to refer to the int(15) object.
x = 5 # Rebind x to refer to the int(5) object.
# Does not modify coordinates.
Now, consider the following:
from fernpy import point
p = point()
x = p(2) # Bind x to refer to the float(p(2)) object.
x = 5 # Rebind x to refer to the int(5) object.
# Does not set p.coordinates[2] to 5.
Thus, boost::python prevents returning reference to types that will be immutable in python because Python does not support it. x does not store the value 5; instead, it contains a reference to the 5 object. If assigning to x did not rebind x, then nonsensical statements, such as 6 = 5 would be possible.
The compile error is a static check that limits return_internal_reference to only work with classes or unions, as those will be mutable types within Python. I imagine that the 'more complicated class template' that works is returning a reference to a user-type.
The short answer is probably that you don't want to return an internal reference to a float. In Python, numbers are immutable, so just returning a copy is safer, and sacrifices no features or speed.
If you want to return something more complicated (such as a list or reference to another wrapped class), you can, but it's still almost always not what you want; the dependency of one object on the other introduces fragility. If you want to be able to modify the internal state of an object, you're probably better off using getters and setters, and copying the data in and out.
I have a C++ library that has a Python wrapper (written with SWIG). This library allows executing small user-defined code (a callback), such as element-wise operations on a vector. I.e. instead of just a + you can do whatever arbitrary binary function. Right now this is accomplished by accepting a callable Python object for the binary function and calling it. It works, but is about 80 times slower than code that doesn't have to bounce up and down into Python at every iteration.
How would I write/build/import a Cython function could be passed into my C++ library so that it can be called directly by the C++ library?
Edit:
If I just stuck to C then I would write something like
EWise(double (*callback)(double, double))
EWise would then callback(10, 20); or such. I want callback to be written in Cython, using whatever name the user wants, and a pointer to it has to be passed to my C++ library through Python somehow. That somehow is where I'm unclear.
The trick with cython is in using the keyword public
cdef public double cython_function( double value, double value2 ):
return value + value2
Then the command cythonize <your_file.pyx> along with <your_file.c> will create header <your_file.h> that you can include.
Alternatively, you can create the header yourself:
#ifdef __cplusplus {
extern "C"
#endif
double cython_function( double value, double value2 );
#ifdef __cplusplus
}
#endif
Update:
Then with a little overlay from Python you can use ctypes's callback mechanism
func_type = CFUNCTYPE(c_double, c_double, c_double)
your_library.set_callback_function ( func_type(user_modules.cython_function) )
You can achieve that by doing pure cdef functions :
# declare the prototype of your function
ctypedef void (*callback_ptr)(int arg)
# declare your function as cdef
cdef void my_callback(int arg):
print 'doing some python here', arg
# now, you can use the cdef func as a callback
# in pure C !
cdef void run():
cdef callback_ptr p = my_callback
p(42)
if __name__ == '__main__':
run()
Note: you can use "cython -a" to see that they are no python code involved for the content of run. So it will work with your c library.
Embedding Python in Another Application may be useful reading.