how to deal with the PyObject* from C++ in Python - c++

I create DLL wrote in C++ , the exporting function returns PyObject * .Then I use ctypes to import the DLL in Python . Now , how can I get the real PyObject ??
here's some part of c++ code:
PyObject* _stdcall getList(){
PyObject * PList = NULL;
PyObject * PItem = NULL;
PList = PyList_New(10);
vector <int> intVector;
int i;
for(int i=0;i<10;i++){
intVector.push_back(i);
}
for(vector<int>::const_iterator it=intVector.begin();it<intVector.end();it++){
PItem = Py_BuildValue("i", &it);
PyList_Append(PList, PItem);
}
return PList;
}
and some python code :
dll = ctypes.windll.LoadLibrary(DllPath)
PList = dll.getList()
*I wanna get the real python list containing 1,2,3,4...10 ? *
Am I clear ?? Thanks advance

You have a number of issues of your code, some modifications:
#include <Python.h>
#include <vector>
extern "C" PyObject* _stdcall getList(){
PyObject *PList = PyList_New(0);
std::vector <int> intVector;
std::vector<int>::const_iterator it;
for(int i = 0 ; i < 10 ; i++){
intVector.push_back(i);
}
for(it = intVector.begin(); it != intVector.end() ; it++ ){
PyList_Append(PList, Py_BuildValue("i", *it));
}
return PList;
}
compile it:
> g++ -Wall -shared lib.cpp -I \Python27\include -L \Python27\libs -lpython27 -o lib.dll -Wl,--add-stdcall-alias
now you can load it as any function and set the getList return type to py_object as:
import ctypes
lib = ctypes.WinDLL('lib.dll')
getList = lib.getList
getList.argtypes = None
getList.restype = ctypes.py_object
getList()
test it:
>>> import ctypes
>>>
>>> lib = ctypes.WinDLL('lib.dll')
>>>
>>> getList = lib.getList
>>> getList.argtypes = None
>>> getList.restype = ctypes.py_object
>>> getList()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>
>>>

With Visual Studio, and Python 64 bits:
1- Create an empty Win32 Project (DLL Type)
2- Right Click on your Solution Project -> Configuration Manager
3- Active Solution configuration (Release)
4- Active Solution Platform -> New, then on the bottom Dropdown list, select x64 -> OK
5- In the Source Files Folder, add an empty C++ file
6- Put your C++ code (One modification for getList to be recognized)
#include <Python.h>
#include <vector>
extern "C" __declspec(dllexport) PyObject* _stdcall getList();
PyObject* _stdcall getList(){
PyObject *PList = PyList_New(0);
std::vector <int> intVector;
std::vector<int>::const_iterator it;
for (int i = 0; i < 10; i++){
intVector.push_back(i);
}
for (it = intVector.begin(); it != intVector.end(); it++){
PyList_Append(PList, Py_BuildValue("i", *it));
}
return PList;
}

I'm not exactly clear what you are asking. But I suppose you mean to ask what you can do now with your DLL.
Well, in order to use it appropriately, you'll have to build a special DLL which directly can be imported as a module in Python. In order to determine what to do in order to use this, it is best you have a look for other modules, how they do it. E. g. MySQLdb could be a candidate.
In short, you have this "wrapper" DLL call your function.
But if I have a second look at your question now, I see that you are trying to load your DLL via ctypes. This is viable as well, maybe even better, and you'll have to use the ctypes.py_object data type.

Related

returning tuple of list of lists of complex from c++ to python

I have a PyObject* outvar object which is basically constructed as:
//some defs
typedef std::complex<double> Pt;
typedef std::vector<Pt> Pgon;
typedef std::vector<Pgon> Pgons;
PyObject* outvar = PyTuple_New(2);
auto outA = pgons2pylist(pA);
auto outB = pgons2pylist(pB);
PyTuple_SET_ITEM(outvar, 0, outp);
PyTuple_SET_ITEM(outvar, 1, outl);
where pgons2pylist
PyObject* pgons2pylist(const Pgons& data) {
PyObject* listObj = PyList_New( data.size() );
for (unsigned int i = 0; i < data.size(); i++) {
PyList_SET_ITEM(listObj, i, pgon2pylist(data[i]));
}
return listObj;
}
and pgon2pylist is:
PyObject* pgon2pylist(const Pgon& data) {
PyObject* listObj = PyList_New( data.size() );
for (unsigned int i = 0; i < data.size(); i++) {
PyList_SET_ITEM(listObj, i, PyComplex_FromDoubles(data[i].real(),data[i].imag()));
}
return listObj;
}
I compile it and run it from py file as:
mylib = ctypes.cdll.LoadLibrary('./mylib.so')
out_data = mylib.call(some_args)
but out_data is always an integer! how can I converted to [[complex]]?
The way you structured your C code, looks closer to an extension module ([Python 3]: Extending Python with C or C++) rather than a simple .dll. Check [SO]: Pass str as an int array to a Python C extended function (extended using SWIG) (#CristiFati's answer) for a comparison between methods.
Then, as a note you need to specify argtypes and restype for an imported function (this is the exact reason why you get an int). Check [SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated for what might happen if you don't.
Also listing [Python 3]: ctypes - A foreign function library for Python page.
Couple of notes about the code:
Since you use Python C API functions in your .dll, you should call it via ctypes.PyDLL (ctypes.pydll.LoadLibrary)
Since you're returning a PyObject*, on Python side you should use py_object (check [SO]: How to cast a ctypes pointer to an instance of a Python class for more details)
So, assuming that you have a working .dll, here's how would you use it (posting the code blindly):
mylib = ctypes.pydll.LoadLibrary('./mylib.so')
outvar = ctypes.py_object.in_dll(mylib, "outvar") # Note that you might have to declare it as extern "C", so its name doesn't get mangled
#EDIT0:
I created a dummy example to test whether everything works.
dll.c:
#include <Python.h>
#if defined(_WIN32)
# define EXPORT __declspec(dllexport)
#else
# define EXPORT
#endif
EXPORT PyObject *tp = NULL;
EXPORT int i = 123;
EXPORT char *s = "Gainarie";
EXPORT float f = -3.14;
EXPORT void initTpl() {
tp = PyTuple_New(2);
PyTuple_SET_ITEM(tp, 0, PyLong_FromLong(7));
PyTuple_SET_ITEM(tp, 1, PyLong_FromLong(-9));
}
code.py:
#!/usr/bin/env python3
import sys
import ctypes
def main():
dll = ctypes.PyDLL("./dll.so")
i = ctypes.c_int.in_dll(dll, "i")
s = ctypes.c_char_p.in_dll(dll, "s")
f = ctypes.c_float.in_dll(dll, "f")
dll.initTpl()
tp = ctypes.py_object.in_dll(dll, "tp")
print(i.value, s.value, f.value, tp.value, type(tp.value))
if __name__ == "__main__":
print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
main()
Notes:
I only tested on Win, as I didn't transfer my files on my Lnx VM, but this shouldn't be a problem
Since it's for demo purposes only, I didn't care about memory leaks (nor did I check whether Py_XDECREF is necessary)
Output:
e:\Work\Dev\StackOverflow\q054429301>dir /b
code.py
dll.c
e:\Work\Dev\StackOverflow\q054429301>"c:\Install\x86\Microsoft\Visual Studio Community\2015\vc\vcvarsall.bat" x64
e:\Work\Dev\StackOverflow\q054429301>cl /nologo /DDLL /MD /I"c:\Install\x64\Python\Python\03.06.08\include" dll.c /link /NOLOGO /DLL /LIBPATH:"c:\Install\x64\Python\Python\03.06.08\libs" /OUT:dll.so
dll.c
Creating library dll.lib and object dll.exp
e:\Work\Dev\StackOverflow\q054429301>dir /b
code.py
dll.c
dll.exp
dll.lib
dll.obj
dll.so
e:\Work\Dev\StackOverflow\q054429301>"e:\Work\Dev\VEnvs\py_064_03.06.08_test0\Scripts\python.exe" code.py
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
123 b'Gainarie' -3.140000104904175 (7, -9) <class 'tuple'>
I understand from your example that you want to call a C++ function from Python which takes a list of complex numbers and return them.
If you use pybind11 you get the conversion from std::vector and std::complex for free. pybind11 is a header only library.
Here is an example
example.cpp
#include <pybind11/pybind11.h>
#include <pybind11/complex.h>
#include <pybind11/stl.h>
namespace py = pybind11;
std::complex<double> foo(std::complex<double> a)
{
return a;
}
std::vector<std::complex<double>> foo2(std::vector<std::complex<double>>& v)
{
return v;
}
PYBIND11_MODULE(samplepy, m) {
m.def("foo", &foo, "Return the complex number entered as an argument");
m.def("foo2", &foo2, "Return the list of complex number entered as an argument");
}
CMakeLists.txt
cmake_minimum_required (VERSION 3.13)
project (samplepy CXX)
set(CMAKE_CXX_STANDARD 17)
file (GLOB_RECURSE SOURCES "*.cpp" "*.h")
if (WIN32)
find_package(PythonInterp)
find_package(PythonLibs 3.6 REQUIRED)
set(PYBIND11_CPP_STANDARD /std:c++latest)
else()
find_package(PythonLibs 3.6 REQUIRED)
set(PYBIND11_CPP_STANDARD -std=c++1z)
endif()
add_library (samplepy SHARED ${SOURCES})
target_link_libraries(samplepy PRIVATE ${PYTHON_LIBRARIES})
set_target_properties(samplepy PROPERTIES PREFIX "${PYTHON_MODULE_PREFIX}" SUFFIX "${PYTHON_MODULE_EXTENSION}")
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${PYTHON_INCLUDE_DIR})
example.py
import samplepy
z = complex(2, -3)
print(type(samplepy.foo(z)))
print(samplepy.foo(z))
l = [complex(1, -2), complex(3, -4)]
print(type(samplepy.foo2(l)))
print(samplepy.foo2(l))
When I run this I get this on the py console
<class 'complex'>
(2-3j)
<class 'list'>
[(1-2j), (3-4j)]

How to have a reference to a typemaped class in swig?

I have a linear algebra matrix class in C++ and want to build a wrapper to Numpy using SWIG. Everything works fine, but if I want to call a function with a reference as argument from python, I get a TypeError.
Here is the swig code:
%{
#define SWIG_FILE_WITH_INIT
%}
%include "typemaps.i"
%include "numpy.i"
%module arraypy
%{
#include "Core/array.h"
#include "Core/array_t.h"
%}
%fragment("NumPy_Fragments");
%init %{
import_array();
%}
%typemap(in) Array<double> {
if(is_array($input)) {
uint size = 1;
for(uint i=0; i<array_numdims($input); ++i)
size *= array_size($input, i);
$1.resize(size);
memcpy($1.p, array_data($input), size*sizeof(double));
$1.nd = array_numdims($input);
$1.N = size;
$1.d0 = array_size($input, 0);
$1.d1 = array_size($input, 1);
$1.d2 = array_size($input, 2);
}
}
%typemap(out) Array<double> {
long dims[3] = { $1.d0, $1.d1, $1.d2 };
PyArrayObject *a = (PyArrayObject*) PyArray_SimpleNew($1.nd, dims, NPY_DOUBLE);
memcpy(PyArray_DATA(a), $1.p, $1.N*sizeof(double));
$result = PyArray_Return(a);
}
%inline %{
void testing(Array<double>& a) {
std::cout << a << endl;
}
%}
If I now run e.g. ipython:
$ import arraypy
$ import numpy
$ arraypy.testing(numpy.array([1, 2, 3]))
I get TypeError: in method testing, argument 1 of type ¨Array< double > &¨
If I add a specific typemap to Array<double> & copying the working one, it doesn compile, since $1 somehow got a pointer. If I account for that, I get a segfault.
How can I make the wrapper work with references as well? (And the same of course is true for pointers instead of references)
As described in http://www.swig.org/Doc2.0/SWIGPlus.html#SWIGPlus_nn18, SWIG translates references back to pointers.
So my first attemp was right: Just copy the typemap for references and pointers. The segfault resulted from something different. To make everything a bit nice a fragmet can be used.

Numpy C++: How to iterate over PyArrayObject without a segfault

For me, the following all result in a segfault:
my_array->descr->subarray->shape;
my_array->dimensions;
PyArray_SHAPE(my_array);
PyArray_DIMS(my_array);
PyArray_ITEMSIZE(my_array);
PyArray_NBYTES(my_array);
My function looks like this:
static PyObject* exterior(PyObject* self, PyArrayObject* old_simplices_array)
{//code here
The rest of my cpp file looks like this:
#include "Python.h"
#include "numpy/arrayobject.h"
/* function */
static PyMethodDef compiled_methods[] =
{
{"_exterior",(PyCFunction)exterior , METH_VARARGS},
{NULL, NULL} /* Sentinel */
};
PyMODINIT_FUNC init_alto(void)
{
(void) Py_InitModule("_alto", compiled_methods);
import_array();
}
The python code that passes the array to "exterior" just passes an NxM uint array. That part works. I can access the array's strides and data. I just cannot determine the bounds of iteration. I am working from within sage if that makes any difference.
How am I supposed to iterate over an array without segfaulting? If the answer is obvious, please idiotproof your answer.
For a better idea of what the function looks like, see here.
In the past I have done the following to iterate over a PyArrayObject:
static PyObject *func1(PyObject *self, PyObject *args) {
PyArrayObject *X;
int ndX;
npy_intp *shapeX;
PyArray_Descr *dtype;
NpyIter *iter;
NpyIter_IterNextFunc *iternext;
PyArg_ParseTuple(args, "O!", &PyArray_Type, &X);
ndX = PyArray_NDIM(X);
shapeX = PyArray_SHAPE(X);
dtype = PyArray_DescrFromType(NPY_DOUBLE);
iter = NpyIter_New(X, NPY_ITER_READONLY, NPY_KEEPORDER, NPY_NO_CASTING, dtype);
if (iter==NULL) {
return NULL;
}
iternext = NpyIter_GetIterNext(iter, NULL);
dataptr = (double **) NpyIter_GetDataPtrArray(iter);
do {
cout << **dataptr << endl;
} while (iternext(iter));
NpyIter_Deallocate(iter);
return Py_BuildValue(something);
}
To find out more information check out this link: http://docs.scipy.org/doc/numpy/reference/c-api.iterator.html

%typemapping of a C++ Library for Python Interface

I want to create a python wrapper for my C++ library. It would be cool, if there is a automatic conversion of std::vector to python lists and the other way round.
Unfortunatly if I add this code to my Interface-file I still get errors in run-time.
%typemap(in) std::vector<float> value (std::vector<float> vIn) {
int iLen = PySequence_Length($input);
for(unsigned int i = 0; i < iLen; i++) {
PyObject *o = PySequence_GetItem($input, i);
if (PyNumber_Check(o)) {
vIn.push_back((float)PyFloat_AsDouble(o) );
}
}
$1 = vIn;
}
%typemap(out) std::vector<float> {
std::vector<float> vOut = $1;
int iLen = vOut.size();
$result = PyList_New(iLen);
for(unsigned int i = 0; i < iLen; i++) {
double fVal = vOut.at(i);
PyObject *o = PyFloat_FromDouble((double) fVal);
PyList_SetItem($result, i, o);
}
}
Class header:
class TrainingSet {
private:
std::vector<std::vector<float> > m_vInputList;
std::vector<std::vector<float> > m_vOutputList;
public:
void AddInput(const std::vector<float> &vIn);
// ..
Python code:
trainSet = TrainingSet()
trainSet.AddInput([0.5, 0.5, 0.5])
Error:
File "runSOMNet.py", line 9, in <module>
trainSet.AddInput([0.5, 0.5, 0.5])
File "/home/dgrat/annetgpgpu/build/ANNet/ANPyNetCPU.py", line 674, in AddInput
def AddInput(self, *args): return _ANPyNetCPU.TrainingSet_AddInput(self, *args)
NotImplementedError: Wrong number or type of arguments for overloaded function 'TrainingSet_AddInput'.
Possible C/C++ prototypes are:
ANN::TrainingSet::AddInput(std::vector< float,std::allocator< float > > const &)
ANN::TrainingSet::AddInput(float *,unsigned int const &)
The std_vector.i library in SWIG provides support for std::vector.
http://www.swig.org/Doc2.0/Library.html#Library_stl_cpp_library
You just need to tell SWIG about the template instantiations you want it to know about:
%include "std_vector.i"
namespace std {
%template(FloatVector) vector<float>;
}
Note that the following Python code will work, but will incur an array copy:
for x in range(0, 3):
list[x] = x
myModule.myFunction(list)
To do the same thing without incurring a copy, construct the list using the SWIG-generated proxy object constructor:
list = myModule.FloatVector()
for x in range(0, 3):
list[x] = x
myModule.myFunction(list)
If you're going to define the %typemaps manually, you will also need a %typemap(check), something like:
%typemap(typecheck) std::vector<float>& {
$1 = PySequence_Check($input) ? 1 : 0;
}
I believe the rule of thumb is that if you define a %typemap(in), you should also define a %typemap(check) --- otherwise, the generated code never gets to where it's put your %typemap(in).

Calling Windows API with libffi on MinGW

I'm trying to have FFI support for my new programming language, which is written in C++ with QT Creator using the MinGW toolchain.
To do this I used a custom-built version of libffi found here: http://ftp.gnome.org/pub/GNOME/binaries/win32/dependencies/libffi-dev_3.0.6-1_win32.zip
I also tried it with another build: http://pkgs.org/fedora-14/fedora-updates-i386/mingw32-libffi-3.0.9-1.fc14.noarch.rpm.html by downloading the SRPM file on Linux, extracting it, and copying the needed files to a Windows partition.
Anyway, I included the required header file, added the import library to the project and put the .dll beside the application's .exe, it compiles and runs, calling MessageBeep() successfully. I tried it next with MessageBoxA(), but it keeps crashing. The debugger doesn't seem to provide much useful information (edit: beside the fact that a call to MessageBoxA did happen) so I keep fiddling with stuff and re-running to no avail.
To isolate the problem from the details of my language, I tried to manually call MessageBoxA by filling myself all the parameters, resulting in the code below, still crashing.
So my question distills to: How can I get the code snippet below to run under QT Creator/MinGW and actually show a message box?
#include "libffi/include/ffi.h"
#include <QLibrary>
void testMessageBox()
{
int n = 4;
ffi_cif cif;
ffi_type **ffi_argTypes = new ffi_type*[n];
void **values = new void*[n];
values[0] = new ulong(0);
values[1] = (void *) "hello";
values[2] = (void *) "mommy";
values[3] = new int32_t(0);
ffi_argTypes[0] = &ffi_type_ulong;
ffi_argTypes[1] = &ffi_type_pointer;
ffi_argTypes[2] = &ffi_type_pointer;
ffi_argTypes[3] = &ffi_type_uint32;
ffi_type *c_retType = &ffi_type_sint32;
int32_t rc; // return value
if (ffi_prep_cif(&cif, FFI_STDCALL, n, c_retType, ffi_argTypes) == FFI_OK)
{
QLibrary lib("user32.dll");
lib.load();
void *msgbox = lib.resolve("MessageBoxA");
ffi_call(&cif, (void (*)()) msgbox, &rc, values);
}
}
you should pass the address to the values array instead of the values. the working code under mingw64 is
#include <stdio.h>
#include <ffi.h>
#include <Windows.h>
int main()
{
ffi_cif cif;
HINSTANCE dllHandle = LoadLibrary("user32.dll");
int n = 4;
ffi_type *ffi_argTypes[4];
void *values[4];
UINT64 a=0;
UINT32 b=0;
TCHAR* s1= "hello";
TCHAR* s2= "hello2";
values[0] = &a;
values[1] = &s1;
values[2] = &s2;
values[3] = &b;
ffi_argTypes[0] = &ffi_type_uint64;
ffi_argTypes[1] = &ffi_type_pointer;
ffi_argTypes[2] = &ffi_type_pointer;
ffi_argTypes[3] = &ffi_type_uint;
ffi_type *c_retType = &ffi_type_sint;
ffi_type rc; // return value
if (ffi_prep_cif(&cif, FFI_DEFAULT_ABI, 4, &ffi_type_sint, ffi_argTypes) == FFI_OK) {
ffi_call(&cif, FFI_FN(GetProcAddress(dllHandle,"MessageBoxA")), &rc, values);
}
return 0;
}