I have a C++ library that has a Python wrapper (written with SWIG). This library allows executing small user-defined code (a callback), such as element-wise operations on a vector. I.e. instead of just a + you can do whatever arbitrary binary function. Right now this is accomplished by accepting a callable Python object for the binary function and calling it. It works, but is about 80 times slower than code that doesn't have to bounce up and down into Python at every iteration.
How would I write/build/import a Cython function could be passed into my C++ library so that it can be called directly by the C++ library?
Edit:
If I just stuck to C then I would write something like
EWise(double (*callback)(double, double))
EWise would then callback(10, 20); or such. I want callback to be written in Cython, using whatever name the user wants, and a pointer to it has to be passed to my C++ library through Python somehow. That somehow is where I'm unclear.
The trick with cython is in using the keyword public
cdef public double cython_function( double value, double value2 ):
return value + value2
Then the command cythonize <your_file.pyx> along with <your_file.c> will create header <your_file.h> that you can include.
Alternatively, you can create the header yourself:
#ifdef __cplusplus {
extern "C"
#endif
double cython_function( double value, double value2 );
#ifdef __cplusplus
}
#endif
Update:
Then with a little overlay from Python you can use ctypes's callback mechanism
func_type = CFUNCTYPE(c_double, c_double, c_double)
your_library.set_callback_function ( func_type(user_modules.cython_function) )
You can achieve that by doing pure cdef functions :
# declare the prototype of your function
ctypedef void (*callback_ptr)(int arg)
# declare your function as cdef
cdef void my_callback(int arg):
print 'doing some python here', arg
# now, you can use the cdef func as a callback
# in pure C !
cdef void run():
cdef callback_ptr p = my_callback
p(42)
if __name__ == '__main__':
run()
Note: you can use "cython -a" to see that they are no python code involved for the content of run. So it will work with your c library.
Embedding Python in Another Application may be useful reading.
Related
When I try to use the following Cython code, I get the error I posted at the end about operator() not being defined. It appears that when I try to use operators Cython does not interpret it as a member function (notice there is no member access in the C++ source). If I try to call prng.operator()() then Cython will fail translation.
Is something special needed to use operator overloading in Cython?
import numpy as np
cimport numpy as np
cdef extern from "ratchet.hpp" namespace "ratchet::detail":
cdef cppclass Ratchet:
Ratchet()
unsigned long get64()
cdef extern from "float.hpp" namespace "prng":
cdef cppclass FloatPRNG[T]:
double operator()()
cdef FloatPRNG[Ratchet] prng
def ratchet_arr(np.ndarray[np.float64_t, ndim=1] A):
cdef unsigned int i
for i in range(len(A)):
A[i] = prng()
def ratchet_arr(np.ndarray[np.float64_t, ndim=2] A):
cdef unsigned int i, j
for i in range(len(A)):
for j in range(len(A[0])):
A[i][j] = prng()
ratchet.cpp: In function ‘PyObject* __pyx_pf_7ratchet_ratchet_arr(PyObject*, PyArrayObject*)’:
ratchet.cpp:1343:162: error: ‘operator()’ not defined
*__Pyx_BufPtrStrided1d(__pyx_t_5numpy_float64_t *, __pyx_pybuffernd_A.rcbuffer->pybuffer.buf, __pyx_t_3, __pyx_pybuffernd_A.diminfo[0].strides) = operator()();
Some more information inspired by Ianh. It appears that operator() cannot be used when the object is stack allocated
cat thing.pyx
cdef extern from 'thing.hpp':
cdef cppclass Thing:
Thing(int)
Thing()
int operator()()
# When this function doesn't exist, thing.so compiles fine
cpdef ff():
cdef Thing t
return t()
cpdef gg(x=None):
cdef Thing* t
if x:
t = new Thing(x)
else:
t = new Thing()
try:
return t[0]()
finally:
del t
cat thing.hpp
#pragma once
class Thing {
int val;
public:
Thing(int v): val(v) {}
Thing() : val(4) {}
int operator()() { return val; }
};
Update: This should be fixed as of Cython 0.24 and later. I've left the workaround here for completeness.
After looking through the C++ compiler errors of examples like yours a little more, what appears to be happening is that Cython has a bug when overloading operator() for a stack allocated object.
It appears to be trying to call operator() as if it were some sort of function you had defined instead of as a method of the C++ object you have defined.
There are two possible workarounds.
You can either alias the call operator and give it a different name in Cython than in C.
You can also just allocate the object on the heap instead.
Depending on your use case, it could be a good idea to just patch the C file generated by Cython.
You'd basically just have to search for hanging calls to operator() change them to method calls on the proper C++ object.
I tried this with my example below and it worked, and it wasn't terribly difficult to trace which objects I needed to insert into the code.
This approach will work well if you are only trying to write Python bindings to the library and will not be making a large number of calls to operator() at the Cython level, but it could become an awful pain if you have a large amount of development you intend to do in Cython.
You could try bug reporting it too.
That would be good regardless of which route you take to get it working.
It seems like the sort of thing that should be easy to fix as well, but I'm not an expert on Cython's internals.
Here's a minimal working example of how to use operator() for a heap allocated object in Cython. It works for me on Cython 0.21.
Thing.hpp
#pragma once
class Thing{
public:
int val;
Thing(int);
int operator()(int);};
Thing.cpp
#include "Thing.hpp"
Thing::Thing(int val){
this->val = val;}
int Thing::operator()(int num){
return this->val + num;}
Thing.pxd
cdef extern from "Thing.hpp":
cdef cppclass Thing:
Thing(int val) nogil
int operator()(int num) nogil
test_thing.pyx
from Thing cimport Thing
cpdef test_thing(int val, int num):
cdef Thing* t = new Thing(val)
print "initialized thing"
print "called thing."
print "value was: ", t[0](num)
del t
print "deleted thing"
setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from os import system
# First compile an object file containing the Thing class.
system('g++ -c Thing.cpp -o Thing.o')
ext_modules = [Extension('test_thing',
sources=['test_thing.pyx'],
language='c++',
extra_link_args=['Thing.o'])]
# Build the extension.
setup(name = 'cname',
packages = ['cname'],
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules)
After running the setup file, I start the Python interpreter in the same directory and run
from test_thing import test_thing
test_thing(1, 2)
and it prints the output
initialized thing
called thing.
value was: 3
deleted thing
showing that the operator is working properly.
Now, if you want to do this for a stack allocated object, you can change the Cython interface as follows:
Thing.pxd
cdef extern from "Thing.hpp":
cdef cppclass Thing:
Thing(int val) nogil
int call "operator()"(int num) nogil
test_thing.pyx
from Thing cimport Thing
cpdef test_thing(int val, int num):
cdef Thing t = Thing(val)
print "initialized thing"
print "called thing."
print "value was: ", t.call(num)
print "thing is deleted when it goes out of scope."
The C++ files and setup file can still be used as they are.
I created a sample ruby extension using C++ class. it works fine when i did not parse the value. but when i parse parameter it show an error. here is my code.
C++ header file
#ifndef CIRCLE_H_
#define CIRCLE_H_
class Circle {
public:
Circle(float radius):_radius(radius) {}
float getArea() { return 3.14159 * _radius * _radius; }
private:
float _radius;
};
CPP file.
#include<ruby.h>
#include"Circle.h"
VALUE method_test(VALUE y){
double x= NUM2DBL(y);
Circle *cir= new Circle(x);
return DBL2NUM(cir->getArea());
}
extern "C" void Init_Test(){
VALUE lemon = rb_define_module("Test");
rb_define_method(lemon, "test", (VALUE(*)(ANYARGS))method_test,1);
}
extconf.rb
require 'mkmf'
have_library( 'stdc++' );
$CFLAGS << " -Wall"
create_makefile( 'Test' );
run.rb
require 'rubygems'
require 'Test'
include Test
puts test(7)
when i execute run.rb it shows an error.
run.rb:7:in `test': can't convert Object into Integer (TypeError)
from run.rb:7:in `<main>'
please help me to solve this problem. thank you.
The line
VALUE method_test(VALUE y) {
Should be
VALUE method_test(VALUE self, VALUE y) {
The error message is thrown by NUM2INT(y) because y is the root Object of the script in your code as-is. The root object is what you got because you mixed in your module at the top level of the script. If you had mixed in to another class, you would have got an instance of that class instead.
All native extension methods should take a self parameter (the first one if you have fixed number of params) which is the context that they are being called in (a Module, Class or instance of a Class). It is how you get foo into the extension when you call foo.test. This might seem odd when both Ruby and C++ do this automatically (self in Ruby, and this in C++): However it is necessary to have self appear as param because the Ruby internal API is written in C, which does not support OO directly - instead the Ruby API defined in ruby.h expects you to write and use C functions that take references to the current object as one of the params.
In addition you are making a call to Circle( int radius ) - which does not exist (although the compiler may be kind and coerce x for you automatically). You probably want to use double variables throughout and use NUM2DBL(y) and DBL2NUM( cir->getArea() )
I've observed that when one types
help
in the Python repl, one gets
Type help() for interactive help, ...
and when one types
help()
one gets kicked into help mode. I'm pretty sure this is because site._Helper defines __repr__() (for the first example) and __call__() (for the second).
I like this behavior (prompt for just the object, and callable syntax), and I'd like to do the same for a C++ class I'm exporting to Python via SWIG. Here is a simple example of what I've tried to do
helpMimic.h
-----------
class HelpMimic
{
public:
HelpMimic() {};
~HelpMimic() {};
char *__repr__();
void operator()(const char *func=NULL);
};
helpMimic.cxx
-------------
char *HelpMimic::__repr__()
{
return "Online help facilities are not yet implemented.";
}
void HelpMimic::operator()(const char *func)
{
log4cxx::LoggerPtr transcriptPtr = oap::getTranscript();
std::string commentMsg("# Online help facilities are not yet implemented. Cannot look up ");
if (func) {
commentMsg += func;
}
else {
commentMsg += "anything.";
}
LOG4CXX_INFO(transcriptPtr, commentMsg);
}
helpMimic.i
-----------
%module sample
%{
#include <helpMimic.h>
%}
class HelpMimic
{
public:
HelpMimic() {};
~HelpMimic() {};
char *__repr__();
void operator()(const char *func=NULL);
};
When I attempt to use this class in my application, I can't seem to get the behavior I see with help (the output below is taken from a C++ application with Python embedded, where each input line is sent through PyEval_String()):
tam = sample.HelpMimic()
tam # echoes 'tam', nothing else
print tam
# _5010b70200000000_p_HelpMimic
print repr(tam)
# <Swig Object of type 'HelpMimic *' at 0x28230a0>
print tam.__repr__()
# Online help facilities are not yet implemented.
That last print shows that the method __repr__() is there, but I can't find it using the simpler object reference or using repr(tam). I also tried defining __str()__ in the hopes that I'd misunderstood which would get called, but still no luck.
I've tried using the %extend directive in the interface file to insert a __str__() or a __repr__() definition into the SWIG interface definition file, instead of defining them directly in C++, but to no avail.
What am I missing?
As #flexo suggested in a comment, if you are using the -builtin flag to the SWIG code generator, repr() will not call your __repr__ method. Instead, you need to define a function that fits in the repr slot.
%feature("python:slot", "tp_repr", functype="reprfunc") HelpMimic::printRepr;
As per HelpMimic::printRepr must have a signature that matches the expected signature (tp_repr in Python docs) - it must return a string or unicode object. Another caveat - you can't put the same function in more than one slot, so don't try to use this for tp_str!
I usually use the %extend feature to avoid tailoring the C/C++ to much for a specific target language. E.g.
%extend MyClass {
%pythoncode %{
def __repr__(self):
# How you want your object to be shown
__swig_getmethods__["someMember"] = SomeMemberGet
__swig_setmethods__["someMember"] = SomeMemberSet
if _newclass:
someMember = property(SomeMemberGet,SomeMemberSet)
def show(self):
# You could possibly visualize your object using matplotlib
%}
};
Where you inside the repr function can call basically any function and format the output to suit your needs. Further, you can add properties and define how they map to setters and getters.
If you want to add a __repr__ in the Python code rather than C/C++, you may need to deal with the default swig definition of __repr__ = _swig_repr.
This turns out to be fairly straightforward:
#if defined(SWIGPYTHON)
%pythoncode %{
del __repr__
def __repr__(self):
return 'object representation'
%}
#endif
As I'm still new to this, I'm facing some problems, here's my C++ code:
#include <python.h>
#define DLLEXPORT extern "C" __declspec(dllexport)
DLLEXPORT PyObject *Add(PyObject *pSelf, PyObject *pArgs)
{
int s,d;
if(!PyArg_ParseTuple(pArgs, "ii" , &s, &d))
{
PyErr_SetString(PyExc_TypeError,
"Add() invalid parameter");
return NULL;
}
return Py_BuildValue("i", s + d);
}
And the Python code:
import ctypes
MyDll = ctypes.cdll.LoadLibrary(r"PyToCppTest.dll")
jj = MyDll.Add(1,2)
I get an error when I run the above Python code:
OSError: exception: access violation reading 0x000000000000000A
I want to pass the data, without converting it, from Python to C++, then convert it inside C++.
Use either an extension or ctypes; you're not supposed to call your extension through ctypes. The point of extensions is to be able to create modules that look native to people using them from Python. ctypes serves to call C code that was written completely oblivious of Python.
There are a few things that are wrong with your code. First and foremost, the proper include is:
#include <Python.h>
Note the capital P. You're probably on Windows, but this wouldn't work on Linux without the capital P.
Also, I don't see the point of the *pSelf pointer in your function declaration, you should get rid of it:
PyObject *Add(PyObject *pArgs)
Now, your main problem is this:
MyDll.Add(1,2)
...does not call MyDll.Add with a tuple. It calls it with two integer arguments, 1 and 2. If you want to pass a tuple, you'd do:
MyDll.Add((1,2))
However, Python's ctypes won't know what to do with this (it normally accepts integer arguments), so you'll need to tell it that Add actually wants a tuple, and returns a Python object:
import ctypes
MyDll = ctypes.cdll.LoadLibrary("PyToCppTest.dll")
MyCFunc = ctypes.PYFUNCTYPE(
ctypes.py_object, # return val: a python object
ctypes.py_object # argument 1: a tuple
)
MyFunc = MyCFunc(('Add', MyDll))
jj = MyFunc((1,2))
I used SWIG to generate a Perl module for a C++ program. I have one function in the C++ code which returns a "char pointer". Now I dont know how to print or get the returned char pointer in Perl.
Sample C code:
char* result() {
return "i want to get this in perl";
}
I want to invoke this function "result" in Perl and print the string.
How to do that?
Regards,
Anandan
Depending on the complexity of the C++ interface, it may be easier, faster, and more maintainable to skip SWIG and write the XS code yourself. XS&C++ is a bit of an arcane art. That's why there is Mattia Barbon's excellent ExtUtils::XSpp module on CPAN. It make wrapping C++ easy (and almost fun).
The ExtUtils::XSpp distribution includes a very simple (and contrived) example of a class that has a string (char*) and an integer member. Here's what the cut-down interface file could look like:
// This will be used to generate the XS MODULE line
%module{Object::WithIntAndString};
// Associate a perl class with a C++ class
%name{Object::WithIntAndString} class IntAndString
{
// can be called in Perl as Object::WithIntAndString->new( ... );
IntAndString();
// Object::WithIntAndString->newIntAndString( ... );
// %name can be used to assign methods a different name in Perl
%name{newIntAndString} IntAndString( const char* str, int arg );
// standard DESTROY method
~IntAndString();
// Will be available from Perl given that the types appear in the typemap
int GetInt();
const char* GetString ();
// SetValue is polymorphic. We want separate methods in Perl
%name{SetString} void SetValue( const char* arg = NULL );
%name{SetInt} void SetValue( int arg );
};
Note that this still requires a valid XS typemap. It's really simple, so I won't add it here, but you can find it in the example distribution linked above.
You must have referred to the SWIG tutorial at www.swig.org/tutorial.html
Anyways, since you just want to invoke the function the C function from perl,
1. Type your interface file(having all the function declarations in the wrapper and the module sections).
2. Compile with swig and options.
3. Compile with gcc to create the objects.
4. Compile with gcc options to create the shared object.
5. run the program as follows:
perl
use moduleName;
$a = moduleName::result();
[NOTE: Look into the generated module file(.pm) for the correct funvtion prototype which points to the correct function in the wrapper file.]