I have C++ code that calls DLL which contains cythonized python functions. The python functions work with DataFrame (create it and then do lookups / stuff on it). I would like to do the DataFrame creation only once, which means I need to keep its state after the python function exits.
I haven't found a way how to return the DataFrame pointer from cython factory to C++ and send it from C++ to the other cython functions later. I want to avoid singleton-like solutions in cython. Please advise.
Edit1:
foo.pyx:
cdef public string Foo( const string& csvpath ) except *:
cdef string c_csvpath = csvpath
#...
foo.h:
__PYX_EXTERN_C DL_IMPORT(std::string) Foo(std::string const &);
I'm going to assume that you want to keep the string return type (if not then you can just return a Python object, which you may find easier). If this is the case then you need to use one of the function arguments to store data. Any mutable Python object could be used in principle but I'm going to demonstrate with a dictionary since I think that makes most sense:
cdef public string Foo( const string& csvpath, dict saved_data ) except *:
cdef string c_csvpath = csvpath
# get the DataFrame if possible, otherwise generate it
try:
df = saved_data['dataframe']
except KeyError:
df = 3.3 # somehow generate your dataframe
# at the end make sure everything's updated
saved_data['dataframe'] = df
return csvpath
The C signature becomes:
__PYX_EXTERN_C DL_IMPORT(std::string) Foo(std::string const &, PyObject *);
In your C++ code you then need to create and save a dictionary:
PyObject* data = PyDict_New();
// good code would check for null here
string out = Foo(string("Hi"),data);
// your data dictionary should now have "dataframe" in it
// a second call, reusing the data frame
string out2 = Foo(string("Hi"),data);
// once you're sure you've done with the data frame
Py_DECREF(data); // frees it, unless Python also has a copy
Related
I'm using Python/ctypes to write an app based on a commercial DLL. This DLL reads a flatfile and returns data through structs. The relevant C structs looks like this:
struct System{
unsigned short user;
unsigned short version;
};
struct Message {
unsigned short header;
unsigned short data;
unsigned int messageType;
};
The API that came with the DLL provides a pointer to a function with the following prototype. This function loops through the flatfile and calls Callback defined next:
typedef int (__stdcall *Process) (const char* filename, const char* base, unsigned int flags, int user, Callback stdcallback);
and the Callback prototype (the fifth argument above) is defined as:
typedef int (__stdcall *Callback) (const System* Sys, const Message* Msg);
In my python file:
from ctypes import *
# using WinDLL for stdcall
lib = WinDLL('CVO.dll')
class System(Structure):
_fields_ = [('user', c_ushort),
('version', c_ushort)]
class Message(Structure):
_fields_ = [('header', c_ushort),
('data', c_ushort),
('messageType', c_uint)]
PROTO = WINFUNCTYPE(c_int, POINTER(System), POINTER(Message))
def py_callbck(Sys, Msg):
print Msg._type_.messageType
# return 0 to read the next line in the flatfile
return 0
Callback = PROTO(py_callbck)
pProcess lib.Process
pProcess.argtypes = [c_char_p,c_char_p,c_uint,c_int,PROTO]
pProcess.restype = c_int
pProcess(c_char_p('flat.ccf'),c_char_p(None),c_uint(0),c_int(0),Callback)
My problem lies in accessing the value the DLL passes to the Msg struct. The code should be printing an c_int (0, 1, 2) but instead is returning a Field type like such:
<Field type=c_ulong, ofs=136, size=4>
I know the flatfile is being processed, because my Python code prints out hundreds of the <Field ...> statements, which it is supposed to do.
My question is how I can access the value the DLL is pushing to the Msg Structure instead of returning <Field ...> statements?
Note I'm relatively new to ctypes and have very little C so if you notice issues with defining anything, please let me know. For example, I know there is a difference between ctypes.pointer which returns and object and ctypes.POINTER which returns a new type. When I use ctypes.pointer the code errors out.
For being relatively new to ctypes you've done a pretty good job here. The _type_ attribute of a pointer is a reference to the pointed at data type. In this case it's Message. The class attribute messageType is the CField data descriptor used by an instance.
What you want to do instead is dereference the pointer to get a Message instance. You can either use a [0] subscript or the contents attribute:
def py_callbck(Sys, Msg):
print Msg[0].messageType # Msg.contents.messageType
# return 0 to read the next line in the flatfile
return 0
Remember to retain a reference to Callback to keep it from being garbage collected. Currently you reference it as a global. That's fine.
Regarding the way you're calling pProcess, it's generally unnecessary to manually create ctypes objects for simple types when you've defined argtypes. You can more simply use the following:
pProcess('flat.ccf', None, 0, 0, Callback)
But pay attention to whether the function needs a writable string buffer. In that case use create_string_buffer. It's unnecessary here since the char * parameters are both const.
I am embedding python in my C++ application using boost python. I am a C++ programmer, with very limited knowledge of Python.
I have a C++ class, PyExpression. Each instance of this class has a string expStr, which is a short user-entered (at runtime) python program, that is executed by calling boost::python::exec. Briefly, I have this set up as:
//import main and its globals
bp::object main = bp::import("__main__");
bp::object main_namespace = main.attr("__dict__");
where main and main_namespace are members of the C++ class PyExpression.
void PyExpression::Run()
{
bp::object pyrun = exec(expStr,main_namespace);
}
The problem here is that different C++ instances of PyExpression modify the same global python namespace, main_namespace, and I want each PyExpression instance to have its own "global" namespace.
If I pass in boost::python::dict class_dict instead of main_namespace above, it works at a basic level. But if PyExpression::expStr imports a module, e.g. import sys, then I get an ImportError. Also, using class_dict, I can no longer call globals(), locals(), vars(), as they all become undefined.
I have also tried exposing PyExpression as a python module. Briefly,
BOOST_PYTHON_MODULE(PyExpModule)
{
bp::class_<PyExpression>("PyExpression", bp::no_init)
//a couple .def functions
}
int pyImport = PyImport_AppendInittab( "PyExpModule", &initPyExpModule );
bp::object thisExpModule = bp::object( (bp::handle<>(PyImport_ImportModule("PyExpModule"))) );
bp::object PyExp_namespace = thisExpModule.attr("__dict__");
Unfortunately, using PyExp_namespace, again I get the ImportError when the string to be executed imports a python module, and again, the namespace is shared between all instances of PyExpression.
In short, I want to be able to use a namespace object/dictionary, that is preferably a class member of PyExpression, have only that instance of PyExpression have access to the namespace, and the namespace to act like a global namespace such that other modules can be imported, and the `globals(), locals(), vars() are all defined.
If anyone can point me to a sketch of working code, I would very much appreciate it. I can't find relevant material on this problem.
Before providing a solution, I want to provide some clarification on Python behavior.
Boost.Python's object is essentially a higher-level handle of a smart pointer. Thus, multiple object instances may point to the same Python object.
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
The above code imports a module named __main__. In Python, modules are essentially singletons due to the import behavior. Therefore, although the C++ main_module may be a member of the C++ PyExpression class, they all point to the same Python __main__ module, as it is a singleton. This results in main_namespace pointing to the same namespace.
Much of Python is built around dictionaries. For example, with an example module:
class Foo:
def __init__(self):
self.x = 42;
def bar(self):
pass
There are 3 dictionaries of interests:
example.__dict__ is the example module's namespace.
>>> example.__dict__.keys()
['__builtins__', '__file__', '__package__', '__name__', 'Foo', '__doc__']
example.Foo.__dict__ is a dictionary that describes the Foo class. Additionally, it would contain the equivalent of C++'s static member variables and functions.
>>> example.Foo.__dict__.keys()
['__module__', 'bar', '__doc__', '__init__']
example.Foo().__dict__ is a dictionary containing instance-specific variables. This would contain the equivalent of C++'s non-static member variables.
>>> example.Foo().__dict__.keys()
['x']
The Python exec statement takes two optional arguments:
The first argument specifies the dictionary that will be used for globals(). If the second argument is omitted, then it is also used for locals().
The second argument specifies the dictionary that will be used for locals(). Variable changes occurring within exec are applied to locals().
To get the desired behavior, example.Foo().__dict__ needs to be used as locals(). Unfortunately, this becomes slightly more complicated because of the following two factors:
Although import is a Python keyword, the CPython implementation is dependent on __builtins__.__import__. Thus, there needs to be a guarantee that the __builtin__ module is assessable as __builtins__ within the namespace passed to exec.
If a C++ class called Foo is exposed as a Python class through Boost.Python, then there is no easy way to access the Python Foo instance from within the C++ Foo instance.
To account for these behaviors, the C++ code will need to:
Get a handle to the Python object's __dict__.
Inject the __builtin__ module into the Python object's __dict__.
Extract the C++ object from the Python object.
Pass the Python object's __dict__ to the C++ object.
Here is an example solution that only sets variables on the instance for which code is being evaluated:
#include <boost/python.hpp>
class PyExpression
{
public:
void run(boost::python::object dict) const
{
exec(exp_.c_str(), dict);
}
std::string exp_;
};
void PyExpression_run(boost::python::object self)
{
// Get a handle to the Python object's __dict__.
namespace python = boost::python;
python::object self_dict = self.attr("__dict__");
// Inject the __builtin__ module into the Python object's __dict__.
self_dict["__builtins__"] = python::import("__builtin__");
// Extract the C++ object from the Python object.
PyExpression& py_expression = boost::python::extract<PyExpression&>(self);
// Pass the Python object's `__dict__` to the C++ object.
py_expression.run(self_dict);
}
BOOST_PYTHON_MODULE(PyExpModule)
{
namespace python = boost::python;
python::class_<PyExpression>("PyExpression")
.def("run", &PyExpression_run)
.add_property("exp", &PyExpression::exp_, &PyExpression::exp_)
;
}
// Helper function to check if an object has an attribute.
bool hasattr(const boost::python::object& obj,
const std::string& name)
{
return PyObject_HasAttrString(obj.ptr(), name.c_str());
}
int main()
{
PyImport_AppendInittab("PyExpModule", &initPyExpModule);
Py_Initialize();
namespace python = boost::python;
try
{
// python: import PyExpModule
python::object py_exp_module = python::import("PyExpModule");
// python: exp1 = PyExpModule.PyExpression()
// python: exp1.exp = "import time; x = time.localtime().tm_year"
python::object exp1 = py_exp_module.attr("PyExpression")();
exp1.attr("exp") =
"import time;"
"x = time.localtime().tm_year"
;
// python: exp2 = PyExpModule.PyExpression()
// python: exp2.exp = "import time; x = time.localtime().tm_mon"
python::object exp2 = py_exp_module.attr("PyExpression")();
exp2.attr("exp") =
"import time;"
"x = time.localtime().tm_mon"
;
// Verify neither exp1 nor exp2 has an x variable.
assert(!hasattr(exp1, "x"));
assert(!hasattr(exp2, "x"));
// python: exp1.run()
// python: exp2.run()
exp1.attr("run")();
exp2.attr("run")();
// Verify exp1 and exp2 contain an x variable.
assert(hasattr(exp1, "x"));
assert(hasattr(exp2, "x"));
// python: print exp1.x
// python: print exp2.x
std::cout << python::extract<int>(exp1.attr("x"))
<< "\n" << python::extract<int>(exp2.attr("x"))
<< std::endl;
}
catch (python::error_already_set&)
{
PyErr_Print();
}
}
And the output:
[twsansbury#localhost]$ ./a.out
2013
5
Due to how libraries are loaded from imports, it may require providing arguments to the linker that will cause all symbols, not only used ones, to the dynamic symbol table. For example, when compiling the above example with gcc, using -rdynamic was required. Otherwise, import time will fail due to an undefined PyExc_IOError symbol.
Python does not provide a 100% reliable isolation mechanism for this kind of task. That said, the essential tool you are looking for is the Python C-API Py_NewInterpreter, which is documented here. You will have to call it upon the creation of your PyExpression object, to create a new (semi)-isolated environment (N.B.: the destructor should call Py_EndInterpreter).
This is untested, but I'd guess something liket this would do the job:
PyThreadState* current_interpreter = Py_NewInterpreter();
bp::object pyrun = exec(expStr);
Py_EndInterpreter(current_interpreter);
You may wrap that into an object. If you wish to do so, you must manage the "thread" state as explained in this other stackoverflow thread.
I've observed that when one types
help
in the Python repl, one gets
Type help() for interactive help, ...
and when one types
help()
one gets kicked into help mode. I'm pretty sure this is because site._Helper defines __repr__() (for the first example) and __call__() (for the second).
I like this behavior (prompt for just the object, and callable syntax), and I'd like to do the same for a C++ class I'm exporting to Python via SWIG. Here is a simple example of what I've tried to do
helpMimic.h
-----------
class HelpMimic
{
public:
HelpMimic() {};
~HelpMimic() {};
char *__repr__();
void operator()(const char *func=NULL);
};
helpMimic.cxx
-------------
char *HelpMimic::__repr__()
{
return "Online help facilities are not yet implemented.";
}
void HelpMimic::operator()(const char *func)
{
log4cxx::LoggerPtr transcriptPtr = oap::getTranscript();
std::string commentMsg("# Online help facilities are not yet implemented. Cannot look up ");
if (func) {
commentMsg += func;
}
else {
commentMsg += "anything.";
}
LOG4CXX_INFO(transcriptPtr, commentMsg);
}
helpMimic.i
-----------
%module sample
%{
#include <helpMimic.h>
%}
class HelpMimic
{
public:
HelpMimic() {};
~HelpMimic() {};
char *__repr__();
void operator()(const char *func=NULL);
};
When I attempt to use this class in my application, I can't seem to get the behavior I see with help (the output below is taken from a C++ application with Python embedded, where each input line is sent through PyEval_String()):
tam = sample.HelpMimic()
tam # echoes 'tam', nothing else
print tam
# _5010b70200000000_p_HelpMimic
print repr(tam)
# <Swig Object of type 'HelpMimic *' at 0x28230a0>
print tam.__repr__()
# Online help facilities are not yet implemented.
That last print shows that the method __repr__() is there, but I can't find it using the simpler object reference or using repr(tam). I also tried defining __str()__ in the hopes that I'd misunderstood which would get called, but still no luck.
I've tried using the %extend directive in the interface file to insert a __str__() or a __repr__() definition into the SWIG interface definition file, instead of defining them directly in C++, but to no avail.
What am I missing?
As #flexo suggested in a comment, if you are using the -builtin flag to the SWIG code generator, repr() will not call your __repr__ method. Instead, you need to define a function that fits in the repr slot.
%feature("python:slot", "tp_repr", functype="reprfunc") HelpMimic::printRepr;
As per HelpMimic::printRepr must have a signature that matches the expected signature (tp_repr in Python docs) - it must return a string or unicode object. Another caveat - you can't put the same function in more than one slot, so don't try to use this for tp_str!
I usually use the %extend feature to avoid tailoring the C/C++ to much for a specific target language. E.g.
%extend MyClass {
%pythoncode %{
def __repr__(self):
# How you want your object to be shown
__swig_getmethods__["someMember"] = SomeMemberGet
__swig_setmethods__["someMember"] = SomeMemberSet
if _newclass:
someMember = property(SomeMemberGet,SomeMemberSet)
def show(self):
# You could possibly visualize your object using matplotlib
%}
};
Where you inside the repr function can call basically any function and format the output to suit your needs. Further, you can add properties and define how they map to setters and getters.
If you want to add a __repr__ in the Python code rather than C/C++, you may need to deal with the default swig definition of __repr__ = _swig_repr.
This turns out to be fairly straightforward:
#if defined(SWIGPYTHON)
%pythoncode %{
del __repr__
def __repr__(self):
return 'object representation'
%}
#endif
I need some advice how can I bind a C/C++ structure to Ruby. I've read some manuals and I found out how to bind class methods to a class, but I still don't understand how to bind structure fields and make them accessible in Ruby.
Here is the code I'm using:
myclass = rb_define_class("Myclass", 0);
...
typedef struct nya
{
char const* name;
int age;
} Nya;
Nya* p;
VALUE vnya;
p = (Nya*)(ALLOC(Nya));
p->name = "Masha";
p->age = 24;
vnya = Data_Wrap_Struct(myclass, 0, free, p);
rb_eval_string("def foo( a ) p a end"); // This function should print structure object
rb_funcall(0, rb_intern("foo"), 1, vnya); // Here I call the function and pass the object into it
The Ruby function seems to assume that a is a pointer. It prints the numeric value of the pointer instead of it's real content (i.e., ["Masha", 24]). Obviously the Ruby function can't recognize this object —I didn't set the object's property names and types.
How can I do this? Unfortunately I can't figure it out.
You have already wrapped your pointer in a Ruby object. Now all you have to do is define how it can be accessed from the Ruby world:
/* Feel free to convert this function to a macro */
static Nya * get_nya_from(VALUE value) {
Nya * pointer = 0;
Data_Get_Struct(value, Nya, pointer);
return pointer;
}
VALUE nya_get_name(VALUE self) {
return rb_str_new_cstr(get_nya_from(self)->name);
}
VALUE nya_set_name(VALUE self, VALUE name) {
/* StringValueCStr returns a null-terminated string. I'm not sure if
it will be freed when the name gets swept by the GC, so maybe you
should create a copy of the string and store that instead. */
get_nya_from(self)->name = StringValueCStr(name);
return name;
}
VALUE nya_get_age(VALUE self) {
return INT2FIX(get_nya_from(self)->age);
}
VALUE nya_set_age(VALUE self, VALUE age) {
get_nya_from(self)->age = FIX2INT(age);
return age;
}
void init_Myclass() {
/* Associate these functions with Ruby methods. */
rb_define_method(myclass, "name", nya_get_name, 0);
rb_define_method(myclass, "name=", nya_set_name, 1);
rb_define_method(myclass, "age", nya_get_age, 0);
rb_define_method(myclass, "age=", nya_set_age, 1);
}
Now that you can access the data your structure holds, you can simply define the high level methods in Ruby:
class Myclass
def to_a
[name, age]
end
alias to_ary to_a
def to_s
to_a.join ', '
end
def inspect
to_a.inspect
end
end
For reference: README.EXT
This is not a direct answer to your question about structures, but it is a general solution to the problem of porting C++ classes to Ruby.
You could use SWIG to wrap C/C++ classes, structs and functions. In the case of a structure, it's burning a house to fry an egg. However, if you need a tool to rapidly convert C++ classes to Ruby (and 20 other languages), SWIG might be useful to you.
In your case involving a structure, you just need to create a .i file which includes (in the simplest case) the line #include <your C++ library.h>.
P.S. Once more, it's not a direct answer to your question involving this one struct, but maybe you could make use of a more general solution, in which case this may help you.
Another option is to use RubyInline - it has limited support for converting C and Ruby types (such as int, char * and float) and it also has support for accessing C structurs - see accessor method in the API.
I used SWIG to generate a Perl module for a C++ program. I have one function in the C++ code which returns a "char pointer". Now I dont know how to print or get the returned char pointer in Perl.
Sample C code:
char* result() {
return "i want to get this in perl";
}
I want to invoke this function "result" in Perl and print the string.
How to do that?
Regards,
Anandan
Depending on the complexity of the C++ interface, it may be easier, faster, and more maintainable to skip SWIG and write the XS code yourself. XS&C++ is a bit of an arcane art. That's why there is Mattia Barbon's excellent ExtUtils::XSpp module on CPAN. It make wrapping C++ easy (and almost fun).
The ExtUtils::XSpp distribution includes a very simple (and contrived) example of a class that has a string (char*) and an integer member. Here's what the cut-down interface file could look like:
// This will be used to generate the XS MODULE line
%module{Object::WithIntAndString};
// Associate a perl class with a C++ class
%name{Object::WithIntAndString} class IntAndString
{
// can be called in Perl as Object::WithIntAndString->new( ... );
IntAndString();
// Object::WithIntAndString->newIntAndString( ... );
// %name can be used to assign methods a different name in Perl
%name{newIntAndString} IntAndString( const char* str, int arg );
// standard DESTROY method
~IntAndString();
// Will be available from Perl given that the types appear in the typemap
int GetInt();
const char* GetString ();
// SetValue is polymorphic. We want separate methods in Perl
%name{SetString} void SetValue( const char* arg = NULL );
%name{SetInt} void SetValue( int arg );
};
Note that this still requires a valid XS typemap. It's really simple, so I won't add it here, but you can find it in the example distribution linked above.
You must have referred to the SWIG tutorial at www.swig.org/tutorial.html
Anyways, since you just want to invoke the function the C function from perl,
1. Type your interface file(having all the function declarations in the wrapper and the module sections).
2. Compile with swig and options.
3. Compile with gcc to create the objects.
4. Compile with gcc options to create the shared object.
5. run the program as follows:
perl
use moduleName;
$a = moduleName::result();
[NOTE: Look into the generated module file(.pm) for the correct funvtion prototype which points to the correct function in the wrapper file.]