For performance reasons I want to port parts of my python program to C++ and I therefore try to write a simple extension for my program. The C++ part will build a dictionary, which then needs to be delivered to the Python program.
One way I found seems to be to build my dict-like object in C++, e.g. a boost::unordered_map, and then translate it to Python using the Py_BuildValue[1] method, which is able to produce Python dicts. But this method which includes converting the container into a string representation and back seems a bit too much 'around the corner' to be the most performant solution!?
So my question is: What is the most performant way to build a Python dictionary in C++? I saw that boost has a Python library which supports mapping containers between C++ and Python, but I didn't find the exact thing I need in the documentation so far. If there is such way I would prefer to directly build a Python dict in C++, so that no copying etc. is needed. But if the most performant way to do this is another one, I'm good with that too.
Here is the (simplified) C++-code I compile into a .dll/.pyd:
#include <iostream>
#include <string>
#include <Python.h>
#include "boost/unordered_map.hpp"
#include "boost/foreach.hpp"
extern "C"{
typedef boost::unordered_map<std::string, int> hashmap;
static PyObject*
_rint(PyObject* self, PyObject* args)
{
hashmap my_hashmap; // DO I NEED THIS?
my_hashmap["a"] = 1; // CAN I RATHER INSERT TO PYTHON DICT DIRECTLY??
BOOST_FOREACH(hashmap::value_type i, my_hashmap) {
// INSERT ELEMENT TO PYTHON DICT
}
// return PYTHON DICT
}
static PyMethodDef TestMethods[] = {
{"rint", _rint, METH_VARARGS, ""},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC
inittest(void)
{
Py_InitModule("test", TestMethods);
}
} // extern "C"
This I want to use in Python like:
import test
new_dict = test.rint()
The dictionary will map strings to integers. Thanks for any help!
Use the CPython API directly yes:
PyObject *d = PyDict_New()
for (...) {
PyDict_SetItem(d, key, val);
}
return d;
Or write a python object that emulate a dict, by overriding __setitem__ and __getitem__. In both method, use your original hashmap. At the end, no copy will happen!
Related
I just successfully created an lua project. (A simple code that runs an lua script so far.)
But how would I make a c++ function and a c++ variable available for the lua script now?
As an example:
int Add(int x, int y) {
return x + y;
}
and
float myFloatValue = 6.0
I'm very new to c++ so I really hope that it won't be too complicated. Here is the code I got so far btw:
#include "stdafx.h"
extern "C" {
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"
}
using namespace System;
int main(array<System::String ^> ^args)
{
lua_State* luaInt;
luaInt = lua_open();
luaL_openlibs (luaInt);
luaL_dofile (luaInt, "abc.lua");
lua_close(luaInt);
return 0;
}
I'll go with John Zwinck's answer as experience has proven to me that using Lua all by itself is a pain in the butt. But, if you want to know the answer check the rest.
For registering C/C++ functions you need to first make your function look like a standard C function pattern which Lua provides:
extern "C" int MyFunc(lua_State* L)
{
int a = lua_tointeger(L, 1); // First argument
int b = lua_tointeger(L, 2); // Second argument
int result = a + b;
lua_pushinteger(L, result);
return 1; // Count of returned values
}
Every function that needs to be registered in Lua should follow this pattern. Return type of int, single parameter of lua_State* L. And count of returned values.
Then, you need to register it in Lua's register table so you can expose it to your script's context:
lua_register(L, "MyFunc", MyFunc);
For registering simple variables you can write this:
lua_pushinteger(L, 10);
lua_setglobal(L, "MyVar");
After that, you're able to call your function from a Lua script. Keep in mind that you should register all of your objects before running any script with that specific Lua state that you've used to register them.
In Lua:
print(MyFunc(10, MyVar))
Result:
20
Rather than doing it using the Lua C API, I suggest using Luabind.
Luabind is a reasonably high-level library specifically built to expose C++ classes and functions to Lua. Without using the Lua C API functions, without manipulating the Lua stack, etc. It's inspired by Boost Python, so if you learn one you'll mostly understand the other.
I've observed that when one types
help
in the Python repl, one gets
Type help() for interactive help, ...
and when one types
help()
one gets kicked into help mode. I'm pretty sure this is because site._Helper defines __repr__() (for the first example) and __call__() (for the second).
I like this behavior (prompt for just the object, and callable syntax), and I'd like to do the same for a C++ class I'm exporting to Python via SWIG. Here is a simple example of what I've tried to do
helpMimic.h
-----------
class HelpMimic
{
public:
HelpMimic() {};
~HelpMimic() {};
char *__repr__();
void operator()(const char *func=NULL);
};
helpMimic.cxx
-------------
char *HelpMimic::__repr__()
{
return "Online help facilities are not yet implemented.";
}
void HelpMimic::operator()(const char *func)
{
log4cxx::LoggerPtr transcriptPtr = oap::getTranscript();
std::string commentMsg("# Online help facilities are not yet implemented. Cannot look up ");
if (func) {
commentMsg += func;
}
else {
commentMsg += "anything.";
}
LOG4CXX_INFO(transcriptPtr, commentMsg);
}
helpMimic.i
-----------
%module sample
%{
#include <helpMimic.h>
%}
class HelpMimic
{
public:
HelpMimic() {};
~HelpMimic() {};
char *__repr__();
void operator()(const char *func=NULL);
};
When I attempt to use this class in my application, I can't seem to get the behavior I see with help (the output below is taken from a C++ application with Python embedded, where each input line is sent through PyEval_String()):
tam = sample.HelpMimic()
tam # echoes 'tam', nothing else
print tam
# _5010b70200000000_p_HelpMimic
print repr(tam)
# <Swig Object of type 'HelpMimic *' at 0x28230a0>
print tam.__repr__()
# Online help facilities are not yet implemented.
That last print shows that the method __repr__() is there, but I can't find it using the simpler object reference or using repr(tam). I also tried defining __str()__ in the hopes that I'd misunderstood which would get called, but still no luck.
I've tried using the %extend directive in the interface file to insert a __str__() or a __repr__() definition into the SWIG interface definition file, instead of defining them directly in C++, but to no avail.
What am I missing?
As #flexo suggested in a comment, if you are using the -builtin flag to the SWIG code generator, repr() will not call your __repr__ method. Instead, you need to define a function that fits in the repr slot.
%feature("python:slot", "tp_repr", functype="reprfunc") HelpMimic::printRepr;
As per HelpMimic::printRepr must have a signature that matches the expected signature (tp_repr in Python docs) - it must return a string or unicode object. Another caveat - you can't put the same function in more than one slot, so don't try to use this for tp_str!
I usually use the %extend feature to avoid tailoring the C/C++ to much for a specific target language. E.g.
%extend MyClass {
%pythoncode %{
def __repr__(self):
# How you want your object to be shown
__swig_getmethods__["someMember"] = SomeMemberGet
__swig_setmethods__["someMember"] = SomeMemberSet
if _newclass:
someMember = property(SomeMemberGet,SomeMemberSet)
def show(self):
# You could possibly visualize your object using matplotlib
%}
};
Where you inside the repr function can call basically any function and format the output to suit your needs. Further, you can add properties and define how they map to setters and getters.
If you want to add a __repr__ in the Python code rather than C/C++, you may need to deal with the default swig definition of __repr__ = _swig_repr.
This turns out to be fairly straightforward:
#if defined(SWIGPYTHON)
%pythoncode %{
del __repr__
def __repr__(self):
return 'object representation'
%}
#endif
As I'm still new to this, I'm facing some problems, here's my C++ code:
#include <python.h>
#define DLLEXPORT extern "C" __declspec(dllexport)
DLLEXPORT PyObject *Add(PyObject *pSelf, PyObject *pArgs)
{
int s,d;
if(!PyArg_ParseTuple(pArgs, "ii" , &s, &d))
{
PyErr_SetString(PyExc_TypeError,
"Add() invalid parameter");
return NULL;
}
return Py_BuildValue("i", s + d);
}
And the Python code:
import ctypes
MyDll = ctypes.cdll.LoadLibrary(r"PyToCppTest.dll")
jj = MyDll.Add(1,2)
I get an error when I run the above Python code:
OSError: exception: access violation reading 0x000000000000000A
I want to pass the data, without converting it, from Python to C++, then convert it inside C++.
Use either an extension or ctypes; you're not supposed to call your extension through ctypes. The point of extensions is to be able to create modules that look native to people using them from Python. ctypes serves to call C code that was written completely oblivious of Python.
There are a few things that are wrong with your code. First and foremost, the proper include is:
#include <Python.h>
Note the capital P. You're probably on Windows, but this wouldn't work on Linux without the capital P.
Also, I don't see the point of the *pSelf pointer in your function declaration, you should get rid of it:
PyObject *Add(PyObject *pArgs)
Now, your main problem is this:
MyDll.Add(1,2)
...does not call MyDll.Add with a tuple. It calls it with two integer arguments, 1 and 2. If you want to pass a tuple, you'd do:
MyDll.Add((1,2))
However, Python's ctypes won't know what to do with this (it normally accepts integer arguments), so you'll need to tell it that Add actually wants a tuple, and returns a Python object:
import ctypes
MyDll = ctypes.cdll.LoadLibrary("PyToCppTest.dll")
MyCFunc = ctypes.PYFUNCTYPE(
ctypes.py_object, # return val: a python object
ctypes.py_object # argument 1: a tuple
)
MyFunc = MyCFunc(('Add', MyDll))
jj = MyFunc((1,2))
I'm trying to do something which should be very simple, but I'm not having much luck figuring out how from the existing documentation.
For a python 2 project I am trying to return a list gettext-translated string as a unicode instances to python. The return value for gettext() is a UTF-8 encoded char*, which should be pretty simple to convert to a python unicode instrance using PyUnicode_FromString. I have a feeling this is trivial to do, but I can't seem to figure out how.
Basd on comments from Ignacio Vazquez-Abrams and Thomas K I did get this working for a single string; for that case you can bypass all the boost.python infrastructure. Here is an example:
PyObject* PyMyFunc() {
const char* txt = BaseClass::MyFunc();
return PyUnicode_FromString(txt);
}
which is exposed with the usual def statement:
class_<MyCclass>("MyClass")
.def("MyFunc", &MyClass::PyMyFunc);
Unfortuantely this does not work when you want to return a list of unicode instances. This is my naive implementation:
boost::python::list PyMyFunc() {
std::vector<std::string> raw_strings = BaseClass::MyFunc();
std::vector<std::string>::const_iterator i;
boost::python::list result;
for (i=raw_strings.begin(); i!=raw_strings.end(); i++)
result.append(PyUnicode_FromString(i->c_str()));
return result;
}
but this does not compile: boost::python::list does seem to handle PyObject values.
With some help from the C++-SIG mailinglist I have this working now. There are two extra steps needed:
use boost::python::handle<> to create a C++ wrapper around the PyObject* which takes care of reference handling
use boost::python::object to create a C++ wrapper around the handle, which allows using a PyObject* instance as a (reasonably) normal C++ class instance, and thus something boost::python::list can handle.
With that knowledge the working code looks like this:
boost::python::list PyMyFunc() {
std::vector<std::string> raw_strings = BaseClass::MyFunc();
std::vector<std::string>::const_iterator i;
boost::python::list result;
for (i=raw_strings.begin(); i!=raw_strings.end(); i++)
result.append(
boost::python::object(
boost::python::handle<>(
PyUnicode_FromString(i->c_str()))));
return result;
}
I used SWIG to generate a Perl module for a C++ program. I have one function in the C++ code which returns a "char pointer". Now I dont know how to print or get the returned char pointer in Perl.
Sample C code:
char* result() {
return "i want to get this in perl";
}
I want to invoke this function "result" in Perl and print the string.
How to do that?
Regards,
Anandan
Depending on the complexity of the C++ interface, it may be easier, faster, and more maintainable to skip SWIG and write the XS code yourself. XS&C++ is a bit of an arcane art. That's why there is Mattia Barbon's excellent ExtUtils::XSpp module on CPAN. It make wrapping C++ easy (and almost fun).
The ExtUtils::XSpp distribution includes a very simple (and contrived) example of a class that has a string (char*) and an integer member. Here's what the cut-down interface file could look like:
// This will be used to generate the XS MODULE line
%module{Object::WithIntAndString};
// Associate a perl class with a C++ class
%name{Object::WithIntAndString} class IntAndString
{
// can be called in Perl as Object::WithIntAndString->new( ... );
IntAndString();
// Object::WithIntAndString->newIntAndString( ... );
// %name can be used to assign methods a different name in Perl
%name{newIntAndString} IntAndString( const char* str, int arg );
// standard DESTROY method
~IntAndString();
// Will be available from Perl given that the types appear in the typemap
int GetInt();
const char* GetString ();
// SetValue is polymorphic. We want separate methods in Perl
%name{SetString} void SetValue( const char* arg = NULL );
%name{SetInt} void SetValue( int arg );
};
Note that this still requires a valid XS typemap. It's really simple, so I won't add it here, but you can find it in the example distribution linked above.
You must have referred to the SWIG tutorial at www.swig.org/tutorial.html
Anyways, since you just want to invoke the function the C function from perl,
1. Type your interface file(having all the function declarations in the wrapper and the module sections).
2. Compile with swig and options.
3. Compile with gcc to create the objects.
4. Compile with gcc options to create the shared object.
5. run the program as follows:
perl
use moduleName;
$a = moduleName::result();
[NOTE: Look into the generated module file(.pm) for the correct funvtion prototype which points to the correct function in the wrapper file.]