Boost::Python- possible to automatically convert from dict --> std::map? - c++

I've got a C++ class, with a member function that can take a small-to-large number of parameters. Lets name those parameters, a-f. All parameters have default values. As a part of the python project I am working on, I want to expose this class to python. Currently, the member function looks something like this:
class myClass {
public:
// Constructors - set a-f to default values.
void SetParameters(std::map<std::string, double> &);
private:
double a, b, c, d, e, f;
}
void myClass::SetParameters(std::map<std::string, double> const& params) {
// Code to iterate over the map, and set any found key/value pairs to their
// corresponding variable. i.e.- "a" --> 2.0, would set myClass::a to 2.0
}
Ideally, in Python, I would like to accomplish this using a dict:
>>> A = myModule.myClass();
>>> A.SetParameters({'a': 2.2, 'd': 4.3, b: '9.3'})
In this way, the user could enter the values in any order, and enter any number of them to be over-ridden. Any thoughts on how this could be accomplished in boost::python? It seems to me that I can do this via changing the map input to a boost::python object, and using the extract functions. However, this would require me to change the interface of my library (I'd prefer to keep the std::map interface, and have some intermediary/auto conversion technique for the python version). Thoughts?

I think there's a couple of ways that are easier to accomplish than writing your own converter. You can use boost::python's map_indexing_suite to do the conversion for you, or you can use keyword arguments in python. I personally prefer keyword arguments, as this is the more "Pythonic" way to do this.
So this is your class (I added a typedef for the map):
typedef std::map<std::string, double> MyMap;
class myClass {
public:
// Constructors - set a-f to default values.
void SetParameters(MyMap &);
private:
double a, b, c, d, e, f;
};
Example using map_indexing_suite:
#include <boost/python/suite/indexing/map_indexing_suite.hpp>
using boost::python;
BOOST_PYTHON_MODULE(mymodule)
{
class_<std::map<std::string, double> >("MyMap")
.def(map_indexing_suite<std::map<std::wstring, double> >() );
class_<myClass>("myClass")
.def("SetParameters", &myClass::SetParameters);
}
Example using keyword arguments. This requires using a raw_function wrapper:
using namespace boost::python;
object SetParameters(tuple args, dict kwargs)
{
myClass& self = extract<myClass&>(args[0]);
list keys = kwargs.keys();
MyMap outMap;
for(int i = 0; i < len(keys); ++i) {
object curArg = kwargs[keys[i]];
if(curArg) {
outMap[extract<std::string>(keys[i])] = extract<double>(kwargs[keys[i]]);
}
}
self.SetParameters(outMap);
return object();
}
BOOST_PYTHON_MODULE(mymodule)
{
class_<myClass>("myClass")
.def("SetParameters", raw_function(&SetParameters, 1));
}
this allows you to write stuff like this in Python:
A.SetParameters(a = 2.2, d = 4.3, b = 9.3)

This blog post has a pretty clear description of how to write these converters. The basic pattern is to define a class that has the form:
struct SomeType_from_PyObject
{
SomeType_from_PyObject();
static void* convertible(PyObject* obj_ptr);
static void construct(PyObject* obj_ptr,
converter::rvalue_from_python_stage1_data* data);
};
Where the constructor is responsible for adding this converter to Boost.Python's registry:
SomeType_from_PyObject::SomeType_from_PyObject()
{
converter::registry::push_back(&convertible,
&construct,
type_id<SomeType>());
}
The function convertible tells Boost whether or not this converter can convert the specified Python object:
void* SomeType_from_PyObject::convertible(PyObject* obj_ptr)
{
if (PyMapping_Check(obj_ptr)) {
return obj_ptr;
} else {
return NULL;
}
}
The construct function actually creates an object of the conversion type:
void SomeType_from_PyObject::construct(PyObject* obj_ptr,
converter::rvalue_from_python_stage1_data* data)
{
typedef converter::rvalue_from_python_storage<SomeType> storage_t;
storage_t* the_storage = reinterpret_cast<storage_t*>(data);
void* memory_chunk = the_storage->storage.bytes;
object obj(handle<>(borrowed(obj_ptr)));
SomeType* output = new (memory_chunk) SomeType();
// Use the contents of obj to populate output, e.g. using extract<>
data->convertible = memory_chunk;
}
and then in your inside your BOOST_PYTHON_MODULE, include the line
SomeType_from_PyObject();

I'm just getting my toes wet with boost::python so can't completely answer your question. But the first roadblock I see is guaranteeing that the py dict's keys are all strings. Python dicts can also be keyed on tuples (and I assume more types).

Related

How to wrap a templated function that follows the callable trait with swig for Python?

I have a C++ class with a method that is templated to follow the callable trait:
// A general data object
struct MyObject
{
// ... hold some data, parameters, ...
};
class MyOptimizationAlgorithm
{
// ...
public:
// An optimization function that uses a user-supplied
// callable to evaluate a data object:
template <class Callable> void optimize(MyObject o, Callable evaluator) {
// ... optimize, optimize, ...
auto value = evaluator(o);
// ... are we good enough yet?
}
};
Here, the MyOptimizationAlgorithm class implements an optimization algorithm. The user supplies a data object (a vector of doubles, no problem here) and an objective function. This function is the user-configurable part on which the optimization algorithm relies. For example, valid callable evaluators could implement Ackley's function, the Cross-in Tray function, etc.
The pattern is actually pretty standard: In C++, the callable/predicate trait allows me to template a method so that I can pass a Functor, or a std::function. E.g.,
struct Ackley
{
double operator()(MyObject const& o)
{
return /* ackley() applied to the values in o */
}
};
MyOptimizationAlgorithm optimizer;
MyObject initialData;
// ... supply data,
// ... tune some parameters of the optimizer, then:
optimizer.optimize(initialData, Ackley());
// ... or:
optimizer.optimize(initalData, [](MyObject const& o) { /* ... */ });
I would now like to create a wrapper for Python with swig. The goal is, of course, to create the evaluator functor in Python and pass it to the C++ routine, like so:
def CrossInTray:
def __call__(self, obj):
# calculate the cross-in tray function.
optimzer = MyOptimizationAlgorithm()
initial_data = MyObject()
# ... again, setup, then:
optimizer.optimize(initial_data, CrossInTray())
I am new to swig. I have gathered that I need to specialize the template (using %template) and that I need to create a director (%director). I have tried to create a Functor wrapper, like so:
%inline %{
struct MyEvaluator
{
virtual double operator()(MyObject const& o) { return 0.0; }
virtual ~MyEvaluator() {}
};
%}
%feature("director") MyEvaluator;
%extend MyAlgorithm {
%template(runPredicated) optimize<MyEvaluator>;
}
I had hoped that I could then create subclasses of my Functor in Python, define __call__ there and have it used, but it only calls MyEvaluator::operator(), which is pretty pointless (and understandable, since I specialized the template to use MyEvaluator).
So: What do I need to add to the interface file to make use of the callable trait of C++ code in Python?
This is a bit too generic. If you want to restrict yourself to std::function interface like
class MyOptimizationAlgorithm
{
// ...
public:
// An optimization function that uses a user-supplied
// callable to evaluate a data object:
template<typename T>
void optimize(MyObject o, std::function<T(MyObject const&)> evaluator) {
// ... optimize, optimize, ...
T value = evaluator(o);
// ... are we good enough yet?
}
};
a possible wrapper can look like
class MyOptimizationAlgorithm:
def `optimize<double>` as optimize(o: MyObject, evaluator: (o: MyObject)->float)
and can be called with any Python function taking MyObject and returning a float. The Python code in the question should just work.
The wrapper above is PyCLIF (instead of SWIG).

Is there a good way to combine or simplify these STL maps?

I'm writing a class that needs to store a bunch of different primitives and classes. I've decided to make a map for each different data type where the key in the map would be the name of the variable, and the value in the map would be the value of the variable. My maps are defined like this:
std::unordered_map<std::string, int> myInts;
std::unordered_map<std::string, float> myFloats;
std::unordered_map<std::string, Something> mySomethings;
For each map, I have to write two methods, one which will get the value of some variable and one which will set the value of some variable like so:
void setMyInt(const std::string &varName, int newVal) {
myInts[varName] = newVal;
}
int getMyInt(const std::string &varName) {
return myInts[varName];
}
This is all fine and easy, however, I ended up with 8 different maps, and 16 of these get set methods. This doesn't seem very efficient or clean to me, not to mention that every time I need to store a new data type I have to define a new map and write 2 new get-set methods.
I considered getting rid of the get set methods and instead writing 2 template methods which would take in the type of the variable which the user needs to get or set, and then accessing the proper set to perform the operation, like so:
template<class Type>
void getVariable<Type>(const std::string &varName) {
// if Type == int -> return myInts[varName];
// else if Type == float -> return myFloats[varName];
}
This seems like a really poor approach since the user could pass in types which are not supported by the class, and the method breaks C++'s rule of not being really generic.
Another idea I had was writing some Variable class which would have all of the fields that this class should store, along with some enum that defines what Variable the class is actually being used for, and then making a map of this Variable class like this:
enum Type {
TYPE_INT,
TYPE_FLOAT,
TYPE_SOMETHING
class Variable {
Type type;
int myInt;
float myFloat;
Something mySomething;
}
std::unordered_map<std::string, Variable> myVariables;
But this also seems like a really poor solution, or at least one which is difficult to understand. Is there some smart way to make this class store different types?
How about a template class like below:
template<typename ValueType>
class UnorderedStringMap : public std::unordered_map<std::string, ValueType> {
public:
...
void setValue(const std::string &varName, ValueType newVal) {
std::unordered_map::operator[](varName) = newVal;
}
const ValueType& getValue(const std::string &varName) {
return std::unordered_map::operator[](varName);
}
...
}
UnorderedStringMap<int> myInts;
UnorderedStringMap<float> myFloats;
You can then use it as a normal std::unordered_map as well.

C++ : nameable objects belonging to an instance of a class, and stored in it

I am trying to make it possible for a programmer (who uses my library) to create nameable instances of type X that are stored inside an instance of class C (or at least are exclusive to that instance).
These are the only two (ugly) solutions I have managed to come up with (needless to say, I am just picking up C++)
1)
class C
{
public:
class XofC
{
public:
XofC() = delete;
XofC(C& mom)
{
mom.Xlist.emplace_front();
ref = Xlist.front();
}
X& access()
{
return ref;
}
private:
X& ref;
};
//etc
private:
std::forward_list<X> Xlist;
friend class XofC;
//etc
}
Problem:
Having to pass everywhere XofC instances.
2)
class C
{
public:
void newX(std::string);
X& getX(std::string);
//etc.
private:
/*possible run-time mapping implementation
std::vector<X> Xvec;
std::unordered_map<std::string, decltype(Xvec.size())> NameMap;
*/
//etc
}
Problem:
This does the job, but since all names of X (std::string) are known at compilation, the overhead of using run-time std::unordered_map<std::string, decltype(Xvec.size())> kind-of bugs me for something this simple.
Possible(?) solution: compile-time replacing of std::string with automatic index (int). Then I could use:
class C
{
public:
void newX(int); //int: unique index calculated at compile time from std::string
X& getX(int); //int: unique index calculated at compile time from std::string
//etc.
private:
std::vector<X> Xvec;
}
Questions:
Is there a 3)?
Is a compile time solution possible for 2)?
This is the real-life situation: I was starting my first C++ "project" and I thought I could use the practice and utility from an awesome user-friendly, simple and fast argument management library. I plan to make an ArgMan class which can parse the argV based on some specified switches. Switches would be named by the programmer descriptively and the trigger strings be specified (e.g. a switch named recurse could have "-r" and "-recursive" as triggers). When necessary, you should be easily able to get the setting of the switch. Implementation detail: ArgMan would have a std::unordered_map<std::string/*a trigger*/, ??/*something linking to the switch to set on*/>. This ensures an almost linear parse of argV relative to argC. How should I approach this?
You could 'abuse' non-type template arguments to get compiletime named instances:
Live on Coliru
Assume we have a data class X:
#include <string>
struct X
{
int has_some_properties;
std::string data;
};
Now, for our named instances, we define some name constants. The trick is, to give them external linkage, so we can use the address as a non-type template argument.
// define some character arrays **with external linkage**
namespace Names
{
extern const char Vanilla[] = "Vanilla";
extern const char Banana [] = "Banana";
extern const char Coconut[] = "Coconut";
extern const char Shoarma[] = "Shoarma";
}
Now, we make a NamedX wrapper that takes a const char* non-type template argument. The wrapper holds a static instance of X (the value).
// now we can "adorn" a `namedX` with the name constants (above)
template <const char* Name>
struct NamedX
{
static X value;
};
template <const char* Name> X NamedX<Name>::value;
Now you can use it like this:
int main()
{
X& vanilla = NamedX<Names::Vanilla>::value;
vanilla = { 42, "Woot!" };
return vanilla.has_some_properties;
}
Note that due to the fact that the template arguments are addresses, no actual string comparison is done. You cannot, e.g. use
X& vanilla = NamedX<"Vanilla">::value;
becuase "Vanilla" is a prvalue without external linkage. So, in fact you could do without some of the complexity and use tag structs instead: Live on Coliru
While Neil's solution did what I asked for, it was too gimmicky to use in my library. Also, sehe's trick is surely useful, but, if I understood correctly, but doesn't seem related to my question. I have decided to emulate the desired behavior using method 1), here is a less broken attempt at it:
class C
{
private:
class X
{
//std::string member;
//etc
};
public:
class XofC
{
public:
XofC(C & _mom) : mom(_mom)
{
mom.Xlist.emplace_front();
tehX = &(Xlist.front());
}
X & get(maybe)
{
if (&maybe != &mom) throw std::/*etc*/;
return &tehX;
}
private:
X * tehX;
C & mom;
};
private:
//etc
std::forward_list<X> Xlist;
friend class XofC;
//etc
};
Usage:
C foo;
bar = C::XofC(foo); //acts like an instance of X, but stored in C, but you have to use:
bar.get(foo)/*reference to the actual X*/.member = "_1_";
Of course, the downside is you have to make sure you pass bar everywhere you need it, but works decently.
This is how it looks like in my tiny argument manager library:
https://raw.github.com/vuplea/arg_manager.h/master/arg_manager.h

C++: Using a string parameter passed in to access something in a type

My goal is to access a class that is passed in as a parameter inside of myFunction.
Here's what I'm trying to do:
void myFunction(string myString)
{
callFunctionOn(OuterType::InnerType::myString);
}
I'm trying to call some function on something that's in a type. For example, my code in some other file might look like:
namespace OuterType {
namespace InnerType {
//stuff here
}
}
However, using myString in that way doesn't work. If myString holds the value "class1", then I want that callFunctionOn part to be interpreted as
callFunctionOn(OuterType::InnerType::class1);
I feel like this is super simple, but I've been programming all day and my mind grows tired...
SOLVED: It looks like in order to this in this way, I'd need a language with reflection. To solve this I took a different approach to the problem and passed in a pointer to the class instead.
C++ doesn't have reflection built in, but it does have pointers to data, functions, and class members. So you can use a std::map or unordered_set to find the pointer with a particular name (you have to add all the name/pointer pairs into the map beforehand).
Your solution is likely to look something like:
namespace Outer
{
namespace Inner
{
void funcA( void ) { std::cout << "called funcA" << std::endl; }
std::map< std::string, void (*)(void) > members;
}
}
// in some initialization function
Outer::Inner::members["funcA"] = &Outer::Inner::funcA;
// later
std::string myString = "funcA";
void (*f)(void) = Outer::Inner::members[myString]; // lookup function by name
(*f)(); // call function via its pointer
Of course the type of the pointer will probably need to change to meet your application requirements.
You're trying to access a variable based on a run-time string that contains its name? That's not possible; the names of variables disappear after compilation and linking. (Except insofar as they are kept around to facilitate debugging).
Do you mean :
OuterType::InnerType::callFunctionOn(myString);
maybe this idea: operator() can take parameters, wrapping it in a class ine can make calls that are resolved in the overloaded operator() based on its parameters.
template<typename TypeSig, class InstanceOf, typename NA,typename Args>
class FuncMap {
public:
typedef TypeSig (InstanceOf:: *cbMethod) ( NA, Args );
FuncMap( InstanceOf & cInst, cbMethod cbM ) : mcInst(cInst) {mcbM = cbM;}
TypeSig operator() ( NA na, Args args) {return (mcInst.*mcbM)(na, args);}
private:
InstanceOf & mcInst;
cbMethod mcbM;
};
you need to build a map of runtime string values as keys and pointers to instance methods as seen above. i used this for re-dispatch tracing and custom runtime dispatch with lesser than RTTI overhead.
this allows you to have default, if no key found, or other logic as you wish.

C++ Code Generation

In my epic quest of making C++ do things it shouldn't, I am trying to put together a compile time generated class.
Based on a preprocessor definition, such as (rough concept)
CLASS_BEGIN(Name)
RECORD(xyz)
RECORD(abc)
RECORD_GROUP(GroupName)
RECORD_GROUP_RECORD(foo)
RECORD_GROUP_RECORD(bar)
END_RECORDGROUP
END_CLASS
While I am fairly sure I generate a class that reads the data from the file system using this sort of structure (Maybe even doing it using Template Metaprogramming), I don't see how I can generate both the functions to access the data and the function to read the data.
I would want to end up with a class something like this
class Name{
public:
xyz_type getxyz();
void setxyz(xyz_type v);
//etc
list<group_type> getGroupName();
//etc
void readData(filesystem){
//read xyz
//read abc
//etc
}
};
Does anyone have any idea if this is even possible?
--EDIT--
To clarify the intended usage for this. I have files in a standard format I want to read. The format is defined already, so it is not open to change. Each file can contain any number records, each of which can contain any number sub records.
The numerous record types each contain a diffrent set of sub records, but they can be are defined. So for example the Heightmap record must contain a Heightmap, but can optional contain normals.
So I would want to define a Record for that like so:
CLASS_BEGIN(Heightmap)
RECORD(VHDT, Heightmap, std::string) //Subrecord Name, Readable Name, Type
RECORD_OPTIONAL(VNML, Normals, std::string)
END_CLASS
For which I would want to output something with the functionality of a class like this:
class Heightmap{
public:
std::string getHeightmap(){
return mHeightmap->get<std::string>();
}
void setHeightmap(std::string v){
mHeight->set<std::string>(v);
}
bool hasNormal(){
return mNormal != 0;
}
//getter and setter functions for normals go here
private:
void read(Record* r){
mHeightmap = r->getFirst(VHDT);
mNormal = r->getFirst(VNML);
}
SubRecord* mHeightmap, mNormal;
}
The issue I am having is that I need every preprocessor definition twice. Once for defining the function definition within the class, and once for creating the read function. As the preprocessor is purely functional, I cannot push the data to a queue and generate the class on the END_CLASS marco definition.
I cannot see a way around this issue, but wondered if anyone who has a greater understanding of C++ did.
If you are looking for a way to serialize/deserialize data with C++ code generation, I would look at Google protobufs (http://code.google.com/p/protobuf/) or Facebook's Thrift (http://incubator.apache.org/thrift/).
For protobufs, you write a data definition like so:
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
A Person C++ class is then generated that lets you load, save and access this data. You can also generate python, java, etc.
You might be able to solve this problem using boost tuples. It will result in a design which is different to what you are thinking of now, but it should allow you to solve the problem in a generic way.
The following example defines a record of the form "std::string,bool" and then reads that data in from a stream.
#include "boost/tuple/tuple.hpp"
#include <iostream>
#include <sstream>
using namespace ::boost::tuples;
The functions are used to read the data from an istream. The first overload stops the iteration through the tuple after we reach the last record type:
//
// This is needed to stop when we have no more fields
void read_tuple (std::istream & is, boost::tuples::null_type )
{
}
template <typename TupleType>
void read_tuple (std::istream & is, TupleType & tuple)
{
is >> tuple.template get_head ();
read_tuple (is, tuple.template get_tail ());
}
The following class implements the getter member for our Record. Using the RecordKind as our key we get the specific member that we're interested in.
template <typename TupleType>
class Record
{
private:
TupleType m_tuple;
public:
//
// For a given member - get the value
template <unsigned int MBR>
typename element <MBR, TupleType>::type & getMember ()
{
return m_tuple.template get<MBR> ();
}
friend std::istream & operator>> (std::istream & is
, Record<TupleType> & record)
{
read_tuple (is, record.m_tuple);
}
};
The next type is the meta description for our record. The enumeration gives us a symbolic name that we can use to access the members, ie. the field names. The tuple then defines the types of those fields:
struct HeightMap
{
enum RecordKind
{
VHDT
, VNML
};
typedef boost::tuple < std::string
, bool
> TupleType;
};
Finally, we construct a record and read in some data from a stream:
int main ()
{
Record<HeightMap::TupleType> heightMap;
std::istringstream iss ( "Hello 1" );
iss >> heightMap;
std::string s = heightMap.getMember < HeightMap::VHDT > ();
std::cout << "Value of s: " << s << std::endl;
bool b = heightMap.getMember < HeightMap::VNML > ();
std::cout << "Value of b: " << b << std::endl;
}
And as this is all template code, you should be able to have records nested in records.
This is a technique I use a lot in C and C++, called "list macro". Suppose you have a list of things like variables, error messages, interpreter opcodes, or anything about which repetitive code needs to be written. In your case it is class member variables.
Suppose it is variables. Put them in a list macro like this:
#define MYVARS \
DEFVAR(int, a, 6) \
DEFVAR(double, b, 37.3) \
DEFARR(char, cc, 512) \
To declare the variables, do this:
#define DEFVAR(typ,nam,inival) typ nam = inival;
#define DEFARR(typ,nam,len) typ nam[len];
MYVARS
#undef DEFVAR
#undef DEFARR
Now you can generate any sort of repetitive code just by redefining DEFVAR and DEFARR, and instantiating MYVARS.
Some people find this rather jarring, but I think it's a perfectly good way to use the preprocessor as a code generator, and accomplish DRY. And, the list macro itself becomes a mini-DSL.
I might play around with a record mixin to do something similar -- add functionality to a class automagically at compile time
template<class Base, class XyzRecType>
class CRecord : public Base
{
protected:
RecType xyz;
public:
CRecord() : Base() {}
RecType Get() {return xyz;}
void Set(const RecType& anXyz) {xyz = anXyz;}
void ReadFromStream( std::istream& input)
{
...
}
};
class CMyClass
{
};
int main()
{
// now thanks to the magic of inheritance, my class has added methods!
CRecord<CMyClass, std::string> myClassWithAStringRecord;
myClassWithAStringRecord.Set("Hello");
}
In general you can accomplish exactly what you want if you merge everything into one macro and then leverage Booost Preprocessor library to define your class. Look at how I implemented the MACE_REFLECT macro which does a partial specialization of an entire class and must reference each name twice in different parts.
This is very similar to how I automatically parse JSON into structs with the help of the pre-processor.
Given your example, I would translate it as such:
struct Name {
xyz_type xyz;
abc_type abc;
boost::optional<foo_type> foo;
boost::optional<bar_type> bar;
};
MACE_REFLECT( Name, (xyz)(abc)(foo)(bar) )
I can now 'visit' the members of Name from my parser:
struct visitor {
template<typename T, T p>
inline void operator()( const char* name )const {
std::cout << name << " = " << c.*p;
}
Name c;
};
mace::reflect::reflector<Name>::visit(visitor());
If your objects can be represented as structs, arrays, key-value-pairs and primitives, then this technique works wonders and gives me instant serialization/deserializtion to/from json/xml or your custom record format.
https://github.com/bytemaster/mace/blob/master/libs/rpc/examples/jsonv.cpp
I'm not exactly sure what you're looking for in some cases.
What happens to foo and bar in the specification?
What does getGroupName actually return? (foo,bar)? or GroupName?
It looks like you're trying to create a mechanism for loading and accessing on-disk structures of arbitrary layout. Is this accurate? (Edit: Just noticed the "set" member function... so I guess you're looking for full serialization)
If you're on a *nix system, specifying your own compiler to compile to .o (likely a perl/python/what-have-you script that finishes with a call to gcc) in the Makefile is a trivial solution. Others might know of ways of doing this on windows.