I have written some physics simulation code in C++ and parsing the input text files is a bottleneck of it. As one of the input parameters, the user has to specify a math function which will be evaluated many times at run-time. The C++ code has some pre-defined function classes for this (they are actually quite complex on the math side) and some limited parsing capability but I am not satisfied with this construction at all.
What I need is that both the algorithm and the function evaluation remain speedy, so it is advantageous to keep them both as compiled code (and preferrably, the math functions as C++ function objects). However I thought of glueing the whole simulation together with Python: the user could specify the input parameters in a Python script, while also implementing storage, visualization of the results (matplotlib) and GUI, too, in Python.
I know that most of the time, exposing C++ classes can be done, e.g. with SWIG but I still have a question concerning the parsing of the user defined math function in Python:
Is it possible to somehow to construct a C++ function object in Python and pass it to the C++ algorithm?
E.g. when I call
f = WrappedCPPGaussianFunctionClass(sigma=0.5)
WrappedCPPAlgorithm(f)
in Python, it would return a pointer to a C++ object which would then be passed to a C++ routine requiring such a pointer, or something similar... (don't ask me about memory management in this case, though :S)
The point is that no callback should be made to Python code in the algorithm. Later I would like to extend this example to also do some simple expression parsing on the Python side, such as sum or product of functions, and return some compound, parse-tree like C++ object but let's stay at the basics for now.
Sorry for the long post and thx for the suggestions in advance.
I do things similar to this all the time. The simplest solution, and the one I usually pick because, if nothing else, I'm lazy, is to flatten your API to a C-like API and then just pass pointers to and from Python (or your other language of choice).
First create your classes
class MyFunctionClass
{
public:
MyFunctionClass(int Param)
...
};
class MyAlgorithmClass
{
public:
MyAlgorithmClass(myfunctionclass& Func)
...
};
Then create a C-style api of functions that creates and destroys those classes. I usually flatted in out to pass void* around becuase the languages I use don't keep type safety anyway. It's just easier that way. Just make sure to cast back to the right type before you actually use the void*
void* CreateFunction(int Param)
{
return new MyFunctionClass(Param);
}
void DeleteFunction(void* pFunc)
{
if (pFunc)
delete (MyFunctionClass*)pFunc;
}
void* CreateAlgorithm(void* pFunc)
{
return new MyAlgorithmClass(*(MyFunctionClass*)pFunc)
}
void DelteAlgorithm(void* pAlg)
{
if (pAlg)
delete (MyAlgorithmClass*)pAlg;
}
No all you need to do is make python call those C-style function. In fact, they can (and probably should) be extern "c" functions to make the linking that much easier.
Related
I'm not sure if my question makes any sense, but I certainly know it is near impossible to get the results from Google. Firstly, what I do not want. I don't want to call some function that prints "hello world" or adds two numbers together, I want to load a Lua script from a C++ program, but allow the script to modify variables of the C++ program, using functions. For example, imagine I have a C++ program like this:
class Foo
{
private:
int number = 0;
public:
void setNumber(const int& newNumber) {number = newNumber;}
}
int main()
{
Foo foo;
//Load and execute Lua script, with foo object
return 0;
}
How could I allow the Lua script to do foo.setNumber() (preferably without foo.)? This may be a very simple question, but as mentioned above, almost all information on Google when searching "Call C++ Function from Lua" assume there is no program, but just a .cpp/hpp file with some functions that you want to call.
I'm on Linux (Ubuntu), but the program needs to compile on all platforms (Windows and Mac)
This is asked here fairly regularly.
To roll your own binding you should:
Master Lua metatables completely.
Read the Programming in Lua stuff on the C API, particularly the part on classes. Alternatively you can read the manual, read the source (API headers especially), and do some googling, but the book will probably save you some time.
Broadly, you expose a C++ class instance to Lua by creating a Lua "userdata" containing a pointer to the class instance and passing this to the Lua script. A userdata is an opaque type; the Lua script can't actually do anything with it (other than pass it around) unless you give it a metatable. At the very least you must implement the __index metamethod on the userdata, which allows your C++ code to intercept attempts to index the userdata and return something meaningful, and the __gc metamethod, which allows your C++ code to delete the exposed C++ object when the corresponding Lua userdata is garbage collected.
For instance, you create a function called createFoo which creates a Foo instance, wraps the pointer as a userdata, applies a metatable implementing __index to it, and returns it to the Lua script.
When the user runs foo.setNumber, your C++ __index metamethod is called with the userdata and the string "setNumber". It's up to you what you return and this determines what foo.setNumber evaluates to in the Lua script. You want foo.setNumber to evaluate to a lua_CFunction which expects a Foo userdata as its first parameter, so that your class methods can be called idiomatically from Lua (i.e. foo:setNumber(12), which is syntax sugar for foo.setNumber(foo, 12)).
It's a very low level and manual process, and once you get the hang of it you're going to end up create a library/templates/macros whatever to do the boilerplate for you. At that point you may want to evaluate the myriad C++ binding libraries that exist. However, thanks to the Law of Leaky Abstractions it's a very good idea to learn to do this manually first.
I had a script with:
Custom language used only for data
Was loaded using a Script class from C++
I had tags like Type, etc
An interface to get a value for a tag - Script::GetValue(Tag, T& value)
The script was used like this:
Script* script("someFile");
script->GetValue("Type", type);
Object* obj = CreateObject(type);
obj->Load(script);
Where Load functions from object was used to load the rest of obj parameters.
Now I changed the script language to lua. My questions is:
Should I keep this way of creating objects(use lua only for data) or should I expose the factory in lua and use it from lua, something like this(in lua):
CreateObject("someType")
SetProperty(someObj, someProperty, someValue)
First of all I want to know which is faster, first or second approach. Do you have other suggestions? Because I'm refactoring this part I'm open to other suggestions. I want to keep lua because is fast, easy to integrate, and small.
You may allow your script environment to create C++ objects or not, depending on your needs.
tolua++ uses all the metatable features to allow a very straightforward manipulation of your c++ types in lua.
For example, this declaration :
// tolua_begin
class SomeCppClass
{
public:
SomeCppClass();
~SomeCppClass();
int some_field;
void some_method();
};
// tolua_end
Will automatically generate the lua bindings to allow this lua scipt :
#!lua
-- obj1 must be deleted manually
local obj1 = SomeCppClass:new()
-- obj1 will be automatically garbage collected
local obj2 = SomeCppClass:new_local()
obj1.some_field = 3 -- direct access to "some_field"
obj2:some_method() -- direct call to "some_method"
obj1:delete()
The advantage of this technique is that your lua code will ve very consistent with the relying C++ code. See http://www.codenix.com/~tolua/tolua++.html
In situations like that, I prefer to setup a bound C function that takes a table of parameters as an argument. So, the Lua script would look like the following.
CreateObject{
Type = "someType"'
someProperty = someValue,
-- ...
}
This table would be on top of the stack in the callback function, and all parameters can be accessed by name using lua_getfield.
You may also want to investigate sandboxing your Lua environment.
The first approach would most likely be faster, but the second approach would probably result in less object initialization code (assuming you're initializing a lot of objects). If you choose the first approach, you can do it manually. If you choose the second approach you might want to use a binding library like Luabind to avoid errors and speed up implementation time, assuming you're doing this for multiple object types and data types.
The simplest approach will probably be to just use Lua for data; if you want to expose the factory and use it via Lua, make sure it's worth the effort first.
I inherited a big application that was originally written in C (but in the mean time a lot of C++ was also added to it). Because of historical reasons, the application contains a lot of void-pointers. Before you start to choke, let me explain why this was done.
The application contains many different data structures, but they are stored in 'generic' containers. Nowadays I would use templated STL containers for it, or I would give all data structures a common base class, so that the container can store pointers to the base class, but in the [good?] old C days, the only solution was to cast the struct-pointer to a void-pointer.
Additionally, there is a lot of code that works on these void-pointers, and uses very strange C constructions to emulate polymorphism in C.
I am now reworking the application, and trying to get rid of the void-pointers. Adding a common base-class to all the data structures isn't that hard (few days of work), but the problem is that the code is full of constructions like shown below.
This is an example of how data is stored:
void storeData (int datatype, void *data); // function prototype
...
Customer *myCustomer = ...;
storeData (TYPE_CUSTOMER, myCustomer);
This is an example of how data is fetched again:
Customer *myCustomer = (Customer *) fetchData (TYPE_CUSTOMER, key);
I actually want to replace all the void-pointers with some smart-pointer (reference-counted), but I can't find a trick to automate (or at least) help me to get rid of all the casts to and from void-pointers.
Any tips on how to find, replace, or interact in any possible way with these conversions?
I actually want to replace all the
void-pointers with some smart-pointer
(reference-counted), but I can't find
a trick to automate (or at least) help
me to get rid of all the casts to and
from void-pointers.
Such automated refactoring bears many risks.
Otherwise, sometimes I like to play tricks by making out of such void* functions the template functions. That:
void storeData (int datatype, void *data);
becomes:
template <class T>
void storeData (int datatype, T *data);
At first implement template by simply wrapping the original (renamed) function and converting the types. That might allow you to see potential problems - already by simply compiling the code.
You probably don't need to get rid of the casts to use shared pointers.
storeData(TYPE_CUSTOMER, myCustomer1->get());
shared_ptr<Customer> myCustomer2(reinterpret_cast<Customer*>fetchData(TYPE_CUSTOMER, "???");
Of course, this assumes that you don't expect to share the same pointer across calls to store/fetch. In other words, myCustomer1 and myCustomer2 don't share the same pointer.
Apparently, there is no automated way/trick to convert or find all uses of void-pointers. I'll have to use manual labor to find all void-pointers, in combination with PC-Lint that will give errors whenever there is an incorrect conversion.
Case closed.
I'm wondering about an idea in my head. I want to ask if you know of any library or article related to this. Or you can just tell me this is a dumb idea and why.
I have a class, and I want to dynamically add methods/properties to it at runtime. I'm well aware of the techniques of using composite/command design pattern and using embedded scripting language to accomplish what I'm talking about. I'm just exploring the idea. Not necessary saying that it is a good idea.
class Dynamic
{
public:
typedef std::map<std::string, boost::function<void (Dynamic&)> > FuncMap;
void addMethod(const std::string& name, boost::function<void (Dynamic&)> func) {
funcMap_[name] = func;
}
void operator[](const std::string& name) {
FuncMap::iterator funcItr = funcMap_.find(name);
if (funcItr != funcMap_.end()) {
funcItr->second(*this);
}
}
private:
FuncMap funcMap_;
};
void f(Dynamic& self) {
doStuffWithDynamic(self);
}
int main()
{
Dynamic dyn;
dyn.addMethod("f", boost::bind(&f, _1));
dyn["f"]; // invoke f
}
The idea is that I can rebind the name "f" to any function at runtime. I'm aware of the performance problem in string lookup and boost::function vs. raw function pointer. With some hard work and non-portable hack I think I can make the performance problem less painful.
With the same kind of technique, I can do "runtime inheritance" by having a "v-table" for name lookup and dispatch function calls base on dynamic runtime properties.
If just want to tell me to use smalltalk or Objective-C, I can respect that but I love my C++ and I'm sticking with it.
What you want is to change C++ into something very different. One of the (many) goals of C++ was efficient implementation. Doing string lookup for function calls (no matter how well you implement it), just isn't going to be very efficient compared to the normal call mechanisms.
Basically, I think you're trying to shoehorn in functionality of a different language. You CAN make it work, to some degree, but you're creating C++ code that no one else is going to be able (or willing) to try to understand.
If you really want to write in a language that can change it's objects on the fly, then go find such a language (there are many choices, I'm sure). Trying to shoehorn that functionality into C++ is just going to cause you problems down the road.
Please note that I'm no stranger to bringing in non-C++ concepts into C++. I once spent a considerable amount of time removing another engineer's attempt at bringing a based-object system into a C++ project (he liked the idea of containers of 'Object *', so he made every class in the system descend from his very own 'Object' class).
Bringing in foreign language concepts almost always ends badly in two ways: The concept runs up against other C++ concepts, and can't work as well as it did in the source language, AND the concept tends to break something else in C++. You end up losing a lot of time trying to implement something that just isn't going to work out.
The only way I could see something like this working at all well, is if you implemented a new language on top of C++, with a cfront-style pre-compiler. That way, you could put some decent syntax onto the thing, and eliminate some of your problems.
If you implemented this, even as a pure library, and then used it extensively, you would in a way be using a new language - one with a hideous syntax, and a curious combination of runtime method resolution and unreliable bounds checking.
As a fan of C/C++ style syntax and apparently a fan of dynamic method dispatch, you may be interested in C# 4.0, which is now in Beta, and has the dynamic keyword to allow exactly this kind of thing to be seamlessly mixed into normal statically typed code.
I don't think it would be a good idea to change C++ enough to make this work. I'd suggest working in another language, such as Lisp or Perl or another language that's basically dynamic, or imbedding a dynamic language and using it.
What you are doing is actually a variation of the Visitor pattern.
EDIT: By the way, another approach would be by using Lua, since the language allows you to add functions at runtime. So does Objective-C++.
EDIT 2: You could actually inherit from FuncMap as well:
class Dynamic;
typedef std::map<std::string, boost::function<void (Dynamic&)> > FuncMap;
class Dynamic : public FuncMap
{
public:
};
void f(Dynamic& self) {
//doStuffWithDynamic(self);
}
int main()
{
Dynamic dyn;
dyn["f"] = boost::bind(&f, _1);
dyn["f"](dyn); // invoke f, however, 'dyn'param is awkward...
return 0;
}
If I understand what you are trying to accomplish correctly, it seems as though dynamic linking (i.e. Dynamically loaded libraries in windows or linux) will do most of what you are trying to accomplish.
That is, you can, at runtime, select the name of the function you want to execute (eg. the name of the DLL), which then gets loaded and executed. Much in the way that COM works. Or you can even use the name of the function exported from that library to select the correct function (C++ name mangling issues aside).
I don't think there's a library for this exact thing.
Of course, you have to have these functions pre-written somehow, so it seems there would be an easier way to do what you want. For example you could have just one method to execute arbitrary code from your favorite scripting language. That seems like an easier way to do what you want.
I keep thinking of the Visitor pattern. That allows you to do a vtable lookup on the visiting object (as well as the visited object, thought that doesn't seem relevant to your question).
And at runtime, you could have some variable which refers to the visitor, and call
Dynamic dynamic;
DynamicVisitor * dv = ...;
dynamic->Accept(dv);
dv = ...; // something else
dynamic->Accept(dv);
The point is, the visitor object has a vtable, which you said you wanted, and you can change its value dynamically, which you said you wanted. Accept is basically the "function to call things I didn't know about at compile time."
I've considered doing this before as well. Basically, however, you'd be on your way to writing a simple VM or interpreter (look at, say, Lua or Topaz's source to see what I mean -- Topaz is a dead project that pre-dates Parrot).
But if you're going that route it makes sense to just use an existing VM or interpreter.
I have a class that is wrapped with swig, and registered with lua. I can create an instance of this class in a lua script, and it all works fine.
But say I have an instance of a class made in my c++ code with a call to new X, and I have la lua_state L with a function in it that I want to call, which accepts one argument, an instance of X... How do I call that function. Here is (some) of the code in question (I've omitted the error handling stuff):
main.cpp
class GuiInst;
extern "C"
{
int luaopen_engine (lua_State *L);
}
int main()
{
GuiInst gui=new GuiInst;
lua_State *L=luaL_newstate();
luaopen_engine(L); //this is swigs module
int error=luaL_loadfile(L,"mainmenu.lua")||
lua_pcall(L, 0, 0, 0);
lua_getglobal(L,"Init");
//Somehow push gui onto lua stack...
lua_pcall(L, 1, 0, 0));
lua_close(L);
}
mainmenu.lua
function Init(gui)
vregion=gui:CreateComponent("GuiRegionVertical");
end
At the moment all I have found that can work is to expose some functionality from the swig generated cpp file, and call that. This is bad for a few reasons... It won't work if I have multiple modulles and I had to change the default linkage specification in the swig file (using -DSWIGRUNTIME=).
I add the following to main.cpp
extern "C"
{
struct swig_module_info;
struct swig_type_info;
int luaopen_engine (lua_State *L);
swig_module_info *SWIG_Lua_GetModule(lua_State* L);
void SWIG_Lua_NewPointerObj(lua_State* L,void* ptr,swig_type_info *type, int own);
swig_type_info *SWIG_TypeQueryModule(swig_module_info *start,swig_module_info *end,const char *name);
}
//and then to push the value...
SWIG_Lua_NewPointerObj(L,gui,SWIG_TypeQueryModule(SWIG_Lua_GetModule(L),SWIG_Lua_GetModule(L),"GuiInst *"),0);
That gets a pointer to the module, then a pointer to the type, then calls swigs function to register it. It was an unreasonable thing to have to dig into a file that's not supposed to be human readable (so it says at the top of the file) and is just MESSY! (but it does work!)
Surely theres a better way to accomplish what I'm trying to do.
PS from a high level pov what I want is to have lua not refcount the Gui components which are created by the Object Factory in GuiInst, in case I'm going about this wrong. This is my first time exposing functionality to a scripting language apart from some very simple (and non-swig) python modules, so I'm prepared to take advice.
Thanks for any advice!
Response to comment by RBerteig
GuiInst's contructor is #defined to private when swig runs to prevent lua constructing instances of it, so that won't work for me. What I was trying to prevent was the following (in lua):
r=engine.GuiRegionVertical()
r:Add(engine.GuiButton())
which would call "g=new GuiButton" then register it with the GuiRegionVertical (which needs to store a pointer for various reasons), then call "delete g", and the GuiRegionVertical is left with a dangling pointer to g.
I suspect what really needs to happen is that GuiRegionVertical::Add(GuiButton*) should increment the ref count of the GuiButton*, and then GuiRegionVertical's destructor should decrement the refcounts of all of its contents, though i'm not sure how this should be done with swig.
That would remove the need for the private constructors, the Gui Object Factory and the nasty externs.
Am I going about this Wrong?
Thanks.
Better late then never, and this solution will help other people.
void handle_web_request(WebRequest *request, WebResponse *response)
{
lua_getfield(rackam->lua_state, LUA_GLOBALSINDEX, "handle_web_request");
SWIG_Lua_NewPointerObj(rackam->lua_state, request, SWIGTYPE_p_WebRequest, 0);
SWIG_Lua_NewPointerObj(rackam->lua_state, response, SWIGTYPE_p_WebResponse, 0);
lua_call(rackam->lua_state, 2, 0);
}
this code must be inside %{}% blocks in your .i file, because SWIGTYPE_p_WebRequest is
#define SWIGTYPE_p_WebResponse swig_types[6]
and swig_types[6] is
static swig_type_info *swig_types[12];
which means that swig_types is only accessable from the C++ file from which it is defined.
this particular snippet is sending in two of my wrappered pointers, so calling handle_web_request(request, response) from the C++ side of things will run the global lua function "handle_web_request" and pass it my two pointers, with the SWIG magic applied.
There is a simple and direct answer, that may not be the most efficient answer. SWIG produces wrappers for manipulating objects from the scripting language side. For objects, it also synthesizes a wrapped constructor. So, the direct solution is to just let the Lua interpreter call SWIG's constructor to create the new object.
For the wrapped engine.GuiInst class, you almost certainly can do something like:
int main()
{
lua_State *L=lua_open();
luaopen_engine(L); //this is swigs module
int error=luaL_loadfile(L,"mainmenu.lua")||
lua_pcall(L, 0, 0, 0);
luaL_dostring(L, "Init(engine.new_GuiInst())");
lua_close(L);
}
For a one-shot case like script startup, the penalty of running a string constant through luaL_dostring() is not bad at all. I'd look harder to avoid it in an event callback or an inner loop, however.
It does seem like there ought to be a way to convert a pointer directly into a wrapped object, I'm not spotting it in my own handful of SWIG generated wrappers.
Edit: Of course, the Lua fragment can be decomposed to API calls that get the engine table global on the stack, extract from it the new_GuiInst member, call it, then call the global Init, but the little bit of efficiency comes at the cost of some clarity.
As for dealing with objects that shouldn't be constructed by accident in user code, as the clarified question indicates, my first impulse would be to let SWIG generate the constructor function, keep a private reference if needed later, and remove it from the table. Even a C module is (usually) just a table whose members contain function values. Being implemented in C doesn't make them read-only unless extra effort is taken.
So, you could always retrieve the value of engine.new_GuiInst and park it in the registry (see luaL_ref() and the discussion in section 3.5 of the Lua Reference Manual of the pseudo-index LUA_REGISTRYINDEX for the details) for later use. Then, before letting any user code run, simply do the equivalent of engine.new_GuiInst = nil. I should note that for the C data types I've been playing with most recently, SWIG created two constructors for each type, named new_TYPE and TYPE. Both were visible in the module's table, and you would want to set both names to nil. If have much less experience with SWIG wrapping C++ classes, and the result may differ...
You might want to check and review the whole content of the engine table returned by SWIG, and create a proxy object that contains only the methods you want available to your users. You can also change the environment seen by the user script so that it only has the proxy available, and names the proxy engine as well. There has been a fair amount of discussion of sandboxing user scripts on the Lua list and at the lua-users wiki.