I have searched the internet for hours at this point. Does anyone know how to parse a namedtuple returned from a python function into a struct or just into separate variables. The part I am having trouble with is getting the data out of the returned pointer. I am calling a python function embedded in C++ using the PyObject_CallFunction() call and I don't know what to do once I have the PyObject* to the returned data.
I am using Python 2.7 for reference.
EDIT: I ended up moving all of the functionality I was trying to do in both Python and C++ to just Python for now. I will update in the near future about attempting the strategy suggested in the comments of this question.
I am calling a python function embedded in C++ using the
PyObject_CallFunction() call and I don't know what to do once I have
the PyObject* to the returned data.
A namedtuple is a tuple subclass that additionally exposes tuple elements as named attributes. This means that you can choose whether to access its data as obj[position] or obj.attribute. The latter is generally more readable, but the former combines well with tuple unpacking. In Python/C, it is probably easier to access it as tuple, since then you can use the convenience function PyArg_ParseTuple, as indicated in the comment.
To extract arbitrary attributes of an object (not necessarily a namedtuple), one would call PyObject_GetAttrString. Given an object describing, say, a point, extracting an attribute such as x might look like this:
PyObject *point = ...; // assume we get a new reference to point
if (!point)
return NULL;
PyObject *x = PyObject_GetAttrString(point, "x");
if (!x) {
// obj.x raised, possibly because point is of a different type
Py_DECREF(point);
return NULL;
}
double x_val = PyFloat_AsDouble(x);
Py_DECREF(x); // x not used below this line
if (x_val == -1 && PyErr_Occurred()) {
// obj.x is not float or float-like
Py_DECREF(point);
return NULL;
}
Py_DECREF(point); // point not used below this line
The error checking and reference counting is quite tedious, but it can be mostly eliminated using guard classes or, better yet, using the classes written by others, such as Boost.Python.
namedtuple is implemented purely in Python. You can see its full source in collections.py. It's very short. The thing to keep in mind is that namedtuple itself is a function which creates a class in the frame in which it is called and then returns this class (not an instance of this class). And it is this returned class that is then used to create instances. So the object which you get is not what you want to pass into C++ if you want to pass individual instances.
C++ creates struct definitions at compile time. namedtuple creates namedtuple classes at run time. If you want to bind them to C++ structs, either use the PyObject to create your newly minted class' instances inside of C++ and assign them to struct elements at compile time. Or create the newly minted class' instances in Python and pass them to C++.
Or you can use _asdict method (provided by namedtuple factory method for all classes it builds) and pass that to C++ to then do the binding of run-time defined data to compile-time defined data.
If you really want to do the bulk of the work in C++, you may also use the Struct module instead of using namedtuple.
namedtuple is really the swiss-army knife of Python for data which stays in Python. It gives positional access, named access, and all the elements are also "properties" (so they have fget accessor method which can be used in maps, filters, etc. instead of having to write your own lambdas).
It's there for things like DB binding (when you don't know which columns will be there at run time). It's less clunky than OrderedDict for converting data from one format into another. When it's used that way, the overhead of processing strings is nothing compared to actual access of the db (even embedded). But I wouldn't use namedtuple for large arrays of structs which are meant to be used in calculations.
Related
While doing a game engine that uses .lua files in order to read parameter values, I got stuck when I had to read these values and assign them to the parameters of each component in C++. I tried to investigate the way Unity does it, but I didn't find it (and I'm starting to doubt that Unity has to do it at all).
I want the parameters to be initialized automatically, without the user having to do the process of
myComponentParameter = readFromLuaFile("myParameterName")
for each one of the parameters.
My initial idea is to use the std::variant type, and storing an array of variants in order to read them automatically. My problems with this are:
First of all, I don't know how to know the type that std::variant is storing at the moment (tried with std::variant::type, but it didn't work for the template), in order to cast from the untyped .lua value to the C++ value. For reference, my component initialization looks like this:
bool init(luabridge::LuaRef parameterTable)
{
myIntParameter = readVariable<int>(parameterTable, "myIntParameter");
myStringParameter = readVariable<std::string>(parameterTable, "myStringParameter");
return true;
}
(readVariable function is already written in this question, in case you're curious)
The second problem is that the user would have to write std::get(myIntParameter); whenever they want to access to the value stored by the variant, and that sounds like something worse than making the user read the parameter value.
The third problem is that I can't create an array of std::variant<any type>, which is what I would like to do in order to automatically initialize the parameters.
Is there any good solution for this kind of situation where I want the init function to not be necessary, and the user doesn't need to manually set up the parameter values?
Thanks in advance.
Let's expand my comment. In a nutshell, you need to get from
"I have some things entered by the user in some file"
to:
"the client code can read the value without std::get"
…which roughly translates to:
"input validation was done, and values are ready for direct use."
…which implies you do not store your variables in variants.
In the end it is a design question. One module somewhere must have the knowledge of which variable names exist, and the type of each, and the valid values.
The input of that module will be unverified values.
The output of the module will probably be some regular c++ struct.
And the body of that module will likely have a bunch of those:
config.foo = readVariable<int>("foo");
config.bar = readVariable<std::string>("bar");
// you also want to validate values there - all ints may not be valid values for foo,
// maybe bar must follow some specific rules, etc
assuming somewhere else it was defined as:
struct Configuration {
int fooVariable;
std::string bar;
};
Where that module lives depends on your application. If all expected types are known, there is no reason to ever use a variant, just parse right away.
You would read to variants if some things do not make sense until later. For instance if you want to read configuration values that will be used by plugins, so you cannot make sense of them yet.
(actually even then simply re-parsing the file later, or just saving values as text for later parsing would work)
How do I create an object in python from a feature class in a geodatabase? I would think the following code would create a featureclass object?
featureclassobject = "C:/path/to/my/featureclass"
But this creates a string object, right? So I am not able to pass this object into an arcpy function later on.
You are correct that it creates a string object. However, whether it will work with a particular ArcPy function depends on the function -- in most cases, the tool simply needs to know the path to the function as a string (which the featureclassobject is).
The help pages are slightly unhelpful in this regard. Buffer, for example, says that input parameter in_features needs to be data type "Feature Layer" -- however, what it really expects is a string that describes where the feature layer can be found.
One significant exception to this is geometry objects:
In many geoprocessing workflows, you may need to run a specific operation using coordinate and geometry information but don't necessarily want to go through the process of creating a new (temporary) feature class, populating the feature class with cursors, using the feature class, then deleting the temporary feature class. Geometry objects can be used instead for both input and output to make geoprocessing easier.
But if you've already got a feature class (or shapefile) on disk, that's much simpler than creating an in-memory geometry object to work with.
In C++ I have the following two classes that I expose (using Boost) to Python:
struct Foo {
// Empty
};
struct FooContainer {
// I use boost::shared_ptr for compatibility with Boost.Python
vector<boost::shared_ptr<Foo>> foos_;
};
In the Python side I might create a special type of Foo that actually does something instead of being just an empty class, and then add it to a FooContainer:
class Useful(Foo):
def __init__(self, a, b):
self.a = a
self.b = b
x = Useful(3, 5);
# Add 'x' to a `FooContainer`
Back in the C++ side, the FooContainer now has some Foos, but it doesn't know or care that they are from Python. The application runs for a while and the data in the Foo objects changes...
Then I decide I want to save the state of my program so I can load it at a later time. But the problem is that FooContainer doesn't know much about its Foo objects, it doesn't even know that they come from Python and I wouldn't want to pollute my FooContainer with data that doesn't really belong in it (single-responsibility principle and all that).
Do you have any advice on how I should organize my application such that saving and loading data, as well as loading fresh data (ie. not from a state that I saved in the past) can be done in a clear way?
You can use boost::python/pickle, and save the data from python. I only have limited experience with the pickling suite, but it should work provided you override appropriate pickling methods in your classes derived in python (see my answer to this question).
You already have python code that creates the Foos, lets call it populateFoos and somehow you have your program call it.
Now the next thing you need is a storeFoos and loadFoos function that does the saving and loading. If you want to keep it generic define them as storeFunc and loadFunc (or callback, depending on the context).
Depending on your program structure you might also need to keep in python a list of all foos created (or associated to a container).
I had a script with:
Custom language used only for data
Was loaded using a Script class from C++
I had tags like Type, etc
An interface to get a value for a tag - Script::GetValue(Tag, T& value)
The script was used like this:
Script* script("someFile");
script->GetValue("Type", type);
Object* obj = CreateObject(type);
obj->Load(script);
Where Load functions from object was used to load the rest of obj parameters.
Now I changed the script language to lua. My questions is:
Should I keep this way of creating objects(use lua only for data) or should I expose the factory in lua and use it from lua, something like this(in lua):
CreateObject("someType")
SetProperty(someObj, someProperty, someValue)
First of all I want to know which is faster, first or second approach. Do you have other suggestions? Because I'm refactoring this part I'm open to other suggestions. I want to keep lua because is fast, easy to integrate, and small.
You may allow your script environment to create C++ objects or not, depending on your needs.
tolua++ uses all the metatable features to allow a very straightforward manipulation of your c++ types in lua.
For example, this declaration :
// tolua_begin
class SomeCppClass
{
public:
SomeCppClass();
~SomeCppClass();
int some_field;
void some_method();
};
// tolua_end
Will automatically generate the lua bindings to allow this lua scipt :
#!lua
-- obj1 must be deleted manually
local obj1 = SomeCppClass:new()
-- obj1 will be automatically garbage collected
local obj2 = SomeCppClass:new_local()
obj1.some_field = 3 -- direct access to "some_field"
obj2:some_method() -- direct call to "some_method"
obj1:delete()
The advantage of this technique is that your lua code will ve very consistent with the relying C++ code. See http://www.codenix.com/~tolua/tolua++.html
In situations like that, I prefer to setup a bound C function that takes a table of parameters as an argument. So, the Lua script would look like the following.
CreateObject{
Type = "someType"'
someProperty = someValue,
-- ...
}
This table would be on top of the stack in the callback function, and all parameters can be accessed by name using lua_getfield.
You may also want to investigate sandboxing your Lua environment.
The first approach would most likely be faster, but the second approach would probably result in less object initialization code (assuming you're initializing a lot of objects). If you choose the first approach, you can do it manually. If you choose the second approach you might want to use a binding library like Luabind to avoid errors and speed up implementation time, assuming you're doing this for multiple object types and data types.
The simplest approach will probably be to just use Lua for data; if you want to expose the factory and use it via Lua, make sure it's worth the effort first.
I have written some physics simulation code in C++ and parsing the input text files is a bottleneck of it. As one of the input parameters, the user has to specify a math function which will be evaluated many times at run-time. The C++ code has some pre-defined function classes for this (they are actually quite complex on the math side) and some limited parsing capability but I am not satisfied with this construction at all.
What I need is that both the algorithm and the function evaluation remain speedy, so it is advantageous to keep them both as compiled code (and preferrably, the math functions as C++ function objects). However I thought of glueing the whole simulation together with Python: the user could specify the input parameters in a Python script, while also implementing storage, visualization of the results (matplotlib) and GUI, too, in Python.
I know that most of the time, exposing C++ classes can be done, e.g. with SWIG but I still have a question concerning the parsing of the user defined math function in Python:
Is it possible to somehow to construct a C++ function object in Python and pass it to the C++ algorithm?
E.g. when I call
f = WrappedCPPGaussianFunctionClass(sigma=0.5)
WrappedCPPAlgorithm(f)
in Python, it would return a pointer to a C++ object which would then be passed to a C++ routine requiring such a pointer, or something similar... (don't ask me about memory management in this case, though :S)
The point is that no callback should be made to Python code in the algorithm. Later I would like to extend this example to also do some simple expression parsing on the Python side, such as sum or product of functions, and return some compound, parse-tree like C++ object but let's stay at the basics for now.
Sorry for the long post and thx for the suggestions in advance.
I do things similar to this all the time. The simplest solution, and the one I usually pick because, if nothing else, I'm lazy, is to flatten your API to a C-like API and then just pass pointers to and from Python (or your other language of choice).
First create your classes
class MyFunctionClass
{
public:
MyFunctionClass(int Param)
...
};
class MyAlgorithmClass
{
public:
MyAlgorithmClass(myfunctionclass& Func)
...
};
Then create a C-style api of functions that creates and destroys those classes. I usually flatted in out to pass void* around becuase the languages I use don't keep type safety anyway. It's just easier that way. Just make sure to cast back to the right type before you actually use the void*
void* CreateFunction(int Param)
{
return new MyFunctionClass(Param);
}
void DeleteFunction(void* pFunc)
{
if (pFunc)
delete (MyFunctionClass*)pFunc;
}
void* CreateAlgorithm(void* pFunc)
{
return new MyAlgorithmClass(*(MyFunctionClass*)pFunc)
}
void DelteAlgorithm(void* pAlg)
{
if (pAlg)
delete (MyAlgorithmClass*)pAlg;
}
No all you need to do is make python call those C-style function. In fact, they can (and probably should) be extern "c" functions to make the linking that much easier.