I am writing code generator and using flatbuffers for generating classes. The rest of the code generator will work with these classes in C++.
I have not been able to figure out how to keep the API consistent for reading data for two different types of classes that flatbuffer may generate. I am using the object api (testRecordT) in the example for whenever an object needs to be written to (and can be read back as well) and flatbuffer overlay for when the data can only be read from.
I have not been able to get any template or free functions to work to give me a consistent api that would work in both the cases.
Below is a snippet of what I am trying to get to work.
struct testRecordT {
int32_t field1;
std::string field2;
};
struct testRecord {
int32_t field1() const {
return 0;
// flatbuffer generated - return GetField<int32_t>(VT_FIELD1, 0);
}
const flatbuffers::String *field2() const {
return nullptr;
// flatbuffer generated - return GetPointer<const flatbuffers::String *>(VT_FIELD3);
}
};
void Test() {
testRecordT * members; // assume pointers are valid
testRecord * memberFunctions;
// Need to be able to create a read function/template that would work. This would simplify the code generation a lot. I can generate either one below, as long as it is consistent in both cases.
auto r = read(members->field1); // or read(members,field1)
auto v = read(memberFunctions->field1); // or read(memberFunctions,field1)
}
The read functions or template functions should be consistent. Any pointers or thoughts would be helpful. I am using C++17 with gcc 7.3.1 .
You can use std::invoke for this. It can both call functions or access members.
auto r = std::invoke(&testRecordT::field1, members);
auto v = std::invoke(&testRecord::field1, memberFunctions);
You can use std::invoke() for this problem.
Related
This is a bit of a doozy and needs some explanation. I'm sorry in advance if I mess up the semantics or if I misunderstand something- I'm no beginner to C++ but some of its more advanced features I have yet to fully wrap my head around.
I'm writing language bindings from a library in C++ (although I am using the C API for this library since it's compiled with MSVC and I'm using MinGW) and I need to write up a binding for callbacks using function pointers. Even if I was using the C++ API, this would be a problem.
Luckily for me, the language I am writing bindings for Ruby (it doesn't really matter, I'm pretty sure other languages have similar C API bindings and this is language agnostic anyway) allows me to get an anonymous function useful for callbacks, in the form of a VALUE.
It's easy enough for me to wrap up these callbacks in a lambda which I can get a function pointer to, like so:
/* This is just a function that is bound to Ruby. It doesn't really matter much. */
VALUE set_some_callback(VALUE callback)
{
/* The types here don't really matter */
/* Create the lambda */
auto lambda = [callback](){
/* Run the callback, the semantics don't matter a lot here */
rb_funcall(callback, rb_intern("call"), 0);
}
/* Takes a function pointer (something like (void*)()) */
library_set_some_callback(fn);
return Qnil; // Return nil
}
This works just fine, in fact I've written bindings for other libraries that do something like this. However, this won't work for the particular library I'm writing bindings for since it exposes callbacks you've set in a struct.
/* Again, semantics don't matter here, this is for sake of example */
typedef struct Foo
{
int data;
(void*)() callback;
} Foo;
This is where the problem lies.
I can bind data (or whatever equivalent) just fine, but I can't bind the callback since it's a function pointer wrapping the actual callback. To bind the callback, I'd somehow need to extract that, and I don't know how...
I've come up with something along the lines of overloading operator() in a custom struct to wrap a callback, but I'm not sure if that's the right approach, or if it would even work.
struct BoundCallback
{
/* Store the callback */
VALUE callback;
/* Overload () */
void operator() const
{
rb_funcall(callback, rb_intern("call"), 0);
}
};
VALUE get_callback_from_foo(VALUE self)
{
/* Implementation of this doesn't matter. */
Foo foo = get_data_from(self);
/* Would this work...? */
BoundCallback callback = static_cast<BoundCallback>(foo.callback);
/* Would this work too? */
return callback.callback;
}
What's the best way to go about solving this problem?
I'm writing some kind of tool that extracts the interface definitions of C++ code.
In process of writing, I decided to restrict the parser to process only the code that was explicitly marked for processing, and I thought that C++ attributes are the best way to do it.
I'd prefer to add e.g. [[export]] annotations to entities I want to export, but I realised that libTooling is unable to see custom attributes without registering them in Clang code itself (I mean adding the attribute to tools/clang/include/clang/Basic/Attr.td).
Thus, my question is: is there a way to register the attribute without modifying that file (e.g. by registering the attribute programmatically or writing own Attr.td file)?
UPD: I'm using ASTMatchers library for source code analysis, so visitor-based approach probably does not work for me.
From what I can tell it is not possible to register custom attributes without directly modifying libtooling.
If you're willing to use pre-processor macros instead of attributes there is a workaround that I've done in the past. The basics are that we'll declare an empty macro, write a pre-processor callback to identify the location of the macro and store it in a queue, then in an AST visitor we'll visit records for either classes, methods, or variables, and check to see if preceeding the entity is our macro.
For the preprocessor you'll need to extend clang::PPCallbacks and implement the MacroExpands method.
void MyPreProcessorCallback::MacroExpands(const clang::Token& MacroNameTok, const clang::MacroDefinition&, const clang::SourceRange Range, const clang::MacroArgs* const Args)
{
// Macro is not named for some reason.
if(!MacroNameTok.isAnyIdentifier())
{ return; }
if(MacroNameTok.getIdentifierInfo()->getName() == "EXPORT")
{
// Save Range into a queue.
}
else
{
return;
}
// If you want arguments you can declare the macro to have varargs and loop
// through the tokens, you can have any syntax you want as they're raw tokens.
// /* Get the argument list for this macro, because it's a
// varargs function all arguments are stored in argument 0. */
// const ::clang::Token* token = Args->getUnexpArgument(0u);
// // All tokens for the argument are stored in sequence.
// for(; token->isNot(::clang::tok::eof); ++token)
// {
// }
}
Inside your RecursiveAstVisitor you can implement visitors that will pop off the top of the queue and check to see if the top macro is before in the translation unit. IIRC visitors of a type are all executed in order of declaration, so the queue should maintain the order. It is worth noting that all Decl's of a type are visited in order, so care has to be taken when distinguishing between function, variables, and classes.
bool MyAstVisitor::VisitFunctionDecl(::clang::FunctionDecl* const function)
{
if(::llvm::isa<::clang::CXXMethodDecl>(function))
{
// If you want to handle class member methods separately you
// can check here or implement `VisitCXXMethodDecl` and fast exit here.
}
if(ourExportTags.empty())
{
return true;
}
const ::clang::SourceLocation tagLoc = ourExportTags.front().getBegin();
const ::clang::SourceLocation declLoc = function->getBeginLoc();
if(getAstContext().getSourceManager().isBeforeInTranslationUnit(tagLoc, declLoc))
{
ourExportTags.pop_front();
// Handle export;
}
return true;
}
EDIT
I haven't used ASTMatchers before, but you could probably accomplish a similar result by writing a matcher, storing all of the declarations to a list, sorting based on location, and then comparing to the original export tag queue.
DeclarationMatcher matcher = functionDecl().bind("funcDecl");
class MyFuncMatcher : public clang::ast_matchers::MatchFinder::MatchCallback
{
public:
virtual void run(const clang::ast_matchers::MatchFinder::MatchResult& Result)
{
if(const FunctionDecl* func = Result.Nodes.getNodeAs<clang::FunctionDecl>("funcDecl"))
{
// Add to list for future processing
}
}
};
void joinTagsToDeclarations()
{
// Sort the declaration list
for(auto decl : myDeclList)
{
if(ourExportTags.empty())
{
break;
}
const ::clang::SourceLocation tagLoc = ourExportTags.front().getBegin();
const ::clang::SourceLocation declLoc = decl->getBeginLoc();
if(getAstContext().getSourceManager().isBeforeInTranslationUnit(tagLoc, declLoc))
{
ourExportTags.pop_front();
// Handle export;
}
}
}
I need to register functions like the following in a list of functions with arguments.
void func1( int a , char* b ) {}
void func2( vec3f a , std::vector<float> b , double c) {}
...
And call them back when I receive data over network with proper arguments. I imagined va_list would solve, but it doesnt work :
void func1(int a, char* b)
{
printf("%d %s",a,b);
}
void prepare(...)
{
va_list argList;
int args = 2;
va_start(argList, args);
((void (*)(va_list))func1)(argList);
va_end(argList);
}
int main(int argc, char **argv)
{
prepare(1, "huhu");
return 0;
}
What is the most elegant way to solve this ?
I know std::bind / std::function has similar abilities, but the internal data is hidden deep in std I assume. I just need a few basic data types, doesnt have to be for arbitrary types. If preprocessor tricks with ##VA_ARGS or using templates would solve, I am also OK with that. Priority is that it is most simple to use.
Edit1 : I found that assembly can solve ( How do I pass arguments to C++ functions when I call them from inline assembly ) - but I would prefer a more platform independent solution.
If your goal is to create your own, small and ad-hoc "rpc" solution, possibly one of the major drivers for making decisions should be: 1. Minimal amount of code 2. Easy as possible.
Keeping that in mind, it is paying off to ponder, what the difference is between the following 2 scenarios:
"Real" RPC: The handlers shall be as you wrote with rpc-method-specific signature.
"Message passing": The handlers receive messages of either "end point-determined type" or simply of a unified message type.
Now, what has to be done to get a solution of type 1?
Incoming byte streams/network packets need to get parsed to some sort of message with regards to some chosen protocol. Then, using some meta-info (contract), according to { serviceContract, serviceMethod }, a specific set of data items needs to be confirmed in the packet and if present, the respective, registered handler function needs to be called. Somewhere within that infrastructure you typically have a (likely code generated) function which does something like that:
void CallHandlerForRpcXYCallFoo( const RpcMessage*message )
{
uint32_t arg0 = message->getAsUint32(0);
// ...
float argN = message->getAsFloat(N);
Foo( arg0, arg1, ... argN );
}
All that can, of course also be packed into classes and virtual methods with the classes being generated from the service contract meta data. Maybe, there is also a way by means of some excessive template voodoo to avoid generating code and having a more generic meta-implementation. But, all that is work, real work. Way too much work to do it just for fun. Instead of doing that, it would be easier to use one of the dozens technologies which do that already.
Worth noting so far is: Somewhere within that piece of art, there is likely a (code generated) function which looks like the one given above.
Now, what has to be done to get a solution of type 2?
Less than for case 1. Why? Because you simply stop your implementation at calling those handler methods, which all take the RpcMessage as their single argument. As such, you can get away without generating the "make-it-look-like-a-function-call" layer above those methods.
Not only is it less work, it is also more robust in the presence of some scenarios where the contract changes. If one more data item is being added to the "rpc solution", the signature of the "rpc function" MUST change. Code re-generated, application code adapted. And that, whether or not the application needs that new data item. On the other hand, in approach 2, there are no breaking changes in the code. Of course, depending on your choices and the kind of changes in the contract, it still would break.
So, the most elegant solution is: Don't do RPC, do message passing. Preferably in a REST-ful way.
Also, if you prefer a "unified" rpc message over a number of rpc-contract specific message types, you remove another reason for code bloat.
Just in case, what I say seems a bit too abstract, here some mock-up dummy code, sketching solution 2:
#include <cstdio>
#include <cstdint>
#include <map>
#include <vector>
#include <deque>
#include <functional>
// "rpc" infrastructure (could be an API for a dll or a lib or so:
// Just one way to do it. Somehow, your various data types need
// to be handled/represented.
class RpcVariant
{
public:
enum class VariantType
{
RVT_EMPTY,
RVT_UINT,
RVT_SINT,
RVT_FLOAT32,
RVT_BYTES
};
private:
VariantType m_type;
uint64_t m_uintValue;
int64_t m_intValue;
float m_floatValue;
std::vector<uint8_t> m_bytesValue;
explicit RpcVariant(VariantType type)
: m_type(type)
{
}
public:
static RpcVariant MakeEmpty()
{
RpcVariant result(VariantType::RVT_EMPTY);
return result;
}
static RpcVariant MakeUint(uint64_t value)
{
RpcVariant result(VariantType::RVT_UINT);
result.m_uintValue = value;
return result;
}
// ... More make-functions
uint64_t AsUint() const
{
// TODO: check if correct type...
return m_uintValue;
}
// ... More AsXXX() functions
// ... Some ToWire()/FromWire() functions...
};
typedef std::map<uint32_t, RpcVariant> RpcMessage_t;
typedef std::function<void(const RpcMessage_t *)> RpcHandler_t;
void RpcInit();
void RpcUninit();
// application writes handlers and registers them with the infrastructure.
// rpc_context_id can be anything opportune - chose uint32_t, here.
// could as well be a string or a pair of values (service,method) or whatever.
void RpcRegisterHandler(uint32_t rpc_context_id, RpcHandler_t handler);
// Then according to taste/style preferences some receive function which uses the registered information and dispatches to the handlers...
void RpcReceive();
void RpcBeginReceive();
void RpcEndReceive();
// maybe some sending, too...
void RpcSend(uint32_t rpc_context_id, const RpcMessage_t * message);
int main(int argc, const char * argv[])
{
RpcInit();
RpcRegisterHandler(42, [](const RpcMessage_t *message) { puts("message type 42 received."); });
RpcRegisterHandler(43, [](const RpcMessage_t *message) { puts("message type 43 received."); });
while (true)
{
RpcReceive();
}
RpcUninit();
return 0;
}
And if RpcMessage then is traded, while packed in a std::shared_ptr, you can even have multiple handlers or do some forwarding (to other threads) of the same message instance. This is one particularly annoying thing, which needs yet another "serializing" in the rpc approach. Here, you simply forward the message.
How did Boost implement Tuple before C++11 and Variadic Templates?
In other words:
Is it possible to implement a Variadic Templates class or function by not using built-in Variadic Templates feature in C++11?
Boost had a limit for the size of the tuple. As in most real-world scenarios you don't need more than 10 elements, you won't mind this limitation. As a library maintainer, I guess, the world became much simpler with variadic templates. No more macro hacks...
Here is an insightful discussion about the size limit of Boost tuple and its implementation:
boost tuple: increasing maximum number of elements
To answer your second question: No, it is not possible. At least not for an unlimited number of elements.
There are 2 common use cases I've seen, as a library developer, for variadic templates. You can build a work around for both.
Case 1: Function objects
std::function<> and lambdas are very nice, but even c++11 only gives you a fairly basic set of things you can do with them "out of the box". To implement really cool things and utilities on top of them, you need to support variadic templates because std::function can be used with any normal function signature.
Workaround:
A recursive call using std::bind is your friend. It IS less efficient than real variadic templates (and some tricks like perfect forwarding probably won't work), but it'll work okay for modest #s of template arguments until you port to c++11.
Case 2: Ordinary classes
Sometimes you need an ordinary class to manage generic std::function<>s (see above) or expose an API like "printf". Workarounds here come down to details and what each API of the class is doing.
APIs that merely manipulate variadic template data but don't need to store it can run as recursive calls. You need to write them so that they "consume" one argument at a time, and stop when they run out of arguments.
APIs (including constructors) that need to STORE variadic template data are harder- you're screwed if the types are really unlimited and could be anything. BUT, if they're always going to be primitives that map deterministically to binary, you can do it. Just write a "Serialize" call taking all the types you support, then use it to serialize the entire set into a binary buffer and build a vector of "type info" data you use to fetch & set them. Its actually a better solution than std::tuple in terms of memory and performance in the special cases its available.
Here's the "serialize tuple" trick:
// MemoryBuffer: A basic byte buffer w/ its size
class MemoryBuffer {
private:
void* buffer;
int size;
int currentSeekPt;
protected:
void ResizeBuffer() {
int newSz = size << 1; // Multiply by 2
void* newBuf = calloc( newSz, 1); // Make sure it is zeroed
memcpy( newBuf, buffer, target->size);
free( buffer);
size = newSz;
buffer = newBuf;
}
public:
MemoryBuffer(int initSize)
: buffer(0), size(initSize), currentSeekPt(0)
{
buffer = calloc( size, 1);
}
~MemoryBuffer() {
if(buffer) {
free( buffer);
}
}
// Add data to buffer
bool AddData(const void* data, int dataSz) {
if(!data || !dataSz) return false;
if(dataSz + currentSeekPt > size) { // resize to hold data
ResizeBuffer();
}
memcpy( buffer, data, dataSz);
return true;
}
void* GetDataPtr() const { return buffer; }
int GetSeekOffset() const { return currentSeekPt; }
int GetTotalSize() const { return size; }
};
struct BinaryTypeInfo {
std::type_info type; // RTTI type_info struct. You can use an "enum"
// instead- code will be faster, but harder to maintain.
ui64 bufferOffset; // Lets me "jump" into the buffer to
}
// Versions of "Serialize" for all 'tuple' data types I support
template<typename BASIC>
bool Serialize(BASIC data, MemoryBuffer* target,
std::vector<BinaryTypeInfo>& types)
{
// Handle boneheads
if(!target) return false;
// Setup our type info structure
BinaryTypeInfo info;
info.type = typeid(data);
info.bufferOffset = target->GetSeekOffset();
int binarySz = sizeof(data);
void* binaryVersion = malloc( binarySz);
if(!binaryVersion) return false;
memcpy( binaryVersion, &data, binarySz); // Data type must support this
if(!target->AddData( binaryVersion, binarySz)) {
free( binaryVersion);
return false;
}
free( binaryVersion);
// Populate type vector
types.push_back( info);
return true;
}
This is just a quick & dirty version; you'd hide the real thing better and probably combine the pieces into 1 reusable class. Note that you need a special version of Serialize() if you wish to handle std::string and more complex types.
I'm using luabind 0.9.1 from Ryan Pavlik's master distribution with Lua 5.1, cygwin on Win XP SP3 + latest patches x86, boost 1.48, gcc 4.3.4. Lua and boost are cygwin pre-compiled versions.
I've successfully built luabind in both static and shared versions.
Both versions pass all the tests EXCEPT for the test_object_identity.cpp test which fails in both versions.
I've tracked down the problem to the following issue:
If an entry in a table is created for NON built-in class (i.e., not int, string, etc), the value CANNOT be retrieved.
Here's a code piece that demonstrates this:
#include "test.hpp"
#include <luabind/luabind.hpp>
#include <luabind/detail/debug.hpp>
using namespace luabind;
struct test_param
{
int obj;
};
void test_main(lua_State* L)
{
using namespace luabind;
module(L)
[
class_<test_param>("test_param")
.def_readwrite("obj", &test_param::obj)
];
test_param temp_object;
object tabc = newtable(L);
tabc[1] = 10;
tabc[temp_object] = 30;
TEST_CHECK( tabc[1] == 10 ); // passes
TEST_CHECK( tabc[temp_object] == 30 ); // FAILS!!!
}
tabc[1] is indeed 10 while tabc[temp_object] is NOT 30! (actually, it seems to be nil)
However, if I use iterate to go over tabc entries, there're the two entries with the CORRECT key/value pairs.
Any ideas?
BTW, overloading the == operator like this:
#include <luabind/operator.hpp>
struct test_param
{
int obj;
bool operator==(test_param const& rhs) const
{
return obj == rhs.obj;
}
};
and
module(L)
[
class_<test_param>("test_param")
.def_readwrite("obj", &test_param::obj)
.def(const_self == const_self)
];
Doesn't change the result.
I also tried switching to settable() and gettable() from the [] operator. The result is the same. I can see with the debugger that default conversion of the key is invoked, so I guess the error arises from somewhere therein, but it's beyond me to figure out what exactly the problem is.
As the following simple test case show, there're definitely a bug in Luabind's conversion for complex types:
struct test_param : wrap_base
{
int obj;
bool operator==(test_param const& rhs) const
{ return obj == rhs.obj ; }
};
void test_main(lua_State* L)
{
using namespace luabind;
module(L)
[
class_<test_param>("test_param")
.def(constructor<>())
.def_readwrite("obj", &test_param::obj)
.def(const_self == const_self)
];
object tabc, zzk, zzv;
test_param tp, tp1;
tp.obj = 123456;
// create new table
tabc = newtable(L);
// set tabc[tp] = 5;
// o k v
settable( tabc, tp, 5);
// get access to entry through iterator() API
iterator zzi(tabc);
// get the key object
zzk = zzi.key();
// read back the value through gettable() API
// o k
zzv = gettable(tabc, zzk);
// check the entry has the same value
// irrespective of access method
TEST_CHECK ( *zzi == 5 &&
object_cast<int>(zzv) == 5 );
// convert key to its REAL type (test_param)
tp1 = object_cast<test_param>(zzk);
// check two keys are the same
TEST_CHECK( tp == tp1 );
// read the value back from table using REAL key type
zzv = gettable(tabc, tp1);
// check the value
TEST_CHECK( object_cast<int>(zzv) == 5 );
// the previous call FAILS with
// Terminated with exception: "unable to make cast"
// this is because gettable() doesn't return
// a TRUE value, but nil instead
}
Hopefully, someone smarter than me can figure this out,
Thx
I've traced the problem to the fact that Luabind creates a NEW DISTINCT object EVERY time you use a complex value as key (but it does NOT if you use a primitive one or an object).
Here's a small test case that demonstrates this:
struct test_param : wrap_base
{
int obj;
bool operator==(test_param const& rhs) const
{ return obj == rhs.obj ; }
};
void test_main(lua_State* L)
{
using namespace luabind;
module(L)
[
class_<test_param>("test_param")
.def(constructor<>())
.def_readwrite("obj", &test_param::obj)
.def(const_self == const_self)
];
object tabc, zzk, zzv;
test_param tp;
tp.obj = 123456;
tabc = newtable(L);
// o k v
settable( tabc, tp, 5);
iterator zzi(tabc), end;
std::cerr << "value = " << *zzi << "\n";
zzk = zzi.key();
// o k v
settable( tabc, tp, 6);
settable( tabc, zzk, 7);
for (zzi = iterator(tabc); zzi != end; ++zzi)
{
std::cerr << "value = " << *zzi << "\n";
}
}
Notice how tabc[tp] first has the value 5 and then is overwritten with 7 when accessed through the key object. However, when accessed AGAIN through tp, a new entry gets created. This is why gettable() fails subsequently.
Thx,
David
Disclaimer: I'm not an expert on luabind. It's entirely possible I've missed something about luabind's capabilities.
First of all, what is luabind doing when converting test_param to a Lua key? The default policy is copy. To quote the luabind documentation:
This will make a copy of the parameter. This is the default behavior when passing parameters by-value. Note that this can only be used when passing from C++ to Lua. This policy requires that the parameter type has an accessible copy constructor.
In pratice, what this means is that luabind will create a new object (called "full userdata") which is owned by the Lua garbage collector and will copy your struct into it. This is a very safe thing to do because it no longer matters what you do with the c++ object; the Lua object will stick around without really any overhead. This is a good way to do bindings for by-value sorts of objects.
Why does luabind create a new object each time you pass it to Lua? Well, what else could it do? It doesn't matter if the address of the passed object is the same, because the original c++ object could have changed or been destroyed since it was first passed to Lua. (Remember, it was copied to Lua by value, not by reference.) So, with only ==, luabind would have to maintain a list of every object of that type which had ever been passed to Lua (possibly weakly) and compare your object against each one to see if it matches. luabind doesn't do this (nor do I think should it).
Now, let's look at the Lua side. Even though luabind creates two different objects, they're still equal, right? Well, the first problem is that, besides certain built-in types, Lua can only hold objects by reference. Each of those "full userdata" that I mentioned before is actually a pointer. That means that they are not identical.
But they are equal, if we define an __eq meta operation. Unfortunately, Lua itself simply does not support this case. Userdata when used as table keys are always compared by identity, no matter what. This actually isn't special for userdata; it is also true for tables. (Note that to properly support this case, Lua would need to override the hashcode operation on the object in addition to __eq. Lua also does not support overriding the hashcode operation.) I can't speak for the authors of Lua why they did not allow this (and it has been suggested before), but there it is.
So, what are the options?
The simplest thing would be to convert test_param to an object once (explicitly), and then use that object to index the table both times. However, I suspect that while this fixes your toy example, it isn't very helpful in practice.
Another option is simply not to use such types as keys. Actually, I think this is a very good suggestion, since this kind of light-weight binding is quite useful, and the only other option is to discard it.
It looks like you can define a custom conversion on your type. In your example, it might be reasonable to convert your type to a Lua number which will behave well as a table index.
Use a different kind of binding. There will be some overhead, but if you want identity, you'll have to live with it. It sounds like luabind has some support for wrappers, which you may need to use to preserve identity:
When a pointer or reference to a registered class with a wrapper is passed to Lua, luabind will query for it's dynamic type. If the dynamic type inherits from wrap_base, object identity is preserved.