Here is what I am trying to do:
I am developing a cross-platform IDE (Linux and Windows) that supports plug-ins. I need to support extensibility using an adapter framework similar to the one that Eclipse provides. See here for more details, but basically I need the following:
Let Adaptee and Adapted be completely unrelated classes which already exist and which we are not allowed to change in any way. I want to create an AdapterManager class which has a method
template <class Adaptee, class Adapted> Adapted* adapt( Adaptee* object);
which will create an instance of Adapted given an instance of Adaptee. How exactly the instance is created depends on an adapter function which will have to be registered with AdapterManager. Each new plug-in should be able to contribute adapter functions for arbitrary types.
Here are my thoughts about a possible solution and why it does not work:
C++11's RTTI functions and the type_info class provide a hash_code() method which returns a unique integer for each type in the program. See here. Thus AdapterManager could simply contain a map that given the hash codes for the Adaptee and Adapter classes returns a function pointer to the adapter function. This makes the implementation of the adapt() function above trivial:
template <class Adaptee, class Adapted> Adapted* AdapterManager::adapt( Adaptee* object)
{
AdapterMapKey mk( typeid(Adapted).hash_code(), typeid(Adaptee).hash_code());
AdapterFunction af = adapterMap.get(mk);
if (!af) return nullptr;
return (Adapted*) af(object);
}
Any plug-in can easily extend the framework by simply inserting an additional function into the map. Also note that any plug-in can try to adapt any class to any other class and succeed if there exists a corresponding adapter function registered with AdapterManager regardless of who registered it.
A problem with this is the combination of templates and plug-ins (shared objects / DLLs). Since two plug-ins can instantiate a template class with the same parameters, this could potentially lead to two separate instances of the corresponding type_info structures and potentially different hash_code() results, which will break the mechanism above. Adapter functions registered from one plug-in might not always work in another plug-in.
In Linux, the dynamic linker seems to be able to deal with multiple declarations of types in different shared libraries under some conditions according to this (point 4.2). However the real problem is in Windows, where it seems that each DLL will get its own version of a template instantiation regardless of whether it is also defined in other loaded DLLs or the main executable. The dynamic linker seems quite inflexible compared to the one used in Linux.
I have considered using explicit template instantiations which seems to reduce the problem, but still does not solve it as two different plug-ins might still instantiate the same template in the same way.
Questions:
Does anyone know of a way to achieve this in Windows? If you were allowed to modify existing classes, would this help?
Do you know of another approach to achieve this functionality in C++, while still preserving all the desired properties: no change to existing classes, works with templates, supports plug-ins and is cross-platform?
Update 1:
This project uses the Qt framework for many things including the plug-in infrastructure. Qt really helps with cross platform development. If you know of a Qt specific solution to the problem, that's also welcome.
Update 2:
n.m.'s comment made me realize that I only know about the problem in theory and have not actually tested it. So I did some testing in both Windows and Linux using the following definition:
template <class T>
class TypeIdTest {
public:
virtual ~TypeIdTest() {};
static int data;
};
template <class T> int TypeIdTest<T>::data;
This class is instantiated in two different shared libraries/DLLs with T=int. Both libraries are explicitly loaded at run-time. Here is what I found:
In Linux everything just works:
The two instantiations used the same vtable.
The object returned by typeid was at the same address.
Even the static data member was the same.
So the fact that the template was instantiated in multiple dynamically loaded shared libraries made absolutely no difference. The linker seems to simply use the first loaded instantiation and ignore the rest.
In Windows the two instantiations are 'somewhat' distinct:
The typeid for the different instances returns type_info objects at different addresses. These objects however are equal when tested with ==. The corresponding hash codes are also equal. It seems like on Windows equality between types is established using the type's name - which makes sense. So far so good.
However the vtables for the two instances were different. I'm not sure how much of a problem this is. In my tests I was able to use dynamic_cast to downcast an instance of TypeIdTest to a derived type across shared library boundaries.
What's also a problem is that each instantiation used its own copy of the static field data. That can cause a lot of problems and basically disallows static fields in template classes.
Overall, it seems that even in Windows things are not as bad as I thought, but I'm still reluctant to use this approach given that template instantiations still use distinct vtables and static storage. Does anyone know how to avoid this problem? I did not find any solution.
I think Boost Extension deals with exactly this problem domain:
http://boost-extension.redshoelace.com/docs/boost/extension/index.html
(in preparation for this library's submission to Boost for review)
In particular you'd be interested in what the author wrote in this blog post: "Resource Management Across DLL Boundaries:
RTTI does not always function as expected across DLL boundaries. Check out the type_info classes to see how I deal with that.
I'm not sure whether his solution is actually robust, but he sure gave this thought, before. In fact, there are some samples using Boost Extensions that you can give a go, you might want to use it.
Related
I'm working on a game engine incorporating ECS. I have a pretty straightforward setup - my engine is in a DLL and my game module is the .EXE which attaches the DLL. I have created an ECS using dense memory block allocations and I'm focusing quite a lot on performance. For the very basis of the ECS i use a type info structure to enumerate different entity component types. I basically use function pointers to templated functions as a way to enumerate different types. It is defined as follows:
struct ComponentTypeInfo
{
typedef (*_InternalId)();
_InternalId TypeId;
size_t TypeSize;
protected:
template<typename>
static void TypeId() {}
}
template<typename TComponent>
struct GetComponentTypeInfo : public ComponentTypeInfo
{
GetComponentTypeInfo() : ComponentTypeInfo(TypeId<TComponent>, sizeof(TComponent))
}
The problem: In practice this works fine, however it breaks when types are compared across modules due to function instances being seperate. This causes the the function pointers to be different, and therefore the Ids to be different. I have tried a bunch of things but they all come to the same issue regarding module boundries.
What I have thought of so far:
Making sure only one module uses type info - Practically impossible since the engine has own component types predefined to use in systems.
Using a counter/increment-style id - Still encounters the same problem - any way of attaching ids to types encounters the problem that templated functionality is always instantiated in its own module, thus the ids would never match.
Using inheritence - since the ECS uses densely packed memory blocks, type erasure happens very early on and the logic working with memory blocks and stacks doesnt have the compile-time information of the types stored. Just the type ids.
I want to stay away from STL unless I absolutely need to use it. However I might end up using a wrapper around it if nothing else works. Libraries (boost, etc) are not a consideration.
Is there any senseble way to enumerate types across modules? Is RTTI only made to be available to STL?
I am searching for a simple, light-weight solution for interface-based runtime object composition in C++. I want to be able to specify interfaces (methods declarations), and objects (creatable through factory pattern) implementing these. At runtime I want mechanisms to instantiate these objects and interconnect these based on interface-connectors. The method calls at runtime should remain fairly cheap, i.e. only several more instructions per call, comparable to functor patterns.
The whole thing needs to be platform independent (at least MS Windows and Linux). And the solution needs to be licensed liberally, like open source LGPL or (even better) BSD or something, especially allowing use commercial products.
What I do not want are heavy things like networking, inter-process-communication, extra compiler steps (one-time code generation is ok though), or dependencies to some heavy libraries (like Qt).
The concrete scenario is: I have such a mechanism in a larger software, but the mechanism is not very well implemented. Interfaces are realized by base classes exported by Dlls. These Dlls also export factory functions to instantiate the implementing objects, based on hand-written class ids.
Before I now start to redesign and implement something better myself, I want to know if there is something out there which would be even better.
Edit: The solution also needs to support multi-threading environments. Additionally, as everything will happen inside the same process, I do not need data serialization mechanisms of any kind.
Edit: I know how such mechanisms work, and I know that several teaching books contain corresponding examples. I do not want to write it myself. The aim of my question is: Is there some sort of "industry standard" lib for this? It is a small problem (within a single process) and I am really only searching for a small solution.
Edit: I got the suggestion to add a pseudo-code example of what I really want to do. So here it is:
Somewhere I want to define interfaces. I do not care if it's C-Headers or some language and code generation.
class interface1 {
public:
virtual void do_stuff(void) = 0;
};
class interface2 {
public:
virtual void do_more_stuff(void) = 0;
};
Then I want to provide (multiple) implementations. These may even be placed in Dll-based plugins. Especially, these two classes my be implemented in two different Dlls not knowing each other at compile time.
class A : public interface1 {
public:
virtual void do_stuff(void) {
// I even need to call further interfaces here
// This call should, however, not require anything heavy, like data serialization or something.
this->con->do_more_stuff();
}
// Interface connectors of some kind. Here I use something like a template
some_connector<interface2> con;
};
class B : public interface2 {
public:
virtual void do_more_stuff() {
// finally doing some stuff
}
};
Finally, I may application main code I want to be able to compose my application logic at runtime (e.g. based on user input):
void main(void) {
// first I create my objects through a factory
some_object a = some_factory::create(some_guid<A>);
some_object b = some_factory::create(some_guid<B>);
// Then I want to connect the interface-connector 'con' of object 'a' to the instance of object 'b'
some_thing::connect(a, some_guid<A::con>, b);
// finally I want to call an interface-method.
interface1 *ia = a.some_cast<interface1>();
ia->do_stuff();
}
I am perfectly able to write such a solution myself (including all pitfalls). What I am searching for is a solution (e.g. a library) which is used and maintained by a wide user base.
While not widely used, I wrote a library several years ago that does this.
You can see it on GitHub zen-core library, and it's also available on Google Code
The GitHub version only contains the core libraries, which is really all the you need. The Google Code version contains a LOT of extra libraries, primarily for game development, but it does provide a lot of good examples on how to use it.
The implementation was inspired by Eclipse's plugin system, using a plugin.xml file that indicates a list of available plugins, and a config.xml file that indicates which plugins you would like to load. I'd also like to change it so that it doesn't depend on libxml2 and allow you to be able to specify plugins using other methods.
The documentation has been destroyed thanks to some hackers, but if you think this would be useful then I can write enough documentation to get you started.
A co-worker gave me two further tips:
The loki library (originating from the modern c++ book):
http://loki-lib.sourceforge.net/
A boost-like library:
http://kifri.fri.uniza.sk/~chochlik/mirror-lib/html/
I still have not looked at all the ideas I got.
I have two dll-exported classes A and B. A's declaration contains a function which uses a std::vector in its signature like:
class EXPORT A{
// ...
std::vector<B> myFunction(std::vector<B> const &input);
};
(EXPORT is the usual macro to put in place _declspec(dllexport)/_declspec(dllimport) accordingly.)
Reading about the issues related to using STL classes in a DLL interface, I gather in summary:
Using std::vector in a DLL interface would require all the clients of that DLL to be compiled with the same version of the same compiler because STL containers are not binary compatible. Even worse, depending on the use of that DLL by clients conjointly with other DLLs, the ''instable'' DLL API can break these client applications when system updates are installed (e.g. Microsoft KB packages) (really?).
Despite the above, if required, std::vector can be used in a DLL API by exporting std::vector<B> like:
template class EXPORT std::allocator<B>;
template class EXPORT std::vector<B>;
though, this is usually mentioned in the context when one wants to use std::vector as a member of A (http://support.microsoft.com/kb/168958).
The following Microsoft Support Article discusses how to access std::vector objects created in a DLL through a pointer or reference from within the executable (http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q172396). The above solution to use template class EXPORT ... seems to be applicable too. However, the drawback summarized under the first bullet point seems to remain.
To completely get rid of the problem, one would need to wrap std::vector and change the signature of myFunction, PIMPL etc..
My questions are:
Is the above summary correct, or do I miss here something essential?
Why does compilation of my class 'A' not generate warning C4251 (class 'std::vector<_Ty>' needs to have dll-interface to be used by clients of...)? I have no compiler warnings turned off and I don't get any warning on using std::vector in myFunction in exported class A (with VS2005).
What needs to be done to correctly export myFunction in A? Is it viable to just export std::vector<B> and B's allocator?
What are the implications of returning std::vector by-value? Assuming a client executable which has been compiled with a different compiler(-version). Does trouble persist when returning by-value where the vector is copied? I guess yes. Similarly for passing std::vector as a constant reference: could access to std::vector<B> (which might was constructed by an executable compiled with a different compiler(-version)) lead to trouble within myFunction? I guess yes again..
Is the last bullet point listed above really the only clean solution?
Many thanks in advance for your feedback.
Unfortunately, your list is very much spot-on. The root cause of this is that DLL-to-DLL or DLL-to-EXE is defined on the level of the operating system, while the the interface between functions is defined on the level of a compiler. In a way, your task is similar (although somewhat easier) to that of client-server interaction, when the client and the server lack binary compatibility.
The compiler maps what it can to the way the DLL importing and exporting is done in a particular operating system. Since language specifications give compilers a lot of liberty when it comes to binary layout of user-defined types and sometimes even built-in types (recall that the exact size of int is compiler-dependent, as long as minimal sizing requirements are met), importing and exporting from DLLs needs to be done manually to achieve binary-level compatibility.
When you use the same version of the same compiler, this last issue above does not create a problem. However, as soon as a different compiler enters the picture, all bets are off: you need to go back to the plainly-typed interfaces, and introduce wrappers to maintain nice-looking interfaces inside your code.
I've been having the same problem and discovered a neat solution to it.
Instead of passing std:vector, you can pass a QVector from the Qt library.
The problems you quote are then handled inside the Qt library and you do not need to deal with it at all.
Of course, the cost is having to use the library and accept its slightly worse performance.
In terms of the amount of coding and debugging time it saves you, this solution is well worth it.
Run into a bit of an issue, and I'm looking for the best solution concept/theory.
I have a system that needs to use objects. Each object that the system uses has a known interface, likely implemented as an abstract class. The interfaces are known at build time, and will not change. The exact implementation to be used will vary and I have no idea ahead of time what module will be providing it. The only guarantee is that they will provide the interface. The class name and module (DLL) come from a config file or may be changed programmatically.
Now, I have all that set up at the moment using a relatively simple system, set up something like so (rewritten pseudo-code, just to show the basics):
struct ClassID
{
Module * module;
int number;
};
class Module
{
HMODULE module;
function<void * (int)> * createfunc;
static Module * Load(String filename);
IObject * CreateClass(int number)
{
return createfunc(number);
}
};
class ModuleManager
{
bool LoadModule(String filename);
IObject * CreateClass(String classname)
{
ClassID class = AvailableClasses.find(classname);
return class.module->CreateObject(class.number);
}
vector<Module*> LoadedModules;
map<String, ClassID> AvailableClasses;
};
Modules have a few exported functions to give the number of classes they provide and the names/IDs of those, which are then stored. All classes derive from IObject, which has a virtual destructor, stores the source module and has some methods to get the class' ID, what interface it implements and such.
The only issue with this is each module has to be manually loaded somewhere (listed in the config file, at the moment). I would like to avoid doing this explicitly (outside of the ModuleManager, inside that I'm not really concerned as to how it's implemented).
I would like to have a similar system without having to handle loading the modules, just create an object and (once it's all set up) it magically appears.
I believe this is similar to what COM is intended to do, in some ways. I looked into the COM system briefly, but it appears to be overkill beyond belief. I only need the classes known within my system and don't need all the other features it handles, just implementations of interfaces coming from somewhere.
My other idea is to use the registry and keep a key with all the known/registered classes and their source modules and numbers, so I can just look them up and it will appear that Manager::CreateClass finds and makes the object magically. This seems like a viable solution, but I'm not sure if it's optimal or if I'm reinventing something.
So, after all that, my question is: How to handle this? Is there an existing technology, if not, how best to set it up myself? Are there any gotchas that I should be looking out for?
COM very likely is what you want. It is very broad but you don't need to use all the functionality. For example, you don't need to require participants to register GUIDs, you can define your own mechanism for creating instances of interfaces. There are a number of templates and other mechanisms to make it easy to create COM interfaces. What's more, since it is a standard, it is easy to document the requirements.
One very important thing to bear in mind is that importing/exporting C++ objects requires all participants to be using the same compiler. If you think that ever could be a problem to you then you should use COM. If you are happy to accept that restriction then you can carry on as you are.
I don't know if any technology exists to do this.
I do know that I worked with a system very similar to this. We used XML files to describe the various classes that different modules made available. Our equivalent of ModuleManager would parse the xml files to determine what to create for the user at run time based on the class name they provided and the configuration of the system. (Requesting an object that implemented interface 'I' could give back any of objects 'A', 'B' or 'C' depending on how the system was configured.)
The big gotcha we found was that the system was very brittle and at times hard to debug/understand. Just reading through the code, it was often near impossible to see what concrete class was being instantiated. We also found that maintaining the XML created more bugs and overhead than expected.
If I was to do this again, I would keep the design pattern of exposing classes from DLL's through interfaces, but I would not try to build a central registry of classes, nor would I derive everything from a base class such as IObject.
I would instead make each module responsible for exposing its own factory functions(s) to instantiate objects.
I was wondering about using or not templates, in other thread I found out that templates must be implement in the header file because of some reasons.
Thats ok, my question is if the source will be need if other programm use it?
from the logic of the other thread's answer, it seems that even other programm would need the full implementation so the compiler can say if a line can or not use the templated function.
if yes, I guess templates are not a good thing for the developer who wants others to use his library?
if no, then we are good and templates will be used.
or if at least there is anyway to save my hard, hours spent, code from others?
(I will use stl vectors and such, but I am asking for my own code... Templates seem to be nice, save you a lot of hardcoded lines or macro abusing, but if others can read your source than it makes almost no sense[lot of sense to open projects xD])
Thanks,
Joe
If you want users of your library to be able to use your templates, their source code needs to be available to those users.
However you can sometimes design your template classes so that most of the logic happens in non-template classes which don't have the full source code in the headers.
It depends on whether your templates are part of your libraries interface or whether they are just part of the implementation.
If they are part of the interface (i.e. perhaps a entry point returns an object of a specific template type), then yes, you need to expose your template definitions to the outside world.
But if the templates are solely part of your implementation, then once you build your library, there is no need to share the template definitions with consumers of your library.
You could write the templates as wrappers around non-template (often non-typesafe) code.
Advantages are...
The source for the non-template implementation code needn't be distributed.
It's a good way to reduce template bloat.
The obvious disadvantages are that you have an extra layer of abstraction and overhead, and non-typesafe implementation code obviously requires some care. I tend to have an abstract 'tool' class defined in the non-template code, specialised in the template wrapper. I call it a tool because methods don't primarily act on the tools state, but on objects passed in as void* parameters. The tool class encapsulates as most type-unsafety issues in a few methods. The template also provides the typesafe wrapper that users actually use, which interfaces to the unsafe code, providing the tool instance and doing typecasts etc.
For example, if I'm implementing a tree data structure, most tree algorithms will be type-unsafe and will see nodes and data items as void* pointers (or perhaps node* and data* pointers with node and data being declared but undefined structs). I'll have an abstract tree-tool-base with pure methods for node creation, disposal and other basic operations, and the wrapper template will basically specialise the tree tool, supplying method implementations that know the precise types of the nodes and data items, and holding an instance of the tool as a class member. To the user, the wrapper is just a typesafe container, same as any other.
BTW - when implementing the template wrapper, watch out for dependent name issues.