TL:DR; Static/Dynamic library and data sharing in UE4.
So all libraries are static or dynamic but no "mixed" solution?
Long description:
I'm browsing the UE4 source code and found that they are using a modular system. So they're building libraries (static or dynamic, based on the IS_MONOLITHIC define) and they are linked based on the build configuration files (like Core.Build.cs).
Let's read the "OpenGLDrv" module's Build file. It depends on the Core and Engine (and so on) libraries, so I guess they are linked to the OpenGLDrv library.
And checking the Launch module, in the Build file they set up dynamic module dependencies for the D3DXX and the OpenGLDrv libraries.
And here comes my problems:
If they are dynamically loaded modules, they have to be dynamic libraries (dll in Windows). If it's not true, are they linked statically?
If linked statically, how are they handling the global/static data sharing? Both the Launch and OpenGLDrv depends on the Core module which is linked statically to both libraries.
If the OpenGLDrv is a dynamic library and the Launch and Core are not, the data sharing problem still exists.
More specifically
I've tried to implement a similar system. I'm using visual studio and my current target platform is Windows only (but I'm writing the code in a platform-independent way).
Currently I have 5 following modules:
Core: It has the really core things like math classes, logger, misc, and the module manager.
Modules: The base classes for the "implementable" modules: InputDevice, GraphicsDevice and so on. The Core is linked to this.
OGLDevice: This supposed to be a dynamic library. It implements the OpenGL-specific GraphicsDevice. Both the Core and Modules are linked to it.
Engine: This is the "main library" which. Both the Core and Modules are linked to this library. It should load the OGLDevice dynamically.
Application: The main executable. The Engine is linked to this.
Here is a pretty cool image drawn by myself in Paint:
The best design-image ever
If all the 4 libraries are static, there aren't any problem with the dependencies. However, this way the OGLDevice is not a dynamically loaded library, because it is linked to the Engine. Maybe the device itself is not instantiated but still linked to the project.
If the Core, Modules and Engine libraries are static but the OGLDevice is dynamic, data sharing problems comes up. First, the Core is "loaded" by the Engine (because it's statically linked to it) but when the Engine loads the OGLDevice library, it will also "load" its own Core. So every static and global are duplicated.
If all modules would be dynamic, I guess it should work. Is there any way to mix the static and dynamic library thing?
Or how is the UE4 doing this?
Related
I just wanted to understand how shared libraries with statically linked libraries are expected to perform. I am writing a shared library (lshared.so) that is statically linked to another library (lstatic). Now, I have another executable helloworld which loads a dynamic version of library lstatic (lstatic.so) and lshared.so as well. Will the static version of library (lstatic) loaded by lshared.so and the dynamic version (lstatic.so) conflict? Will they share any global state? Tried to show it better below.
helloworld
--> lshared.so (statically linked to lstatic at compile time)
--> lstatic.so (hello world loads this at runtime)
To throw some context, I've to use a shared library for logging. It is possible that lshared.so is statically linked to a different version that the one available to helloworld at runtime.
Will the static version of library (lstatic) loaded by lshared.so and the dynamic version (lstatic.so) conflict.
Possibly.
Will they share any global state.
Possibly.
The answers depend on how exactly lshared.so is built (which symbols it exports), and how the main helloworld binary is linked.
The answers also tend to be somewhat complicated, and may depend on inlining and other optimizations, and possibly on compiler versions.
It is best to avoid doing that by using shared version of lstatic.so, where all the answers become simple.
How can a shared library that is loaded at runtime from my main program use classes from another shared library that is also loaded at runtime?
I've modularized my game engine architecture. I've created a Core module that contains classes such as a custom String library. My Engine module uses the String class contained in Core throughout itself but currently can't compile because it doesn't know about the String class.
Is it possible to compile the Core module in a way that can provide Engine with enough detail to compile, while also preserving the value of a modules? If I statically linked Core to Engine, that would defeat the purpose.
If I dynamically linked Core to Engine, wouldn't that also defeat the purpose? The only thing I can think of is that this is necessary for modules to compile, but that I would still dynamically load them at runtime.
I ask because the .lib from DLL version of the Core module is enough to get Engine to compile, but will cause the DLL to be loaded at runtime if I kept with this. Is there a way to get engine to compile from basically what the .lib provides without actually causing them to be automatically loaded?
The way I understand it native addons for node are just shared objects or dynamically linked libraries on the operating system. Normally though, dynamically linked libraries are linked at load time. But in node you can require() modules dynamically. So how does this work? Is it more like dynamic loading with dlopen and the function pointers?
nodejs supports addons that are written in other languages (such as C/C++). As such, those other languages can use all OS facilities to dynamically load other DLLs, either system DLLs or their own DLLs.
If you want to read more about the nodejs add-on process, you can see here: Nodejs Addons.
Built in library support (such as the fs module) has the same ability.
Indeed native addons for node are dynamically loaded. Taken from this post:
C/C++ addons are just dynamic libraries, and Node loads those libraries with a function in libuv called uv_dlopen. That provides a uniform interface for loading dynamic libraries on both Windows and Unix-like systems.
First, we have a system project, which provides some abstract architecture classes. Then, several module projects with different names inherit from one of system's classes to implement specialized features. Last, an application brings both together. It instantiates the base architecture from system and attaches modules to it.
This gives the following dependencies.
system has no dependencies and builds to a static library.
All modules depend on system and build to static libraries.
application depends on system and on all modules and builds the executable.
When I build this setup, do all module libraries contain copies of the system library? If so, can I prevent this behavior? In the end, they are, anyway, statically linked together in the executable along with the system library.
When you build a library, nothing gets linked. A library is just a bunch of object files stuck together in an archive. It's only when you build an application that the linker gets involved, pulling in the pieces that are needed by the various components of the application.
I am creating an application that supports modules in the form of dlls that are loaded dynamically at runtime. The code is laid out in the following way:
core - static library
This has a mechanism to load shared libraries and call a "create" function that returns a new Module object (uses a shared header).
module shared library (linked against core static library)
This module uses the shared Module header and also uses other classes from the core library (hence why it is linked against the core library). It is built to include all symbols from static libraries.
test application executable (linked against core static library)
I am getting funky, and seemingly sporadic behavior. They always end up in access violations but it seems that member variables that I very explicitly set (integers) will print out in later functions as garbage (i have verified that they are not being deleted earlier). This only ever seems to happen if they dynamic library is loaded (even if I never call the create function).
My main question is, is there are danger here that the symbols in the shared library will conflict with the symbols in the executable (since they came from the same static library) and cause problems even though they are from the exact same static library?
I can't speak for Linux and OS X behavior, but on Windows, the following is exactly what is happening. Since you say you also want to compile on Windows, this is relevant.
The problem you are experiencing is that you actually have multiple versions of everything in the core. Each module and the application itself has its own copy of the core, and their variables are not shared. This includes the C runtime, so things like new/delete across module boundaries are fraught with peril.
To verify that this is what is happening, create a simple test: set a global in the core to a value in your test application, then from from your dynamically loaded code try to access that global and see what you get. I will wager that you will see that your store to the global will not be reflected!
Solutions:
1) Make core a shared dynamic library. This may or may not be an option for you.
2) Operate extremely carefully with the knowledge of the above; All CRT and/or your own core state will not be shared, so you must make sure things will be allocated/destroyed on their own side of the module boundaries, among other things.
My own application is designed almost identically to yours; ie a static library with shared code needed by both the application and the modules, and then dynamically loaded plugins loaded by the application core.
What I do for all shared core state that must be accessed across modules is that the first thing each module does after loading is have its "core pointer" set to an instantiation of the core libraries in the application. This ensures that all modules are working with the same data.