I have a cpp file from which I am generating a shared library (using autofoo and the like). Within the cpp file, I have declared a couple of static fields that I use throughout the library functions.
My question is 2-part:
1) Where are these fields stored in memory? It's not as if the system instantiates the entire library and keeps it in memory... the library, after all, really is just a bunch of hooks.
2) Is there a better way to do this? The reason I did it to begin with is that I want to avoid requiring the user to pass the fields into every library function call as parameters.
Thanks!
The code used to load shared libraries:
Generally (each has minor technical differences):
Loads the shared lib into memory
Walks the symbol table and updates the address of function in the DLL
Initializes any global static members using their constructor.
Note: The shared lib loader need not do all this at the load point.
It may do some of these jobs lazily (implementation detail). But they will be done before use.
Any Global staic POD variables (things with no constructor). Will be stored in special memory segments depending on weather they are initialized or not (again an implementation detail). If they were initialized then they will be loaded with the from disk (or shared lib source) with that value already defined.
So the answer to your questions:
undefined.
The library is code segments
Initialized data segments
Uninitialized data segments
Some utility code that knows how to link it into a running application.
Better than what exactly
Good practice would suggest passing values to a function rather than relying on global state. But to be honest that is an over generalization and really down to the problem.
Logically speaking, it is as if the system instantiates the entire library. In practice, only the code is really "shared" in a shared library, anybody who links against it will get a copy of the data. (Well maybe not read-only data). So, as far your questions go:
1) Your process will get a copy of the variable somehow (dependent on how the shared library system on your OS works).
2) I don't see a problem with this approach.
Related
I have a program, code-named foo. foo depends on common.so and is linked to it in the normal way (sorry I don't know the technical way to say that). When foo is running it then dynamically loads bar.so using dlopen(). So far so good.
But, bar.so also depends on common.so. Will dlopen() re-load common.so (from what I've read it loads any required dependencies recursively), or will it detect that it is already loaded? If it does re-load it, could that cause problems in my program? Both foo and bar.so need to see the changes in common.so that either of them make to static variables there.
Maybe my design needs to be changed or requires use of -rdynamic (which I also don't quite understand properly yet)?
The POSIX spec for dlopen() says:
Only a single copy of an executable object file shall be brought into
the address space, even if dlopen() is invoked multiple times in
reference to the executable object file, and even if different
pathnames are used to reference the executable object file.
On Linux, this is implemented using a reference count; until dlclose is called an equal number of times, the shared object will remain resident.
[update]
I realize you are asking about shared objects implicitly loaded as dependencies, but the same principle applies. Otherwise, many things would break... In particular, global constructors in the shared object would run multiple times, which would wreak havoc.
TL;DR: Is it possible to load a class object from a library at runtime, close the library and then use the object as a "normal" object (after closing)?
I am trying to implement a plug-in system with some sort of "hot swap" functionality. Suppose my program expects a doSomething() function from its plugins. My idea would be to scan the fileystem for any libs in a specific folder, extract the functions and then close the lib (before using the functions!). This way, a monitor thread could just monitor changes on the filesystem and reset the function pointer in case something changed and thus plug-ins could be "hot swapped".
I believe that the function pointer would become invalid as soon as I close the library (Is that so?). Therefore my idea is to let the library return a copy of an object which does the desired functionality. In this case, I would call the lib to create the object before closing it and save the copy of the object in my program. However, since the object can use other objects/functions of the library, I am not sure if this would work, since these objects/functions would not be available, would they?
You cannot copy the object and close the library, since only data, but not the code of those objects is copied. Instead of it OS loads code of the library to the memory and all function pointers points to this region of memory. What will be if OS unloads the library?
You can implement something like this. You can have a Proxy object that contains a pointer to current loaded implementation. If a new library is detected, you can load new library, create instance of a new implementation, delete old instance of implementation, close old library. In this way you implement a "hot swap" mechanism and avoid problem with shared libraries code.
If you chose way described in item 2, beware of concurrency problems (what if another thread is scheduled when old implementation is deleted, but before the pointer is changed?).
An object is data, not code. A copy of an object is a copy of the data, but it still refers to the original code. As soon as you unload a dynamic library, its code is gone from memory, and any objects still referencing that code (i.e. of a type provided by the library) will be in trouble as soon as they are asked to execute a member function (such as the destructor).
So no, it's not possible to unload a library and keep using its code.
I'm still learning C++. I have one problem. Lets say that your project has global object which always exists e.g ApiManager and all other modules have access to it (by #include). For now I'm doing it by:
Header:
class ApiManager : public QObject
{
Q_OBJECT
public:
explicit ApiManager(QObject *parent = 0);
signals:
public slots:
};
extern ApiManager apiMng;
Source:
ApiManager apiMng;
The problem is that other objects need to have access when initialized too and I noticed that C++ global objects are created alphabetically. I'm wondering how do you deal with it? Exists some trick for this? For example in Free Pascal world each class module has initialization and finalization sections:
Type
TApiManager = class
end;
var ApiMng: TApiManager;
initialization
ApiMng := TApiManager.Create;
finalization
ApiMng.Free;
... and initialization order of project modules can be sorted in project source in uses clause (like #include in C++). I know that there is a lot of ways to do this (for example initialize everything in main.cpp with custom order) but want to know what is a "good habit" in C++ world
Edit: Solved by Q_GLOBAL_STATIC (introduced in Qt 5.1 but work for Qt 4.8 too) but still have two issues:
Still don't know how to manage constructor orders (and where to initialize it). Because global objects created by Q_GLOBAL_STATIC are not created at application startup. They are created on first usage. So I need to "touch" these object somewhere (in main.cpp?) with my custom order.
Documentation is saying that Q_GLOBAL_STATIC must be called in body .cpp file, not in header. But then other classes do not see this object. So I created static function which expose reference to this object:
.cpp:
Q_GLOBAL_STATIC(ApiManager, apiMng)
ApiManager *ApiManager::instance()
{
return apiMng();
}
But from this topic: http://qt-project.org/forums/viewthread/13977 Q_GLOBAL_STATIC should expose instance automatically, but it doesn't
They are not initialized in alphabetical order, and the initialization order among the translation units are undefined as nothing is guaranteed by the standard about it.
Why global variables are evil
Global variables should be avoided for several reasons, but the primary reason is because they increase your program’s complexity immensely. For example, say you were examining a program and you wanted to know what a variable named g_nValue was used for. Because g_nValue is a global, and globals can be used anywhere in the entire program, you’d have to examine every single line of every single file! In a computer program with hundreds of files and millions of lines of code, you can imagine how long this would take!
Second, global variables are dangerous because their values can be changed by any function that is called, and there is no easy way for the programmer to know that this will happen.
Why Global Variables Should Be Avoided When Unnecessary
Non-locality -- Source code is easiest to understand when the scope of its individual elements are limited. Global variables can be read or modified by any part of the program, making it difficult to remember or reason about every possible use.
No Access Control or Constraint Checking -- A global variable can be get or set by any part of the program, and any rules regarding its use can be easily broken or forgotten. (In other words, get/set accessors are generally preferable over direct data access, and this is even more so for global data.) By extension, the lack of access control greatly hinders achieving security in situations where you may wish to run untrusted code (such as working with 3rd party plugins).
Implicit coupling -- A program with many global variables often has tight couplings between some of those variables, and couplings between variables and functions. Grouping coupled items into cohesive units usually leads to better programs.
Concurrency issues -- if globals can be accessed by multiple threads of execution, synchronization is necessary (and too-often neglected). When dynamically linking modules with globals, the composed system might not be thread-safe even if the two independent modules tested in dozens of different contexts were safe.
Namespace pollution -- Global names are available everywhere. You may unknowingly end up using a global when you think you are using a local (by misspelling or forgetting to declare the local) or vice versa. Also, if you ever have to link together modules that have the same global variable names, if you are lucky, you will get linking errors. If you are unlucky, the linker will simply treat all uses of the same name as the same object.
Memory allocation issues -- Some environments have memory allocation schemes that make allocation of globals tricky. This is especially true in languages where "constructors" have side-effects other than allocation (because, in that case, you can express unsafe situations where two globals mutually depend on one another). Also, when dynamically linking modules, it can be unclear whether different libraries have their own instances of globals or whether the globals are shared.
Testing and Confinement - source that utilizes globals is somewhat more difficult to test because one cannot readily set up a 'clean' environment between runs. More generally, source that utilizes global services of any sort (e.g. reading and writing files or databases) that aren't explicitly provided to that source is difficult to test for the same reason. For communicating systems, the ability to test system invariants may require running more than one 'copy' of a system simultaneously, which is greatly hindered by any use of shared services - including global memory - that are not provided for sharing as part of the test.
In general, please avoid global variables as a rule of thumb. If you do need to have them, please use Q_GLOBAL_STATIC.
Creates a global and static object of type QGlobalStatic, of name VariableName and that behaves as a pointer to Type. The object created by Q_GLOBAL_STATIC initializes itself on the first use, which means that it will not increase the application or the library's load time. Additionally, the object is initialized in a thread-safe manner on all platforms.
You can also use Q_GLOBAL_STATIC_WITH_ARGS. Here you can find some inline highlight from the documentation:
Creates a global and static object of type QGlobalStatic, of name VariableName, initialized by the arguments Arguments and that behaves as a pointer to Type. The object created by Q_GLOBAL_STATIC_WITH_ARGS initializes itself on the first use, which means that it will not increase the application or the library's load time. Additionally, the object is initialized in a thread-safe manner on all platforms.
Some people also tend to create a function for wrapping them, but they do not reduce the complexity significantly, and they eventually either forget to make those functions thread-safe, or they put more complexity in. Forget about doing that as well when you can.
The initialization order of global objects is only defined within a translation unit (there it is top to bottom). There is no guarantee between translation units. The typical work-around is to wrap the object into a function and return a reference to a local object:
ApiManager& apiMng() {
static ApiManager rc;
return rc;
}
The local object is initialized the first time the function is called (and, when using C++11 also in a thread-safe fashion). This way, the order of construction of globally accessed objects can be ordered in a useful way.
That said, don't use global objects. They are causing more harm than good.
Good habit in C++ world would be to avoid global objects at all costs - the more localized is the object the better it is.
If you absolutely have to have global object, I think the best would be to initialize objects in custom order in main - to be explicit about initialization order. Fact that you are using qt is one more argument towards initializing in main - you probably would want to initialize QApplication (which requires argc and argv as input arguments) prior to any other QObject.
I have read about GCC's Options for Code Generation Conventions, but could not understand what "Generate position-independent code (PIC)" does. Please give an example to explain me what does it mean.
Position Independent Code means that the generated machine code is not dependent on being located at a specific address in order to work.
E.g. jumps would be generated as relative rather than absolute.
Pseudo-assembly:
PIC: This would work whether the code was at address 100 or 1000
100: COMPARE REG1, REG2
101: JUMP_IF_EQUAL CURRENT+10
...
111: NOP
Non-PIC: This will only work if the code is at address 100
100: COMPARE REG1, REG2
101: JUMP_IF_EQUAL 111
...
111: NOP
EDIT: In response to comment.
If your code is compiled with -fPIC, it's suitable for inclusion in a library - the library must be able to be relocated from its preferred location in memory to another address, there could be another already loaded library at the address your library prefers.
I'll try to explain what has already been said in a simpler way.
Whenever a shared lib is loaded, the loader (the code on the OS which load any program you run) changes some addresses in the code depending on where the object was loaded to.
In the above example, the "111" in the non-PIC code is written by the loader the first time it was loaded.
For not shared objects, you may want it to be like that because the compiler can make some optimizations on that code.
For shared object, if another process will want to "link" to that code it must read it to the same virtual addresses or the "111" will make no sense. But that virtual-space may already be in use in the second process.
Code that is built into shared libraries should normally be position-independent code, so that the shared library can readily be loaded at (more or less) any address in memory. The -fPIC option ensures that GCC produces such code.
The link to a function in a dynamic library is resolved when the library is loaded or at run time. Therefore, both the executable file and dynamic library are loaded into memory when the program is run.
The memory address at which a dynamic library is loaded cannot be determined in
advance, because a fixed address might clash with another dynamic library requiring the same address.
There are two commonly used methods for dealing with this problem:
1.Relocation. All pointers and addresses in the code are modified, if necessary, to fit the actual load address. Relocation is done by the linker and the loader.
2.Position-independent code. All addresses in the code are relative to the current position. Shared objects in Unix-like systems use position-independent code by default. This is less efficient than relocation if program run for a long time, especially in 32-bit mode.
The name "position-independent code" actually implies the following:
The code section contains no absolute addresses that need relocation, but only self relative
addresses. Therefore, the code section can be loaded at an arbitrary memory address and shared between multiple processes.
The data section is not shared between multiple processes because it often contains
writeable data. Therefore, the data section may contain pointers or addresses that
need relocation.
All public functions and public data can be overridden in Linux. If a function
in the main executable has the same name as a function in a shared object, then the
the version in main will take precedence, not only when called from main, but also when
called from the shared object. Likewise, when a global variable in the main has the same
name as a global variable in the shared object, then the instance in main will be
used, even when accessed from the shared object. This so-called symbol interposition is intended to mimic the behavior of static libraries.
A shared object has a table of pointers to its functions, called procedure linkage table (PLT), and a table
of pointers to its variables called global offset table (GOT) in order to implement this "override" feature.
All accesses to functions and public variables go through these tables.
p.s. Where dynamic linking cannot be avoided, there are various ways to avoid the time-consuming features of the position-independent code.
You can read more from this article: http://www.agner.org/optimize/optimizing_cpp.pdf
Adding further...
Every process has same virtual address space (If randomization of virtual address is stopped by using a flag in linux OS)
(For more details Disable and re-enable address space layout randomization only for myself)
So if its one exe with no shared linking (Hypothetical scenario), then we can always give same virtual address to same asm instruction without any harm.
But when we want to link shared object to the exe, then we are not sure of the start address assigned to shared object as it will depend upon the order the shared objects were linked.That being said, asm instruction inside .so will always have different virtual address depending upon the process its linking to.
So one process can give start address to .so as 0x45678910 in its own virtual space and other process at the same time can give start address of 0x12131415 and if they do not use relative addressing, .so will not work at all.
So they always have to use the relative addressing mode and hence fpic option.
A minor addition to the answers already posted: object files not compiled to be position independent are relocatable; they contain relocation table entries.
These entries allow the loader (that bit of code that loads a program into memory) to rewrite the absolute addresses to adjust for the actual load address in the virtual address space.
An operating system will try to share a single copy of a "shared object library" loaded into memory with all the programs that are linked to that same shared object library.
Since the code address space (unlike sections of the data space) need not be contiguous, and because most programs that link to a specific library have a fairly fixed library dependency tree, this succeeds most of the time. In those rare cases where there is a discrepancy, yes, it may be necessary to have two or more copies of a shared object library in memory.
Obviously, any attempt to randomize the load address of a library between programs and/or program instances (so as to reduce the possibility of creating an exploitable pattern) will make such cases common, not rare, so where a system has enabled this capability, one should make every attempt to compile all shared object libraries to be position independent.
Since calls into these libraries from the body of the main program will also be made relocatable, this makes it much less likely that a shared library will have to be copied.
I was wondering how to share some memory between different program modules - lets say, I have a main application (exe), and then some module (dll). They both link to the same static library. This static library will have some manager, that provides various services. What I would like to achieve, is to have this manager shared between all application modules, and to do this transparently during the library initialization.
Between processes I could use shared memory, but I want this to be shared in the current process only.
Could you think of some cross-platform way to do this? Possibly using boost libraries, if they provide some facilities to do this.
Only solution I can think of right now, is to use shared library of the respective OS, that all other modules will link to at runtime, and have the manager saved there.
EDIT:
To clarify what I actually need:
I need to find out, if the shared manager was already created (the answers below already provided some ways to do that)
Get the pointer to the manager, if it exists, or Set the pointer somewhere to the newly created manager object.
I think you're going to need assistance from a shared library to do this in any portable fashion. It doesn't necessarily need to know anything about the objects being shared between modules, it just needs to provide some globally-accessible mapping from a key (probably a string) to a pointer.
However, if you're willing to call OS APIs, this is feasible, and I think you may only need two implementations of the OS-specific part (one for Windows DLLs and GetProcAddress, one for OSes which use dlopen).
As each module loads, it walks the list of previously loaded modules looking for any that export a specially-named function. If it finds one (any, doesn't matter which, because the invariant is that all fully-loaded modules are aware of the common object), it gets the address of the common object from the previously loaded module, then increments the reference count. If it's unable to find any, it allocates new data and initializes the reference count. During module unload, it decrements the reference count and frees the common object if the reference count reached zero.
Of course it's necessary to use the OS allocator for the common object, because although unlikely, it's possible that it is deallocated from a different library from the one which first loaded it. This also implies that the common object cannot contain any virtual functions or any other sort of pointer to segments of the different modules. All its resources must by dynamically allocated using the OS process-wide allocator. This is probably less of a burden on systems where libc++ is a shared library, but you said you're statically linking the CRT.
Functions needed in Win32 would include EnumProcessModules, GetProcAddress, HeapAlloc, and HeapFree, GetProcessHeap and GetCurrentProcess.
Everything considered, I think I would stick to putting the common object in its own shared library, which leverages the loader's data structures to find it. Otherwise you're re-inventing the loader. This will work even when the CRT is statically linked into several modules, but I think you're setting yourself up for ODR violations. Be really particular about keeping the common data POD.
For use from the current process only, you don't need to devise any special function or structure.
You could do it even without any function but it is more safe and cross platform friendly to define set of functions providing access to the shared data. And these functions could be implemented by the common static library.
I think, only concern of this setup is that: "Who will own the data?". There must exist one and only one owner of the shared data.
With these basic idea, we could sketch the API like this:
IsSharedDataExist // check whether of not shared data exist
CreateSharedData // create (possibly dynamically) shared data
DestroySharedData // destroy shared data
... various data access API ...
Or C++ class with the Singleton pattern will be appropriate.
UPDATE
I was confused. Real problem can be defined as "How to implement a Singleton class in a static library that will be linked with multiple dynamic loading library (will be used in the same process) in platform independent way".
I think, basic idea is not much different but make sure the singleton is the really single is the additional problem of this setup.
For this purpose, you could employ Boost.Interprocess.
#include <boost/config.hpp>
#include <boost/interprocess/sync/named_mutex.hpp>
...
boost::interprocess::named_mutex* singleton_check = 0;
// in the Create function of the singleton
try {
singleton_check = new boost::interprocess::named_mutex(boost::interprocess::create_only, "name_of_the_mutex" );
// if no exception throw, this is the first time execution
}
catch (...)
{
}
Freeing the named_mutex is as simple as delete singleton_check.
UPDATE#2
Another suggestion.
I think, we should not place shared data in the common static library. If we can not ensure globally unique data, it is not only tricky platform dependent implementation problems but also waste of memory and global resources.
If you prefer static library implementation you should make two static libraries. One for the server/creator of the shared data, one for users of that shared data. Server library define and provide access to the Singleton. Client library provide various data access method.
This is effectively same as the Singleton implementation without static libraries.
You can use boost::interprocess http://www.boost.org/doc/libs/1_45_0/doc/html/interprocess.html
and on Windows you can create a shared segment in your DLL that will be shared by all processes using #pragma's: http://www.codeproject.com/KB/DLL/data_seg_share.aspx
As per MSDN I see there are only two ways to share data between modules
Using data_seg pragma
Use shared memory.
As someone pointed out Shared Segment works only for two instances of the same dll so we are left with only one choice to use Memory-Mapped Files technique.