I have a shared library that I've created that references a lot of C++ template functions. These symbols get entered into the shared library's export table as weak references (e.g. they show as type W when I view the shared library's symbols using nm). This means that at runtime, these symbols can possibly be interposed by copies from a different shared library that got loaded first.
It's important for my application that my shared library use the copies of these functions that are contained within the library itself, not from any other library. Is there any way to ensure this? It sounds to me like it would be tantamount to statically linking all of the various template instantiations into the shared library.
This means that at runtime, these symbols can possibly be interposed
by copies from a different shared library that got loaded first.
Note that they can be interposed regardless of weak attribute (see this GCC post which says that dynamic linker treats weaks similar to strongs, unless LD_DYNAMIC_WEAK is set, which usually isn't).
It's important for my application that my shared library
use the copies of these functions that are contained
within the library itself, not from any other library.
Is there any way to ensure this?
There are several things you can do.
The usually recommended approach is to add fvisibility=hidden to your CFLAGS to prevent exporting any symbols from your library and then mark the (hopefully very few) exported functions with __attribute__((visibility("default"))). This would also allow for better optimization at compile-time and faster start-up as rtld will need to process fewer symbols.
A poor man's limited solution would be to employ -fvisibility-inlines-hidden which is a limited form of -fvisility=hidden. It will only hide inline functions (e.g. resulting from STL templates).
In case you do not want to mess with source code, link with -Wl,-Bsymbolic - this would force references to be resolved within the library whenever possible.
-- EDIT --
Actually you'll need -Bsymbolic even if you enable -fvisibility=hidden to prevent other libraries (or executable itself) from dynamically interposing intra-library references to exported functions.
Related
Let's say we have a program that is statically linked against MyLib1.0. Also there is a shared library which is linked against MyLib1.1.
Now what happens if the program loads this shared library? My assumption is that during runtime we will have multiple different definitions of the same symbols.
Does both of the following scenarios violates ODR?
Shared lib exports all the symbols including the symbols of MyLib1.1
Shared lib hides every symbols of MyLib1.1
Does it matter at all if shared lib is dynamically linked or dynamically loaded?
There are some similar questions but I couldn't find a very clear answer so far.
From the point of view of the ISO C++ standard which defines the "one-definition-rule" there is no concept of static or dynamic linking. There are only translation units.
If your program contains translation units which define the same entity in two different ways (e.g. because the definition changed with version 1.1 of MyLib), then that is an ODR violation and the program has undefined behavior, irregardless of how you are linking the program.
Everything beyond that is specific to how linking works on the given platform, although of course a lot of it is common behavior, e.g. shared libraries can override symbols and have symbols local to itself.
Compilers have flags and attributes to specify this behavior, e.g. GCC has the -fvisibility flag and the __attribute__((visbility(/*...*/))) attribute. Specifing that a symbol is local to the shared library doesn't imply though that it will be compatible when another version is used in the program. For example if the memory layout of a class changed between versions and an object is passed between two parts of the program using the different versions, they are likely to access memory in an incompatible manner, resulting in undefined behavior in the practical sense (rather then the standard's one).
Whether a certain combination of linking different library versions works depends on a lot of factors. So it should only be done if the library author states that this is supported. They need to explicitly take care that changes to the library are binary/ABI-compatible. This is already true when using headers of a different version, not only when actually linking it.
Will linking a static cpp lib and a dynamic cpp lib, both containing different versions of boost, violate ODR?
I am working on an iphone application. For final executable, I need to link a static library say libstatic1.a and a dynamic framwork say libdyanamic1.
libstatic1.a contains some version of boost, say boost 1.x and libdynamic1 contains another version of boost say boost 1.y. Now will final executable which links both of these, violate ODR rule?
Symbol visibility in libdynamic1:
I inspected symbols present in libdynamic1 using nm -g -C libdynamic1 and observed that symbols of boost threadpool and boost filesystem are present in the list.
If I am violating ODR, what are my options to handle the situation?
(So far I have tested the executable on multiple devices and have not experienced any issue.)
The standard only talks about "programs" where a "program" is a set of translation units "linked together", each consisting of a sequence of declarations [basic.link]. Arguing the ODR, which also concerns itself only with "programs", when it comes to questions involving dynamic libraries is not that straight forward. Since a "program" is required to contain a main function [basic.start.main]/1, a dynamic link library will generally not qualify as a "program" on its own.
Strictly speaking, I think a dynamic library would have to be viewed as just another set of translation units that are "linked" with the rest to form the final program. Thus, the program would really only be complete once all images have been loaded into memory and dynamic linking is finished (run-time dynamic linking would seem to further complicate the matter, but can be ignored here I guess). In this sense, the program described in your question (linking both the static and the dynamic library where each is using a different version of boost) will almost certainly be violating the ODR since you are going to have multiple translation units which are, e.g., using different definitions of the same entities [basic.def.odr]/12.
In practice, however, this issue is highly platform- and toolchain-dependent. At the ABI level, you typically find that the types of linkage a symbol can have are more differentiated than what you find at the language level within C++. On Windows, for example, you typically have to explicitly specify which symbols should be exported when building a dynamic library, all names are internal to the library by default. On ELF-based Linux, on the other hand, that is famously not the case. It would seem, however, that you can use the -fvisibility=hidden option to switch GCC to a more Windows-like default where your library will only export what you explicitly tell it to. Note that you must not have anything to do with boost in the interface your library exports as that will obviously lead to undefined behavior in your case…
I have a limited knowledge of dynamic libraries and I usually have problems related to libraries that I do not understand.
I recently learned of libraries from google search and especially from the following links:
Difference between shared objects (.so), static libraries (.a), and DLL's (.so)?.
http://www.ibm.com/developerworks/library/l-dynamic-libraries/. That article was very useful in understanding the dynamic libraries and their usage:
If I understood well (correct me if I am wrong), there are two possible usages of shared objects:
dynamic linking: the shared object is automatically loaded by the dynamic linker when the program starts.
dynamic loading: the share object is loaded and used under the program control at runtime through the dynamic loading API (dlopen, dlerror, dlsym and dlclose). That option is useful for plugins.
If I got everything right, in the case of dynamic linking, all the symbols are verified at compilation time. This allows the compiler/linker to know exactly which shared object is effectively used by the program and which one is not used.
Now, it happens that the dynamic linker is always invoked at runtime even if the shared object is not used. It can be verified by linking an empty program against libraries that are not in locations searchable at runtime, and the execution will fail. Linking a program against library that is not actually used in the program can happen when there are updates and the use of a library is no longer necessary. It also happen when one isolates a part of the program for debugging, and link against all the libraries of the main program.
My question is: is there an option to ask the compiler/linker to not include reference to shared objects that do not have symbols referred to in the program?
Is there any issue that prevent the compiler from doing that?
The following posts share some similarities with the present question, but none of them has an accepted answer, nor an answer that satisfies my curiosity:
https://stackoverflow.com/questions/22617744/how-to-disable-the-runtime-checking-of-shared-object-if-they-are-not-used
Delay-Load equivalent in unix based systems
If you happen to use g++/ld there are a few suggestions spelled out on How to remove unused C/C++ symbols with GCC and ld?
For example:
gcc -Os -fdata-sections -ffunction-sections test.cpp -o test.o -Wl,--gc-sections
-dead_strip
-dead_strip_dylibs
However I'm actually not sure it's possible for the compiler to do this in the general case. Consider a dependent shared library that has a weak reference to the library that you want to remove from your link line: How would the compiler know that it's safe to remove the library and/or symbols at that point?
I'm checking a makefile, and see that the libraries used are:
LIBS = -lcppunit -ldl
lcppunit is the unit testing library. What is ldl then?
This is the interface to the dynamic loader, which provides a client program with ability to do things such as explicitly load other libraries, lookup symbols within, etc.
Most programs do not need to do such things explicitly, since the linker does what is needed to enable ordinary usage of shared libraries while loading the program and libraries themselves. However programs that are clever or try to explore and manipulate the dynamic linking system and its data need explicit access. Some of the capabilities are distantly similar to reflection in Java, though with major limitations (such as applying only to dynamic symbols)
libdl is the dynamic linking library.
libdl is the dynamic link library used in plugin architectures with well defined interfaces. At least that's how I've seen it used.
I have a library consisting of some 300 c++ files.
The program that consumes the library does not want to dynamically link to it. (For various reasons, but the best one is that some of the supported platforms do not support dynamic linking)
Then I use g++ and ar to create a static library (.a), this file contains all symbols of all those files, including ones that the library doesn't want to export.
I suspect linking the consuming program with this library takes an unnecessary long time, as all the .o files inside the .a still need to have their references resolved, and the linker has more symbols to process.
When creating a dynamic library (.dylib / .so) you can actually use a linker, which can resolve all intra-lib symbols, and export only those that the library wants to export. The result however can only be "linked" into the consuming program at runtime.
I would like to somehow get the benefits of dynamic linking, but use a static library.
If my google searches are correct in thinking this is indeed not possible, I would love to understand why this is not possible, as it seems like something that many c and c++ programs could benefit from.
Static libraries are just archives (hence ".a"), a collection of .o files. Like a tar archive, just even more plain. Since ar is not a linker, there is no conglomeration (as "ld -r" would do) and thus no intralibrary symbol elimination.
That's why shared libraries were invented in the first place, and they are pretty common now, so people just ignore the drawbacks of static libraries. They simply go by "it compiles? ship it.".
I haven't tried or tested this, but it looks like ld's ability to perform incremental or partial linking might be what you're looking for. Check if the --relocatable option (you might also need to look at the -Ur option if dealing with C++) when applied to the object files that would go into the library will do what you want.
I think you should then be able to use the output of that operation as an object file (or have it in a static library itself) for your program's final link step.