I want to understand the following scenario as what exactly happens.
I have a shared library(A) which loads another shared library(B) by dynamic loading.
Library A exposes few methods(F1) as well.
The main program loads the library, A and then library A loads library B at later point.
Now When library B tries to uses function F1 from the library A, it fails to load its symbol, and compile stops execution silently without giving any warning/error or core. Even though I can see these symbols exported from the library A(using nm command).
I am not very sure what exactly happening. I observed this on Linux.64.
The same scenario on windows work.
My psychic debugging powers tell me that because you tagged this as C++ you're compiling C++ code. I also guess that you used slightly different options to compile libraries A and B, causing their name decoration and/or ABI to be incompatible, and thus the method(s) can't be found.
Have you considered just letting the linker apply all the shared objects to your application rather than using dlopen? Then you'd know straight away that there was a problem because the linker couldn't match the symbols.
Alternately perhaps Linux just doesn't like your circular library references: can you refactor your libraries to eliminate the circular dependency?
Related
I have a situation more or less as illustrated here; it exists both on Windows and Linux, but I’ll draw it using Linux notation which is slightly clearer:
libbar.so.1 and libbar.so.2 contain procedures with the same names and signatures but different behaviours. If I link the applications ‘naïvely’, then symbols from the library that is loaded last will ‘hide’ the ones from the one that is loaded first (at least, if I understand this reference correctly) and I have no way of using both libraries at the same time nor determining which library’s symbols actually get used on the day.
Certainly, the error messages I’m getting at runtime are consistent with this mental model.
Now, if I could recompile all the libraries, then as described in the reference I could use the –default-symver option (at least when compiling with and linking with gcc) to disambiguate fully the symbols from each library, and force libfoo.so to look for the function baz##libbar.so.1 and the application to look for the function baz##libbar.so.2 with no ambiguity.
What, though, if I can only recompile my application and libbar.so.2? Will libfoo.so ignore the function baz##libbar.so.2, or will it look no further than the baz part of it and, potentially, still use it instead of the one in libbar.so.1?
It’s a full cross-platform application so I have exactly the same issue to address on Windows. Here, I have an inkling that I may be able to use the SxS mechanism as the situation isn’t all that different from, say, having an application compiled in VS2019 that links to a library compiled in VS2013; so that the application would need to link to one version of the runtimes and the library would need to link to the other. But whilst I know how to handle this with system libraries, I’m less sure how to proceed with other libraries.
Suppose you have
a pre-built iOS executable app (for simulator or device).
a pre-built static archive library static library which among other things contains c++ static initializers.
Now it should be possible to merge the two built products to produce the a new iOS executable which is like the old one, except that it is now also linked with the additional static library, and on execution will run the static library's static initializers.
Which tool (if any) could help solve this merge problem?
Edit: An acceptable solution is also to dynamically load the library using dlopen. The whole purpose of this is for application testing, so the re-linked app will never see app store.
How a compiler work (in a simple explanation)
The most popular C++ compilers (like say, GCC), work by translating all the C++ (and Obj-C, C, etc...) code to ASM.
Then it calls the appropriate assembler for the target processor, and create the object binaries.
Then it calls the linker, that search on those binaries for the symbols that explain what links with what. A common optimisation that linkers can do, is also strip of the final binary anything from the statically linked libraries that was not used, other common optimisation is not attempt to link at all unused libraries.
Also finally, the linker removes the things that only it needed.
What this mean in your case
You have a library, the library has the linking symbols. You also has a executable, that one had its linking symbols stripped, in fact depending on how it was optimised the internal jumps might be only a couple of jmp instructions to arbitrary addresses on the code. No machine, can do what you want in a automatic manner, because you don't have the needed information on the executable.
How to do it anyway
You need to disassemble the executable, figure on your own where are the function calls, and then manually reassemble it with your library, changing those functions call to jump to addresses in your library instead.
This process is sometimes used by game moders to change the video drivers of old games (for example to update their OpenGL version, or to force Glide games to use some newer drivers, and so on).
So if you want to do that anyway (I warn you: it is absurdly crazy to do though...) ask those guys :) I don't remember right now anyone to point to you, but they exist.
Analogy
When you are in normal linking phase, the compiled object files are like a source code that the machine understands, full of function calls as needed.
After it is compiled, all function calls became goto.
So if you are a linker tasked in doing what you want to do, imagine that you would be reading a source code filled with goto to random places in the code (sometimes even to inside loops) and that you have to somehow figure what ones of those you want to change to jump to the new part you are trying to paste there.
I know next to nothing about the linking process, and it almost always gets in the way when I am trying to start a new project or add a new library. Whenever I search for fixes to these type of errors, I will find people with a similar problem but rarely any sort of fix.
Is there any generalized way of going about finding what the problem is, and fixing it?
I'm using visual studio 2010, and am statically linking my libraries into my program. My problems always seem to stem from conflicts with LIBCMT(D).lib, MSVCRT(D).lib, and a few other libraries doublely defining certain functions. If it matters at all, my intent is to avoid using "managed" C++.
If your error is related to LIBCMT(D).lib and the like, usually that depends from the fact that you are linking against a library that uses a different CRT version than yours. The only real fix is to either use the library compiled for the same version of the CRT you use (often there is the "debug" and "release" version also for this reason), either (if you are desperate) change the CRT version you use to match the one of the library.
What is happening behind the scenes is that both your program and your library need the CRT functions to work correctly, and each one already links against it. If they are linking against the same version of it nothing bad happens (the linker sees that it's the same and doesn't complain), otherwise there are multiple conflicting implementations of the same functions, so the linker doesn't know which are right for which object modules (and also, since they are probably not binary compatible, internal data structures of the two CRTs will be incompatible).
The specific link errors you mentioned (with LIBCMT(D).lib, MSVCRT(D).lib libraries) are related to conflicts in code generation options between modules/libraries in your program.
When you compile a module, the compiler automatically inserts in the resulting .obj some references to the runtime libraries (LIBCMT&MSVCRT). Now, there is one version of these libraries for each code generation mode (I'm referring to the option at Configuration properties -> C/C++ -> Code Generation -> Runtime Library). So if you have two modules compiled with a different mode, each of them will reference a different version of the library, the linker will try to include both, and of course there'll be duplicated symbols, since essentially all the symbols are the same in these libraries, only their implementations differ.
The solution comes in three parts. First, make sure all the modules in a project use the same mode. Second, if you have dependencies between projects, all of them have to use the same mode. Third, if you use third-party libraries, you have to either know which mode they use (and adopt it) or be able to recompile them with the desired mode.
The last one is the most difficult. Sometimes, libraries come pre-compiled, and not always the provider gives information about the mode used. Worse, if you're using more than one third-party library, they may have conflicting modes. In those cases, you have no better option than trial-and-error.
Also notice that each Visual Studio version has its own set of runtime libraries, so when using third-party libraries you have to use those compiled with the same version of Visual Studio you're using. If the provider doesn't offer it, your only choice is to recompile yourself.
I'm writing a library for Linux which has some functionality which relies on the a shared library libfoo.so.
I'm trying to case where libfoo.so doesn't exist into consideration. I have a well defined behavior for such a case, but I don't know how to properly implement it, linkage wise.
Currently, my library is shipped compiled against libfoo.so, when a client tries to compile it's code against it on an environment which doesn't include libfoo.so he gets linkage errors.
My question is, how can I build my library in such a way that it would compile even if libfoo.so doesn't exist, but behave differently. The only solution I was able to come up with myself is to branch from it a version which doesn't support foo, but there must be a better way...
Thanks in advance
A follow up due to the given answer:
It seems that not linking against libfoo but rather dynamically loading it with dlopen solves the problem, but it requires me to manually export all the symbols, and is limited in scope... Is there any "less painful" to achieve that?
There are two different problems that kind of fit the description, depending on whether the presence of the library is to be detected at compile time or at runtime.
At compile time you will have to use some tool to detect whether the library is present and modify the build scripts to pass that information down to the code (think defines).
At runtime, you can avoid linking against the library, and rather dynamically load it. The code should handle failures to locate/load the library and fall back to the alternative version.
The compilation is done on the fly :
only CImg functionalities really used
by your program are compiled and
appear in the compiled executable
program. This leads to very compact
code, without any unused stuffs.
Any one knows the principle?
CImg is a header-only library, and they use templates liberally, which is what they're referring to.
If they used a precompiled library of some kind (.dll/.lib/.a/.so) the library file would have to contain the entire CImg library, regardless of which bits of it you actually use.
In the case of a statically linked library (.lib or .a), the linker can then strip out unused symbols, but that may depend on optimization settings.
When the entire library is included in one or two headers, it is only actually compiled when you #include it, and so it is part of the same compilation process as the rest of your program, and the compiler can easily determine which parts of the library are used, and which ones aren't.
And because the CImg API uses templates, no code is generated for functions that are never called.
They are overselling it a bit though, because as the other answers point out, unused symbols will usually be stripped out anyway.
Sounds like fairly standard behaviour to me - a C++ linker will usually throw away any unused library references rather than including uncallable code. Similarly, an optimized build will not include uncallable code.
This sounds like MSVC's Eliminate Unreferenced Data (/OPT:REF) linker command, GCC should have something similar too this too