I've got a project working on that's using FreeImage and openCV, currently we're using the jpeg support from both of these (I am working towards fixing that, but for now it's got to stay). Anyhow, FreeImage compiles libjpeg 7.0 into its static libraries, and openCV's highgui library links it in as a shared library (on my system, Ubuntu 9, I've got libjpeg 6.2 installed).
They link into a final library that's used to link into various programs, java wrappers, etc. All of that works fine, no symbol conflicts or anything during compile/link time. However, when I go to open an image using the openCV cvLoadImage function, it dies when reading the header, most likely due to differences between headers in 6.2 and 7.0.
If I unlink FreeImage (and comment out the code that requires it), the openCV calls start working again, so clearly the static libjpeg symbols from FreeImage are conflicting with symbols that would be loaded from the libjpeg shared library. What I can't figure out is why my compiler isn't throwing an error during linking because of the two sets of libjpeg symbols. Additionally, I've tried replacing my system's jpeglib.h header with the 7.0 version temporarily to see if openCV compiled with that would then sync up with the symbols that freeimage brings to the table, to no avail it seems.
Lastly I put a printf in jpeg_read_header in the libjpeg that freeimage compiles, and rebuilt it to see if openCV is using the freeimage libjpeg definition. It didn't print out so I have to assume not.
So I guess my questions are
1) Why doesn't linking a static libjpeg and a shared libjpeg generate linking errors due to duplicate symbols?
2) Does anyone know why these two things are conflicting with one another?
Edit: Compiling openCV in debug mode and then in regular mode again seems to have knocked something loose and made it work again, no idea what's going on.
Generally speaking the linker is fine about being passed multiple libraries that all resolve the same symbol(s). It just uses the first one it finds. The order of the libraries on your linker command line will determine which one "wins".
This, by the way, is NOT true of object files. Every linker I've ever used assumes that you want to use all of the objects that you specify, and will complain if more than one has the same symbol.
You need to modify the linking options to export only the symbols that you want, then all the symbols in conflict will be hidden internally in the static linking, then no conflicts. By default all the symbols are exported, that's the problem. See this link: http://www.gnu.org/software/gnulib/manual/html_node/Exported-Symbols-of-Shared-Libraries.html
it is like this
Static libraries are compiled in, dynamic libraries are loaded during runtime, but only those symbols which are missing (I think). you can compile shared libraries in, and then you probably get symbol collision.
so opencv uses symbols which are compiled in, since they already present, rather than those from dynamic libraries. you end up using static symbols, possibly with different signatures, from prospective of opencv.
Related
I'm building a executable in a big project, one of its dependency, a .so file, is linked against boost 1.6.2, and the executable itself must be static linked to a different version of boost, I don't know the version of that, maybe 1.6.0, when I link to the .so directly, it's able to compile and link, but when I run that binary, the behaviour of that binary becomes strange when calling to the functions of classes of the external .so file, such as it can run into infinite lock, and core dump inside boost. But if I don't link a different version of boost, it works. I guess the version of boost caused this problem. Is that true? How can I fix this?
The safest thing to do is to rename the namespace of your statically linked version of boost to ensure there are no symbol clashes. Unfortunately I don't think boost has any macros for changing its namespace so you'll have to do a manual find and replace in the source code.
I have a C++ file a.cpp with the library dependency in the path /home/name/lib and the name of the library abc.so.
I do the compilation as follows:
g++ a.cpp -L/home/name/lib -labc
This compiles the program with no errors.
However while running the program, I get the ERROR:
./a.out: error while loading shared libraries: libabc.so.0: cannot open shared object file: No such file or directory
However if before running the program, I add the library path as
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/name/lib;
and compile and run now, it works fine.
Why am I not able to link the library by giving it from the g++ command?
Because shared object libraries are linked at runtime - you either need to add the location to the library search path (as you've done), place it somewhere within an already existing path (like /usr/lib), or add a symbolic link to an existing library path that links to the location where it exists.
If you want to compile the library in so there is no runtime dependency, you'll need a static library (which would be abc.a) - note that this has numerous downsides, however (if the library is updated, you'll need to recompile your executable to incorporate that update).
Why am I not able to link the library by giving it from the g++ command?
You are able to link, and you did link the library succesfully. Otherwise you would not be able to build executable (in your case a.out). The problem you mixed 2 different things: linking with shared libraries and loading them at runtime. Loading shared libraries is a pretty complex concept and described pretty well here Program-Library-HOWTO read from 3.2.
You are linking dynamically, is the default behavior with GCC. LD_LIBRARY_PATH is used to specify directories where to look for libraries (is a way of enforce using an specific library), read: Program-Library-HOWTO for more info. There is also an ld option -rpath to specify libraries search path for the binary being compiled (this info is written in the binary and only used for that binary, the LD_LIBRARY_PATH affect other apps using the same library, probably expecting a new or old version).
Linking statically is possible (but a little tricky) and no dependency would be required (but sometimes is not recommended, because prevent the update of the dependent libraries, for example for security reason, in static linking your always are using the versions of the libraries you have when compiled the binary).
I can't be the only one to run into this.
I have a C++ application that needs to link with one third-party and another static library set in an SDK. The SDK has, for some hideously frustrating reason, recompiled a subset of that same third-party library into their own (renamed) lib, although the symbols themselves are named the same and they are not encapsulated within a namespace. My application itself depends upon the same third-party library.
I've considered a few options, but maybe I'm missing something and hopefully a fresh look will help me out. Perhaps I'm close and someone will know the next step for one of these . I'll enumerate what I've tried and the shortcomings of each solution so far:
Link with both.
I get about 2500 lines of symbol redefinition / size change warnings and errors. This is when I first found that they defined the same symbols. I'm trying to recompile OpenSSL with g++ and drop it into a namespace at the moment...see edit below...
Link with the SDK only.
I get undefined symbols that my own code depends upon - this is when I found that their recompile of the third party lib is a subset, or at least was configured with one module disabled.
Link with the third party lib only.
I have a couple of undefined symbols reported by the SDK - one of them is actually a #define in a header file within the third party lib, so all references in the third party lib resolve to the definition, but references outside there do not. I moved that into the c file, which resolves that, however I still have two unresolved functions I can't find anywhere. This is the closest I've gotten so far.
Strip conflicting symbols from one lib and link in both.
So far this hasn't worked. It could be a version issue between the lib statically linked in the SDK and the versions I've tried using of the third-party lib, but it looks like some functions were moved between symbols, so by removing a symbol, I inadvertently remove a function that I need elsewhere. There doesn't seem to be a perfect mapping between functions in symbols in the SDK vs functions in symbols in the third-party lib. Is it plausible to strip functions without having to manually adjust addresses?
I've been examining symbols in libs with:
nm -C --defined-only lib<name>.a
And extracting entire objects with:
ar -x lib<name>.a <objname>.o
Hopefully this will also help others who have had to link with third-party libs that conflict with one another. For the sake of specifics, the third-party lib is OpenSSL, and the SDK is Opsec - libcpopenssl.a is the offending lib in Opsec.
**EDIT- A late entry possible workaround may be to recompile OpenSSL with g++ and put the whole thing in a namespace, and then link both libs. I'm trying that now...more to come...
A Google search indicate that SSL_get_peer_dh and DH_dup are really additions from libcpopenssl.a, and they don't exist in my copy of OpenSSL either. So you'll really have to link that library in. Mixing both libraries together (Approach 4 above) at binary level is unlikely to work -- OpenSSL is very picky about its ABI (they usually have .so files versioned down to the minor number) so you'd have to be very lucky to have an .so that's ABI-compatible to their .a file.
My suggestion is a variation of Approach 4, but at source level: you'll have link in the Opsec libcpopenssl.a, since it's a modified version of OpenSSL which include extra symbols (and possibly other modifications), and grab the extra functions you need from the OpenSSL sources and recompile those objects with libcpopenssl.a, so they can use the functions from the Opsec version. If you're only using a few OpenSSL functions which are not exported by libcpopenssl.a, this is quite doable.
Granted, that's still a cumbersome approach, but it's a guaranteed way to get symbol compatibility, provided of course that the Opsec SDK didn't make semantic changes to OpenSSL that will break the additional OpenSSL functions that you're pulling into your project.
(I'm new at StackOverflow so I don't know if this suggestion qualifies as a proper answer, but I don't have reputation points to post comments, anyway. I'll remove this if it's inappropriate.)
If you're morbidly curious, 249 files were modified in the most recent version of OpenSSL to get it to compile. The most common issue by far was the abundance of C-style pointer casts, particularly with void*. Now I see "reinterpret_cast" in my dreams.
This didn't solve it alone though - it still needs to be placed into a namespace in its entirety, which means modifying all of the files again as well as my own internal references to it. I think I'm going to pass on this for now.
Thanks for the help everyone.
I've tried searching A LOT for this with no luck (possibly because I'm not using the right technical terms). My issue is mainly to do with linking static libs, compiling and deploying. Before I get into details, my executables compile fine on my system; the main issue is how to deploy these as a working solution to others.
I've written a basic c++ image processing exe that uses OpenCV static libraries (I link these in VC++ using the Project>Properties>Linker> add additional dependencies, as standard). I compile by pointing to the right include files by setting the VC++ Options... basically, it all compiles fine. I now want to be able to deploy this on another PC. I understand I'll need the release version of the exe + static libs... is there anything else?
Some of the libs rely on using libjpeg and libpng; I don't think these are included as standard. Also, I've set the linker path to the static libs to be relative (e.g. resources/libs) so it's not system dependent so it knows where to find the libs. The basic OpenCV data strucs are working fine (e.g. CvPoint), but when I try to load an image using CvLoadImage, the application crashes. If I use standard ifstream fopen instead, I can open the file with no problems (but can't seem to get it into the IplImage OpenCV image strut - does anyone know how to do these? Probably to do with IplImage->imageData.).
Any help very much appreciated. Thanks!
Static libraries do not have to (and should not) be distributed with the application. Static libraries are built into the exe file by the linker.
The reason why OpenCV crashes is that it cannot find libpng/libjpeg dlls. OpenCV doesn't link them as static dependencies but uses LoadLibrary/dlopen APIs at runtime instead. If these calls fail, there's probably no nice recovery and the application crashes. Your problems should be fixed if you include the libpng/libjpeg libraries.
Also beware - some .lib files aren't truly static libraries but are just a thin layer that allows the linker to find the appropriate functions in a DLL and generate the dynamic linking code so that the programmer doesn't have to do that by hand. You will usually see that from the .lib file size that is pretty small and that your application cries that it cannot find a DLL entry point at the startup of the exe..
I have a C++ executable and I'm dynamically linking against several libraries (Boost, Xerces-c and custom libs).
I understand why I would require the .lib/.a files if I choose to statically link against these libraries (relevant SO question here). However, why do I need to provide the corresponding .lib/.so library files when linking my executable if I'm dynamically linking against these external libraries?
The compiler isn't aware of dynamic linking, it just knows that a function exists via its prototype. The linker needs the lib files to resolve the symbol. The lib for a DLL contains additional information like what DLL the functions live in and how they are exported (by name, by ordinal, etc.) The lib files for DLL's contain much less information than lib files that contain the full object code - libcmmt.lib on my system is 19.2 MB, but msvcrt.lib is "only" 2.6 MB.
Note that this compile/link model is nearly 40 years old at this point, and predates dynamic linking on most platforms. If it were designed today, dynamic linking would be a first class citizen (for instance, in .NET, each assembly has rich metadata describing exactly what it exports, so you don't need separate headers and libs.)
Raymond Chen wrote a couple blog entries about this specific to Windows. Start with The classical model for linking and then follow-up with Why do we have import libraries anyway?.
To summarize, history has defined the compiler as the component that knows about detailed type information, whereas the linker only knows about symbol names. So the linker ends up creating the .DLL without type information, and therefore programs that want to link with it need some sort of metadata to tell it about how the functions are exported and what parameter types they take and return.
The reason .DLLs don't have all the information you need to link with them directly is is historic, and not a technical limitation.
For one thing, the linker inserts the versions of the libraries that exist at link time so that you have some chance of your program working if library versions are updated. Multiple versions of shared libraries can exist on a system.
The linker has the job of validating that all your undefined symbols are accounted for, either with static content or dynamic content.
By default, then, it insists on all your symbols being present.
However, that's just the default. See -z, and --allow-shlib-undefined, and friends.
Perhaps this dynamic linking is done via import libraries (function has __declspec(dllimport) before definition).
If this is the way than compilator expects that there's __imp_symbol function declared and this function is responsible for forwarding call to the right library dynamically loaded.
Those functions are generated during linkage of symbols with __declspec(dllimport) keyword
Here is a very SIMPLIFIED description that may help. Static linking puts all of the code needed to run your program into the executable so everything is found. Dynamic linking means some of the required code does not get put into the executable and will be found at runtime. Where do I find it? Is function x() there? How do I make a call to function x()? That is what the library tells the linker when you are dynamically linking.