I am working in a very big C++ project to create a big shared object where we are using an external SDK which have several header files and several shared libraries which belong to each other. This means that the declaration of SDK classes are in the header files but their definitions are in the shared objects.
I understand that because of the declarations in header files I can compile this code.
But what I do not understand exactly is when do I have to specify the used shared objects for the linker explicitly?
Namely if I specify it (e.g. in cmake with target_link_libraries command) then the linker can check that a symbol will be in the shared library or not. But what happens if I do not specify it (i.e. there is not any -l[shared_object_name] flags in linkage)? My experience is (which surprised me) that is work properly (i.e. the whole building process finished). How can it possible?
In POSIX shared libraries, you can have undefined symbols in a shared library, and all will link just fine. As long as the executable is fully linked, there will be no linker errors.
That's done this way because dynamic libraries mimic the behaviour of static libraries, and static libraries can have undefined symbols (static libraries are not linked, to begin with).
If you come from a Windows background, then it will surprise you, because Windows DLLs cannot have undefined symbols.
If you are worried about this, you can check the linker options --no-undefined and --no-allow-shlib-undefined.
My experience is (which surprised me) that is work properly (i.e. the whole building process finished).
That seems unlikely.
In fact…
How can it possible?
It's not.
The only explanation is that you weren't using symbols defined inside those library files. They were either in header-only parts of the third-party code, or they weren't part of the third-party code at all.
Or you were building a shared library of your own. The ultimate executable would still need the third-party libraries linked in, though.
Related
I am working on a little project to understand the chain of compiler and linker better.
Suppose that i have the libraries libfoo.a and libbar.a. I want to create a library libmy.a, that acts like a wrapper or top level API to both libraries. The target is, that only libmy.a should be required to build an executable, that uses my defined wrapper functions. I created a cmake project and set up the library
cmake_minimum_required(VERSION 3.14)
project(Wrapper)
set(CMAKE_CXX_STANDARD 11)
add_library(my STATIC ${SOME_SRC_FILES})
#set up the lib/inc paths and libs to link
target_include_directories(my PUBLIC /path/to/Foo/inc/ /path/to/Bar/inc/)
target_link_directories(my PUBLIC /path/to/Foo/lib/ /path/to/Bar/lib)
target_link_libraries(my PUBLIC foo bar)
That works fine and there is no problem in compilation. However, if I try to reference the object from an external project, it tells me, that I have undefined references to the functions in libfoo.a and libbar.a. As far as I understand the problem, the linker only creates a declaration in the libmy.a, without including its definition from the external library. I checked this by opening libmy.a with the nm libmy.a command, where the used functions of the external libraries are declared, but undefined.
I came across one solution that used ar to combine multiple library files. However I would like to avoid such methods, because if it is not a single library, but a bunch of, say 10 libraries, it is not suitable to search each library for a definition and copy it into libmy.a. Just throwing all libraries together isn't a solution either, because the file will get too big.
Is it importand to note, that one of these library packages is CUDA?
I am sure there is a solution, but I was not able to find one. Any help would be appreciated
The target is, that only libmy.a should be required to build an executable
This is already an unconventional goal for static libraries.
Static libraries normally only contain the object code built from the source code for that library. Users of that library must also link to the libraries that your library requires, because the definitions have not been copied in to your library when it was built.
Tools like ar can be used to combine multiple static libraries together, since they are just archives of object code. The tool can not predict which object code the end-user will use though, so it will bundle entire libraries. Otherwise, the end user may be looking for a definition that you left out and then need to link in a 2nd copy of the dependency lib anyways.
If you want to provide a library that has everything the end-user needs, cut down to what your wrapper actually uses, you can build a shared library. Shared libraries are considered executable, so the compiler knows that any unreferenced object code is not going to be used, and it will not be included in the shared library.
You can force the entire static libraries to be included in shared libraries though.
On GCC you can use the linker argument: --whole-archive to ensure that all of the object code from the following libraries is included.
On MSVC, you can use the /WHOLEARCHIVE:<library file name> argument to do the same.
Let's say you compile a C++ shared library libBeta.so which makes use of pre-existing C++ shared libraries libAlpha1.so, libAlpha2.so, libAlpha3.so, etc. If I then write a C++ application which uses libBeta.so directly (and therefore indirectly uses the other libraries), should I link my application to libBeta.so only, or should I link my application to all libraries?
My intuition tells me that I should only link to libBeta.so, because linking to all libraries seems redundant as libBeta.so is already linked to the other libraries. However, undefined reference to errors are proving my intuition wrong.
Could someone explain me why my intuition might be wrong in particular cases?
p.s.:
OS: Linux
Compiler: g++
EDIT
As it turns out the tool I was using for compiling has different behaviour for compiling an executable and compiling a shared library. Linkage to sub-libraries were being omitted when compiling a shared library :(
Shared libraries are fully linked entities, and you don't need to explicitly link to their dependencies.
This is unlike static libraries which is only a collection of object files. When you use a static library you must link to its dependencies. But for share libraries, no you don't need that.
If you get undefined references, then it's not for the dependencies of the shared libraries you link to. It's either that you are missing linking with your own code, or you actually link with a static library.
You only need to link with your direct dependency, libBeta.so.
Actually, a few years ago on some Linux distributions you could get away with having indirect dependencies in your executable -- in this case, say, on libAlpha1.so -- and as long as the dependency gets loaded at runtime, directly or indirectly, the dependency would get satisfied.
This is no longer the case.
I wrote a static library (compiled with TDM-gcc 4.8.1 in Windows 7 for x64) that has dependencies on other static libraries. Boost libraries (locale and system) to be specific.
Since I'm building a static library I assumed that the libraries I'm dependend on would automatically included in my final .a, especially since I'm using them in my code.
But when I'm trying to build an executable that statically links to my aforementioned library there are still undefined references to some boost parts, that are definitely used in my library.
Is there a way to fix that?
Any help is gladly appreciated. Thank you
Edit:
I haven't been careful enough, because I now know what causes the problem. I'm using codeblocks and all the necessary arguments for building the archive are declared in the project prooperties. But codeblocks doesn't even call the linker when building my library. Instead it calls ar.exe and passes all object files of my project. That way, no external library are ever included. So, I have too look for away to tell codeblocks to build the library in the right way..
Your executable needs to link against all the relevant libraries, including the ones it directly depends on, plus the ones it indirectly depends on. When you link a static library you typically do not embed other static libraries within it.
I'm having difficult understanding my text, and I want to know each section extremely well. Please, be as descriptive as possible. Thank you, very much.
Programmers often find they have some code they would like to use in several programs, or code they would like to allow other programmers to make use of. They could copy-and-paste the logic into every program, or they could place the logic in common header and implementation files that they include in every project. Neither of those approaches is terribly elegant (or efficient).
Static libraries are one way to share logic so that other programs and programmers can make use of it. A static library is a binary file (often with a .LIB extension) suitable for input to the linker. A programmer consumes the static library by writing code to call one or more functions implemented in the static library and by arranging for the linker to read that static library file as an additional input. During the build, the linker will resolve all references including those to functions implemented within the static library. Often a static library is paired with a header file that describes the functions implemented in the library.
Dynamic Link Libraries are another way to share logic. Dynamic libraries are also binary files (often with a .DLL extension). Unlike static libraries, DLLs are not used as input to the linker. Instead, DLLs are loaded dynamically during program execution. DLLs may be explicitly loaded by calls to LoadLibraryEx and GetProcAddress or implicitly loaded when the consuming program is loaded.
A static library is a library that is included into your binary. It's said to be static because it's linked against your program when the linker runs. A DLL is a dynamically linking library, so it's shared code that is linked against your binary program when the program is loaded into memory, not when it's compiled and linked.
The names say it all - static libraries are static (you compile with them linked using a linker and they become part of your program) and DLLs (dynamically linked libraries) are linked to dynamically at runtime, by filename, then you call their methods. It's external.
In my quest to learn C++, I have come across dynamic and static libraries.
I generally get the gist of them: compiled code to include into other programs.
However, I would like to know a few things about them:
Is writing them any different than a normal C++ program, minus the main() function?
How does the compiled program get to be a library? It's obviously not an executable, so how do I turn, say 'test.cpp' into 'test.dll'?
Once I get it to its format, how do I include it in another program?
Is there a standard place to put them, so that whatever compilers/linkers need them can find them easily?
What is the difference (technically and practically) between a dynamic and static library?
How would I use third party libraries in my code (I'm staring at .dylib and .a files for the MySql C++ Connector)
Everything I have found relating to libraries seems to be targeting those who already know how to use them. I, however, don't. (But would like to!)
Thanks!
(I should also note I'm using Mac OS X, and although would prefer to remain IDE-neutral or command-line oriented, I use QtCreator/Netbeans)
Is writing them any different than a normal C++ program, minus the main() function?
No.
How does the compiled program get to be a library? It's obviously not an executable, so how do I turn, say 'test.cpp' into 'test.dll'?
Pass the -dynamiclib flag when you're compiling. (The name of the result is still by default a.out. On Mac OS X you should name your dynamic libraries as lib***.dylib, and on Linux, lib***.so (shared objects))
Once I get it to its format, how do I include it in another program?
First, make a header file so the the other program can #include to know what functions can be used in your dylib.
Second, link to your dylib. If your dylib is named as libblah.dylib, you pass the -lblah flag to gcc.
Is there a standard place to put them, so that whatever compilers/linkers need them can find them easily?
/usr/lib or /usr/local/lib.
What is the difference (technically and practically) between a dynamic and static library?
Basically, for a static lib, the whole library is embedded into the file it "links" to.
How would I use third party libraries in my code (I'm staring at .dylib and .a files for the MySql C++ Connector)
See the 3rd answer.
Is writing them any different than a normal C++ program, minus the main() function?
Except for the obvious difference that a library provides services for other programs to use, usually (*) there isn't a difference.
* in gcc classes/functions are exported by default - this isn't the case in VC++, there you have to explicitly export using __declspec(export).
How does the compiled program get to be a library? It's obviously not an executable, so how do I turn, say 'test.cpp' into 'test.dll'?
This depends on your compiler. In Visual Studio you specify this in your project configuration. In gcc to create a static library you compile your code normally and then package it in an archive using ar. To create a shared you compile first (with the -fpic flag to enable position independent code generation, a requirement for shared libraries), then use the -shared flag on the object files. More info can be found in the man pages.
Once I get it to its format, how do I include it in another program?
Again this is a little compiler-dependant. In VS, if it's a shared library, when including the class/function you wish to use it should be marked with a __declspec(import) (this is usually done with ifdefs) and you have to specify the .lib file of the shared library for linkage. For a static library you only have to specify the .lib file (no export/import needed since the code will end up in your executable).
In gcc you only need to specify the library which you link against using -llibrary_name.
In both cases you will need to provide your client some header files with the functions/classes that are intended for public use.
Is there a standard place to put them, so that whatever compilers/linkers need them can find them easily?
If it's your own library then it's up to you. Usually you can specify the linker additional folders to look in. We have a lib folder in our source tree where all .lib (or .a/.so) files end up and we add that folder to the additional folder to look in.
If you're shipping a library on UNIX the common place is usually /usr/lib (or /usr/local/lib), this is also where gcc searches in by default.
What is the difference (technically and practically) between a dynamic and static library?
When you link a program to static libraries the code of the libraries ends up in your executable. Practically this makes your executable larger and makes it harder to update/fix a static library for obvious reasons (requires a new version of your executable).
Shared libraries are separate from your executable and are referenced by your program and (usually) loaded at runtime when needed.
It's also possible to load shared libraries without linking to them. It requires more work since you have to manually load the shared library and any symbol you wish to use. On Windows this is done using LoadLibrary/GetProcAddress and on POSIX systems using dlsym/dlopen.
How would I use third party libraries in my code?
This is usually accomplished by including the necessary header files and linking with the appropriate library.
A simple example to link with a static library foo would look like this: gcc main.cpp -o main.o -L/folder/where/foo.a/is/at -lfoo.
Most open source projects have a readme that gives more detailed instructions, I'd suggest to take a look at it if there is one.
Is writing [libraries] any different than a normal C++ program, minus the main() function?
That depends on your definition of "different." From the language's point of view, you write a file or collection of files, don't put in a main() and you tell the compiler to generate a library instead of an executable.
However, designing libraries is much harder because you have no control over the code that calls you. Libraries need to be more robust against failure than normal code. You can't necessarily delete pointers somebody passes to your function. You can't tell what macros will mess with your code. You also can't accidentally pollute the global namespace (eg., don't put using namespace std at the beginning of your header files).
How does the compiled program get to be a library? It's obviously not an executable, so how do I turn, say 'test.cpp' into 'test.dll'?
That depends on the compiler. In Visual C++ this is a project config setting. In gcc (going from memory) it's something like gcc -c foo.c -shared.
Once I get it to its format, how do I include it in another program?
That depends on your compiler and linker. You make sure the header files are available via a project setting or environment variable, and you make sure the binaries are available via a different project setting or compiler variable.
Is there a standard place to put them, so that whatever compilers/linkers need them can find them easily?
That depends on the operating system. In UNIX you're going to put things in places like /usr/lib, /usr/local/lib. On Windows people used to put DLLs in places like C:\WINDOWS but that's no longer allowed. Instead you put it in your program directory.
What is the difference (technically and practically) between a dynamic and static library?
Static libraries are the easier, original model. At compile time the linker puts all the functions from the library into your executable. You can ship the executable without the library, because the library is baked in.
Dynamic libraries (also called shared libraries) involve the compiler putting enough information in the executable that at runtime the linker will be able to find the correct libraries and call the methods in there. The libraries are shared across the whole system among the programs that use them. Using dynamic linking (dlsym(), et. al.) adds a few details to the picture.
How would I use third party libraries in my code (I'm staring at .dylib and .a files for the MySql C++ Connector)
That's going to depend on your platform, and unfortunately I can't tell you much about .dylib files. .a files are static libraries, and you simply need to add them to your final call to gcc (gcc main.c foo.a -o main if you know where foo.a is, or gcc main.c -lfoo -o main if the system knows where foo.a, foo.la, or foo.so are). Generally you make sure the compiler can find the library and leave the linker to do the rest.
The difference between a static and dynamic library is that the linking is done at compile time for static libraries, embedding the executable code into your binary, while for dynamic libraries linking is done dynamically at program start. The advantages are that the libraris can be separately distributed, updated and the code (memory) can be shared among several programs.
To use a library you simply provide -l to g++ for a lib.a or lib.so
I'm writing this to be more pragmatic than technically correct. It's enough to give you the general idea of what you're after.
Is writing them any different than a normal C++ program, minus the main() function?
For a static library, there's really not much difference.
For a dynamic library, the most likely difference you'll need to be aware of is that you may need to export the symbols you want to be available outside your library. Basically everything you don't export is invisible to users of your library. Exactly how you export, and whether you even need to by default, depends on your compiler.
For a dynamic library you also need to have all symbols resolved, which means the library can't depend on a function or variable that comes from outside the library. If my library uses a function called foo(), I need to include foo() in my library by writing it myself or by linking to another library that supplies it. I can't use foo() and just assume the user of my library will supply it. The linker won't know how to call a foo() that doesn't yet exist.
How does the compiled program get to be a library? It's obviously not an executable, so how do I turn, say 'test.cpp' into 'test.dll'?
It's similar to how you turn test.cpp into test.exe - compile and link. You pass options to the compiler to tell it whether to create an executable, a static library, or a dynamic library.
Once I get it to its format, how do I include it in another program?
In your source code, you include header files necessary to use the library, much as you would include a header file for code that's not in a library. You'll also need to include the library on your link line, telling the linker where to find the library. For many systems, creating a dynamic library generates two files, the shared library and a link library. It's the link library that you include on the link line.
Is there a standard place to put them, so that whatever compilers/linkers need them can find them easily?
There is an environment variable that tells the linker where to look for libraries. The name of that variable is different from one system to another. You can also tell the linker about additional places to look.
What is the difference (technically and practically) between a dynamic and static library?
A static library gets copied into the thing it is linked to. An executable will include a copy of the static library and can be run on another machine without also copying the static library.
A dynamic library stays in a separate file. The executable loads that separate file when it runs. You have to distribute a copy of the dynamic library with your program or it won't run. You can also replace the dynamic library with a new version, and as long as the new library has the same interface it will still run with the old executable. It also may save space if several executables use the same dynamic library. In fact dynamic libraries are often called shared libraries.
How would I use third party libraries in my code
Same as you would use one you created yourself, as described above.