I can build my application against the shared library but I'm getting the unresolved symbol errors when linking it against the static version of the same library:
I can build my application this way:
g++ -lutils application.cpp -o application.exe
The above command links in the shared version of an utils library.
I'm trying link against the static version of the library like this:
g++ -l:utils.a application.cpp -o application.exe
Both times I'm using
export LD_LIBRARY_PATH=path/to/utils:$LD_LIBRARY_PATH
to inform g++ where utils.a is placed.
The unresolved symbol reported by ld is present in the output of the nm:
nm --defined-only path/to/utils.a
and is marked with the "T" (meaning that it is from the code section).
I'm trying to figure out what can be the reason of the problem.
Is it correct to use LD_LIBRARY_PATH to specify where to search for utils.a?
What is the exact command to verify that a static library defines (resolves) the symbol? Is the command
nm --defined-only path/to/utils.a
enough or should I use any additional options like
nm --defined-only --demangle path/to/utils.a
e.g.?
Just option -static should be enough for compiler. In case only one library has to be static, then -static- and lib name is short name not file name.
Is it correct to use LD_LIBRARY_PATH to specify where to search for utils.a?
As mentioned by #user10605163, LD_LIBRARY_PATH is not to find path to static library at compile and link time. It is an environment variable used in some Linux distribution to search shared libraries during run time. Please find more documentation here It is useful for build and test environment but not a recommended way of linking in production systems.
What is the exact command to verify that a static library defines (resolves) the symbol? Is the command nm --defined-only path/to/utils.a
Yes, that is correct. However based on the information provided this error is not likely an error with symbols not present in utils(as it worked with shared library), but with the linking.
Refer GNU documentation GCC link options
Excerpt:
-l library : Search the library named library when linking. The linker searches a standard list of directories for the library. The directories searched include several standard system directories plus any that you specify with -L.
Also, with -l link option you need to provide the library name (without 'lib' and extension) or full file name.
-lutils or -llibutils.a
You can also provide direct full path here only, if required.
Related
I have the following problem:
I have two separate c++ projects, and want to use certain functions from one of them in the other. I am compiling using g++, on Linux.
I decided to do this by creating a shared library out of the project from which to use the functions. I added -fPIC to the compiler flags in the Makefile and then created a shared library like this:
g++ -shared -Wl,-soname,libmyproject.so.1 -o libmyproject.so a.o b.o c.o -lc
Then I simply copied the .so file and the header files into the (parent) directory of the new project and added the following to its Makefile:
To LIBS:
-L/../external_proj/libmyproject.so
To CXXFLAGS:
-I/../external_proj
Then I #include the appropriate header file in the destination project code and try to call certain functions from the original project. However, when I compile the destination project I get an error "undefined reference" for the imported function.
My question is then: is there something I'm missing in this setup? Is there perhaps something that needs to be added to the headers in the original project in order to export the functions I want to use?
I should note this is the first time I have attempted to use a shared library in this way. Thanks.
The -L option only specifies the directory where the linker will search for libraries to link with. Then you will need to use the -l option to specify the base name of the shared library (without the "lib" prefix and the ".so" suffix).
But even that will unlikely to be enough. The runtime loader needs to find the shared library, when you attempt to try to execute it. -L and -l will be sufficient to successfully link, but the runtime loader only searches /usr/lib(64)?, and a few other places by default. It does NOT search the current directory, and the ELF binary only records the names of the shared libraries that must be loaded, and not their full pathnames. You have to explicitly record any extra directories to search for any shared libraries, which is the -rpath option.
To finish the job you will also need to pass -rpath to the linker, but g++ does not support this option directory, you will have to use -W to do that.
The full set of options you will likely need are:
-L/../external_proj -lmyproject -Wl,-rpath -Wl,`cd ../external_proj && pwd`
See gcc documentation for more information on the -W option.
Absolute pathnames should be used with -rpath, hence the need to obtain the full pathname to the directory where the shared library is.
The -L flag is to add a path to search libraries in. The -l (lower-case L) is for linking with a library in the search path.
Or you can skip the flags and link with the library directly, almost like you do now (but without the -L option).
If you use the -l option, then remember that for a file libname.so you use only name as the library name. As in -lname. The linker will search for the correct files with the added prefix and suffix.
And lastly an important note about the paths used when linking: If you use -L and -l to link with a shared library, it's only the linker which will find the library. The OS runtime-loader will not be able to see the path used and will not find the library, if it's in a non-standard location. For that you must also set the runtime-path using the special linker option -rpath.
Unfortunately the GCC frontend program g++ doesn't recognize that option, you have to use -Wl to tell g++ to pass on an option to the actual linker. As in -Wl,-rpath,/path/to/libraries.
To summarize, here are the different variants you can use:
Link directly with the library: g++ your_source.cpp ../external_proj/libmyproject.so
Use the -L and -l options: g++ your_source.cpp -L../external_proj -lmyproject
To set the runtime linker path: g++ your_source.cpp -L../external_proj -lmyproject -Wl,-rpath,../external_proj
I make a program using a dynamic library, libexample.so. The dynamic library depends on another dynamic library, libtool.so.
It looks linker succeeded linking the libexample.so because of message from gcc.
Building target: libexample.so
Invoking: GCC C++ Linker
g++ -L/home/takehiro/Documents/documents/code/lib/tool -shared -o "libexample.so" ./classes/example.o ./classes/example_template.o ./classes/example_test.o ./classes/impl.o -ltool
Finished building target: libexample.so
cp libexample.so /home/takehiro/Documents/documents/code/lib/example
However, it failed to link it with the libtool.so.
ldd /home/takehiro/Documents/documents/code/lib/example/libexample.so
...
libtool.so => not found
...
I checked existence of the libtool.so under /home/takehiro/Documents/documents/code/lib/tool which is pointed by -L optoin at above linker by
ls /home/takehiro/Documents/documents/code/lib/tool
libtool.so
This is the first time to use a dynamic library depending another dynamic library. So I am so confused. Is it normal or malfunction? Why it cannot link them?
Does someone have a suggestion or a solution to me? I am very glad it.
Thank you very much.
All that the -L option does is tell the linker where the shared library is, at link time.
This does not have any effect on where the runtime loader searches for shared libraries. That's why the shared library fails to be loaded at runtime.
You also need to pass the -rpath option to the linker, when you're linking your shared library, in order to set the RPATH attribute on the shared library, that specifies where its dependencies are to be searched for. Something like
g++ -L/home/takehiro/Documents/documents/code/lib/tool \
-Wl,-rpath=/home/takehiro/Documents/documents/code/lib/tool \
... remaining options
I created a cpp project, which used a lib file named: libblpapi3_64.so
This file comes from a library which I download it from Internet.
My project runs without any error. So I update it to bitbucket.
Then my colleague downloads it and runs it at his own computer. But he gets an error:
usr/bin/ld: cannot find -lblpapi3_64.
In fact, I have copied it into my project repository. I mean I created a file named lib under my project and all lib files that I used are in it.
There are also other lib files such as liblog4cpp.a, but they are all good. Only the libblpapi3_64.so gets the error.
Is it because it's a .so file not .a file? Or there is other reason?
Btw, the file name of libblpapi3_64.so is green and others files(.a) is white. I think it's not a link file, it's the original file.
Briefly:
ld does not know about where your project libs are located. You have to place it into ld's known directories or specify the full path of your library by -L parameter to the linker.
To be able to build your program you need to have your library in /bin/ld search paths and your colleague too. Why? See detailed answer.
Detailed:
At first, we should understand what tools do what:
The compiler produces simple object files with unresolved symbols (it does not care about symbols so much at it's running time).
The linker combines a number of object and archive files, relocates their data and ties up symbol references into a single file: an executable or a library.
Let's start with some example. For example, you have a project which consists of 3 files: main.c, func.h and func.c.
main.c
#include "func.h"
int main() {
func();
return 0;
}
func.h
void func();
func.c
#include "func.h"
void func() { }
So, when you compile your source code (main.c) into an object file (main.o) it can't be run yet because it has unresolved symbols. Let's start from the beginning of producing an executable workflow (without details):
The preprocessor after its job produces the following main.c.preprocessed:
void func();
int main() {
func();
return 0;
}
and the following func.c.preprocessed:
void func();
void func() { }
As you may see in main.c.preprocessed, there are no connections to your func.c file and to the void func()'s implementation, the compiler simply does not know about it, it compiles all the source files separately. So, to be able to compile this project you have to compile both source files by using something like cc -c main.c -o main.o and cc -c func.c -o func.o, this will produce 2 object files, main.o and func.o. func.o has all it's symbols resolved because it has only one function which body is written right inside the func.c but main.o does not have func symbol resolved yet because it does not know where it is implemented.
Let's look what is inside func.o:
$ nm func.o
0000000000000000 T func
Simply, it contains a symbol which is in text code section so this is our func function.
And let's look inside main.o:
$ nm main.o
U func
0000000000000000 T main
Our main.o has an implemented and resolved static function main and we are able to see it in the object file. But we also see func symbol which marked as unresolved U, and thus we are unable to see its address offset.
For fixing that problem, we have to use the linker. It will take all the object files and resolve all these symbols (void func(); in our example). If the linker somehow is unable to do that it throws a error like unresolved external symbol: void func(). This may happen if you don't give the func.o object file to the linker. So, let's give all the object files we have to the linker:
ld main.o func.o -o test
The linker will go through main.o, then through func.o, try to resolve symbols and if it goes okay - put it's output to the test file. If we look at the produced output we will see all symbols are resolved:
$ nm test
0000000000601000 R __bss_start
0000000000601000 R _edata
0000000000601000 R _end
00000000004000b0 T func
00000000004000b7 T main
Here our job is done. Let's look the situation with dynamic(shared) libraries. Let's make a shared library from our func.c source file:
gcc -c func.c -o func.o
gcc -shared -fPIC -Wl,-soname,libfunc.so.1 -o libfunc.so.1.5.0 func.o
Voila, we have it. Now, let's put it into known dynamic linker library path, /usr/lib/:
sudo mv libfunc.so.1.5.0 /usr/lib/ # to make program be able to run
sudo ln -s libfunc.so.1.5.0 /usr/lib/libfunc.so.1 #creating symlink for the program to run
sudo ln -s libfunc.so.1 /usr/lib/libfunc.so # to make compilation possible
And let's make our project depend on that shared library by leaving func() symbol unresolved after compilation and static linkage process, creating an executable and linking it (dynamically) to our shared library (libfunc):
cc main.c -lfunc
Now if we look for the symbol in its symbols table we still have our symbol unresolved:
$ nm a.out | grep fun
U func
But this is not a problem anymore because func symbol will be resolved by dynamic loader before each program start. Okay, now let's back to the theory.
Libraries, in fact, are just the object files which are placed into a single archive by using ar tool with a single symbols table which is created by ranlib tool.
Compiler, when compiling object files, does not resolve symbols. These symbols will be replaced to addresses by a linker. So resolving symbols can be done by two things: the linker and dynamic loader:
The linker: ld, does 2 jobs:
a) For static libs or simple object files, this linker changes external symbols in the object files to the addresses of the real entities. For example, if we use C++ name mangling linker will change _ZNK3MapI10StringName3RefI8GDScriptE10ComparatorIS0_E16DefaultAllocatorE3hasERKS0_ to 0x07f4123f0.
b) For dynamic libs it only checks if the symbols can be resolved (you try to link with correct library) at all but does not replace the symbols by address. If symbols can't be resolved (for example they are not implemented in the shared library you are linking to) - it throws undefined reference to error and breaks up the building process because you try to use these symbols but linker can't find such symbol in it's object files which it is processing at this time. Otherwise, this linker adds some information to the ELF executable which is:
i. .interp section - request for an interpreter - dynamic loader to be called before executing, so this section just contains a path to the dynamic loader. If you look at your executable which depends on shared library (libfunc) for example you will see the interp section $ readelf -l a.out:
INTERP 0x0000000000000238 0x0000000000400238 0x0000000000400238
0x000000000000001c 0x000000000000001c R 1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
ii. .dynamic section - a list of shared libraries which interpreter will be looking for before executing. You may see them by ldd or readelf:
$ ldd a.out
linux-vdso.so.1 => (0x00007ffd577dc000)
libfunc.so.1 => /usr/lib/libfunc.so.1 (0x00007fc629eca000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fefe148a000)
/lib64/ld-linux-x86-64.so.2 (0x000055747925e000)
$ readelf -d a.out
Dynamic section at offset 0xe18 contains 25 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libfunc.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
Note that ldd also finds all the libraries in your filesystem while readelf only shows what libraries does your program need. So, all of these libraries will be searched by dynamic loader (next paragraph).
The linker works at build time.
Dynamic loader: ld.so or ld-linux. It finds and loads all the shared libraries needed by a program (if they were not loaded before), resolves the symbols by replacing them to real addresses right before the start of the program, prepares the program to run, and then runs it. It works after the build and before running the program. Less speaking, dynamic linking means resolving symbols in your executable before each program start.
Actually, when you run an ELF executable with .interp section (it needs to load some shared libraries) the OS (Linux) runs an interpreter at first but not your program. Otherwise you have an undefined behavior - you have symbols in your program but they are not defined by addresses which usually means that the program will be unable to work properly.
You may also run dynamic loader by yourself but it is unnecessary (binary is /lib/ld-linux.so.2 for 32-bit architecture elf and /lib64/ld-linux-x86-64.so.2 for 64-bit architecture elf).
Why does the linker claim that /usr/bin/ld: cannot find -lblpapi3_64 in your case? Because it tries to find all the libraries in it's known paths. Why does it search the library if it will be loaded during runtime? Because it needs to check if all the needed symbols can be resolved by this library and to put it's name into the .dynamic section for dynamic loader. Actually, the .interp section exists in almost every c/c++ elf because the libc and libstdc++ libraries are both shared, and compiler by default links any project dynamically to them. You may link them statically as well but this will enlarge the total executable size. So, if the shared library can't be found your symbols will remain unresolved and you will be UNABLE to run your application, thus it can't produce an executable. You may get the list of directories where libraries are usually searched by:
Passing a command to the linker in compiler arguments.
By parsing ld --verbose's output.
By parsing ldconfig's output.
Some of these methods are explained here.
Dynamic loader tries to find all the libraries by using:
DT_RPATH dynamic section of an ELF file.
DT_RUNPATH section of the executable.
LD_LIBRARY_PATH environment variable.
/etc/ld.so.cache - own cache file which contains a compiled list of candidate libraries previously found in the augmented library path.
Default paths: In the default path /lib, and then /usr/lib. If the binary was linked with -z nodeflib linker option, this step is skipped.
ld-linux search algorithm
Also, note please, that if we are talking about shared libraries, they are not named .so but in .so.version format instead. When you build your application the linker will look for .so file (which is usually a symlink to .so.version) but when you run your application the dynamic loader looks for .so.version file instead. For example, let's say we have a library test which version is 1.1.1 according to semver. In the filesystem it will look like:
/usr/lib/libtest.so -> /usr/lib/libtest.so.1.1.1
/usr/lib/libtest.so.1 -> /usr/lib/libtest.so.1.1.1
/usr/lib/libtest.so.1.1 -> /usr/lib/libtest.so.1.1.1
/usr/lib/libtest.so.1.1.1
So, to be able to compile you must have all of versioned files (libtest.so.1, libtest.so.1.1 and libtest.so.1.1.1) and a libtest.so file but for running your app you must have only 3 versioned library files listed first. This also explains why do Debian or rpm packages have devel-packages separately: normal one (which consists only of the files needed by already compiled applications for running them) which has 3 versioned library files and a devel package which has only symlink file for making it possible to compile the project.
Resume
After all of that:
You, your colleague and EACH user of your application code must have all the libraries in their system linker paths to be able to compile (build your application). Otherwise, they have to change Makefile (or compile command) to add the shared library location directory by adding -L<somePathToTheSharedLibrary> as argument.
After successful build you also need your library again to be able to run the program. Your library will be searched by dynamic loader (ld-linux) so it needs to be in it's paths (see above) or in system linker paths. In most of linux program distributions, for example, games from steam, there is a shell-script which sets the LD_LIBRARY_PATH variable which points to all shared libraries needed by the game.
You could look at our Rblapi package which uses this very library too.
Your basic question of "how do I make a library visible" really has two answers:
Use ld.so. The easiest way is to copy blpapi3_64.so to /usr/local/lib. If you then call ldconfig to update the cache you should be all set. You can test this via ldconfig -p | grep blpapi which should show it.
Use an rpath instruction when building your application; this basically encodes the path and makes you independent of ld.so.
I want to start with a simple linking usage to explain my problem. Lets assume that there is a library z which could be compiled to shared library libz.dll(D:/libs/z/shared/libz.dll) or to static library libz.a (D:/libs/z/static/libz.a).
Let I want to link against it, then I do this:
gcc -o main.exe main.o -LD:/libs/z/static -lz
According to this documentation, gcc would search for libz.a, which is
archive files whose members are object files
I also can do the following:
gcc -o main.exe main.o -LD:/libs/z/shared -lz
It is not mentioned in the documentation above that -l flag will search for lib<name>.so.
What will happen if I libz.a and libz.dll will be in the same directory? How the library will be linked with a program? Why I need the flags -Wl,-Bstatic and -Wl,-Bdynamic if -l searches both for shared and static libraries?
Why some developers provide .a files with .dll files for the same modules, if I compile a shared library distribution?
For example, Qt provides .dll files in bin directory with .a files in lib directory. Is it the same library, but built like shared and static, respectively? Or .a files are some kind of dummy libraries which provide linking with shared libraries, where there are real library implementations?
Another example is OpenGL library on Windows. Why every compiler must provide the static OpenGL lib like libopengl32.a in MingW?
What are files with .dll.a and .la extensions used for?
P.S. There are a lot of questions here, but I think each one depends on the previous one and there is no need to split them into several questions.
Please, have a look at ld and WIN32 (cygwin/mingw). Especially, the direct linking to a dll section for more information on the behavior of -l flag on Windows ports of LD. Extract:
For instance, when ld is called with the argument -lxxx it will attempt to find, in the first directory of its search path,
libxxx.dll.a
xxx.dll.a
libxxx.a
cygxxx.dll (*)
libxxx.dll
xxx.dll
before moving on to the next directory in the search path.
(*) Actually, this is not cygxxx.dll but in fact is <prefix>xxx.dll, where <prefix> is set by the ld option -dll-search-prefix=<prefix>. In the case of cygwin, the standard gcc spec file includes -dll-search-prefix=cyg, so in effect we actually search for cygxxx.dll.
NOTE: If you have ever built Boost with MinGW, you probably recall that the naming of Boost libraries exactly obeys the pattern described in the link above.
In the past there were issues in MinGW with direct linking to *.dll, so it was advised to create a static library lib*.a with exported symbols from *.dll and link against it instead. The link to this MinGW wiki page is now dead, so I assume that it should be fine to link directly against *.dll now. Furthermore, I did it myself several times with the latest MinGW-w64 distribution, and had no issues, yet.
You need link flags -Wl,-Bstatic and -Wl,-Bdynamic because sometimes you want to force static linking, for example, when the dynamic library with the same name is also present in a search path:
gcc object1.o object2.o -lMyLib2 -Wl,-Bstatic -lMyLib1 -Wl,-Bdynamic -o output
The above snippet guarantees that the default linking priority of -l flag is overridden for MyLib1, i.e. even if MyLib1.dll is present in the search path, LD will choose libMyLib1.a to link against. Notice that for MyLib2 LD will again prefer the dynamic version.
NOTE: If MyLib2 depends on MyLib1, then MyLib1 is dynamically linked too, regardless of -Wl,-Bstatic (i.e. it is ignored in this case). To prevent this you would have to link MyLib2 statically too.
I use log4cxx logging library. I need to link with its static version to avoid additional binary dependencies. I use it in my dynamic library. Default build of log4cxx produces static library but I cannot link with it because it was compiled w/o -fPIC flag. So I changed log4cxx bulding as:
CPPFLAGS="-fPIC -static" ./configure
make
As a result I received a liblog4cxx.a that I can link with my .so library. Linking was done by Cmake, something like:
target_link_libraries(my_dynamic_lib log4cxx)
link_directories(relative_path_to_dir_where_liblog4cxx.a_lives)
Everything looked fine until runtime. I cannot load my_dynamic_lib.so because of undefined symbol "logger"
Please explain me what's wrong and how to resolve this problem.
thanks
You can verify whether the shared library contains the symbol by using
nm -g my_dynamic_lib.so | grep logger
If it is shown with symbol type U it means it's undefined.
Normally a shared library will not resolve all the symbols it needs until run-time, so it is possible (and perfectly normal) to link a shared library with missing symbols.
If you put -llog4cxx at the start of the linker command line for my_dynamic_lib.so then it won't link to any of the code in there, and will leave the logger symbol unresolved until run-time. To force it to use the symbols in the static library make sure you list the static library after the objects that need it:
g++ -fPIC -shared -o my_dynamic_lib.so obj1.o obj2.o -llog4cxx ...
I don't know how to do that with cmake, but it looks as though your CMakefile only links to log4cxx when linking the main executable, not the dynamic library.
Usually you would link liblog4cxx.a with your executable and not with your my_dynamic_lib.so. I don't think you can link like in your example unless you can provide documentation that says otherwise.