I created a library that I'd like others to use.
To Compile My library:
/usr/bin/g++ -fPIC -shared -Wl,-soname,libMYLIB.so [inputs] -lboost_system -lboost_thread
To Compile a Binary:
/usr/bin/g++ myTest.cpp -lMYLIB -lboost_system
I would like that line to only be:
/usr/bin/g++ myTest.cpp -lMYLIB
How do I avoid having to specify my libraries dependencies later? What flag in the linker or compiler am I looking for?
There is a linker option (I mean ld linker http://linux.die.net/man/1/ld) --unresolved-symbols=ignore-all or -unresolved-symbols=ignore-in-object-files:
Determine how to handle unresolved symbols. There are four possible values for method:
* ignore-all
Do not report any unresolved symbols.
* report-all
Report all unresolved symbols. This is the default.
* ignore-in-object-files
Report unresolved symbols that are contained in shared libraries, but ignore them if they come from regular object files.
* ignore-in-shared-libs
Report unresolved symbols that come from regular object files, but ignore them if they come from shared libraries. This can be useful
when creating a dynamic binary and it is known that all the shared
libraries that it should be referencing are included on the linker's
command line.
And this is an example. I have a library libmylib.so and an application main:
So first I build the library:
$ g++ -fpic -shared mylib.cpp -o libmylib.so
When I build the application but I do not add -lmylib on command line. Normally it results in error Unresolved external symbols but since I add -Wl,--unresolved-symbols=ignore-in-object-files to the command line I get no errors:
$ g++ -fpic -g main.cpp -Wl,--unresolved-symbols=ignore-in-object-files -Wl,-rpath,.
Then I run my program:
$ ./a.out
./a.out: symbol lookup error: ./a.out: undefined symbol: _Z7my_funcd
It doesn't work as expected but then I use LD_PRELOAD:
$ LD_PRELOAD=./libmylib.so ./a.out
2
So with LD_PRELOAD it works
Related
(Note: This question had been closed, citing that this had an answer. However, my question is not generic, I am asking why this works in ubuntu/redhat, but not in macos/cygwin. So I have edited this question, by modifying the title, mentioning the words macos and ubuntu.)
I have the following c++ code:
// main.cpp
#include<iostream>
#include<cstdio>
#include "defs.h" // has the function headers only
int func0(int a0) {
printf("func0-%d\n", a0);
return a0+1;
}
int func1(int a1) {
int x;
x=func0(a1);
printf("func1-%d\n", x);
return a1+1;
}
int func2(int a2) {
int x;
x=func1(a2);
printf("func2-%d\n", x);
return x+5;
}
int main() {
func1(5);
func2(8);
}
I can compile and run this code as:
g++ main.cpp; ./a.out
Now I would like to move the functions to different files (func1 to f1.cpp,
fun0 and func2 to f2.cpp, and main to main.cpp), and create shared libraries
like this:
g++ -c -pipe -std=c++11 -fPIC main.cpp
g++ -c -pipe -std=c++11 -fPIC f1.cpp
g++ -c -pipe -std=c++11 -fPIC f2.cpp
g++ -shared -o libx1.so f1.o
g++ -shared -o libx2.so f2.o
g++ main.o -L. -lx1 -lx2 -o exe
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
./exe
The above instructions work in redhat linux and ubuntu. But when I run the same commands in other variants of linux (eg macos or cygwin) I get errors during creation of the shared library like this:
g++ -shared -o libx1.so f1.o
undefined reference to func0(int)
g++ -shared -o libx2.so f2.o
undefined reference to func1(int)
Why is this error happening only in some OS versions, and not happening in redhat/ubuntu? Is it due to the gcc versions, or something to do with the OS?
(The above instructions work with g++ in redhat(gcc version 8.3.1) and ubuntu (9.4.0). It does not work with g++ in cygwin(11.3.0) and in macos(11.2.0).)
The problem is caused by cyclic dependencies between the two libraries. Before doing anything else, you should ask yourself whether it is acceptable to have cyclic dependencies for your project. It is never a good idea, but if the alternative involves massive refactoring, it could be the lesser of two evils. Still, refactoring should probably be the default answer in most cases. If you cannot refactor, the rest of this answer is for you.
How are cyclic dependencies handled on different OSes?
On both Linux and Mac OS X (and on FreeBSD and on most commercial Unixes of old), references are resolved at load time. The loader uses the first suitable symbol definition it encounters, be it it in the main executable, in the shared object itself, or in a different shared object. It is not known until load time where that definition will be found.
So when the executable from the question is loaded, the dynamic loader finds the definition of func1 in libx1 and the definitions of func0 and func2 in libx2, and all is well.
The difference between Linux and Mac OS X lies in the linker (ld) behaviour. Both GNU ld and LLVM ld by default allow unresolved references when building a shared library. Mac OS X ld appears to be of a different breed and unresolved references are not allowed by default. One can either list the dependencies on the link line, or explicitly allow unresolved references using the Mac-specific ld option -undefined dynamic_lookup. But of course when the dependencies are cyclic, the first option is problematic. For code in question:
g++ -shared -o libx1.so f1.o -Wl,-undefined,dynamic_lookup
g++ -shared -o libx2.so f2.o -Wl,-undefined,dynamic_lookup
Windows DLLs work very differently. Each symbol must be resolved at link time. Unlike the Unix-y loaders, the loader must know exactly which DLL to search for each imported symbol. There is no option to allow unresolved references in DLLs because there is no mechanism to resolve them at load time from an unknown source.
Windows still allows cyclic dependencies between DLLs, but the mechanism is a bit different. The linker must use separate import libraries in this case (they are usually optional when using GNU or LLVM toolchains). The linking is done in two phases. First, the .lib files are generated for each future .dll, and then .dll themselves are produced using the .lib files from the first stage. For code in question:
# first stage
g++ -shared -Wl,--out-implib=x1.lib -o x1.dll f1.o
g++ -shared -Wl,--out-implib=x2.lib -o x2.dll f2.o
# second stage
g++ -shared -o x1.dll f1.o x2.lib
g++ -shared -o x2.dll f2.o x1.lib
The first stage will report undefined symbols but will still produce the .lib file needed for the second stage.
printme() and getme() defined in top.cpp and top.h
I used printme() function in test.cpp (test cpp file) in main function
g++34 -c top.cpp -fPIC
ar rcs libtop.a top.o
g++34 -c test.cpp -fPIC
g++34 -shared -o ltop.so -ltop -L. -fPIC
getme is not getting exported in ltop.so
How i can force getme function exported in ltop.so
When i do nm ltop.so
it is not showing getme symbol
i want to force this
Note: file can have multiple unused function like - getme()
I want to force all to export to so library
Normally, when linking with a static library only the modules in the static library that contain an unresolved symbol end up getting linked.
Here, since there is no unresolved reference to getme(), hence this module does not get linked from the static library. The solution is to explicitly make it unresolved.
A minor complicating factor is the C++ symbol name mangling. It is necessary to figure out what is the mangled symbol name for the getme() function. The easiest way is to look at the library with the nm command:
$ nm libtop.a
top.o:
0000000000000000 T _Z5getmei
Ok, so the mangled symbol name is _Z5getmei. The -u linker flag forces an unresolved reference to the indicated symbol to be used when linking:
g++ -shared -o ltop.so -L. -ltop -Wl,-u -Wl,_Z5getmei
The documentation for the -u option is found in the ld man page. This includes the module in the shared library:
$ nm ltop.so | grep getme
0000000000000680 T _Z5getmei
Don't use your static library to create your dynamic one. Instead, use the component object file directly:
g++34 -shared -o ltop.so -fPIC top.o
The reason is that when you specify a library with -l when compiling a binary, only unresolved external symbols from earlier in the compilation line are picked up. In your case, this is nothing, so precisely nothing is picked up in creating libtop.so from libtop.a
UPDATE: As an alternative, if the original object files are no longer available, youc an use the --whole-archive linker option to force it to include everything from a static library, rather than just unresolved externals:
g++34 -shared -o ltop.so -fPIC -Wl,--whole-archive ./libtop.a
Or:
g++34 -shared -o ltop.so -fPIC -Wl,--whole-archive -L. -ltop
I'm trying to build some example c++ code that use boost library. I use this as reference example of static linking.
And everything is fine when I build with dynamic libs.
g++ -Wall -std=c++0x -O3 -Wfatal-errors -I/usr/include/boost/include -c -o src/main.o src/main.cpp
g++ -Wall -std=c++0x -O3 -Wfatal-errors -I/usr/include/boost/include -c -o src/ThreadExample.o src/ThreadExample.cpp
g++ -Wall -std=c++0x -O3 -Wfatal-errors -I/usr/include/boost/include -c -o src/Utils.o src/Utils.cpp
g++ src/main.o src/ThreadExample.o src/Utils.o -lboost_thread -lboost_filesystem -lboost_system -lboost_timer -o ThreadExampleBinary
But when I use static libs I get lots of undefined reference errors:
g++ -Wall -std=c++0x -O3 -Wfatal-errors -I/usr/include/boost/include -c -o src/main.o src/main.cpp
g++ -Wall -std=c++0x -O3 -Wfatal-errors -I/usr/include/boost/include -c -o src/ThreadExample.o src/ThreadExample.cpp
g++ -Wall -std=c++0x -O3 -Wfatal-errors -I/usr/include/boost/include -c -o src/Utils.o src/Utils.cpp
g++ -static src/main.o src/ThreadExample.o src/Utils.o -lboost_thread -lboost_filesystem -lboost_system -lboost_timer -o ThreadExampleBinary
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libboost_timer.a(cpu_timer.o): In function `boost::timer::cpu_timer::start()':
(.text+0x7fd): undefined reference to `boost::chrono::steady_clock::now()'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libboost_timer.a(cpu_timer.o): In function `boost::timer::cpu_timer::stop()':
(.text+0x94c): undefined reference to `boost::chrono::steady_clock::now()'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libboost_timer.a(cpu_timer.o): In function `boost::timer::cpu_timer::elapsed() const':
(.text+0xa59): undefined reference to `boost::chrono::steady_clock::now()'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libboost_timer.a(cpu_timer.o): In function `boost::timer::cpu_timer::resume()':
(.text+0xb60): undefined reference to `boost::chrono::steady_clock::now()'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libboost_timer.a(cpu_timer.o): In function `boost::timer::auto_cpu_timer::auto_cpu_timer(std::ostream&, short)':
(.text+0xca5): undefined reference to `boost::chrono::steady_clock::now()'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libboost_timer.a(cpu_timer.o):(.text+0xd4e): more undefined references to `boost::chrono::steady_clock::now()' follow
collect2: error: ld returned 1 exit status
make: *** [ThreadExampleBinary] Error 1
Seems this can be fixed adding additional -lboost_chrono library.
But why it works in dinamic setting?
With static linking, you have to also statically link to any libraries depended on by the libraries you are linking to.
The difference is that shared libraries have an entry in the ELF header, called NEEDED that lists other shared libraries that are to be included when you link in this one.
You can see them with this command:
$ objdump -p /usr/lib/libboost_timer.so | grep NEEDED
NEEDED libboost_chrono.so.1.60.0
NEEDED libboost_system.so.1.60.0
NEEDED librt.so.1
NEEDED libstdc++.so.6
NEEDED libgcc_s.so.1
NEEDED libc.so.6
But for static libraries there is no such system, as they are simply collection of object files.
It is worth noting that the NEEDED entry in the shared objects are entirely optional, and if they are not available, then they will behave exactly like the static ones. But most shared libraries include them.
Many libraries use the pkg-config infrastructure to provide the full command line needed, but AFAIK boost is not one of them.
How to automate this process? Well, you do not. You just include what is needed and follow the linker errors to discover further needs.
You can find which static library includes a symbol with something like:
$ nm --print-file-name --defined-only --demangle /usr/lib/*.a 2> /dev/null | \
grep -q 'boost::chrono::steady_clock::now()'
/usr/lib/libboost_chrono.a:chrono.o:0000000000000090 T boost::chrono::steady_clock::now()
But why it works in dinamic setting?
Most of my make files have the following notes (from when I found the answer back when I needed it ... sorry, I don't know where I found it.)
Note - when a build using '-l' finds the .so version of that library (so - shared object) and the same .a archive is also there, g++ prefers the .so over the .a.
However, you can still achieve static link by fully specifying the path to .a.
Example:
$(CC) $(CC_FLAGS) $< /usr/local/lib/libboost_chrono.a -o $# ...
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sometimes, archive code refers to symbols in yet another archive.
i.e. -lyyy_i686 uses some function from -lxxx_i686, and xxx was listed first in the build command. It is possible that a symbol might remain unresolved and the linker fails.
When this happens, try adding xxx a second time to the end of the archive list
from: -lxxx_i686 -lyyy_i686 -lrt -pthread
becomes -lxxx_i686 -lyyy_i686 -lrt -pthread -lxxx_i686
^^^^^^^^^_____________________________^^^^^^^^^^
The preceeding presumes only .a libraries are findable (no xxx.so)
Another alternative, you can command the linker to search an archive multiple times
-lyyy_i686 -lrt -pthread -(-lxxx_i686-)
tell gcc to link this library as many times as needed __^^__________^^
Again, only .a is findable
Finally, if you choose to link with the .so, note that this does not pull in the entire lib into the current build target. Instead, at runtime, the .so is loaded into memory, if it is not already there. This happens after the program starts. For most applications, this is considered a 'small' start-up performance hit as the program adjusts its memory map (automagically behind the scenes). I believe I once found that the app itself can control when to load the .so.
The use of .so pulls in the entire library to memory (if not already installed). Thus, there would not be any missing symbols, in my xxx vs yyy scenario above, the .so pulls all the symbols, whether used or not.
And once again, when trying to load xxx using "-lxxx_i686", the linker prefers to pull in libxxx_i686.so even when the libxxx_i686.a is in the same directory.
I have a program structure that has
static library(ACE)
static library(common.a)
dynamic library plugin 1(1.so)
plugin 2(2.so) and executable
plugin1, plugin2 and executable all use both common.a and libACE.a
Follow the tutorial here: http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html.
I only link those two static library when compiling the executable as shown below:
g++ -g -DUNIX -DLINUX -Wall -D__NUMBER_FIELD_ID__ -I/opt/ACE_wrappers -Ilib/ -I. -I./common -I./common/lib -I../inc -I/opt/pct/pctlib/inc -o acs_d acs_d.o -L../lib -Wl,--export-dynamic -rdynamic -Wl,--whole-archive /opt/ACE_wrappers/ace/libACE.a common/libcommon_d.a -Wl,--no-whole-archive -ldl -lrt -lpthread
The point is, when I use dlopen to open those two plugins, one succeeds and one fails
The successful one use more ACE functions and the error is complaining undefined symbol as shown below:
[CModuleMgr] loadCModule(): Errors occurred when opening the module. nCModuleId[1] pLibHandle[(nil)] sCModulePath[/opt/acs/adapter/libadapter_d.so] sError[/opt/acs/adapter/libadapter_d.so: undefined symbol: _ZN17ACE_Event_Handler10set_handleEi]
For the main program, I have tried to use command nm to find the symbol
$ nm acs_d | grep _ZN17ACE_Event_Handler10set_handleEi
000000000048f240 t _ZN17ACE_Event_Handler10set_handleEi
It is there, but the plugin cannot find it! I have used option like -Wl,--export-dynamic -rdynamic -Wl,--whole-archive. But it still cannot find this symbol. any idea?
It is there, but the plugin cannot find it!
No, the symbol is not there!
Or rather, the symbol has internal linkage (t), and is not visible or usable outside of the ELF image into which it is linked. Globally visible symbols have external (T) linkage.
The most likely cause for the symbol to have t linkage is that the symbol has __attribute__((visibility("hidden"))) at the source level. Documentation here.
Why ACE developers marked it as such, I don't know.
I'm trying to load a shared library (plugin) I was provided (closed source) with dlopen under a Linux ARM platform. I'm trying to load this way:
void* handle = dlopen(<library_path>/<library_name>, RTLD_NOW);
The result is a failure with this message:
Failed to load <library_path>/<library_name>: undefined symbol: <symbol_name>.
I tried to look inside the library with nm, but it seems the lib was stripped, no symbol could be found. I also tried using readelf -s, and, in fact, I got this result:
12663: 00000000 0 NOTYPE GLOBAL DEFAULT UND <symbol_name>
By reading around, I get that readelf -s returns all the symbols, including those symbols defined in libraries referenced by it.
The answers to this question are not completely clear to me: is this a symbol which is supposed to be in the library and which is not there because it was compiled the wrong way or is this a symbol I'm supposed find somewhere else? The output of readelf -d seems to suggest I'm providing all the needed shared libraries. May this error be related to a mistake in the way I'm compiling my executable or is this something not related to the loader?
Also, I read about the meaning of each column, but those values are quite strange. How do you interpret that symbol description? Why is address 0? Why is type NOTYPE?
undefined symbol: X means always that X should be exported from one of loaded libraries, but it's not. You should find out in which library requested symbol is and link to it.
You should know that this message is always result of problem with library, it's not fault. Library should know how to get all it's symbols. If it doesn't you can link your executable to required library so when you load your plugin, requested symbol is already known.
This error might have more complex reason. In case when both plugin and main app are linking to library, then attempts to link it might end with undefined symbols anyway. This might happen if main app and plugin are using different version of library (namely plugin uses newer one). Then at the point of loading plugin older version is already loaded, so loader assumes everything is ok, but newer version might contain new symbols. If plugin uses them, you will get undefined symbol errors.
This problem appears also if the order of the static libraries in the linking command is wrong for the app. The Unix ld linker requires that the library which implements a function is specified after the library which refers the function.
I got this trouble when I was trying to build libtesseract shared library taking libz library from a custom location (not a standard libz from the host, but manually built from source as well). I have put an example below:
Wrong linking order (-lz before -llept):
$ g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/5/crtbeginS.o -Wl,--whole-archive ....(some libs) -Wl,--no-whole-archive -L/home/build/jenkins/workspace/tesseract/zlib/bin/lib -L/home/build/jenkins/workspace/tesseract/leptonica/bin/lib -L/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/x86_64-linux-gnu -lz -llept -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/5/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crtn.o -g -O2 -Wl,-soname -Wl,libtesseract.so.4 -o .libs/libtesseract.so.4.0.1
Check with "nm -D":
$ nm -D .libs/libtesseract.so.4.0.1 | grep deflateInit
U deflateInit_
Check with "dlopen":
Cannot load ./tesseract/src/api/.libs/libtesseract.so.4.0.1 (./tesseract/src/api/.libs/libtesseract.so.4.0.1: undefined symbol: deflateInit_)
It happens because the linker is processing in the loop all static libraries passed in the command line and skipping those which are not used by any of the preceeding ones. Since on the moment of checking of libz.a the linker sees that all of already checked libraries do not use any function from libz.a the linker just "forgets" libz.a.
Proper linking order (-lz after -llept):
$ g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/5/crtbeginS.o -Wl,--whole-archive ....(some libs) -Wl,--no-whole-archive -L/home/build/jenkins/workspace/tesseract/zlib/bin/lib -L/home/build/jenkins/workspace/tesseract/leptonica/bin/lib -L/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/x86_64-linux-gnu -llept -lz -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/5/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crtn.o -g -O2 -Wl,-soname -Wl,libtesseract.so.4 -o .libs/libtesseract.so.4.0.1
Check with "nm -D":
$ nm -D .libs/libtesseract.so.4.0.1 | grep deflateInit
000000000041fb5b T deflateInit_
000000000041fba3 T deflateInit2_
"dlopen" did not show this error this time.