Removing linker dependencies - c++

I'm working on a c++ app on linux using g++. My app links in a shared library that exposes a simple API I need. This library internally references a large number of additional shared libs. I've had to find each and every one and add them to my Makefile to get them linked in.
I assume my app has to link to any of the libs the primary lib depends on. Is the only way around this linking requirement, having the primary lib compile in the static libs of all its dependencies? Does this apply to using the plug-in model via dlopen/dlsym?
ty


Is the only way around this linking requirement, having the primary lib compile in the static libs of all its dependencies?
No. The shared library could itself link to the shared libraries it depends on. Most linkers would pick up those libraries as well, and link your executable to those libraries without you mentioning them at the linker stage.
In your case, it sounds like the shared library did not link to the libraries it required. the ldd tool can be useful in this regard.
As an example, say you produce this shared library:
gcc -shared foo.o -o libfoo.so -lm
Now the libfoo.so is linked to the math library (libm). Any application linked to libfoo.so
would be linked to libm as well, i.e. you'd just have to do
gcc -o prog main.o -lfoo
If on the other hand the shared library was not linked to lib, but was produced with only
gcc -shared foo.o -o libfoo.so
You'd have to explicitly link to libm when linking your application:
gcc -o prog main.o -lfoo -lm
When you dlopen() a shared library, the runtime linker will load all the libraries that shared library was also linked against - unless they're already loaded. So - if a library you dlopen() is not linked against libraries it depends on, and your executable are not linked against those libraries either, dlopen() will fail (unless you specify RTLD_LAZY, in which case things fail later on)


I assume my app has to link to any of the libs the primary lib depends on.
It sounds as though either your app is using those symbols directly (not just indirectly through the API shared lib) or that shared lib hasn't been linked properly, so it wasn't linked to the libraries it depends on. If when that shared library was created its dependencies were linked to with -l then it would cause them to be linked to automatically when linking your app to the API lib.
Is the only way around this linking requirement, having the primary lib compile in the static libs of all its dependencies?
That's one way, but not the only way.
Does this apply to using the plug-in model via dlopen/dlsym?
As long as the plugins are correctly linked to the libs they depend on ... No, in that case the linker cannot possibly know what libs you will dlopen at run-time, so can't know what their dependencies will be, so you don't need to name them at link-time. It wouldn't be possible to without knowing in advance all the plugins that might get loaded.
If the plugin libraries weren't properly linked you'd get the same problem when trying to dlopen them.


I assume my app has to link to any of the libs the primary lib depends on. Is the only way around this linking requirement, having the primary lib compile in the static libs of all its dependencies?
Off course. There are no ways around : either static linking, or having libraries (correct versions, if not ABI compatible) in libraries run-time search path.
Does this apply to using the plug-in model via dlopen/dlsym?
No. For that, you need to have shared library in a path from which you want to load it.

Related

Shared lib with link to static libs internally

I have a shared library called MyLib. I have linked MyLib to a static version of Boost and three other libraries.
MyLib does not expose any feature of those linked libraries. In other words, the public interface headers of MyLib does not mention or include Boost or any other library.
I have built MyLib and got .lib(.a) and .dll(.so) files.
Now, I have MyEXE which is a program that is linked to MyLib. Everything is smooth till now.
If I need some feature of Boost or the other three libraries to be used in MyEXE:
Will it be safe to link MyEXE to Boost or the other libraries? Will I face any ODR problem?
If it is OK to link to them, should I link to the exact same version of Boost and other libraries that were used with MyLIB?
Does the previous scenario seems bad? Should I follow a better scenario? (Distributing the source and keeping the link process to the end developer is not an option for me)

C++ linking custom .so libraries and then with JNI

I have a problem linking C++ libraries and I could really use some advice.
I have two custom .so libraries, which I am using in my cpp project, which I am then linking to Java using JNI.
In my cpp project I am including the header files for the two libraries and JNI. Then I am creating an object file, like this:
g++ -I/usr/lib/jvm/java-7-openjdk-amd64/include -I/usr/lib/jvm/java-7-openjdk-amd64/include/linux -std=c++11 -fPIC -c my_file.cpp
The two includes specify the path to JNI libraries. I am using cpp11 and then a switch -fpic I found that may be necessary during creation of shared libraries.
Next I tried to link my object file: my_file.o with the two shared libraries like this:
g++ -fpic -L/<absolute path>/library1.so -L/<absolute path>/library2.so my_file.o -shared -o my_file.so
I am building a shared library which will be called from Java using JNI. I tried several different variations of assembling the linking parameters but none worked. I am further calling System.load() in Java on an absolute paths to my_file.so and library1.so
Currently I am receiving an error:
/usr/lib/jvm/java-7-openjdk-amd64/bin/java: symbol lookup error: /<absoulte path>/my_file.so: undefined symbol: <method from library 1>
I have read several post here about linking and required parameter order of libraries in order to ensure the correct linking and I have gone through much of g++ documentation, yet no matter how I order them and no matter what parameters I use, it does not seem to work. I will be greatfull for any advice as to how to link these files.

Figured it out with some help.
Firstly, for those who do not know about the command "ldd" on linux, it shows the linked symbols (linked library dependencies). When I saw that my libraries weren't there I tried to rearrange the linking command.
I had to place the object file my_file.o before the other two libraries. Then I saw that ldd pointed out that the libraries were not found. I moved them to a system library and with some more symlinks everything worked. Thanks to everyone for their effort

Code::Blocks and boost 1.55: static library is not used when dynamic library is present

I have built the boost 1.55 serialzation library with the following command:
b2 --build-dir=build toolset=gcc --with-serialization --layout=tagged link=static threading=multi stage
and got libboost_serialization-mt.a and libboost_wserialization-mt.a in my stage/lib directory - fine. Then I added boost_serialization to my C::B project's linker library list and compiled the boost serialization example and it ran fine from the command line. I then built the dynamic and single thread variants additionally using
b2 --build-dir=build toolset=gcc --with-serialization --layout=tagged link=static,shared threading=multi,single stage
and got more libraries in my stage/lib directory, as expected. What puzzles me is that there is a .so file for every library, even those that should be static. Why is it there? What is it needed for?
When I now compile the project, the executable complains:
error while loading shared libraries: libboost_serialization.so.1.55.0: cannot open shared object file: No such file or directory
The library is definitely there and I might just need to add the path to it to LD_LIBRARY_PATH, but I want to link statically for now. How can I do that?
I also don't quite understand the library naming: I have some libboost_wserialization... libraries in my lib folder and the w prefix to serialization is not described in the library naming section of the current boost getting started page.
Your answers gave me a better understanding of what was going on - now I know where the boost_wserialization libraries came from. I turned out that after doing the second build, all present libraries were shared, and the static libraries were overwritten. That's why I got confused by the "extra" .so files for those libraries which previously were indeed static.

Ok, first question:
Why is there a boost_serialization and boost_wserialization library?
the wserialization library is wchar_t oriented. Put into a separate library because it may not actually be needed.
Why are there multiple shared/static libraries?
The reason you're seeing all those extra shared libraries is because you're invoking b2 with link=static,shared, which instructs boost to build the shared libraries as well as the static libraries. Additionally, adding in the thread=multi causes the building of the mt libraries, which are libraries that should be used when linking to multi-threaded applications.
Why am I getting the run-time link error about libboost_serialization.so.1.55.0?
By default, most unix/linux systems will prefer the use of shared libraries over static libraries when linking, so when you try to link it will prefer to use the shared libraries over the static ones. If you want to force the link of the static library rather than the shared one, you tell the compile-time linker to do that, using:
-Wl,-Bstatic -lboost_serialization -Wl,-Bdynamic
This will cause the linker to look for the static variant of the boost_serialization library, rather than the dynamic one.
Now, because you're using code::blocks, I'd have to look up how to specify these flags on a case-by-case basis, but the most sensible thing to do is to clean the boost build using ./b2 clean and then rebuild, specifying only link=static, then you should end up with only .a files, which will produce stand-alone executables again.
If you want to specify this option for code::blocks, you would need to put them into the Build Settings -> Linker settings -> Other Linker Options field for the project. Simply specifying the library in the libraries field will not work for this case. Additionally, forgetting to pass in the -Wl,-Bdynamic option will cause it to try to link in static versions of some platform libraries, which can lead to build failure if the library in question is not present.
If you want to avoid having to set LD_LIBRARY_PATH to run the binary, you can add the option -Wl,-rpath,/path/to/boost/libraries, to the linker flags which will cause the compiled program to search that directory when trying to resolve the location of libraries.

What puzzles me is that there is a .so file for every library, even
those that should be static. Why is it there? What is it needed for?
You apparently are using someone else's make file. I wrote my own. My build command does not create a ".so" (shared object library). It only creates the ".a" (archive library). The linker knows how to use either.
See man ar. The utility ar builds archives.
See man ld. The utility ld can build shared objects.
You might look for these utility invocations in your build sequence, or ask someone where they are and comment out the use of ld, as you most likely do not need both (and building both will extend your build time unnecessarily). Alternatively, you might temporarily rename the ld command, and try your build. When it can not find the ld command, you might get a useful hint as to where the ld is invoked.
In my make file, the commands look like the following. The comment char is a # at beginning of line. (The string expansions $(AR) and $(LD) allow the use of non-standard utilities.)
$(TARGET_ARCHIVE): $(OBJ)
#echo R10: $(TARGET_ARCHIVE) :from: $(OBJ)
$(AR) crs $(TARGET_ARCHIVE) $(OBJ)
# $(TARGET_OLB) : $(OBJ)
# #echo R00: $(TARGET_OLB) :from: $(OBJ)
# $(LD) -o $(TARGET_OLB) -r $(OBJ)
The archive (.a), when used, is linked directly to and included in your executable. When the executable is loaded, all the referenced symbols of the .a are already in it. (un-referenced symbols and code are not linked in)
The shared object (.so) is not directly linked, but rather your executable gets a handle (or perhaps a file name) to the .so. It is my belief that when your executable is loaded, the .so is not immediately loaded. The .so does not load until the first time your executable references a symbol that is in the .so. At that loading, your app will experience a delay, but probably this late loading is reasonable for most applications.
It is also possible that the .so is already loaded in system memory before you activated your process. In that case, when your executable first references a symbol in the .so, some system code will 'map' the existing in-memory .so to your application -- probably faster than loading it, but I suppose the big benefit is that a .so that is used / referenced by many processes need only be loaded once, saving memory space. The loaded .so has all of its symbols, even if your app does not need all of them.
In either case, your executable will be smaller with .so's, bigger with .a's, but the .so's have some small performance hit for each .so that needs to be loaded or mapped in. With 4 GB in my desktop, the desktop has never felt 'crowded'. It's swap has never been used (afaik). So I generally use .a's.
NOTE: When the linker has access to both an archive (.a) and a shared object (.so) file, the linker will use the .so (and ignore the .a). Probably you can override that preference, but I have not tried. I find it easier to simply move the archive (.a) into a separate (from the .so's) directory, and inform the linker via the -L build option.

How to compile a standalone OpenCV executable?

I compile my my OpenCV programs as follows:
g++ `pkg-config --cflags opencv --libs opencv` <filename>.cpp
It works perfectly on my computer. Can I complile the shared libraries alone with the program so that it can be run on other computers which doesnt have opencv on it? If so, how do I do it?

The program during compilation are dynamically linked to Shared Libraries (.so files) on our computer. The executable compiled use these shared libraries during run-time. But these shared libraries may not be present on other computers, hence might not be able to run the executable.
Solution to this will be to statically link Archive Libraries (.a files) instead of dynamically linking Shared Libraries. OpenCV does not distribute archive libraries as such. So one will have to compile archive library from the its source using cmake -DBUILD_SHARED_LIBS=OFF. This archive library can be used to create standalone executable.

Yes to some degree: lookup static linking. Your opencv copy must have .a versions of the libraries (not just .so you mention) and that is also true for any other dependencies.

No. You need to use static libraries.

Why doesn't Libtool want to link with a static library?

I want to build a shared library that uses ZipArchive using GNU Autotools but I'm having this problem:
Warning: linker path does not have real file for library -lziparch.
I have the capability to make that library automatically link in when
you link to this library. But I can only do this if you have a
shared version of the library, which you do not appear to have
because I did check the linker path looking for a file starting
with libziparch and none of the candidates passed a file format test
using a file magic. Last file checked: /usr/local/ZipArchive/ZipArchive/libziparch.a
The inter-library dependencies that have been dropped here will be
automatically added whenever a program is linked with this library
or is declared to -dlopen it.
If I build a static library or if I use a shared library of ZipArchive it works but the problem is that the makefile that comes with ZipArchive source code only builds a static library.
How can I force Libtool to link with a static library?

Generally, static archives are created with non-pic object files and they cannot be put into shared libraries.
What this message is telling you though, is that when a program links to YOUR library using Libtool, that -lziparch will get added to the link. So you don't need to change anything unless you're building a module for an interpreted language. In that case, you will have to build ZipArchive as a shared library. In addition, this wouldn't work on a platform like MS Windows where shared libraries (DLLs) have to resolve all their symbols at link time.
All that said, if your ziparch static lib is PIC code, you can use the -whole-archive flag when linking it to your library. This would be the least portable solution though.