question
How can I compile a shared library linking with version-independent boost shared library?
My cmakelists.txt is like as follows
find_package(Boost REQUIRED COMPONENTS serialization)
...
target_link_libraries(_omplpy PRIVATE ${Boost_LIBRARIES} ${otherdeps})
And, after compiling this, I checked dependency by ldd command and it shows that only the dependency of boost libraries are too specific (seems that version 1.71.0 is specified, though other libraries does not care about the minor version)
h-ishida#stone-jsk:~/python/ompl-python-thin$ ldd build/_omplpy.cpython-38-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffd34ca9000)
libboost_serialization.so.1.71.0 => /lib/x86_64-linux-gnu/libboost_serialization.so.1.71.0 (0x00007f208012f000)
libboost_filesystem.so.1.71.0 => /lib/x86_64-linux-gnu/libboost_filesystem.so.1.71.0 (0x00007f2080111000)
distir libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f20800ee000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f207ff0c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f207fdbd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f207fda0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f207fbae000)
/lib64/ld-linux-x86-64.so.2 (0x00007f20812a6000)
The problem is, boost libraries version are different for different ubuntu distributions, thus my compiled shard library _omplpy cannot be used in different distribution.
context (maybe unrelated)
I am trying to distribute a python package where a shared library linked with boost stuff is inside. Because python wheel (binary version of package) is only python-version (like 2.7, 3.8), os (mac, windows, ldistirinux), and archtecture dependent (like x86_64, aarch64), it seems impossible to distribute packages dependent on specific ubuntu distribution. For your information, the package mentioned is https://github.com/HiroIshida/ompl-thin-python and corresponding CMakeLists.txt is here https://github.com/HiroIshida/ompl-thin-python/blob/master/CMakeLists.txt
How can I compile a shared library linking with version-independent boost shared library?
You can't, because the different boost libraries are not ABI-compatible. If you somehow succeeded in linking your library with "version-independent boost", the result would be a runtime crash.
Read about the reasons for external library versioning here.
P.S.
I decided to build static boost library with -fPIC option and compile the target library with linking that.
That is much easier, but unless you are very careful with symbol hiding, this approach will cause runtime crashes (if you are lucky) as soon as someone tries to use your python package in program which uses a different version of boost libraries (due to symbol collision and ABI incompatibility). And if you are unlucky, symbol collision may cause other very hard to find bugs.
Related
I have an issue which sounds annoyingly simple, but I feel I've tried everything to resolve it and I'm out of ideas.
I have a program I wrote which uses boost/program_options.hpp using the include statement
#include <boost/program_options.hpp>
This program works on my laptop and now I am trying to put it onto a server where I don't have sudo privileges. On my laptop, where the whole thing works I have boost versions 1.56 and 1.58 and on the server I found 1.53.0. I didn't install or build this though.
The shared library file that exists on the server is:
/usr/lib64/libboost_program_options.so.1.53.0
Running ldd on that gives me the output:
ldd /usr/lib64/libboost_program_options.so.1.53.0
linux-vdso.so.1 => (0x00007ffc54a63000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fc0dcad8000)
libm.so.6 => /lib64/libm.so.6 (0x00007fc0dc7d5000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc0dc5bf000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc0dc1fe000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc0dd071000)
Which I assume means it's found all its dependencies.
I have made a symbolic link to my home directory to create a libboost_program_options.so link, which I guess means the -lboost_program_options linker flag should work.
The way I compile is by:
g++ -L/home/homeDir code.cpp -o code.o -std=c++11 -Wall -lboost_program_options
And all I get is:
fatal error: boost/program_options.hpp: No such file or directory
#include "boost/program_options.hpp"
^
compilation terminated.
I've tried this with quotes and <>, and without "boost/" etc, but I always get the same complaint. I assume this means that it finds the library as there is no complaint about the -lboost_program_options flag?
While it works on my machine, I may not be using the shared program options library at all. On my laptop I have both a static version of libboost_program_options in my library path which it may be using, and the program_options.hpp header outside of a library which may be in my include path. I feel like I should be able to use the shared library that's on the server and not annoy the server admin for the 5th time (things work really slowly with these things here).
I'm sorry if this is my oversight. I'm fairly new to boost, but I feel like I've tried everything to resolve this myself at the moment.
I'd be happy with any suggestions as to how I could change this... or if I'm just being a bit stupid with using shared libraries.
Thanks in advance.
EDIT:
After more research I feel like I'm just getting confused between static and shared libraries. I'm trying to link a shared library at compile time, which I guess doesn't make sense. Also including the header file of something which should not be linked at compile time makes little sense. My code is clearly designed for the static library.
The error message is produced by the pre-compiler which does not find the Boost header file (as it says in the error message). You need to set the correct include path (-I...) where the Boost library header files are found, obviously they are not under /usr/include.
Path to the Boost libraries (-L...) must be specified accordingly.
Finally: If you are linking against the Boost shared libraries on your system, then you should use the same version of Boost as installed on the server. Otherwise, link against the static libraries, this may make you executable larger, but it will run on any server, regardless of the Boost version installed there (even none).
sudo apt install libboost-program-options-dev
I am writing a shared library that I am deploying under Windows, Linux and Mac. On the Linux side, I am attempting to make sure that my library has as few dependencies as possible. I don't want the end developer to have to worry at all about what my library uses internally, and in particular I don't want to force them to install anything.
When I run ldd on my library at the moment, I see:
linux-gate.so.1 => (0xf57fe000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb773d000)
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb7654000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb74a1000)
/lib/ld-linux.so.2 (0xb7782000)
libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb745d000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb7440000)
This looks fairly reasonable to me, but some of these libraries I am not really sure what they are. Can anyone tell me whether this list of dependencies is reasonable, or whether I can get rid of some of these? With this list of dependencies, will my library run on a wide array of Linux configurations and distros? That is what I am aiming for, maximum portability.
When compiling, I am specifying the flag -static-libgcc. Are there any more flags I can specify to link in the C++ standard library as well, for example? Internally my library uses std::thread in C++11, but I don't want to force the application writer to necessarily have that available (if they are using an older version of GCC for instance).
Update:
I am now specifying -static-libstdc++, in addition to -static-libgcc. My dependency list now looks as follows:
linux-gate.so.1 => (0xf57fe000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb7737000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb7584000)
/lib/ld-linux.so.2 (0xb77a2000)
The only ones that cause me concern are libc.so.6, and linux-gate.so.1. I don't know what these are. Are they old, and if so have they remained backwards compatible for a long time? If so I will just keep them linking dynamically, but otherwise I have to continue investigating. Any tips would be appreciated.
linux-gate.so.1 is a virtual DSO, meaning it doesn't really exist. The best way to explain it is just to read this link Here.
To answer your question, I think your best choice is to continue linking dynamically to both of them and do a few different builds to target different distros. I have found that typically when you build for Ubuntu, it will work on several of the Debian linux systems.
I have CentOS 6.2 (64bit with gcc 4.4.6 as default). Unfortunately, my code only compiles with gcc 3.4.6, so I installed gcc separately (from source) under /home/rajat/local. On linking a simple "Hello World" program, I get the following.
>ldd a.out
linux-vdso.so.1 => (0x00007fff215ff000)
libstdc++.so.6 => /home/rajat/local/lib64/libstdc++.so.6 (0x00007f11853e7000)
libm.so.6 => /lib64/libm.so.6 (0x00000033be400000)
libgcc_s.so.1 => /home/rajat/local/lib64/libgcc_s.so.1 (0x00007f11851ce000)
libc.so.6 => /lib64/libc.so.6 (0x00000033bd000000)
/lib64/ld-linux-x86-64.so.2 (0x00000033bcc00000)
While stdc++ and gcc link to my 3.4.6 libraries, libm and libc still link to default libraries. Is this OK?? The 3.4.6 installation also did not produce libm or libc libraries?
Yes, that's ok.
The libc/libm is part of glibc, not the gcc compiler. libstdc++ on the other hand ls part of gcc.
Libraries in /lib and /usr/lib as well as their 64-bit counterparts are system-wide libraries. They are supposed to be compiler neutral and other compilers besides GCC can also link to them and they actually do, especially since libc is the only portable way to interface with the operating system kernel.
libc writers take great care of writing their header files in such a way as to make different compilers produce the same binary memory objects in order to properly interface with the code in the library. Besides, there is a well-defined OS ABI interface that all compilers adhere to in order to be compatible with one another. This might not be true for languages other than C, e.g. object files from different C++ compiler versions rarely go well with one another.
Compiler-specific libraries are not installed in /lib or /usr/lib and sometimes can be linked statically so that the dependency can be removed.
I have a requirement that I link all my libraries statically including libstdc++, libc, pthread etc. There is one omniorb library which I want to link dynamically.
Currently I have dynamically linked all the libraries.
ldd shows the following
linux-vdso.so.1 => (0x00007fff251ff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f291cc47000)
libomniDynamic4.so.1 (0x00007f291c842000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f291c536000)
libm.so.6 => /lib64/libm.so.6 (0x00007f291c2e0000)
libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f291c0d7000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f291bebf000)
libc.so.6 => /lib64/libc.so.6 (0x00007f291bb66000)
/lib64/ld-linux-x86-64.so.2 (0x00007f291ce63000)
librt.so.1 => /lib64/librt.so.1 (0x00007f291b95d000)
libomniORB4.so.1 (0x00007f291b6aa000)
libomnithread.so.3 (0x00007f291cf35000
I need ldd to show libomniDynamic4.so.1 as the only dynamically linked library.
How do I achieve this?
Trying to make a linux executable that runs on all distros eh? Good luck...But I digress...
You want to look at the -v flag output for g++. It shows the internal link commands executed by g++/ld. Specifically, you'll want to inspect the the final link command collect2 and all of its arguments. You can then specify the exact paths to the .a libs you want to link against. You'll also have to track down static libs of everything. My libstdc++.a is in /usr/lib/gcc/x86_64-linux-gnu/4.4/libstdc++.a
rant on: My biggest complaint about linux is the fractured state of executables. Why cant I compile a binary on one machine and copy it to another and run it!? Even the Ubuntu distros one release apart will produce binary files that cannot be run on the other due to libc/libstdc++ ABI incompatibilites
edit #1 I just wanted to add that The script on this page produces a .png of an executables .so dependencies. This is very useful when attempting to do what you describe.
Be aware ldd <exename> will list all dependencies down the chain, not just immediate dependencies of the executable. So even if your executable only depended upon omniorb.so, but omniorb.so depended upon, libphread.so, ldd's output would list that. Look up the manpage of readelf to find only the immediate dependencies of a binary.
One other item that to be aware of. if omniorb.so depends upon libstdc++.so, you'll have no choice but to be dependant on that same lib. Otherwise ABI incompatibilities will break RTTI between your code and omniorb's code.
I need ldd to show libomniDynamic4.so.1 as the only dynamically linked library.
That is impossible.
First, ldd will always show ld-linux-x86-64.so.2 for any (x86_64) binary that requires dynamic linking. If you use dynamic linking (which you would with libomniDynamic4.so.1), then you will get ld-linux-x86-64.so.2.
Second, linux-vdso.so.1 is "injected" into your process by the kernel. You can't get rid of that either.
Next, the question is why you want to minimize use of dynamic libraries. The most common reason is usually mistaken belief that "mostly static" binaries are more portable, and will run on more systems. On Linux this is the opposite of true.
If in fact you are trying to achieve a portable binary, several methods exist. The best one so far (in my experience) has been to use apgcc.
It is very difficult to build a single binary that runs on a lot of Linux distros and linking statically is not the key point.
Please note that a binary built with an older glibc version--i.e., an old Linux distro--may run on newer Linux distros as well. This works because glibc is back-compatible.
A possible way to attain the desired result is:
compile the binary on an old Linux OS
find out all the required libraries for your compiled binary using the command ldd or lsof
(when running) on the binary, details here
copy the required libraries of the old Linux OS in a 'custom-lib' folder
always bundle/release this custom-lib folder with your binary
create a bash script that puts the custom-lib folder on top of the folders list in LD_LIBRARY_PATH environment variable, and then invokes your binary.
In this way, by executing the binary with the bash script, I was able to execute binaries on a wide range of embedded devices with very different Linux versions.
But there are always problematic cases where this fails.
Please note, I always tested this with cli applications/binaries.
Other possible ways..
There also seems to be elegant ways to compile glibc-back-compatible binaries, for example this that seems to compile binaries compatible with older ABI. But I have not checked this route.
when linking, use -static before specifying the libraries you want to link statically to, and use -dynamic before the libraries you want to link dynamically to. You should end up with a command line looking like this:
g++ <other options here> -dynamic -lomniDynamic4 -static -lpthread -lm -lgomp <etc>
Of course, you'll need .a versions of the libraries you want to link statically (duh).
I know this might be a very stupid question but I am new to compiled languages (my domain is mostly scripting languages like PHP, Python or JavaScript).
I am learning C++ for one project where it is the only language I can use.
I wrote a program in Ubuntu 10.10 and then compiled it. I can run the generated binary file from cmd like this and it works:
sudo ./compiled-program
But, I have used some external libraries in the program (OpenCV). Does that mean that all computers where I will run the program will have to have OpenCV installed? Or is OpenCV bundled inside the compiled binary file? Will it work on PCs without OpenCV installed?
You should read a few things about libraries, and particularly what makes the difference between static and dynamic libraries. To quote the basic definitions so you get the point :
A static library, also known as an
archive, consists of a set of routines
which are copied into a target
application by the compiler, linker,
or binder, producing object files and
a stand-alone executable file.
[...]
Dynamic linking involves loading the
subroutines of a library (which may be
referred to as a DLL, especially under
Windows, or as a DSO (dynamic shared
object) under Unix-like systems) into
an application program at load time or
runtime, rather than linking them in
at compile time.
Not a stupid question at all!
The "normal" way this works - is that your program has been linked against a "Shared Library" - in which case, yes, the user needs the OpenCV (or whatever bundle includes the shared library) to work.
If you compiled as a static executable, (using the -static) flag, then it, and all libraries would be included directly into your executable, making a bit of a larger executable that wastes more memory, because it isn't using a shared library.
There are ways that you could compile your program to link only your OpenCV libraries as static - but that only can be done if the bundle included a static library ".a" vs. a shared one ".so".
If you had to build your code against dependencies, like OpenCV, it depends on if you did static or dynamic linking.
See here which has sections covering these ideas: http://en.wikipedia.org/wiki/Library_(computing)
For starters, try doing this on the command line:
ldd compiled-program
You will get output like this (as an example, I did ldd on my python binary in /usr/bin):
birryree#lilun:/usr/bin$ ldd python
linux-gate.so.1 => (0xb7ff7000)
libpthread.so.0 => /lib/i686/cmov/libpthread.so.0 (0xb7fd5000)
libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb7fd1000)
libutil.so.1 => /lib/i686/cmov/libutil.so.1 (0xb7fcd000)
libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb7f82000)
libcrypto.so.0.9.8 => /usr/lib/i686/cmov/libcrypto.so.0.9.8 (0xb7e2a000)
libz.so.1 => /usr/lib/libz.so.1 (0xb7e16000)
libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7df0000)
libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb7caa000)
/lib/ld-linux.so.2 (0x80000000)
Python wants a lot of additional stuff, like libssl (part of OpenSSL), the GNU C library (libc), and some others.
Now if you're going to be moving this thing around to other systems, you either hope they have an environment similar to yours, distribute it as source and use something like the autotools/GNU Build System to build it, or you can forego all that and statically link everything into your binary, which will bring in all the stuff your executable needs without need for a dynamic link.
If you've "compiled against" OpenCV, then machines running your app need it too. You need to copy the libs when you install your app, or ensure that they're already installed.
It depends on wether you are compiling against a shared (Dynamic) library or compiling it into your executable (compiling against a static library). If you are compiling against a shared library you need to distribute the shared library ... otherwise .. you don't.
There are two kinds of libraries, static and dynamically loaded.
Statically loaded libraries are joined with your binary file, while dynamically loaded libraries are loaded at runtime.
It depends on whether or not the executable is statically-built or dynamically linked. In a statically-built executable it is the case that the library files the executable needs are compiled into the executable and there isn't an need to carry around additional library files. In a dynamically linked executable it is the case that the library files the executable needs are linked at runtime and therefore a copy of the library files are needed at runtime.