Custom-built and default boost libraries on the same system - c++

Question

I've built some boost libraries (particularly system and serialization) with the _GLIBCXX_USE_CXX11_ABI=0 macro (https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html) due to the fact that I'm using boost::asio to share information (by mean) between my code and an old application (Webots 7.4.3, pre GCC-5). However, I already have standard boost libraries installed in my system (required by several other applications). I've opted to install the custom-compiled versions to /usr/local/lib (while the default versions are kept in /usr/lib) and renamed the libraries with the suffix _oldabi:
$ ls -lash /usr/local/lib | grep boost
1,1M -rw-r--r-- 1 root root 1,1M jun 21 14:17 libboost_serialization_oldabi.a
416K -rw-r--r-- 2 root root 415K jun 21 14:17 libboost_serialization_oldabi.so
416K -rw-r--r-- 2 root root 415K jun 21 14:17 libboost_serialization_oldabi.so.1.64.0
0 lrwxrwxrwx 1 root root 39 jun 21 14:18 libboost_serialization.so.1.64.0 -> libboost_serialization_oldabi.so.1.64.0
48K -rw-r--r-- 1 root root 46K jun 21 14:17 libboost_system_oldabi.a
20K -rw-r--r-- 2 root root 20K jun 21 14:17 libboost_system_oldabi.so
20K -rw-r--r-- 2 root root 20K jun 21 14:17 libboost_system_oldabi.so.1.64.0
0 lrwxrwxrwx 1 root root 32 jun 21 14:18 libboost_system.so.1.64.0 -> libboost_system_oldabi.so.1.64.0
708K -rw-r--r-- 1 root root 706K jun 21 14:17 libboost_wserialization_oldabi.a
300K -rw-r--r-- 2 root root 299K jun 21 14:17 libboost_wserialization_oldabi.so
300K -rw-r--r-- 2 root root 299K jun 21 14:17 libboost_wserialization_oldabi.so.1.64.0
0 lrwxrwxrwx 1 root root 40 jun 21 14:18 libboost_wserialization.so.1.64.0 -> libboost_wserialization_oldabi.so.1.64.0
The code works well, socket communications work as intended. However, when trying to set up the code on another computer without the default boost libraries installed (only custom-compiled versions of boost::system and boost::serialization), I get (system:9) Bad file descriptor when performing write/read operations. While trying to figure out the problem, I went back for the original computer and by looking at /proc/XXXX/maps I noticed that my executable was actually loading the default boost libraries under /usr/lib at runtime, despite being linked with the custom-built versions at compilation.
Thus my issues are 1) how to make the system prioritize /usr/local/lib over /usr/lib or force it to identify the _boost_oldabi suffix as different libraries; and 2) figure out why does adding the GCC ABI macro seems to break the code; The source code, make files and libraries are exactly the same, the only thing that differs between the computers is the OS (Arch Linux and Ubuntu).

Answer 1

Preamble: If you need your custom boost libraries only for a single tool, it is better to install them into a place which is specific to this single tool. Installing custom libs into a system-wide location (/usr/local/lib) is not a very good idea, because
It increases the risk that also your system-wide apps will use the bad one.
If you want once cleanup /usr/local/lib, you won't be able to know, what is using the libs there.
It would be better if you would make them available only for your software. For example, if your software is in /srv/niceproject/bin, then the libs could exist in /srv/niceproject/lib.
Note, the shared library handling is enough smart in Linux to handle libraries with different versions available, all the binaries will map in the libs according to their needed versions. But, it depends on special symbols inserted by the linker into the binaries. If you change only the compiler flags, these symbols may be the same, thus a false lib may be linked in.
You can prioritize /usr/local/lib by changing the shared library order in /etc/ld.so.conf and in /etc/ld.so.conf.d, and then calling an ldconfig. To get the list of the currently visible libraries, use ldconfig -p.
There is a lot, so an ldconfig -p|grep libboost could be a nicer result.
The shared library directory order is affected also by the LD_LIBRARY_PATH environment variable, what can be specified for every process. Thus, calling your tool with
LD_LIBRARY_PATH=/srv/niceproject/lib /srv/niceproject/bin/mytool
will call mytool on a way, that it tries to find the libs in /srv/niceproject/lib first, and looks the system-wide ones only after it. It is a colon-separated list of pathes, like $PATH.
Linking your binary with a library given by an absolute path will hardcode the absolute path into the library. So:
gcc -o mytool -l/usr/local/lib/libboost_oldapi_anything.so
Also this is a dirty solution in general, but it would solve your problem.
You can see, what exactly an actual library is loading, with the ldd tool:
# ldd /bin/bash
linux-vdso.so.1 => (0x00007ffd1c1cf000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f4baa499000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4baa295000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4ba9eca000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4baa6d0000)
ldd is a special binary, you can specify any LD_LIBRARY_PATH, it won't affect it (but it will affect the binary you are checking with it). In my example, with an LD_LIBRARY_PATH=/srv/niceproject/lib ldd /srv/niceproject/bin/mytool will show, how the mytool is affected by this environment variable.