Library not linked even though present in link targets - c++

I'm trying to build an application using cmake but the executable is unable to link to the libraries, the ldd of my executable looks like this
linux-vdso.so.1 (0x00007ffd447a0000)
libavcodec.so.58 => /usr/local/ffmpeg/libavcodec.so.58 (0x00007f5673121000)
libavformat.so.58 => /usr/local/ffmpeg/libavformat.so.58 (0x00007f5672cdb000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5671ffb000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5671de3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56719f2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5671654000)
/lib64/ld-linux-x86-64.so.2 (0x00007f567629b000)
libswresample.so.3 => not found
libavutil.so.56 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5671435000)
libavutil.so.56 => not found
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5671231000)
while in my make --trace the library is present
/usr/bin/c++ -I/home/arunabh-compute/sensor-drivers -isystem
/usr/local/include -isystem /usr/local/GenICam/include -Wall -O3 -
std=gnu++14 -o CMakeFiles/save_cpp.dir/stdafx.cpp.o -c /home/arunabh-
compute/sensor-drivers/stdafx.cpp
CMakeFiles/save_cpp.dir/build.make:178: update target 'save_cpp' due
to: CMakeFiles/save_cpp.dir/link.txt
CMakeFiles/save_cpp.dir/Cpp_Save.cpp.o
CMakeFiles/save_cpp.dir/stdafx.cpp.o
CMakeFiles/save_cpp.dir/build.make /usr/local/ffmpeg/libavcodec.so
/usr/local/ffmpeg/libavcodec.so.58
/usr/local/ffmpeg/libavcodec.so.58.18.100
/usr/local/ffmpeg/libavdevice.so /usr/local/ffmpeg/libavdevice.so.58
/usr/local/ffmpeg/libavdevice.so.58.3.100
/usr/local/ffmpeg/libavfilter.so /usr/local/ffmpeg/libavfilter.so.7
/usr/local/ffmpeg/libavfilter.so.7.16.100
/usr/local/ffmpeg/libavformat.so /usr/local/ffmpeg/libavformat.so.58
/usr/local/ffmpeg/libavformat.so.58.12.100
/usr/local/ffmpeg/libavutil.so /usr/local/ffmpeg/libavutil.so.56
/usr/local/ffmpeg/libavutil.so.56.14.100
/usr/local/ffmpeg/libswresample.so
/usr/local/ffmpeg/libswresample.so.3
/usr/local/ffmpeg/libswresample.so.3.1.100
/usr/local/ffmpeg/libswscale.so /usr/local/ffmpeg/libswscale.so.5
/usr/local/ffmpeg/libswscale.so.5.1.100
even though libswresample.so.3, libavutil.so.56 is clearly present in the make --trace the application doesn't link to it for some reason, additionally if i put the /usr/local/ffmpeg folder in my LD_LIBRARY_PATH everything starts working just fine. Additionally libavcodec.so.58 is found. Trying to understand what's happening and how to fix it. Thanks.

By linking with these libraries, you marked them as needed in your binary, without recording any paths.
By default, your system only looks for dynamic libraries in a handful of locations.
You have a few options:
Add /usr/local/ffmpeg to a file in /etc/ld.so.conf.d/ (don't forget to call ldconfig). This will tell your dynamic linker (ld.so) to look there.
Add /usr/local/ffmpeg to the LD_LIBRARY_PATH environment variable as you did. You can make this permanent by editing /etc/environment on some machines. This has the same effect, but can be overridden by users. Most games on Linux rely on this to find the bundled libraries, for example, but this does necessitate a launcher script.
Link your binary with -rpath=/usr/local/ffmpeg. This informs the dynamic linker that you would like it to look there too. You can do this in CMake by setting the INSTALL_RPATH target property. You probably want to set BUILD_WITH_INSTALL_RPATH as well, so you do not have to set LD_LIBRARY_PATH during development or set a separate BUILD_RPATH.
Finally, you can just move the ffmpeg libraries to a directory in the default search path. However, this may clash with system libraries and requires root access. This is more a last-resort or convenience-over-everything approach.

Related

CLion can't find shared library when running an executable

I am working on a project. I've been using a simple editor so far and my own Makefile to build it. I would like to switch to CLion, though.
According to this question you can tell CMake to run your Makefile. So my CMake.txt looks like this:
cmake_minimum_required(VERSION 3.6)
project(rekotrans_testbed_simulator)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
add_custom_target(rekotrans_testbed_simulator COMMAND make -C ${rekotrans_testbed_simulator_SOURCE_DIR} CLION_EXE_DIR=${PROJECT_BINARY_DIR})
It builds fine. I also set the working directory and pointed at the correct executable.
In my project I test using cppunit 1.13. However it can't find the shared library:
/home/kunterbunt/dev/comnets/git-repository/rekotrans-testbed-simulator/rekotrans-testbed-simulator-tests: error while loading shared libraries: libcppunit-1.13.so.0: cannot open shared object file: No such file or directory
LD_LIBRARY_PATH points to
echo $LD_LIBRARY_PATH
/usr/local/lib
and /usr/local/lib contains the library:
ls /usr/local/lib/
libcppunit-1.13.so.0# libcppunit-1.13.so.0.0.2* libcppunit.a libcppunit.la* libcppunit.so# pkgconfig/
ldd shows this:
ldd /home/kunterbunt/dev/comnets/git-repository/rekotrans-testbed-simulator/rekotrans-testbed-simulator-tests
linux-vdso.so.1 (0x00007ffc257e8000)
libboost_thread.so.1.63.0 => /usr/lib/libboost_thread.so.1.63.0 (0x00007f1c73254000)
libboost_system.so.1.63.0 => /usr/lib/libboost_system.so.1.63.0 (0x00007f1c73050000)
libboost_date_time.so.1.63.0 => /usr/lib/libboost_date_time.so.1.63.0 (0x00007f1c72e3f000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1c72c22000)
libboost_program_options.so.1.63.0 => /usr/lib/libboost_program_options.so.1.63.0 (0x00007f1c729a4000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f1c727a0000)
libcppunit-1.13.so.0 => /usr/local/lib/libcppunit-1.13.so.0 (0x00007f1c72563000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f1c721db000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f1c71ed7000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f1c71cc0000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f1c71922000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f1c7171a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1c7347c000)
So why can't CLion find it? Everything works if I run the binary from console.
Have a look at How to set the environmental variable LD_LIBRARY_PATH in linux:
If you add your custom library path to the LD configuration, then CLion will find your libraries automatically and you don't have to add them to the run configurations.
On Ubuntu/Debian, you can configure LD by creating a new .conf file
sudo nano /etc/ld.so.conf.d/myLocalLibs.conf
that simply contains the path to your libraries: /usr/local/lib. Finally, call
sudo ldconfig
to update the LD configuration.
Note that on some systems (Ubuntu/Debian) you cannot set the LD_LIBRARY_PATH in /etc/profile or /etc/environment:
Since Ubuntu 9.04 Jaunty Jackalope, LD_LIBRARY_PATH cannot be set in
$HOME/.profile, /etc/profile, nor /etc/environment files. You must use
/etc/ld.so.conf.d/*.conf configuration files. See Launchpad bug #366728
for more information.
(help.ubuntu.com)
As oLen correctly pointed out, CLion doesn't seem to start as my user when launched via the GUI (Gnome in my case). I don't know as what it starts, but setting LD_LIBRARY_PATH=/usr/local/lib in /etc/profile and rebooting (or resourcing) it makes it work - in a nutshell the variable wasn't set for whatever user is running CLion.
Another way is Run -> Edit Configurations -> (select your application) -> Environment variables. Here you can manually set LD_LIBRARY_PATH to whatever you need, in my case to /usr/local/lib.
As an alternative to the accepted answer you can go to Run -> Edit Configurations -> Templates, choose CMake Application (and/or Google Test) and set Environment variables: to:
LD_LIBRARY_PATH=/usr/local/gcc-latest/lib64
Any new application created from now on will inherit these settings.

How can I programatically determine where my C++ runtime libraries are?

I'm building C++11 with GCC 4.8 on CentOS 6 using "hax", then deploying on arbitrary CentOS 6 targets (on which anything C++-related out of the box will have been built as C++03 with GCC 4.3) (ref).
To make this work, I'm going to ship all of my third-party libs as well as the g++ runtimes, and rpath my executables so the newer libs are assuredly found in the proper place. For the runtimes, by my count I need to ship libstdc++ and libgcc_s. But I need to know where they are on my build system so that I can package them up.
Is there some neat way I can query for their location from within my packaging script?
(If the best approach is too awkward I'll just link them statically, but I'd like to avoid that if I can as my project includes several executables. Also if I were to statically link everything I believe I'd run the risk of GPL-ing my whole project, e.g. by statically linking the MySQL C API via my C++ MySQL wrapper lib. Could do a mixture of both, I suppose, though some sources warn against this…)
For bonus points, do I need to add anything to this list, out of libssl, libcrypto, libm, libpthread, libc, librt, libz and ld-linux-x86-64?
If I understand correctly, you have already built your binaries and just want to get a list of runtime libraries to package them along with binaries? You can try using ldd for this, like:
> ldd /usr/bin/ls
linux-vdso.so.1 (0x00007ffe76dd2000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc97131f000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007fc97111a000)
libacl.so.1 => /lib64/libacl.so.1 (0x00007fc970f10000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc970b68000)
libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fc970902000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc9706fd000)
/lib64/ld-linux-x86-64.so.2 (0x000055c4ba4ed000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007fc9704f8000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc9702db000)
This way you will see all libraries needed except, of course, the ones that are used via dlopen().
In my Makefile:
GCC_INSTALL_DIR := $(shell $(CXX) -print-search-dirs | grep install | cut -d' ' -f2)
…then my main build target will perform:
ln -sf $(GCC_INSTALL_DIR)/libstdc++.so $(BIN_DIR)/deps/
…and I can dump everything in $(BIN_DIR)/deps into the right place on install.
I think.

Conflicting boost versions

I have compiled boost::system and boost::serialization from with the older GCC ABI (https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html) due to some older third-party code I'm using.
I had them built to /usr/local/lib which is already set as a valid path to the linker (I use other libraries there as well), and renamed them to:
$ ls /usr/local/lib/libboost_*
/usr/local/lib/libboost_serialization_old_abi.so
/usr/local/lib/libboost_serialization_old_abi.so.1.60.0
/usr/local/lib/libboost_system_old_abi.so
/usr/local/lib/libboost_system_old_abi.so.1.60.0
/usr/local/lib/libboost_wserialization_old_abi.so
/usr/local/lib/libboost_wserialization_old_abi.so.1.60.0
Default, mainstream boost libraries are under /usr/lib as usual. What happens is that when I link any piece of code to these specific libraries with the custom flags -lboost_system_old_abi and -lboost_serialization_old_abi, the resulting binary will be linked to the default boost libraries:
$ ldd darwin_socket
linux-vdso.so.1 (0x00007ffd137ea000)
/usr/local/webots/resources/projects/robots/darwin-op/libraries/darwin/libdarwin.so (0x00007fcb9edaa000)
libipsocket.so.1 => /usr/local/lib/libipsocket.so.1 (0x00007fcb9eb7b000)
libboost_system.so.1.60.0 => /usr/lib/libboost_system.so.1.60.0 (0x00007fcb9e977000)
libboost_serialization.so.1.60.0 => /usr/lib/libboost_serialization.so.1.60.0 (0x00007fcb9e739000)
libController.so => not found
libCppController.so => not found
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fcb9e3b7000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fcb9e0b2000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fcb9de9c000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fcb9dafb000)
/lib64/ld-linux-x86-64.so.2 (0x00007fcb9efc1000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007fcb9d8f3000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fcb9d6d6000)
which is quite strange, because, if use original -lboost_system and -lboost_serialization flags, gcc can't even link to the default boost due to new/old ABI incompatibilities.
So what exactly is happening here?
The problem is that just renaming your custom built libraries is not enough. The library name is embedded into the library as soname (you can see it with the readelf -d command) and is used when your application is liked with the library. Basically, the sonames from the custom libraries are put as the dependencies into your application binary, and since they are the same as the official Boost library names, the wrong binaries are loaded in run time.
You have to make sure the custom built Boost libraries are properly named in the build process. You can try doing this by adding --buildid=old_abi option to your b2 command line.

Error when using python to wrap up my c++ code into *.so file

I have a small project wrote in c++, and now I am going to build it into *.so file.
it works fine in the building process, no errors and generate the *.so I want.
file I use for building *.so file
module1 = Extension('classmod',
include_dirs=['/usr/lib/python2.7/dist-packages/numpy/','/home/******/OpenCV/opencv/release/include','/home/******/OpenCV/opencv/release/include/opencv','/home/******/OpenCV/opencv/release/include/opencv2','/home/******/modulepipe/eigen-eigen-b23437e61a07'],
library_dirs=['/usr/local/lib/'],
libraries=['opencv_calib3d','opencv_contrib','opencv_core','opencv_features2d','opencv_flann','opencv_gpu',
'opencv_highgui','opencv_imgproc','opencv_legacy','opencv_ml','opencv_nonfree','opencv_objdetect',
'opencv_photo','opencv_stitching','opencv_superres','opencv_ts','opencv_video','opencv_videostab'],
sources=['classmodule.cpp','patch_match.cpp','compute_jaccard.cpp','find_diff.cpp','refine_match.cpp','tps.cpp','book_pipeline_single.cpp','imgfeatures.c','kdtree.c','minpq.c','sift.c','utils.c'],
language='c++',
extra_compile_args=['-O3'])
setup(name='classmod',
version='1.0',
description='blah',
author='blah',
url='blah',
ext_modules=[module1]
)
However when I am trying to load this .so lib use lbb, it tells me about can not find several opencv *.so lib.
command I use for testing:
~/CDH/modulepipe$ ldd classmod.so
linux-gate.so.1 => (0xb77b4000)
libopencv_calib3d.so.2.4 => not found
libopencv_core.so.2.4 => not found
libopencv_features2d.so.2.4 => not found
libopencv_highgui.so.2.4 => not found
libopencv_imgproc.so.2.4 => not found
libopencv_nonfree.so.2.4 => not found
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb7690000)
libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb7663000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb7645000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb762a000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb7480000)
/lib/ld-linux.so.2 (0xb77b5000)
This is the problem I have. Can anyone help me on explaining this?
I had this "lib not found problem" before, and it was in the build phase. But in this case, in the building process it passed, which means ldd could find *.so's it needed. So how it happens that, when I am trying to run "ldd classmod.so", it could not find the *.so's.
Your loader can't find those shared objects. Either place them somewhere the loader expects to find them, modify the loader configuration (man 8 ldconfig) to look for shared objects where they are, or use one of the loader's environment variables (man 8 ld.so) to specify where to look for them.

Qt program does not detect libraries

I built a Qt application and included the necessary libraries needed in the build directory. When I try to run the application from a different computer, it doesn't work.
This is the ldd output:
linux-vdso.so.1 => (0x00007fff8c7fe000)
libQt5Widgets.so.5 => not found
libQt5Gui.so.5 => not found
libQt5Core.so.5 => not found
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5cb4143000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5cb3f2d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5cb3b6c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5cb3870000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5cb4456000)
The files not found are in the same directory as the binary. How do I make them detectable?
On Linux current binary directiry is not the place where the loader looks for shared libraries. See this question: https://serverfault.com/questions/279068/cant-find-so-in-the-same-directory-as-the-executable
You can add the following to your .pro file to force the dynamic linker to look in the same directory as your Qt application at runtime in Linux :
unix:{
# suppress the default RPATH if you wish
QMAKE_LFLAGS_RPATH=
# add your own with quoting gyrations to make sure $ORIGIN gets to the command line unexpanded
QMAKE_LFLAGS += "-Wl,-rpath,\'\$$ORIGIN\'"
}