I have a numerical C++ model of thousands of LOC that outputs different results on different machines and I want to fully understand why that is. Those differences start out in like the 14th significant digit but then propagate to produce quite different results.
I've spent many days on this now and I finally came to this point: When I build it on my machine and copy it to machine B, I can get the exact same results on both machines if I compiled it via
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXE_LINKER_FLAGS="-static" -DCMAKE_FIND_LIBRARY_SUFFIXES=".a" ..
So that is already great. But I want to know more specifically what causes the differences. So I checked which libraries are possible culprits:
$ ldd my_software
linux-vdso.so.1 (0x00007ffdbd99c000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f08720b4000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0871d16000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0871afe000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f087170d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0872724000)
So my idea was that I would ideally find one library that changes it and then think about what part of this library that is and if I can do something about this.
Only linking a single library statically worked for, e.g., libgcc:
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXE_LINKER_FLAGS="-static-libgcc" ..
But the results were still different. So that was not it. The problem now is, that I seem to not be able to link libc or libm statically:
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXE_LINKER_FLAGS="-static-libm" .. # works
cd ..
cmake --build statically_linked_m/ --target my_software
...
[100%] Linking CXX executable my_software
g++: error: unrecognized command line option ‘-static-libm’; did you mean ‘-static-libgo’?
So I am wondering, why can't I do that? I mean, it apparently got linked statically when I compiled it with the -DCMAKE_EXE_LINKER_FLAGS="-static" -DCMAKE_FIND_LIBRARY_SUFFIXES=".a" flags. So why is this not working?
I hope I made my issue clear. Any pointers are highly appreciated!
Related
I'm trying to build an application using cmake but the executable is unable to link to the libraries, the ldd of my executable looks like this
linux-vdso.so.1 (0x00007ffd447a0000)
libavcodec.so.58 => /usr/local/ffmpeg/libavcodec.so.58 (0x00007f5673121000)
libavformat.so.58 => /usr/local/ffmpeg/libavformat.so.58 (0x00007f5672cdb000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5671ffb000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5671de3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56719f2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5671654000)
/lib64/ld-linux-x86-64.so.2 (0x00007f567629b000)
libswresample.so.3 => not found
libavutil.so.56 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5671435000)
libavutil.so.56 => not found
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5671231000)
while in my make --trace the library is present
/usr/bin/c++ -I/home/arunabh-compute/sensor-drivers -isystem
/usr/local/include -isystem /usr/local/GenICam/include -Wall -O3 -
std=gnu++14 -o CMakeFiles/save_cpp.dir/stdafx.cpp.o -c /home/arunabh-
compute/sensor-drivers/stdafx.cpp
CMakeFiles/save_cpp.dir/build.make:178: update target 'save_cpp' due
to: CMakeFiles/save_cpp.dir/link.txt
CMakeFiles/save_cpp.dir/Cpp_Save.cpp.o
CMakeFiles/save_cpp.dir/stdafx.cpp.o
CMakeFiles/save_cpp.dir/build.make /usr/local/ffmpeg/libavcodec.so
/usr/local/ffmpeg/libavcodec.so.58
/usr/local/ffmpeg/libavcodec.so.58.18.100
/usr/local/ffmpeg/libavdevice.so /usr/local/ffmpeg/libavdevice.so.58
/usr/local/ffmpeg/libavdevice.so.58.3.100
/usr/local/ffmpeg/libavfilter.so /usr/local/ffmpeg/libavfilter.so.7
/usr/local/ffmpeg/libavfilter.so.7.16.100
/usr/local/ffmpeg/libavformat.so /usr/local/ffmpeg/libavformat.so.58
/usr/local/ffmpeg/libavformat.so.58.12.100
/usr/local/ffmpeg/libavutil.so /usr/local/ffmpeg/libavutil.so.56
/usr/local/ffmpeg/libavutil.so.56.14.100
/usr/local/ffmpeg/libswresample.so
/usr/local/ffmpeg/libswresample.so.3
/usr/local/ffmpeg/libswresample.so.3.1.100
/usr/local/ffmpeg/libswscale.so /usr/local/ffmpeg/libswscale.so.5
/usr/local/ffmpeg/libswscale.so.5.1.100
even though libswresample.so.3, libavutil.so.56 is clearly present in the make --trace the application doesn't link to it for some reason, additionally if i put the /usr/local/ffmpeg folder in my LD_LIBRARY_PATH everything starts working just fine. Additionally libavcodec.so.58 is found. Trying to understand what's happening and how to fix it. Thanks.
By linking with these libraries, you marked them as needed in your binary, without recording any paths.
By default, your system only looks for dynamic libraries in a handful of locations.
You have a few options:
Add /usr/local/ffmpeg to a file in /etc/ld.so.conf.d/ (don't forget to call ldconfig). This will tell your dynamic linker (ld.so) to look there.
Add /usr/local/ffmpeg to the LD_LIBRARY_PATH environment variable as you did. You can make this permanent by editing /etc/environment on some machines. This has the same effect, but can be overridden by users. Most games on Linux rely on this to find the bundled libraries, for example, but this does necessitate a launcher script.
Link your binary with -rpath=/usr/local/ffmpeg. This informs the dynamic linker that you would like it to look there too. You can do this in CMake by setting the INSTALL_RPATH target property. You probably want to set BUILD_WITH_INSTALL_RPATH as well, so you do not have to set LD_LIBRARY_PATH during development or set a separate BUILD_RPATH.
Finally, you can just move the ffmpeg libraries to a directory in the default search path. However, this may clash with system libraries and requires root access. This is more a last-resort or convenience-over-everything approach.
I'm building C++11 with GCC 4.8 on CentOS 6 using "hax", then deploying on arbitrary CentOS 6 targets (on which anything C++-related out of the box will have been built as C++03 with GCC 4.3) (ref).
To make this work, I'm going to ship all of my third-party libs as well as the g++ runtimes, and rpath my executables so the newer libs are assuredly found in the proper place. For the runtimes, by my count I need to ship libstdc++ and libgcc_s. But I need to know where they are on my build system so that I can package them up.
Is there some neat way I can query for their location from within my packaging script?
(If the best approach is too awkward I'll just link them statically, but I'd like to avoid that if I can as my project includes several executables. Also if I were to statically link everything I believe I'd run the risk of GPL-ing my whole project, e.g. by statically linking the MySQL C API via my C++ MySQL wrapper lib. Could do a mixture of both, I suppose, though some sources warn against this…)
For bonus points, do I need to add anything to this list, out of libssl, libcrypto, libm, libpthread, libc, librt, libz and ld-linux-x86-64?
If I understand correctly, you have already built your binaries and just want to get a list of runtime libraries to package them along with binaries? You can try using ldd for this, like:
> ldd /usr/bin/ls
linux-vdso.so.1 (0x00007ffe76dd2000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc97131f000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007fc97111a000)
libacl.so.1 => /lib64/libacl.so.1 (0x00007fc970f10000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc970b68000)
libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fc970902000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc9706fd000)
/lib64/ld-linux-x86-64.so.2 (0x000055c4ba4ed000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007fc9704f8000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc9702db000)
This way you will see all libraries needed except, of course, the ones that are used via dlopen().
In my Makefile:
GCC_INSTALL_DIR := $(shell $(CXX) -print-search-dirs | grep install | cut -d' ' -f2)
…then my main build target will perform:
ln -sf $(GCC_INSTALL_DIR)/libstdc++.so $(BIN_DIR)/deps/
…and I can dump everything in $(BIN_DIR)/deps into the right place on install.
I think.
I have compiled boost::system and boost::serialization from with the older GCC ABI (https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html) due to some older third-party code I'm using.
I had them built to /usr/local/lib which is already set as a valid path to the linker (I use other libraries there as well), and renamed them to:
$ ls /usr/local/lib/libboost_*
/usr/local/lib/libboost_serialization_old_abi.so
/usr/local/lib/libboost_serialization_old_abi.so.1.60.0
/usr/local/lib/libboost_system_old_abi.so
/usr/local/lib/libboost_system_old_abi.so.1.60.0
/usr/local/lib/libboost_wserialization_old_abi.so
/usr/local/lib/libboost_wserialization_old_abi.so.1.60.0
Default, mainstream boost libraries are under /usr/lib as usual. What happens is that when I link any piece of code to these specific libraries with the custom flags -lboost_system_old_abi and -lboost_serialization_old_abi, the resulting binary will be linked to the default boost libraries:
$ ldd darwin_socket
linux-vdso.so.1 (0x00007ffd137ea000)
/usr/local/webots/resources/projects/robots/darwin-op/libraries/darwin/libdarwin.so (0x00007fcb9edaa000)
libipsocket.so.1 => /usr/local/lib/libipsocket.so.1 (0x00007fcb9eb7b000)
libboost_system.so.1.60.0 => /usr/lib/libboost_system.so.1.60.0 (0x00007fcb9e977000)
libboost_serialization.so.1.60.0 => /usr/lib/libboost_serialization.so.1.60.0 (0x00007fcb9e739000)
libController.so => not found
libCppController.so => not found
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fcb9e3b7000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fcb9e0b2000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fcb9de9c000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fcb9dafb000)
/lib64/ld-linux-x86-64.so.2 (0x00007fcb9efc1000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007fcb9d8f3000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fcb9d6d6000)
which is quite strange, because, if use original -lboost_system and -lboost_serialization flags, gcc can't even link to the default boost due to new/old ABI incompatibilities.
So what exactly is happening here?
The problem is that just renaming your custom built libraries is not enough. The library name is embedded into the library as soname (you can see it with the readelf -d command) and is used when your application is liked with the library. Basically, the sonames from the custom libraries are put as the dependencies into your application binary, and since they are the same as the official Boost library names, the wrong binaries are loaded in run time.
You have to make sure the custom built Boost libraries are properly named in the build process. You can try doing this by adding --buildid=old_abi option to your b2 command line.
I have a small project wrote in c++, and now I am going to build it into *.so file.
it works fine in the building process, no errors and generate the *.so I want.
file I use for building *.so file
module1 = Extension('classmod',
include_dirs=['/usr/lib/python2.7/dist-packages/numpy/','/home/******/OpenCV/opencv/release/include','/home/******/OpenCV/opencv/release/include/opencv','/home/******/OpenCV/opencv/release/include/opencv2','/home/******/modulepipe/eigen-eigen-b23437e61a07'],
library_dirs=['/usr/local/lib/'],
libraries=['opencv_calib3d','opencv_contrib','opencv_core','opencv_features2d','opencv_flann','opencv_gpu',
'opencv_highgui','opencv_imgproc','opencv_legacy','opencv_ml','opencv_nonfree','opencv_objdetect',
'opencv_photo','opencv_stitching','opencv_superres','opencv_ts','opencv_video','opencv_videostab'],
sources=['classmodule.cpp','patch_match.cpp','compute_jaccard.cpp','find_diff.cpp','refine_match.cpp','tps.cpp','book_pipeline_single.cpp','imgfeatures.c','kdtree.c','minpq.c','sift.c','utils.c'],
language='c++',
extra_compile_args=['-O3'])
setup(name='classmod',
version='1.0',
description='blah',
author='blah',
url='blah',
ext_modules=[module1]
)
However when I am trying to load this .so lib use lbb, it tells me about can not find several opencv *.so lib.
command I use for testing:
~/CDH/modulepipe$ ldd classmod.so
linux-gate.so.1 => (0xb77b4000)
libopencv_calib3d.so.2.4 => not found
libopencv_core.so.2.4 => not found
libopencv_features2d.so.2.4 => not found
libopencv_highgui.so.2.4 => not found
libopencv_imgproc.so.2.4 => not found
libopencv_nonfree.so.2.4 => not found
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb7690000)
libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb7663000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb7645000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb762a000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb7480000)
/lib/ld-linux.so.2 (0xb77b5000)
This is the problem I have. Can anyone help me on explaining this?
I had this "lib not found problem" before, and it was in the build phase. But in this case, in the building process it passed, which means ldd could find *.so's it needed. So how it happens that, when I am trying to run "ldd classmod.so", it could not find the *.so's.
Your loader can't find those shared objects. Either place them somewhere the loader expects to find them, modify the loader configuration (man 8 ldconfig) to look for shared objects where they are, or use one of the loader's environment variables (man 8 ld.so) to specify where to look for them.
Problem: I am building an external project in CMake. The project has a Makefile that ultimately produces a shared object. I want to link against and install this object in my super project, just as if it were one of the libraries in the project. The problem is that the ExternalProject lib is getting linked into my applications and libraries with a relative path, not an absolute path which causes problems when running from any directory besides where CMake puts it.
I have created a sample SSCCE example project to demonstrate my overall setup. Feel free to peruse and compile if needed (git clone https://github.com/calebwherry/cmake-SO-question-main --recursive && cd cmake-SO-question-main && mkdir build && cd build && cmake .. && make && cd src/app/testApp && ldd testApp).
Whenever I run ldd on the executable and libs, I get output like this:
linux-vdso.so.1 => (0x00007fff8b5a2000)
libTestLib.so => /home/jwherry3/repos/cmake-superprj-main-test/build/src/lib/TestLib/libTestLib.so (0x00007f592da57000)
../../lib/libExtLib.so (0x00007f592d855000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f592d539000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f592d2b7000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f592d0a0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f592cd14000)
/lib64/ld-linux-x86-64.so.2 (0x00007f592dc5a000)
I have tried all kinds of things dealing with RPATHS but can't get the ExtLib to link right. The lib that is local to the project (libTestLib.so) links just fine.
I also tried to set the LD_LIBRARY_PATH to override the relative path when I run the app but even when I do that, it still doesn't find the library. I suppose because it is relative it does not follow the normal linking order? The outcome is that the binary will not run unless I am in the directory where it resides.
I feel like I'm doing something really dumb when creating dependencies with the ExternalProject and that is my issue but I have beat my head for 3 days and haven't come up with anything.
System setup: Debian Wheezy 64-bit, CMake 3.0.2, g++-4.9.2.
Took awhile but I have finally arrived at an answer with help from the CMake Users listserv, specifically #brad-king.
The main issue is that I am not compiling my shared object correctly in my external project. Brad's answer to my question:
Thanks for the complete/simple example. The problem is that
the libExtLib.so file built by the external project does not
have DT_SONAME set by the linker. When a linker is given a
path to a shared library file that has no soname then the path
is copied into the DT_NEEDED field of the consumer since it
has no soname to use for that.
There are two solutions:
Make sure DT_SONAME gets set properly by the external build system.
Tell CMake that the imported library file has no soname:
set_property(TARGET ExtLib PROPERTY IMPORTED_NO_SONAME 1)
CMake will then link it via -lExtLib and the linker will not
store the path to the file in DT_NEEDED but only the file name.
Either of these should resolve the issue. Number 1 is cleaner.
Since I have control over the Make files in the external library, I opted for the cleaner first solution by compiling the shared object like so:
$(SHARED_TARGET): $(OBJECTS)
$(CXX) $(CXXFLAGS) $(OBJECTS) -o $# $(LDFLAGS) -Wl,-soname,$#
I have revised my original example and made it a lot simpler: https://github.com/calebwherry/cmake-SO-question-main. I will leave it up as a reference for anyone stumbling across this post at a later date.
P.S.
There is another fix that is sub-optimal. If I hadn't done any of the above, I could have just given target_link_library the full path to the library I was linking against and not used the imported library target. I have noted this in the CMake file in the repo and commented it out. Not a great solution but does solve the issue another way.