I am using this code as a test.
#include <sstream>
#include <iostream>
int main () {
std::stringstream ss;
ss << "This is a test\n";
std::cout << ss.str();
}
I compile with
g++ -O0 -g test.cpp. When I run the program in gdb and stop at a breakpoint on the cout line, trying to print ss or ss.str() fails.
(gdb) p ss
$1 = <incomplete type>
(gdb) p ss.str()
Couldn't find method std::stringstream::str
gdb also gives me a ton of warnings about the debug information for libstdc++ and libc not matching their respective libraries, followed by an additional warning suggesting I install separate debuginfos.
warning: the debug information found in "/usr/lib/debug/usr/lib64/libc-2.17.so.debug" does not match "/lib64/libc.so.6" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug/usr/lib64/libstdc++.so.6.0.19.debug" does not match "/lib64/libstdc++.so.6" (CRC mismatch).
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7_6.6.x86_64 libstdc++-4.8.5-36.el7_6.2.x86_64
However, these packages are already installed according to rpm -qa. I don't have permissions to attempt to reinstall these or try other suggestions from this similar question.
I also found this question and checked the debug-file-directory that gdb is using.
(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/lib/debug".
I am running RHEL7 with the included versions of g++ and gdb.
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
The command debuginfo-install glibc-2.17-260.el7_6.6.x86_64 libstdc++-4.8.5-36.el7_6.2.x86_64 will install debugging information for those packages, not the packages themselves.
It seems that a different version of the debugging information has already been installed on the system, so you should be able to request that your system administrator installs matching package versions. (Some companies have policies against installing compilers and debuggers in production, but that does not seem to apply here.)
If you cannot get the correct debuginfo package versions installed on the system, you can download the packages from the Red Hat Customer Portal, copy it to the machine, unpack them using rpm2cpio … | cpio -id, and point GDB to the extracted debugging information. As of this writing, Red Hat does not offer a public symbol server unfortunately.
Note that packages from CentOS will not work even if they have the same name/version/release because they are not binary-identical due to different build environments.
My linux (SLES-8) server currently has glibc-2.2.5-235, but I have a program which won't work on this version and requires glibc-2.3.3.
Is it possible to have multiple glibcs installed on the same host?
This is the error I get when I run my program on the old glibc:
./myapp: /lib/i686/libc.so.6: version `GLIBC_2.3' not found (required by ./myapp)
./myapp: /lib/i686/libpthread.so.0: version `GLIBC_2.3.2' not found (required by ./myapp)
./myapp: /lib/i686/libc.so.6: version `GLIBC_2.3' not found (required by ./libxerces-c.so.27)
./myapp: /lib/ld-linux.so.2: version `GLIBC_2.3' not found (required by ./libstdc++.so.6)
./myapp: /lib/i686/libc.so.6: version `GLIBC_2.3' not found (required by ./libstdc++.so.6)
So I created a new directory called newglibc and copied the following files in:
libpthread.so.0
libm.so.6
libc.so.6
ld-2.3.3.so
ld-linux.so.2 -> ld-2.3.3.so
and
export LD_LIBRARY_PATH=newglibc:$LD_LIBRARY_PATH
But I get an error:
./myapp: /lib/ld-linux.so.2: version `GLIBC_PRIVATE' not found (required by ./newglibc/libpthread.so.0)
./myapp: /lib/ld-linux.so.2: version `GLIBC_2.3' not found (required by libstdc++.so.6)
./myapp: /lib/ld-linux.so.2: version `GLIBC_PRIVATE' not found (required by ./newglibc/libm.so.6)
./myapp: /lib/ld-linux.so.2: version `GLIBC_2.3' not found (required by ./newglibc/libc.so.6)
./myapp: /lib/ld-linux.so.2: version `GLIBC_PRIVATE' not found (required by ./newglibc/libc.so.6)
So it appears that they are still linking to /lib and not picking up from where I put them.
It is very possible to have multiple versions of glibc on the same system (we do that every day).
However, you need to know that glibc consists of many pieces (200+ shared libraries) which all must match. One of the pieces is ld-linux.so.2, and it must match libc.so.6, or you'll see the errors you are seeing.
The absolute path to ld-linux.so.2 is hard-coded into the executable at link time, and can not be easily changed after the link is done (Update: can be done with patchelf; see this answer below).
To build an executable that will work with the new glibc, do this:
g++ main.o -o myapp ... \
-Wl,--rpath=/path/to/newglibc \
-Wl,--dynamic-linker=/path/to/newglibc/ld-linux.so.2
The -rpath linker option will make the runtime loader search for libraries in /path/to/newglibc (so you wouldn't have to set LD_LIBRARY_PATH before running it), and the -dynamic-linker option will "bake" path to correct ld-linux.so.2 into the application.
If you can't relink the myapp application (e.g. because it is a third-party binary), not all is lost, but it gets trickier. One solution is to set a proper chroot environment for it. Another possibility is to use rtldi and a binary editor. Update: or you can use patchelf.
This question is old, the other answers are old. "Employed Russian"s answer is very good and informative, but it only works if you have the source code. If you don't, the alternatives back then were very tricky. Fortunately nowadays we have a simple solution to this problem (as commented in one of his replies), using patchelf. All you have to do is:
$ ./patchelf --set-interpreter /path/to/newglibc/ld-linux.so.2 --set-rpath /path/to/newglibc/ myapp
And after that, you can just execute your file:
$ ./myapp
No need to chroot or manually edit binaries, thankfully. But remember to backup your binary before patching it, if you're not sure what you're doing, because it modifies your binary file. After you patch it, you can't restore the old path to interpreter/rpath. If it doesn't work, you'll have to keep patching it until you find the path that will actually work... Well, it doesn't have to be a trial-and-error process. For example, in OP's example, he needed GLIBC_2.3, so you can easily find which lib provides that version using strings:
$ strings /lib/i686/libc.so.6 | grep GLIBC_2.3
$ strings /path/to/newglib/libc.so.6 | grep GLIBC_2.3
In theory, the first grep would come empty because the system libc doesn't have the version he wants, and the 2nd one should output GLIBC_2.3 because it has the version myapp is using, so we know we can patchelf our binary using that path. If you get a segmentation fault, read the note at the end.
When you try to run a binary in linux, the binary tries to load the linker, then the libraries, and they should all be in the path and/or in the right place. If your problem is with the linker and you want to find out which path your binary is looking for, you can find out with this command:
$ readelf -l myapp | grep interpreter
[Requesting program interpreter: /lib/ld-linux.so.2]
If your problem is with the libs, commands that will give you the libs being used are:
$ readelf -d myapp | grep Shared
$ ldd myapp
This will list the libs that your binary needs, but you probably already know the problematic ones, since they are already yielding errors as in OP's case.
"patchelf" works for many different problems that you may encounter while trying to run a program, related to these 2 problems. For example, if you get: ELF file OS ABI invalid, it may be fixed by setting a new loader (the --set-interpreter part of the command) as I explain here. Another example is for the problem of getting No such file or directory when you run a file that is there and executable, as exemplified here. In that particular case, OP was missing a link to the loader, but maybe in your case you don't have root access and can't create the link. Setting a new interpreter would solve your problem.
Thanks Employed Russian and Michael Pankov for the insight and solution!
Note for segmentation fault: you might be in the case where myapp uses several libs, and most of them are ok but some are not; then you patchelf it to a new dir, and you get segmentation fault. When you patchelf your binary, you change the path of several libs, even if some were originally in a different path. Take a look at my example below:
$ ldd myapp
./myapp: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./myapp)
./myapp: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./myapp)
linux-vdso.so.1 => (0x00007fffb167c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9a9aad2000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9a9a8ce000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9a9a6af000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9a9a3ab000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a99fe6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9a9adeb000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9a99dcf000)
Note that most libs are in /lib/x86_64-linux-gnu/ but the problematic one (libstdc++.so.6) is on /usr/lib/x86_64-linux-gnu. After I patchelf'ed myapp to point to /path/to/mylibs, I got segmentation fault. For some reason, the libs are not totally compatible with the binary. Since myapp didn't complain about the original libs, I copied them from /lib/x86_64-linux-gnu/ to /path/to/mylibs2, and I also copied libstdc++.so.6 from /path/to/mylibs there. Then I patchelf'ed it to /path/to/mylibs2, and myapp works now. If your binary uses different libs, and you have different versions, it might happen that you can't fix your situation. :( But if it's possible, mixing libs might be the way. It's not ideal, but maybe it will work. Good luck!
Use LD_PRELOAD:
put your library somewhere out of the man lib directories and run:
LD_PRELOAD='mylibc.so anotherlib.so' program
See: the Wikipedia article
First of all, the most important dependency of each dynamically linked program is the linker. All so libraries must match the version of the linker.
Let's take simple exaple: I have the newset ubuntu system where I run some program (in my case it is D compiler - ldc2). I'd like to run it on the old CentOS, but because of the older glibc library it is impossible. I got
ldc2-1.5.0-linux-x86_64/bin/ldc2: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by ldc2-1.5.0-linux-x86_64/bin/ldc2)
ldc2-1.5.0-linux-x86_64/bin/ldc2: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ldc2-1.5.0-linux-x86_64/bin/ldc2)
I have to copy all dependencies from ubuntu to centos.
The proper method is following:
First, let's check all dependencies:
ldd ldc2-1.5.0-linux-x86_64/bin/ldc2
linux-vdso.so.1 => (0x00007ffebad3f000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f965f597000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f965f378000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f965f15b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f965ef57000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f965ec01000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f965e9ea000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f965e60a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f965f79f000)
linux-vdso.so.1 is not a real library and we don't have to care about it.
/lib64/ld-linux-x86-64.so.2 is the linker, which is used by the linux do link the executable with all dynamic libraries.
Rest of the files are real libraries and all of them together with the linker must be copied somewhere in the centos.
Let's assume all the libraries and linker are in "/mylibs" directory.
ld-linux-x86-64.so.2 - as I've already said - is the linker. It's not dynamic library but static executable. You can run it and see that it even have some parameters, eg --library-path (I'll return to it).
On the linux, dynamically linked program may be lunched just by its name, eg
/bin/ldc2
Linux loads such program into RAM, and checks which linker is set for it. Usually, on 64-bit system, it is /lib64/ld-linux-x86-64.so.2 (in your filesystem it is symbolic link to the real executable).
Then linux runs the linker and it loads dynamic libraries.
You can also change this a little and do such trick:
/mylibs/ld-linux-x86-64.so.2 /bin/ldc2
It is the method for forcing the linux to use specific linker.
And now we can return to the mentioned earlier parameter --library-path
/mylibs/ld-linux-x86-64.so.2 --library-path /mylibs /bin/ldc2
It will run ldc2 and load dynamic libraries from /mylibs.
This is the method to call the executable with choosen (not system default) libraries.
Setup 1: compile your own glibc without dedicated GCC and use it
This setup might work and is quick as it does not recompile the whole GCC toolchain, just glibc.
But it is not reliable as it uses host C runtime objects such as crt1.o, crti.o, and crtn.o provided by glibc. This is mentioned at: https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location Those objects do early setup that glibc relies on, so I wouldn't be surprised if things crashed in wonderful and awesomely subtle ways.
For a more reliable setup, see Setup 2 below.
Build glibc and install locally:
export glibc_install="$(pwd)/glibc/build/install"
git clone git://sourceware.org/git/glibc.git
cd glibc
git checkout glibc-2.28
mkdir build
cd build
../configure --prefix "$glibc_install"
make -j `nproc`
make install -j `nproc`
Setup 1: verify the build
test_glibc.c
#define _GNU_SOURCE
#include <assert.h>
#include <gnu/libc-version.h>
#include <stdatomic.h>
#include <stdio.h>
#include <threads.h>
atomic_int acnt;
int cnt;
int f(void* thr_data) {
for(int n = 0; n < 1000; ++n) {
++cnt;
++acnt;
}
return 0;
}
int main(int argc, char **argv) {
/* Basic library version check. */
printf("gnu_get_libc_version() = %s\n", gnu_get_libc_version());
/* Exercise thrd_create from -pthread,
* which is not present in glibc 2.27 in Ubuntu 18.04.
* https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291 */
thrd_t thr[10];
for(int n = 0; n < 10; ++n)
thrd_create(&thr[n], f, NULL);
for(int n = 0; n < 10; ++n)
thrd_join(thr[n], NULL);
printf("The atomic counter is %u\n", acnt);
printf("The non-atomic counter is %u\n", cnt);
}
Compile and run with test_glibc.sh:
#!/usr/bin/env bash
set -eux
gcc \
-L "${glibc_install}/lib" \
-I "${glibc_install}/include" \
-Wl,--rpath="${glibc_install}/lib" \
-Wl,--dynamic-linker="${glibc_install}/lib/ld-linux-x86-64.so.2" \
-std=c11 \
-o test_glibc.out \
-v \
test_glibc.c \
-pthread \
;
ldd ./test_glibc.out
./test_glibc.out
The program outputs the expected:
gnu_get_libc_version() = 2.28
The atomic counter is 10000
The non-atomic counter is 8674
Command adapted from https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location but --sysroot made it fail with:
cannot find /home/ciro/glibc/build/install/lib/libc.so.6 inside /home/ciro/glibc/build/install
so I removed it.
ldd output confirms that the ldd and libraries that we've just built are actually being used as expected:
+ ldd test_glibc.out
linux-vdso.so.1 (0x00007ffe4bfd3000)
libpthread.so.0 => /home/ciro/glibc/build/install/lib/libpthread.so.0 (0x00007fc12ed92000)
libc.so.6 => /home/ciro/glibc/build/install/lib/libc.so.6 (0x00007fc12e9dc000)
/home/ciro/glibc/build/install/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc12f1b3000)
The gcc compilation debug output shows that my host runtime objects were used, which is bad as mentioned previously, but I don't know how to work around it, e.g. it contains:
COLLECT_GCC_OPTIONS=/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crt1.o
Setup 1: modify glibc
Now let's modify glibc with:
diff --git a/nptl/thrd_create.c b/nptl/thrd_create.c
index 113ba0d93e..b00f088abb 100644
--- a/nptl/thrd_create.c
+++ b/nptl/thrd_create.c
## -16,11 +16,14 ##
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
+#include <stdio.h>
+
#include "thrd_priv.h"
int
thrd_create (thrd_t *thr, thrd_start_t func, void *arg)
{
+ puts("hacked");
_Static_assert (sizeof (thr) == sizeof (pthread_t),
"sizeof (thr) != sizeof (pthread_t)");
Then recompile and re-install glibc, and recompile and re-run our program:
cd glibc/build
make -j `nproc`
make -j `nproc` install
./test_glibc.sh
and we see hacked printed a few times as expected.
This further confirms that we actually used the glibc that we compiled and not the host one.
Tested on Ubuntu 18.04.
Setup 2: crosstool-NG pristine setup
This is an alternative to setup 1, and it is the most correct setup I've achieved far: everything is correct as far as I can observe, including the C runtime objects such as crt1.o, crti.o, and crtn.o.
In this setup, we will compile a full dedicated GCC toolchain that uses the glibc that we want.
The only downside to this method is that the build will take longer. But I wouldn't risk a production setup with anything less.
crosstool-NG is a set of scripts that downloads and compiles everything from source for us, including GCC, glibc and binutils.
Yes the GCC build system is so bad that we need a separate project for that.
This setup is only not perfect because crosstool-NG does not support building the executables without extra -Wl flags, which feels weird since we've built GCC itself. But everything seems to work, so this is only an inconvenience.
Get crosstool-NG, configure and build it:
git clone https://github.com/crosstool-ng/crosstool-ng
cd crosstool-ng
git checkout a6580b8e8b55345a5a342b5bd96e42c83e640ac5
export CT_PREFIX="$(pwd)/.build/install"
export PATH="/usr/lib/ccache:${PATH}"
./bootstrap
./configure --enable-local
make -j `nproc`
./ct-ng x86_64-unknown-linux-gnu
./ct-ng menuconfig
env -u LD_LIBRARY_PATH time ./ct-ng build CT_JOBS=`nproc`
The build takes about thirty minutes to two hours.
The only mandatory configuration option that I can see, is making it match your host kernel version to use the correct kernel headers. Find your host kernel version with:
uname -a
which shows me:
4.15.0-34-generic
so in menuconfig I do:
Operating System
Version of linux
so I select:
4.14.71
which is the first equal or older version. It has to be older since the kernel is backwards compatible.
Setup 2: optional configurations
The .config that we generated with ./ct-ng x86_64-unknown-linux-gnu has:
CT_GLIBC_V_2_27=y
To change that, in menuconfig do:
C-library
Version of glibc
save the .config, and continue with the build.
Or, if you want to use your own glibc source, e.g. to use glibc from the latest git, proceed like this:
Paths and misc options
Try features marked as EXPERIMENTAL: set to true
C-library
Source of glibc
Custom location: say yes
Custom location
Custom source location: point to a directory containing your glibc source
where glibc was cloned as:
git clone git://sourceware.org/git/glibc.git
cd glibc
git checkout glibc-2.28
Setup 2: test it out
Once you have built he toolchain that you want, test it out with:
#!/usr/bin/env bash
set -eux
install_dir="${CT_PREFIX}/x86_64-unknown-linux-gnu"
PATH="${PATH}:${install_dir}/bin" \
x86_64-unknown-linux-gnu-gcc \
-Wl,--dynamic-linker="${install_dir}/x86_64-unknown-linux-gnu/sysroot/lib/ld-linux-x86-64.so.2" \
-Wl,--rpath="${install_dir}/x86_64-unknown-linux-gnu/sysroot/lib" \
-v \
-o test_glibc.out \
test_glibc.c \
-pthread \
;
ldd test_glibc.out
./test_glibc.out
Everything seems to work as in Setup 1, except that now the correct runtime objects were used:
COLLECT_GCC_OPTIONS=/home/ciro/crosstool-ng/.build/install/x86_64-unknown-linux-gnu/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crt1.o
Setup 2: failed efficient glibc recompilation attempt
It does not seem possible with crosstool-NG, as explained below.
If you just re-build;
env -u LD_LIBRARY_PATH time ./ct-ng build CT_JOBS=`nproc`
then your changes to the custom glibc source location are taken into account, but it builds everything from scratch, making it unusable for iterative development.
If we do:
./ct-ng list-steps
it gives a nice overview of the build steps:
Available build steps, in order:
- companion_tools_for_build
- companion_libs_for_build
- binutils_for_build
- companion_tools_for_host
- companion_libs_for_host
- binutils_for_host
- cc_core_pass_1
- kernel_headers
- libc_start_files
- cc_core_pass_2
- libc
- cc_for_build
- cc_for_host
- libc_post_cc
- companion_libs_for_target
- binutils_for_target
- debug
- test_suite
- finish
Use "<step>" as action to execute only that step.
Use "+<step>" as action to execute up to that step.
Use "<step>+" as action to execute from that step onward.
therefore, we see that there are glibc steps intertwined with several GCC steps, most notably libc_start_files comes before cc_core_pass_2, which is likely the most expensive step together with cc_core_pass_1.
In order to build just one step, you must first set the "Save intermediate steps" in .config option for the intial build:
Paths and misc options
Debug crosstool-NG
Save intermediate steps
and then you can try:
env -u LD_LIBRARY_PATH time ./ct-ng libc+ -j`nproc`
but unfortunately, the + required as mentioned at: https://github.com/crosstool-ng/crosstool-ng/issues/1033#issuecomment-424877536
Note however that restarting at an intermediate step resets the installation directory to the state it had during that step. I.e., you will have a rebuilt libc - but no final compiler built with this libc (and hence, no compiler libraries like libstdc++ either).
and basically still makes the rebuild too slow to be feasible for development, and I don't see how to overcome this without patching crosstool-NG.
Furthermore, starting from the libc step didn't seem to copy over the source again from Custom source location, further making this method unusable.
Bonus: stdlibc++
A bonus if you're also interested in the C++ standard library: How to edit and re-build the GCC libstdc++ C++ standard library source?
#msb gives a safe solution.
I met this problem when I did import tensorflow as tf in conda environment in CentOS 6.5 which only has glibc-2.12.
ImportError: /lib64/libc.so.6: version `GLIBC_2.16' not found (required by /home/
I want to supply some details:
First install glibc to your home directory:
mkdir ~/glibc-install; cd ~/glibc-install
wget http://ftp.gnu.org/gnu/glibc/glibc-2.17.tar.gz
tar -zxvf glibc-2.17.tar.gz
cd glibc-2.17
mkdir build
cd build
../configure --prefix=/home/myself/opt/glibc-2.17 # <-- where you install new glibc
make -j<number of CPU Cores> # You can find your <number of CPU Cores> by using **nproc** command
make install
Second, follow the same way to install patchelf;
Third, patch your Python:
[myself#nfkd ~]$ patchelf --set-interpreter /home/myself/opt/glibc-2.17/lib/ld-linux-x86-64.so.2 --set-rpath /home/myself/opt/glibc-2.17/lib/ /home/myself/miniconda3/envs/tensorflow/bin/python
as mentioned by #msb
Now I can use tensorflow-2.0 alpha in CentOS 6.5.
ref: https://serverkurma.com/linux/how-to-update-glibc-newer-version-on-centos-6-x/
Can you consider using Nix http://nixos.org/nix/ ?
Nix supports multi-user package management: multiple users can share a
common Nix store securely, don’t need to have root privileges to
install software, and can install and use different versions of a
package.
I am not sure that the question is still relevant, but there is another way of fixing the problem: Docker. One can install an almost empty container of the Source Distribution (The Distribution used for development) and copy the files into the Container. That way You do not need to create the filesystem needed for chroot.
If you look closely at the second output you can see that the new location for the libraries is used. Maybe there are still missing libraries that are part of the glibc.
I also think that all the libraries used by your program should be compiled against that version of glibc. If you have access to the source code of the program, a fresh compilation appears to be the best solution.
"Employed Russian" is among the best answer, and I think all other suggested answer may not work. The reason is simply because when an application is first created, all its the APIs it needs are resolved at compile time. Using "ldd" u can see all the statically linked dependencies:
ldd /usr/lib/firefox/firefox
linux-vdso.so.1 => (0x00007ffd5c5f0000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f727e708000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f727e500000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f727e1f8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f727def0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f727db28000)
/lib64/ld-linux-x86-64.so.2 (0x00007f727eb78000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f727d910000)
But at runtime, firefox will also load many other dynamic libraries, eg (for firefox) there are many "glib"-labelled libraries loaded (even though statically linked there are none):
/usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2.2.2
/lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0
/usr/lib/x86_64-linux-gnu/libavahi-glib.so.1.0.2
Manytimes, you can see names of one version being soft-linked into another version. Eg:
lrwxrwxrwx 1 root root 23 Dec 21 2014 libdbus-glib-1.so.2 -> libdbus-glib-1.so.2.2.2
-rw-r--r-- 1 root root 160832 Mar 1 2013 libdbus-glib-1.so.2.2.2
This therefore means different version of "libraries" exists in one system - which is not a problem as it is the same file, and it will provide compatibilities when applications have multiple versions dependencies.
Therefore, at the system level, all the libraries are almost interdependent on one another, and just changing the libraries loading priority via manipulating LD_PRELOAD or LD_LIBRARY_PATH will not help - even it can load, runtime it may still crash.
http://lightofdawn.org/wiki/wiki.cgi/-wiki/NewAppsOnOldGlibc
Best alternative is chroot (mentioned by ER briefly): but for this you will need to recreate the entire environment in which is the original binary execute - usually starting from /lib, /usr/lib/, /usr/lib/x86 etc. You can either use "Buildroot", or YoctoProject, or just tar from an existing Distro environment. (like Fedora/Suse etc).
When I wanted to run a chromium-browser on Ubuntu precise (glibc-2.15), I got the
(typical) message "...libc.so.6: version `GLIBC_2.19' not found...".
I considered the fact, that files are not needed permamently, but only for start.
So I collected the files needed for the browser and sudo and created a mini-glibc-2.19-
environment, started the browser and then copied the original files back
again. The needed files are in RAM and the original glibc is the same.
as root
the files (*-2.15.so) already exist
mkdir -p /glibc-2.19/i386-linux-gnu
/glibc-2.19/ld-linux.so.2 -> /glibc-2.19/i386-linux-gnu/ld-2.19.so
/glibc-2.19/i386-linux-gnu/libc.so.6 -> libc-2.19.so
/glibc-2.19/i386-linux-gnu/libdl.so.2 -> libdl-2.19.so
/glibc-2.19/i386-linux-gnu/libpthread.so.0 -> libpthread-2.19.so
mkdir -p /glibc-2.15/i386-linux-gnu
/glibc-2.15/ld-linux.so.2 -> (/glibc-2.15/i386-linux-gnu/ld-2.15.so)
/glibc-2.15/i386-linux-gnu/libc.so.6 -> (libc-2.15.so)
/glibc-2.15/i386-linux-gnu/libdl.so.2 -> (libdl-2.15.so)
/glibc-2.15/i386-linux-gnu/libpthread.so.0 -> (libpthread-2.15.so)
the script to run the browser:
#!/bin/sh
sudo cp -r /glibc-2.19/* /lib
/path/to/the/browser &
sleep 1
sudo cp -r /glibc-2.15/* /lib
sudo rm -r /lib/i386-linux-gnu/*-2.19.so
I am trying to install caffe following this tutorial
Basically I have the following error when I type the last make command:
me#dl-01:/home/me/caffe-master$ make runtest
.build_release/tools/caffe
caffe: command line brew
usage: caffe command args
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-gpu (Run in GPU mode on given device ID.) type: int32 default: -1
-iterations (The number of iterations to run.) type: int32 default: 50
-model (The model definition protocol buffer text file..) type: string
default: ""
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-weights (Optional; the pretrained weights to initialize finetuning. Cannot
be set simultaneously with snapshot.) type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle
ERROR: something wrong with flag 'flagfile' in file '/root/glog-0.3.3/gflags-master/src/gflags.cc'. One possibility: file '/root/glog-0.3.3/gflags-master/src/gflags.cc' is being linked both statically and dynamically into this executable.
make: *** [runtest] Error 1
I don't understand how to solve this error. Did anybody find this error before? how can I solve it?
Whether or not you've already solved this somewhere else, I'm posting the answer here in-case others run into the same problem.
Primarily, this problem seems to have come about because we don't always read things properly and blindly follow all instructions thinking they all apply to our case. hint: they don't.
In the installation instructions for Caffe (presuming Ubuntu instructions), there is a section which states:
Everything is packaged in 14.04.
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev protobuf-compiler
Blindly ignoring the next title, which states clearly:
Remaining dependencies, 12.04
we go on to install these dependencies, building and installing as required, resulting in the unfortunate side-effect of having 2 versions of libgflags, one dynamic (in /usr/lib[/x86_x64] and one static in /usr/local/lib
Resolution
Promise ourselves failthfully we'll read instructions properly next time around.
Uninstall libgflags
sudo apt-get remove -y libgflags
Delete make install versions
sudo rm -f /usr/local/lib/libgflags.a /usr/local/lib/libgflags_nothreads.a
sudo rm -rf /usr/local/include/gflags
Clean Caffe build
cd <path>/<to>/caffe
make clean
Re-install libgflags package
sudo apt-get install -y libgflags-dev
Rebuild Caffe
make all
make test
make runtest
Et Voila. All tests should now run and you're ready to rock the deep-learning boat.
I've worked out a way to debug this issue analytically. In my case, I was cross-compiling for an older ABI, so apt-get wasn't an option and I was compiling all dependencies manually.
First let's take a look at what this issue actually is. In the Google GFlags library, flags are declared through global objects. When the global object's constructor is run, it calls into the GFlags library to register that command line flag. If the global constructor gets run multiple times (due to multiple versions of the library containing it being loaded into memory), then the GFlags register method dies with an error.
What does GLog have to do with this? Well, GLog uses GFlags, and it has globally declared flag objects. Even if GFlags is linked correctly, if the GLog library gets loaded multiple times, you get an error pointing to logging.cc in GLog.
Sounds like quite a mess, huh. Even if GLog and GFlags are linked as shared in most cases, if another library links to a static version or some other version, kaboom.
Luckily, we can debug this issue using GDB and other tools, if you're willing to delve through some tricky symbol analysis.
First, you'll want to run GDB on the Python interpreter when it tries to import caffe:
gdb --args python -c 'import caffe'
Now, run the program once through so that GDB can pick up all the libraries it imports:
(gdb) r
Now, we can set a breakpoint on the place in the function (FlagRegistry::RegisterFlag()) that prints the error message, and run it again. Note that this line number is from my version of GFlags (2.2.2), you may have to look at the source code of your GFlags version and get the line number.
(gdb) break gflags.c:728
(gdb) r
Hopefully, GDB should then break on the first instance of the error (if not, check that gflags has been built with debugging symbols).
Look at the backtrace:
(gdb) bt
#0 google::(anonymous namespace)::FlagRegistry::RegisterFlag (this=0xa33b30, flag=0x1249d20) at dev/gflags-2.2.2/src/gflags.cc:728
#1 0x00007ffff0f3247a in _GLOBAL__sub_I_logging.cc () from prefix/lib/libcaffe2.so
#2 0x00007ffff7de76ca in call_init (l=<optimized out>, argc=argc#entry=3, argv=argv#entry=0x7fffffffdb08, env=env#entry=0x7fffffffdb28) at dl-init.c:72
#3 0x00007ffff7de77db in call_init (env=0x7fffffffdb28, argv=0x7fffffffdb08, argc=3, l=<optimized out>) at dl-init.c:30
#4 _dl_init (main_map=main_map#entry=0xd9c2a0, argc=3, argv=0x7fffffffdb08, env=0x7fffffffdb28) at dl-init.c:120
#5 0x00007ffff7dec8f2 in dl_open_worker (a=a#entry=0x7fffffffcf70) at dl-open.c:575
#6 0x00007ffff7de7574 in _dl_catch_error (objname=objname#entry=0x7fffffffcf60, errstring=errstring#entry=0x7fffffffcf68, mallocedp=mallocedp#entry=0x7fffffffcf5f,
operate=operate#entry=0x7ffff7dec4e0 <dl_open_worker>, args=args#entry=0x7fffffffcf70) at dl-error.c:187
#7 0x00007ffff7debdb9 in _dl_open (file=0x9aee70 "prefix/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state.so", mode=-2147483646,
caller_dlopen=0x51bb39 <_PyImport_GetDynLoadFunc+233>, nsid=-2, argc=<optimized out>, argv=<optimized out>, env=0x7fffffffdb28) at dl-open.c:660
#8 0x00007ffff75ecf09 in dlopen_doit (a=a#entry=0x7fffffffd1a0) at dlopen.c:66
#9 0x00007ffff7de7574 in _dl_catch_error (objname=0xabf9f0, errstring=0xabf9f8, mallocedp=0xabf9e8, operate=0x7ffff75eceb0 <dlopen_doit>, args=0x7fffffffd1a0) at dl-error.c:187
#10 0x00007ffff75ed571 in _dlerror_run (operate=operate#entry=0x7ffff75eceb0 <dlopen_doit>, args=args#entry=0x7fffffffd1a0) at dlerror.c:163
#11 0x00007ffff75ecfa1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#12 0x000000000051bb39 in _PyImport_GetDynLoadFunc ()
<snip>
Well that's a lot to deal with, but let's focus on the line that's actually important:
#1 0x00007ffff0f3247a in _GLOBAL__sub_I_logging.cc () from prefix/lib/libcaffe2.so
This is the call to the constructor for the global variables in logging.cc (which is part of GLog). As you can see, this call is in libcaffe2.so, meaning that GLog has been statically linked to libcaffe2.so [I was using caffe2, but this procedure should be the same for both].
You can then set a breakpoint on google::(anonymous namespace)::FlagRegistry::RegisterFlag and rerun the program from the start. Look at each call to RegisterFlag(), and figure out where this particular flag was registered the first time. If the library providing the flag is a shared library, then it should only ever get registered from that .so file, and nowhere else.
To confirm the diagnosis, you can use
nm <library> | grep _GLOBAL__sub_I_logging.cc
to check for that init function in a library file. Once you've found your culprit, you'll need to rebuild it so that it doesn't link to GFlags/GLog statically.
I also had two libraries installed, a shared .so library and a static .a library. I removed them all as well as the /usr/local/include/glog folder.
The .so file I had brought over when I (cross) compiled the system, while the .a was from a native and up-to-date build.
Ultimately it came down to building glog (natively) in such a way that it provided the .so files.
I started with a clean download:
git clone git://github.com/google/glog
Then I edited CMakeLists.txt.
Where it says:
add_library (glog
${GLOG_SRCS}
)
I changed it to:
add_library (glog SHARED
${GLOG_SRCS}
)
Next you should be able to follow the other instructions. For my particular case I had to use slightly different instructions, not saying you have to do this. For me it was:
mkdir build
cd build
export CXXFLAGS="-fPIC"
cmake ..
make
sudo make install
This gave me the .so files and put them in the right place. Then I started over with caffe and it fixed the error for me.
the program under live env, segmentation fault some time, i try to gdb the coredump file,
but can't found the code line cause coredump.
Program terminated with signal 11, Segmentation fault.
#0 0x00000038f3a41bf5 in __correctly_grouped_prefixwc () from /lib64/libc.so.6
(gdb) bt
#0 0x00000038f3a41bf5 in __correctly_grouped_prefixwc () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()
(gdb) info r
rax 0x1ac1b108 448901384
rbx 0x2add423b4ff0 47129787322352
rcx 0x2add48128640 47129885312576
rdx 0x0 0
rsi 0x1 1
rdi 0x2add48000020 47129884098592
rbp 0x2add3f1aef50 0x2add3f1aef50
rsp 0x2add423b4ff0 0x2add423b4ff0
r8 0x2 2
r9 0x2 2
r10 0x0 0
r11 0x0 0
r12 0x0 0
r13 0x3 3
r14 0x1000 4096
r15 0x2add3f1b0000 47129734873088
rip 0x38f3a41bf5 0x38f3a41bf5 <__correctly_grouped_prefixwc+165>
eflags 0x10246 [ PF ZF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
fctrl 0x37f 895
fstat 0x0 0
ftag 0xffff 65535
fiseg 0x0 0
fioff 0xc54f06 12930822
foseg 0x2add 10973
fooff 0x423b3f00 1111179008
fop 0x0 0
mxcsr 0x1fa1 [ IE PE IM DM ZM OM UM PM ]
cat /etc/redhat-release
CentOS release 5.5 (Final)
and i want to debug glibc at the source level, run yum install yum-utils to install the debuginfo-install program.
then, run sudo debuginfo-install glibc, the result following
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* addons: centos.ustc.edu.cn
* base: mirror.bit.edu.cn
* extras: centos.ustc.edu.cn
* updates: centos.ustc.edu.cn
Checking for new repos for mirrors
Could not find debuginfo for main pkg: glibc-2.5-123.x86_64
Could not find debuginfo for main pkg: glibc-2.5-123.i686
No debuginfo packages available to install
and then i try to run yum search glibc-debuginfo
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* addons: centos.ustc.edu.cn
* base: mirrors.163.com
* extras: centos.ustc.edu.cn
* updates: centos.ustc.edu.cn
Warning: No matches found for: glibc-debuginfo
No Matches found
no matches found again.
i try to run yum search glibc
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* addons: centos.ustc.edu.cn
* base: mirrors.163.com
* extras: centos.ustc.edu.cn
* updates: centos.ustc.edu.cn
================================================================================ Matched: glibc =================================================================================
compat-glibc.i386 : Compatibility C library
compat-glibc.x86_64 : Compatibility C library
compat-glibc-headers.x86_64 : Header files for development using standard C libraries.
glibc.i686 : The GNU libc libraries.
glibc.x86_64 : The GNU libc libraries.
glibc-common.x86_64 : Common binaries and locale data for glibc
glibc-devel.i386 : Object files for development using standard C libraries.
glibc-devel.x86_64 : Object files for development using standard C libraries.
glibc-headers.x86_64 : Header files for development using standard C libraries.
glibc-utils.x86_64 : Development utilities from GNU C library
kernel-headers.x86_64 : Header files for the Linux kernel for use by glibc
nss_db.i386 : An NSS library for the Berkeley DB.
nss_db.x86_64 : An NSS library for the Berkeley DB.
yp-tools.x86_64 : NIS (or YP) client programs.
yum-protect-packages.noarch : Yum plugin to prevents Yum from removing itself and other protected packages
i try to run sudo yum install glibc-devel.x86_64, and gdb the coredump file again,
but it display the following
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
how can i find the code line cause coredump? i try to google, but not found, any ideas?
First, although __correctly_grouped_prefixwc caused the segmentation fault, it's likely that it was passed incorrect arguments from some other piece of code, perhaps strtod or strtol or something that called them. That being said, here is how to set things up so that gdb can show the line of source code in __correctly_grouped_prefixwc that caused the segmentation fault.
To do source-level debugging, you need an executable or shared object's debug info, and its source code. Linux and Unix distributions in general do not include these by default, to conserve storage space, but they make them available as packages.
On CentOS, you just need to install the debuginfo package for each executable or library you're interested in. To do this, run
sudo yum install yum-utils
which will install the debuginfo-install program, then run
sudo debuginfo-install glibc
to download and install the glibc-debuginfo-2.5-123 package (your version number may vary). This will install, among many other files, /usr/lib/debug/lib64/libc.so.6.debug, /usr/lib/debug/lib64/libc-2.5.so.debug, and /usr/src/debug/glibc-2.5-20061008T1257/stdlib/grouping.c, which are what you need.
debuginfo-install is a short python program that enables the debuginfo repositories and downloads and installs the debuginfo package corresponding to the package you give as an argument, plus all its dependencies. As an alternative, you can download the debuginfo packages directly from http://debuginfo.centos.org (or any mirrors) and install them using rpm -i.
You mentioned that you got the error No debuginfo packages available to install. Perhaps you don't have the debuginfo repo configured. On my CentOS 5 system, the configuration is in the file /etc/yum.repos.d/CentOS-Debuginfo.repo
# All debug packages from all the various CentOS-5 releases
# are merged into a single repo, split by BaseArch
#
# Note: packages in the debuginfo repo are currently not signed
#
[base-debuginfo]
name=CentOS-5 - Debuginfo
baseurl=http://debuginfo.centos.org/5/$basearch/
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
enabled=0
For other releases, general instructions for adding the debuginfo repo are in this CentOS wiki article.