G++ 4.9.2 and 8.2.0 - final binary's dependency with default lib path searching ./../lib directory (implicit RPATH issue) - c++

I encountered an odd difference between the binary produced by a 4.9.2 G++ and a 8.2.0 G++ on a Linux, when upgrading from the former to the latter. I was able to narrow it down to the below minimized snippet
//main.cpp
int main(){}
//emptylib.cpp
#!/bin/sh
#to build and verify output
rm -rf bin lib *.o
gcc_bin=/path/to/gcc_4.9.2/bin
#gcc_bin=/path/to/gcc_8.2.0/bin
${gcc_bin}/g++ -B${gcc_bin} -c main.cpp
${gcc_bin}/g++ -B${gcc_bin} -c emptylib.cpp
${gcc_bin}/g++ -B${gcc_bin} -shared emptylib.o -o libemptylib.so
${gcc_bin}/g++ -B${gcc_bin} main.o -o main -L. -lemptylib
mkdir bin
mkdir lib
mv libemptylib.so lib/
mv main bin/
ldd bin/main
The result is a bit strange, because I have provided no -rpath=../lib, but all the same, with the 4.9.2 compiler driver I end up with (in the output of ldd):
libemptylib.so => /path/to/bin/../lib/libemptylib.so
and with 8.2.0
libemptylib.so => not found
the lib does not resolve, as I would expect. I haven't been able to see any documentation that hints at a search for ../lib in any version, but I would not expect this to be a behavior by any version. Was this a bug in 4.9.2 or some earlier version or something that shouldn't have happened in the first place?
As a side note, the 8.2.0 and the 4.9.2 don't have an ld inside of their bin directory (despite me putting the -B in the code snippet), so both compiler drivers are using the /usr/bin/ld (I've verified as much) and producing different results based on the compiler driver alone.
Update:
On request, I ran readelf -d bin/main on both binaries and I did receive different output for rpath. GCC 4.9.2 gave me:
0x000000000000000f (RPATH) Library rpath: [$ORIGIN:$ORIGIN/../lib64:$ORIGIN/../lib:/path/to/gcc_libs/lib64:/path/to/gcc_libs/lib:/path/to/gcc_4.9.2/lib64:/path/to/gcc_4.9.2/lib]
where 8.2.0 did not have that
0x0000000000000001 (NEEDED) Shared Library: [libemptylib.so]
So I guess I have an answer to what happened. So how did this end up in my binary in 4.9.2, and why did it change between the two versions where now it doesn't in 8.2.0?

I did receive different output for rpath.
rpath: [$ORIGIN:$ORIGIN/../lib64:$ORIGIN/../lib:/path/to/gcc_libs/lib64:/path/to/gcc_libs/lib:/path/to/gcc_4.9.2/lib64:/path/to/gcc_4.9.2/lib]
So the difference is in how GCC 4.9.2 vs. 8.2.0 were configured.
I am guessing that 8.2 came from your distribution (or at least was configured with default installation paths), whereas 4.9 was configured to be installed into /path/to/gcc_4.9.2.
That explains why 4.9 adds -rpath with /path/to/gcc_4.9.2/lib64 for a 64-bit builds. It does not however explain where $ORIGIN of /path/to/gcc_4.9.2/lib (the latter shouldn't be part of a 64-bit build) come from.
Configuring GCC so that it puts exactly what's necessary, no more and no less, is a bit of a black magic. I wouldn't be surprised if 4.9 had some bugs in that area.

Related

How to change default GCC compiler to be used with MPI on Linux CentOS

I have two GCC compilers installed on a Linux (CentOS) machine. The old version of GCC (4.4.7) is in the default folder (came with CentOS) and the newer one that I intend to use is in /usr/local/gcc/4.9.3/. My code utilizes MPI and LAPACK/LAPACKE/BLAS libraries and with the old GCC I used to compile source (for example “main.cpp”) like this:
mpiCC main.cpp -o main -L/home/USER1/lapack-3.6.1 -llapacke -llapack -lblas -lm –Wall
This still invokes the old GCC 4.4.7. What should I modify so the above MPI compilation (mpiCC) invokes GCC 4.9.3 executable from the new location at /usr/local/gcc/4.9.3/el6/bin/ ?
From MPICH Installer's Guide version 3.2 (page 6):
"The MPICH configure step will attempt to find the C, C++, and Fortran compilers for you, but if you either want to override the default or need to specify a compiler that configure doesn't recognize, you can specify them on the command line [...]. For example, to select the Intel compilers instead of the GNU compilers on a system with both, use"
./configure CC=icc CXX=icpc F77=ifort FC=ifort ...
Is there a way to dicriminate between different version of GCC compilers in ./configure ?
I guess mpiCC uses the first gcc compiler found in the $PATH variable.
You should be able to set the new version of gcc by running:
PATH="/usr/local/gcc/4.9.3/el6/bin:$PATH" mpiCC main.cpp -o main -L/home/USER1/lapack-3.6.1 -llapacke -llapack -lblas -lm –Wall
If you really want two versions of GCC installed at the same time and use both of them here is a good link that explains how to do this:
http://gcc.gnu.org/faq.html#multiple
Finally found how. Here is the recipe:
1) check your if you shell is bash, if not set it to bash: $ echo $SHELL
/bin/tcsh
It was tcsh and needed to be set to bash.
2) Switch to bash: $ bash
bash-4.1$
3) Add new version of GCC to the front of the PATH:
bash-4.1$ export PATH=/usr/local/gcc/4.9.3/el6/bin:$PATH
4) Check the PATH: bash-4.1$ echo $PATH
/usr/local/gcc/4.9.3/el6/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin
5) Check version of GCC used (It picks up the first GCC from the PATH):
bash-4.1$ gcc --version
gcc (GCC) 4.9.3
Note: this is just for the current session.

why self built g++ compiler fails to compile my code

I wanted to use latest g++ compiler(4.9.1) on suse linux, but suse only supports an older g++ version. So, I took a latest source code from one of the gnu mirror sites and compiled it myself. Everything went fine. But when I tried to compile my test code using the built g++, the compilation fails with error,
"/root/home/include/c++/4.9.1/x86_64-unknown-linux-gnu/bits/os_defines.h:39:22: fatal error: features.h: No such file or directory".
I can find a "features.h" in "/root/home/include/c++/4.9.1/parallel", but I feel that it should be there in "/root/home/include/c++/4.9.1/" itself.
I copied "/root/home/include/c++/4.9.1/parallel/features.h" to "/root/home/include/c++/4.9.1/" just to see what happens. Now it complains with error "whcar.h" not found.
Have I missed something.
Here are the steps I followed to build g++.
1. /root/home/gcc_build/objdir# ../gcc-4.9.1/configure --prefix=/root/home/ --disable-multilib
2. /root/home/gcc_build/objdir# make -j16
3. /root/home/gcc_build/objdir# make install
4. /root/home/gcc_build/test# /root/home/bin/g++ --sysroot /root/home -m64 test.cpp
I resolved the issue by removing sysroot option and pointing c++ include and library path to my home directory. One thing I noticed was that the g++ source does not come with libc and c header files, and libc has to be installed seperately. But with sysroot option, g++ was trying to look all the header files in my home directory.
Below is the command I used to successfully compile the code.
/root/home/bin/g++ -I /root/home/include/c++/4.9.1 -L /root/home/lib64 -Wl,--rpath=/root/home/lib64 --std=c++0x -m64 test.cpp
Take a look at the GCC Directory Options. It is important to use the correct "specifier" as well (-isystem, -L, -B, -I etc)

Linking g++ 4.8 to libstdc++

I downloaded and built gcc 4.8.1 on my desktop, running 64-bit Ubuntu 12.04. I built it out of source, like the docs recommend, and with the commands
../../gcc-4.8.1/configure --prefix=$HOME --program-suffix=-4.8
make
make -k check
make install
It seemed to pass all the tests, and I installed everything into my home directory w/ the suffix -4.8 to distinguish from the system gcc, which is version 4.6.3.
Unfortunately when I compile c++ programs using g++-4.8 it links to the system libc and libstdc++ rather than the newer ones compiled from gcc-4.8.1. I downloaded and built gcc 4.8 because I wanted to play around with the new C++11 features in the standard library, so this behaviour is definitely not what I wanted. What can I do to get gcc-4.8 to automatically link to the standard libraries that came with it rather than the system standard libraries?
When you link with your own gcc you need to add an extra run-time linker search path(s) with -Wl,-rpath,$(PREFIX)/lib64 so that at run-time it finds the shared libraries corresponding to your gcc.
I normally create a wrapper named gcc and g++ in the same directory as gcc-4.8 and g++-4.8 which I invoke instead of gcc-4.8 and g++-4.8, as prescribed in Dynamic linker is unable to find GCC libraries:
#!/bin/bash
exec ${0}SUFFIX -Wl,-rpath,PREFIX/lib64 "$#"
When installing SUFFIX and PREFIX should be replaced with what was passed to configure:
cd ${PREFIX}/bin && rm -f gcc g++ c++ gfortran
sed -e 's#PREFIX#${PREFIX}#g' -e 's#SUFFIX#${SUFFIX}#g' gcc-wrapper.sh > ${PREFIX}/bin/gcc
chmod +x ${PREFIX}/bin/gcc
cd ${PREFIX}/bin && ln gcc g++ && ln gcc c++ && ln gcc gfortran
(gcc-wrapper.sh is that bash snippet).
The above solution does not work with some versions of libtool because g++ -Wl,... -v assumes linking mode and fails with an error.
A better solution is to use specs file. Once gcc/g++ is built, invoke the following command to make gcc/g++ add -rpath to the linker command line (replace ${PREFIX}/lib64 as necessary):
g++ -dumpspecs | awk '/^\*link:/ { print; getline; print "-rpath=${PREFIX}/lib64", $0; next } { print }' > $(dirname $(g++ -print-libgcc-file-name))/specs
I just had the same problem when building gcc-4.8.2. I don't have root access on that machine and therefore need to install to my home directory. It took several attempts before I figured out the magic required to get this to work so I will reproduce it here so other people will have an easier time. These are the commands that I used to configure gcc:
prefix=/user/grc/packages
export LDFLAGS=-Wl,-rpath,$prefix/lib
export LD_RUN_PATH=$prefix/lib
export LD_LIBRARY_PATH=$prefix/lib
../../src/gmp-4.3.2/configure --prefix=$prefix
../../src/mpfr-2.4.2/configure --prefix=$prefix
../../src/mpc-0.8.1/configure --prefix=$prefix --with-mpfr=$prefix --with-gmp=$prefix
../../src/gcc-4.8.2/configure --prefix=$prefix --with-mpfr=$prefix --with-gmp=$prefix --with-mpc=$prefix --enable-languages=c,c++
That got me a working binary but any program I built with that version of g++ wouldn't run correctly unless I built it with the -Wl,-rpath,$prefix/lib64 option. It is possible to get g++ to automatically add that option by providing a specs file. If you run
strace g++ 2>&1 | grep specs
you can see which directories it checks for a specs file. In my case it was $prefix/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/specs so I ran g++ -dumpspecs to create a new specs file:
cd $prefix/lib/gcc/x86_64-unknown-linux-gnu/4.8.2
$prefix/bin/g++ -dumpspecs > xx
mv xx specs
and then edited that file to provide the -rpath option. Search for the lines like this:
*link_libgcc:
%D
and edit to add the rpath option:
*link_libgcc:
%D -rpath /user/grc/packages/lib/%M
The %M expands to either ../lib or ../lib64 depending on whether you are building a 32-bit or a 64-bit executable.
Note that when I tried this same trick on an older gcc-4.7 build it didn't work because it didn't expand the %M. For older versions you can remove the %M and just hardcode lib or lib64 but that is only a viable solution if you only ever build 32-bit executables (with lib) or only ever build 64-bit executables (with lib64).
gcc -print-search-dirs will tell you where your compiler is looking for runtime libraries, etc. You can override this with the -B<prefix> option.

Ubuntu 11.10 linking perftools library

I can't get gcc in Ubuntu 11.10 to properly link in the google perftools -lprofiler.
The problem seems to be that the linker discards libraries which are not directly used in a program.
An example will help.
Let's call this main.cpp:
#include <math.h>
int main()
{
double value;
for (int i=0; i < 1000000; i++)
{
for (int j=0; j < 1000; j++)
value = sqrt(100.9);
}
return 0;
}
Compile using:
g++ -c main.cpp -o main.o
g++ main.o -o main -lm -lprofiler
Check the executable using ldd ./main:
linux-vdso.so.1 => (0x00007fff5a9ff000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f32bc1c9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f32bc593000)
Normally, I would run:
CPUPROFILE=/tmp/profile ./main
to produce profile output. But since the profile library is not linked in no profile output is generated.
I've made sure the profiler library is in my search path, and have tried directly linking against the shared library and static library.
The above test works fine on Ubuntu 10.04, Ubuntu 10.10, Ubuntu 11.04, SUSE 12.1, and Fedora 16.
Also, once I include function calls that use the profiler (such as ProfilerStart() and ProfilerStop()), then the profiler library gets linked into the executable.
Any ideas on how to get gcc to link in the profiler library?
Thanks.
g++ main.o -o main -lm -lprofiler
As another.anon.coward commented, you are likely falling victim of your g++ using --as-needed linker flag. Try this instead:
g++ main.o -Wl,--no-as-needed -lprofiler -Wl,--as-needed
Notes:
g++ already adds -lm, no need to add it again
It is important to turn --as-needed back on. Not doing so will likely cause you to link to additional libraries that you don't really need.
In my case, the problem was that there was only a libprofiler.so.0, and no libprofiler.so in /usr/lib/:
user#compy:/usr/include$ dpkg -L libgoogle-perftools4
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libgoogle-perftools4
/usr/share/doc/libgoogle-perftools4/README.Debian
/usr/share/doc/libgoogle-perftools4/copyright
/usr/lib
/usr/lib/libprofiler.so.0.4.5
/usr/lib/libtcmalloc.so.4.2.6
/usr/lib/libtcmalloc_debug.so.4.2.6
/usr/lib/libtcmalloc_and_profiler.so.4.2.6
/usr/share/doc/libgoogle-perftools4/AUTHORS
/usr/share/doc/libgoogle-perftools4/TODO
/usr/share/doc/libgoogle-perftools4/README.gz
/usr/share/doc/libgoogle-perftools4/NEWS.gz
/usr/share/doc/libgoogle-perftools4/changelog.Debian.gz
/usr/lib/libtcmalloc.so.4
/usr/lib/libtcmalloc_and_profiler.so.4
/usr/lib/libprofiler.so.0
/usr/lib/libtcmalloc_debug.so.4
I don't know what the official fix to this is, but I simply created a symlink in /usr/lib:
user#compy:/usr/lib$ sudo ln -s libprofiler.so.0 libprofiler.so
This will make -lprofiler work.
If you don't mind changing your Makefile you can alternatively specify -l:libprofiler.so.0 instead of -lprofiler (note the extra colon) (source).
EDIT: The official way to get the .so is apparently to install the libgoogle-perftools-dev package as explained here:
user#compy:/usr/lib$ dpkg -S libprofiler.so
libgoogle-perftools-dev: /usr/lib/libprofiler.so
libgoogle-perftools4: /usr/lib/libprofiler.so.0.4.5
libgoogle-perftools4: /usr/lib/libprofiler.so.0
I understand that if you want to link to a certain lib, you should install the libx-dev package, which will contain the /usr/lib/libx.so. This file will only be a symlink to a specific version, such as /usr/lib/libx.so.1.2. When you link against /usr/lib/libx.so by specifying -lx to your linker, you will actually create a link in your program against the specific version linked at the time by recording a SONAME of libx.so.1 (the last version number is stripped as oulined here). So when you run your program at a later point in time the dynamic linker will look only for /usr/lib/libx.so.1, which is symlinked to /usr/lib/libx.so.1.2, and no /usr/lib/libx.so hence no dev package needs to exist.
So the libx-dev packages is for compiling and linking against libx, and the libx package is for running a precompiled program against libx.

how to port c/c++ applications to legacy linux kernel versions

Ok, this is just a bit of a fun exercise, but it can't be too hard compiling programmes for some older linux systems, or can it?
I have access to a couple of ancient systems all running linux and maybe it'd be interesting to see how they perform under load. Say as an example we want to do some linear algebra using Eigen which is a nice header-only library. Any chance to compile it on the target system?
user#ancient:~ $ uname -a
Linux local 2.2.16 #5 Sat Jul 8 20:36:25 MEST 2000 i586 unknown
user#ancient:~ $ gcc --version
egcs-2.91.66
Maybe not... So let's compile it on a current system. Below are my attempts, mainly failed ones. Any more ideas very welcome.
Compile with -m32 -march=i386
user#ancient:~ $ ./a.out
BUG IN DYNAMIC LINKER ld.so: dynamic-link.h: 53: elf_get_dynamic_info: Assertion `! "bad dynamic tag"' failed!
Compile with -m32 -march=i386 -static: Runs on all fairly recent kernel versions but fails if they are slightly older with the well known error message
user#ancient:~ $ ./a.out
FATAL: kernel too old
Segmentation fault
This is a glibc error which has a minimum kernel version it supports, e.g. kernel 2.6.4 on my system:
$ file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, for GNU/Linux 2.6.4, not stripped
Compile glibc myself with support for the oldest kernel possible. This post describes it in more detail but essentially it goes like this
wget ftp://ftp.gnu.org/gnu/glibc/glibc-2.14.tar.bz2
tar -xjf glibc-2.14.tar.bz2
cd glibc-2.14
mkdir build; cd build
../configure --prefix=/usr/local/glibc_32 \
--enable-kernel=2.0.0 \
--with-cpu=i486 --host=i486-linux-gnu \
CC="gcc -m32 -march=i486" CXX="g++ -m32 -march=i486"
make -j 4
make intall
Not sure if the --with-cpu and --host options do anything, most important is to force the use of compiler flags -m32 -march=i486 for 32-bit builds (unfortunately -march=i386 bails out with errors after a while) and --enable-kernel=2.0.0 to make the library compatible with older kernels. Incidentially, during configure I got the warning
WARNING: minimum kernel version reset to 2.0.10
which is still acceptable, I suppose. For a list of things which change with different kernels see ./sysdeps/unix/sysv/linux/kernel-features.h.
Ok, so let's link against the newly compiled glibc library, slightly messy but here it goes:
$ export LIBC_PATH=/usr/local/glibc_32
$ export LIBC_FLAGS=-nostdlib -L${LIBC_PATH} \
${LIBC_PATH}/crt1.o ${LIBC_PATH}/crti.o \
-lm -lc -lgcc -lgcc_eh -lstdc++ -lc \
${LIBC_PATH}/crtn.o
$ g++ -m32 -static prog.o ${LIBC_FLAGS} -o prog
Since we're doing a static compile the link order is important and may well require some trial and error, but basically we learn from what options gcc gives to the linker:
$ g++ -m32 -static -Wl,-v file.o
Note, crtbeginT.o and crtend.o are also linked against which I didn't need for my programmes so I left them out. The output also includes a line like --start-group -lgcc -lgcc_eh -lc --end-group which indicates inter-dependence between the libraries, see this post. I just mentioned -lc twice in the gcc command line which also solves inter-dependence.
Right, the hard work has paid off and now I get
$ file ./prog
./prog: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, for GNU/Linux 2.0.10, not stripped
Brilliant I thought, now try it on the old system:
user#ancient:~ $ ./prog
set_thread_area failed when setting up thread-local storage
Segmentation fault
This, again, is a glibc error message from ./nptl/sysdeps/i386/tls.h. I fail to understand the details and give up.
Compile on the new system g++ -c -m32 -march=i386 and link on the old. Wow, that actually works for C and simple C++ programmes (not using C++ objects), at least for the few I've tested. This is not too surprising as all I need from libc is printf (and maybe some maths) of which the interface hasn't changed but the interface to libstdc++ is very different now.
Setup a virtual box with an old linux system and gcc version 2.95. Then compile gcc version 4.x.x ... sorry, but too lazy for that right now ...
???
Have found the reason for the error message:
user#ancient $ ./prog
set_thread_area failed when setting up thread-local storage
Segmentation fault
It's because glibc makes a system call to a function which is only available since kernel 2.4.20. In a way it can be seen as a bug of glibc as it wrongly claims to be compatible with kernel 2.0.10 when it requires at least kernel 2.4.20.
The details:
./glibc-2.14/nptl/sysdeps/i386/tls.h
[...]
/* Install the TLS. */ \
asm volatile (TLS_LOAD_EBX \
"int $0x80\n\t" \
TLS_LOAD_EBX \
: "=a" (_result), "=m" (_segdescr.desc.entry_number) \
: "0" (__NR_set_thread_area), \
TLS_EBX_ARG (&_segdescr.desc), "m" (_segdescr.desc)); \
[...]
_result == 0 ? NULL \
: "set_thread_area failed when setting up thread-local storage\n"; })
[...]
The main thing here is, it calls the assembly function int 0x80 which is a system call to the linux kernel which decides what to do based on the value of eax, which is set to
__NR_set_thread_area in this case and is defined in
$ grep __NR_set_thread_area /usr/src/linux-2.4.20/include/asm-i386/unistd.h
#define __NR_set_thread_area 243
but not in any earlier kernel versions.
So the good news is that point "3. Compiling glibc with --enable-kernel=2.0.0" will probably produce executables which run on all linux kernels >= 2.4.20.
The only chance to make this work with older kernels would be to disable tls (thread-local storage) but which is not possible with glibc 2.14, despite the fact it is offered as a configure option.
The reason you can't compile it on the original system likely has nothing to do with kernel version (it could, but 2.2 isn't generally old enough for that to be a stumbling block for most code). The problem is that the toolchain is ancient (at the very least, the compiler). However, nothing stops you from building a newer version of G++ with the egcs that is installed. You may also encounter problems with glibc once you've done that, but you should at least get that far.
What you should do will look something like this:
Build latest GCC with egcs
Rebuild latest GCC with the gcc you just built
Build latest binutils and ld with your new compiler
Now you have a well-built modern compiler and (most of a) toolchain with which to build your sample application. If luck is not on your side you may also need to build a newer version of glibc, but this is your problem - the toolchain - not the kernel.