boost testing fpic linking error - c++

I have been staring and googling this but I cannot see what I have done.
I have a working project on a 32 bit machine. I have just pulled the repository to a 64 bit machine (which was the original development machine for the project) and I am now getting the following linking errors when trying to build the testing binary
/usr/bin/ld: error: /usr/lib/libboost_test_exec_monitor-mt.a(unit_test_log.o): requires dynamic R_X86_64_PC32 reloc against 'std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld: error: /usr/lib/libboost_test_exec_monitor-mt.a(unit_test_log.o): requires unsupported dynamic reloc 11; recompile with -fPIC
I really can't see what I could have changed. The boost libraries are pulled straight from the ubuntu repositories. Anyone with any clues.

You are linking a static library (the Boost one) into a dynamic library. Static libraries are not typically built with -fPIC, as they are assumed to be linked into a program only, not another library.
On 32 bit x86, such code is silently fixed up by relocating the portions of the code that are not position-independent to the load address; this makes the affected pages unshareable. For this to work, the relocation entry needs to be converted from a link time to a run time relocation.
This conversion fails on x86 64 bit; the two error messages mean
The relocation is applied to a 32 bit value, but the displacement may be larger than that (shared libraries live at random addresses for security reasons, which places them wide apart on 64 bit platforms, and
for this reason, there is no dynamic relocation type corresponding to the relocation entry from the static library.
Thus, the linker cannot generate code that would be loadable, and rightfully refuses to do so.
To solve this problem, you need to link against the shared libboost_test_exec_monitor-mt, or build a static library yourself.

Shared libraries can be set up in two ways. One is with absolute addresses, so that each binary that loads the shared object gets it own copy of the shared code, but the calls have no extra indirection and are as fast as possibly. The other way is with "PIC" or position independent code. This adds an extra layer of indirection but then one copy of the shared library code can serve all applications that need it (because the extra layer of indirection is per application binary).
What you're seeing is that when you try to build in 64-bit, the absolute addresses from the first option aren't able to force a particular 64-bit address (possibly some object file in your code doesn't support 64-bit addresses) and the compiler is telling you that you have to use option 2 with PIC enabled. In order to do this you'll need to compile all your code and libraries with -fPIC assuming g++/gcc. You may also need to link the library with -shared but I can't recall the precise times you have to do that.

Okay, Simon's answer really helped me along the way.
The ultimate solution to this particular problem was to use
libboost_unit_test_framework
(which comes with a shared library) in place of
libboost_test_exec_monitor
(which does not)

Related

Is the address of a C function symbol constant between compiles

I have been experimenting with symbol visibility in my shared library and noticed that the address / value of an exported function symbol does not seem to change. Are these addresses constant between compiles, or is this a coincidence?
The addresses where obtained on a Virtual Machine running Arch Linux using the command readelf with option -W and --dyn-syms.
The reason I'm asking is that I am wondering if the address of a templated C++ function could be used as an uuid for an object type. This is of interest in my serialization routine where I would like to setup an id system which is constant between compiles (object types are registered statically at initialization time, so order is not defined).
If build process is unchanged (i.e. compiler, linker, Makefiles and code remain the same) the static address in ELF file will not change either. But if any component changes, all bets are off.
More importantly, dynamic address (assigned by dynamic loader) will be different on each run due to address-space randomization in modern Linux distros so you should not rely on it.
When you build your code you can choose either to build it position dependent or position independent this has nothing to do with static build (though you can't build a position independent static binary). Position dependent binaries (given the same sources, compiler and build flags) will always generate the same addresses, but as I say further down, I wouldn't rely on it in release.
This is supplied by GCC's options -fPIE (Position independent executable), -fPIC (Position independent code), -pie. ELF executable files can be built as either position dependent or independent but shared objects (libraries) will always be built as position independent as you need to be able to load them in a random location given to you by the OS. From GCC's MAN page:
-fPIC
If supported for the target machine, emit position-independent code, suitable for dynamic linking and avoiding any limit on the size of the global offset table.
-fpie
-fPIE
These options are similar to -fpic and -fPIC, but generated position independent code can be only linked into executables. Usually these options are used when -pie GCC option will be used during linking.
-pie
Produce a position independent executable on targets which support it. For predictable results, you must also specify the same set of options that were used to generate code (-fpie, -fPIE, or model suboptions) when you specify this option.
When loading a PIC shared object you cannot assume it will reside in the same place for each run, as it might be affected by ASLR that is driven by the kernel.
In any way I don't think it's a good practice to use memory addresses as uuids to classes as these might change, even more so if these template classes are implemented as part of a shared object.

How do I compile gnu libstdc++.a with PIC?

I'm trying to compile a shared library for systems that don't have a libstdc++ new enough to run it, which means I need to statically link libstdc++ into the library. I'm trying to link directly to libstdc++.a (-Wl,-Bstatic -L/path/to/lib64 -lstdc++), but I'm getting an error that seems to indicate that my libstdc++.a wasn't linked with PIC:
/usr/bin/ld: lib64/libstdc++.a(compatibility.o): relocation R_X86_64_32 against `typeinfo for __cxxabiv1::__forced_unwind' can not be used when making a shared object; recompile with -fPIC
lib64/libstdc++.a: could not read symbols: Bad value
I tried recompiling GCC with CFLAGS="-fPIC" and CXXFLAGS="-fPIC", but that didn't seem to do anything.
Help?
The idea is to pass --with-pic option to the configure. GCC build is a very complex beast with multiple stages, so simply passing a CFLAGS variable may not work as intended.
Reading GCC configure documentation. I used this flag --enable-host-shared when building GCC as it said in the description:
Specify that the host code should be built into position-independent
machine code (with -fPIC), allowing it to be used within shared
libraries, but yielding a slightly slower compiler.

position independent executable (-pie) for arm(cortex-m3)

I'm programming for stm32 (Cortex-m3) with codesourcery g++ lite(based on gcc4.7.2 version). And I want the executables to be loaded dynamically.
I knew I have two options available:
1. relocatable elf, which needs a elf parser.
2. position independent code (PIC) with a global offset register
I prefer PIC with global offset register, because it seems it's easier to implement and I'm not familiar with elf or any elf library. Also, It's easy to generate a .bin file from an elf file with some tools.
I've tried building my program with "-msingle-pic-base -fpic" compiling options and "-pie" linking options, but then I got a linking error:
...path...ld.exe: ...path...thumb2\libstdc++.a(pure.o): relocation
R_ARM_THM_MOVW_ABS_NC against `a local symbol' can not be used when
making a shared object; recompile with -fPIC
I don't quite understand the error message. It seems the default standard c/c++ library can't go with my options and I need to get the source of the library and rebuild for my own purpose.
So,
1. Could anyone provide me any useful information/link on how to work with the position independent executable ?
2. with the -msingle-pic-base option, I don't need to care too much about the GOT and ld script anymore, right?
Note: Without the "-pie" linking option I can build the program. But the program fails when calling a c++ virtual function (when I'm using the IDE(keil)'s simulator to debug my program). I don't understand what's going on and what I've been missing.
----------------------------------------------------------------------
-- added 20130314
with the -msingle-pic-base option, I don't need to care too much about the GOT and ld script anymore, right?
From my experiments, the register (r9 is used in my program) should point to the beginning of the got.plt sections. Delete the "-pie" option, the linking will success, (with r9 properly set) then the c++ virtual function is called successfully. However, I still think the "-pie" option is important, which may ensure that the current standard library is position independent. Could anyone explain this for me?
----------------------------------------------------------------------
-- added 20130315
I took a look at the documents on ABI from ARM's website. But it was of little help because they are not targeting a specific platform. There seems to be a concept of EABI (I'm using sourcery's arm-none-eabi edition), but I couldn't find any documentation on "EABI" from arm's website. I can't neither find documentation on this topic from sourcery and gcc's. There're more than one implementation of PIC, so which one is the sourcery g++ using in the none-eabi case? I think the behaviors of the "-msingle-pic-base", "-fpie", "-pie" options are so poorly documented !
-----------------------------------------------------------------------
From the dis-assembly code, I just figured out that, whit the "-msingle-pic-base", the r9 should point to the base address of the ".got" section, the pointers in the .got sections are absolute pointer and the addressing of variable is similar to the description in the article : Position Independent Code (PIC) in shared libraries. So I still need to modify the ".got" sections on loading. I don't know what is the ".got.plt" section used for in my program. It seems that function calls are using PC-relative addressing.
How to build with the "-pie" or how to link a standard library compiled with "-fpic" is still a problem for me.
The error message tells you to recompile the libstdc++ library, which is most often built, when the gcc compiler is built.
Thus you must recompile your standard libraries (libstdc++, libgcc_*, libc, libm and the all) with -fPIC and link your project against them.
If you rely on prebuilt compiler packages, you're mostly out of the game in the microcontroller world. If you build your compiler yourself (which is, by the way, not too difficult, but an advanced/expert task) you are on the go.
It is also possible to compile your stdandard libraries yourself with the compiler you have. You will need the sources of libraries and figure out, how the compiler package build system builds them and you have to mimic this. Perhaps here are some experts, who can advise you on this way.
There's a nice blog post on this topic, eight years after asking the question initially, but it's there: https://mcuoneclipse.com/2021/06/05/position-independent-code-with-gcc-for-arm-cortex-m/
The general outline is that you have to:
Set up GOT from linker-generated information
Set up PLT from Program Header information
Implement a binder based on the GOT entries
Compile your library as a shared relocatable binary: -msingle-pic-base -mpic-register=r9 -mno-pic-data-is-text-relative -fPIC
Set R9 accordingly

Static linking - working with GTKmm application? - revised

Is it possible to make a static linking (compilation) on Gtk(mm) program? I need the program to be less relaying on dependences in user's system.
I try:
g++ -static data/Area.h data/Picture.cpp data/GLScene.cpp data/KBDialog.cpp data/Dialogs.h data/FilePreview.cpp data/MainWindow.cpp prog.cpp -o prog `pkg-config --cflags --libs gtkmm-2.4 gtkglextmm-1.2 exiv2`
but It fails:
/usr/bin/ld: cannot find -lgtkmm-2.4
/usr/bin/ld: cannot find -lGL
/usr/bin/ld: cannot find -latkmm-1.6
/usr/bin/ld: cannot find -lgdkmm-2.4
/usr/bin/ld: cannot find -lpangomm-1.4
/usr/bin/ld: cannot find -lgdk_pixbuf-2.0
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libgio-2.0.a(glocalfileinfo.o): In function `lookup_gid_name':
(.text+0x207a): warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libgio-2.0.a(glocalvfs.o): In function `g_local_vfs_parse_name':
(.text+0x26c): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libglib-2.0.a(gutils.o): In function `g_get_any_init_do':
(.text+0x1244): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libglib-2.0.a(gutils.o): In function `g_get_any_init_do':
(.text+0x1237): warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libglib-2.0.a(gutils.o): In function `g_get_any_init_do':
(.text+0x124f): warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libglib-2.0.a(gutils.o): In function `g_get_any_init_do':
(.text+0xf6e): warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libgio-2.0.a(glocalfileinfo.o): In function `lookup_uid_data':
(.text+0x1eea): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libX11.a(xim_trans.o): In function `_XimXTransSocketUNIXConnect':
(.text+0xe23): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libgio-2.0.a(gnetworkaddress.o): In function `g_network_address_parse':
(.text+0xe3c): warning: Using 'getservbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib/gcc/i686-linux-gnu/4.4.5/../../../../lib/libgio-2.0.a(gnetworkaddress.o): In function `g_network_address_parse':
(.text+0xe4c): warning: Using 'endservent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
collect2: ld returned 1 exit status
I would rather avoid doing that, because GTK depends on tricky low level libraries which are really very system specific (perhaps libfontconfig.so etc), and contains system-specific information (e.g. builtin paths for fonts...).
I also think that GTK needs dynamic shared libraries to implement theming or styling (so GTK itself is calling dlopen, and having a statically linked libdl is not reasonable).
I suggest at least linking dynamically gtk and all its dependencies.
(Your question asked twice pisses me off, so here is a more detailed answer that I might edit and complete further)
Why dynamically linked shared libraries are useful?
First, almost every binary is dynamically linked on today's Linux systems. On my Debian/Sid system, I only have /sbin/ldconfig /bin/sash and /usr/bin/rar statically linked executables, but about seven thousand other dynamically linked executables (under /bin & /usr/bin). Even essential programs like /sbin/init are today dynamically linked.
There are several wins in having mostly dynamically linked ELF executables using shared libraries
Avoid wasting disk space. When dynamically linked executables did not exist (1986 era, SunOS3.5, because the kernel was not able to mmap file segments), people took a lot of time mixing several binaries in a single one (I remember textedit and cmdtool being the same binary, a mix of several programs on SunOS3.5) to win disk space. Ok, disk space is cheaper today, but if my seven thousand programs each had to link statically libc that would consume several gigabytes of disk space (and that would mean an extra DVD or hours of networking upload when installing a Linux distribution).
Enabling an easier update. When the packaging system (apt-get, dpkg and friends on Debian) upgrades a common shared library (like the GLibC or Gtk), it replaces the dynamically linked shared libraries (*.so files, called ELF shared objects) and all the future executions of binary using them take profit. So if /usr/lib/libgtk-3.so is updated, there is no need to update /usr/bin/gedit to take advantage of the bug fixes inside libgtk-3.so; just restarting gedit will make it profit of improvements in libgtk-3.so
More efficient overall RAM usage. A file like libc.so is used by almost every process, and even libgtk-3.so is used by dozens of processes. Most of it is mmap-ed read-only "text" segment (notably containing the executable binary machine code and read-only constants like string); this mapping is using the same RAM cells for every process using it. So the memory is shared
Legal compliance with LGPL license
The LGPL-2.1 license of GTK libraries is the only reason why you are legally allowed to use GTK (i.e. run GTK programs, and link your own program with GTK). This license gives you rights, in particular the one to improve GTK or take advantage of GTK improvements, but you should not prohibit users of your (e.g. proprietary) program linking /usr/lib/gtk-3.so to take advantage of improvements inside GTK itself. The section 6 of LGPL2.1 mention explicitly dynamic linking. You are not allowed to distribute statically linked GTK binaries without giving the user the mean to upgrade his GTK library. The most convenient way is having your GTK program dynamically linked against libgtk-3.so. A less easy alternative would be to distribute your statically linked executable with its object *.o files and instructions on how to re-link it statically against an hypothetical improved libgtk.a (which don't exist).
Plugin ability to dynamically load other library modules
A program can load some shared object at run time using the dlopen function (based upon the mmap system call, thru the -ldl library). This is how plugins are possible on Linux. GTK uses itself very actively this ability: theming, styling, and perhaps fonts are using dlopen and implemented by dlopen-ing appropriate stuff. Since dlopen is a public interface to the dynamic loader /lib64/ld-linux-x86-64.so.2, the -ldl library is a dynamically shared object libdl.so.2 sharing functionality and code with the dynamic loader (itself referenced in every dynamically linked executable as the "ELF interpreter"). It is uncommon and unwise to link -ldl statically. Even the libc.so library may load other modules (perhaps for DNS support, etc...); some functionalities are restricted in statically linked executables (see file /etc/nsswitch.conf etc.).
dynamic linking is slightly slower at startup time, since a program has to initiate and dynamically load (this is the role of ld-linux-x86-64.so.2) at startup all the dynamic libraries it needs. Code inside a dynamic library needs to be position independent code otherwise the relocation part of dynamically loaded libraries would be too big (and the relocation effort at start-up too long), which may cost an extra register (and this is mostly true on 32 bits x86 processors, much less on x86-64 or AMD64 64 bits ones) so makes up slightly bigger machine code (on 32 bits x86 machines, we are speaking of a few percents of size increase and runtime slowdown; on 64 bits machines, it is negligible). Of course, relocating hundred of thousands of external calls may take some time (and happens more with C++ code than with C code, perhaps because of name mangling issues).
Why you (Marco) should not statically link your GTK binary?
The five first points above should convince you that linking statically GTK is an evil thing to do. In particular, take attention of the legal aspects (LGPL): making knowingly an LGPL violation is a huge professional mistake, don't do that.
If you really wanted, with weeks of effort, you probably could be technically able (by recompiling and hacking GTK source code) to link statically your binary with GTK (with some reduced functionalities, like no theming), but that is probably unethical and useless. If your boss is stupid enough to require you that, try to convince him (or else find another job). And the very fact that you've asked on a public forum how to link statically GTK (which I am understanding as "how to violate the LGPL license") put you at risk. There are organizations -like gpl-violations- which take attention to that.
I don't see any useful reason to statically link a GTK program. Even proprietary programs using a GUI library are dynamically linked (a good example is AMD FGLRX driver and its companion programs like amdccle providing a Qt based graphical interface for installation).
Of course, you may want to deal with dependencies. Leave that to the package manager of your linux distribution.
If you want more help, please explain much more what you really want to do, and convince us that you don't ask help in violating a license. Better yet, try to distribute your software with a free license like e.g. GPLv3

What does -fPIC mean when building a shared library?

I know the '-fPIC' option has something to do with resolving addresses and independence between individual modules, but I'm not sure what it really means. Can you explain?
PIC stands for Position Independent Code.
To quote man gcc:
If supported for the target machine, emit position-independent code, suitable for dynamic linking and avoiding any limit on the size of the global offset table. This option makes a difference on AArch64, m68k, PowerPC and SPARC.
Use this when building shared objects (*.so) on those mentioned architectures.
The f is the gcc prefix for options that "control the interface conventions used
in code generation"
The PIC stands for "Position Independent Code", it is a specialization of the fpic for m68K and SPARC.
Edit: After reading page 11 of the document referenced by 0x6adb015, and the comment by coryan, I made a few changes:
This option only makes sense for shared libraries and you're telling the OS you're using a Global Offset Table, GOT. This means all your address references are relative to the GOT, and the code can be shared accross multiple processes.
Otherwise, without this option, the loader would have to modify all the offsets itself.
Needless to say, we almost always use -fpic/PIC.
man gcc says:
-fpic
Generate position-independent code (PIC) suitable for use in a shared
library, if supported for the target machine. Such code accesses all
constant addresses through a global offset table (GOT). The dynamic
loader resolves the GOT entries when the program starts (the dynamic
loader is not part of GCC; it is part of the operating system). If
the GOT size for the linked executable exceeds a machine-specific
maximum size, you get an error message from the linker indicating
that -fpic does not work; in that case, recompile with -fPIC instead.
(These maximums are 8k on the SPARC and 32k on the m68k and RS/6000.
The 386 has no such limit.)
Position-independent code requires special support, and therefore
works only on certain machines. For the 386, GCC supports PIC for
System V but not for the Sun 386i. Code generated for the
IBM RS/6000 is always position-independent.
-fPIC
If supported for the target machine, emit position-independent code,
suitable for dynamic linking and avoiding any limit on the size of
the global offset table. This option makes a difference on the m68k
and the SPARC.
Position-independent code requires special support, and therefore
works only on certain machines.