Related
I recently read the CSAPP and had some doubts about the compilation system part of it.
Now we have a sample using HelloWorld.c(just print hello world). The book said in Pre-processor phase, they replace the "#include " line with the content of this header file. But when I open the stdio.h, I find that there is only a declaration for printf() and there is no concrete implementation. So in the compilation system, when will the specific implementation of printf() be introduced?
And the book also said, in linking phase, the linker(ld) linked helloworld.o and printf.o . Why the linker knows to link my object file to printf.o? In a compilation system, why does it declare this function in the first step(Pre-processor phase) and link the concrete implementation in the last step(linking phase)?
Practically, over-simplified:
You can compile a function into a library (ex. .a or .so file on unix).
The library has a function body (assembly instructions) and a function name. Ex. the library libc.so has printf function that starts at character number 0xaabbccdd in the library file libc.so.
You want to compile your program.
You need to know what arguments printf takes. Does it take int ? Does it take char *? Does it take uint_least64_t? It's in the header file - int printf(const char *, ...);. The header tells the compiler how to call the function (what parameters does the function take and what type it returns). Note that each .c file is compiled separately.
The function declaration (what arguments the function takes and what does it return) is not stored in the library file. It is stored in the header (only). The library has function name (only printf) and compiled function body. The header has int printf(const char *, ...); without function body.
You compile your program. The compiler generates the code, so that arguments with proper size are pushed onto the stack. And from the stack your code takes variable returned from the function. Now your program is compiled into assembly that looks like push pointer to "%d\n" on the stack; push some int on the stack; call printf; pop from the stack the returned "int"; rest of the instructions;.
Linker searches through your compiled program and it sees call printf. It then says: "Och, there is no printf body in your code". So then it searches printf in the libraries, to see where it is. The linker goes through all the libraries you link your program with and it finds printf in the standard library - it's in libc.so at address 0xaabbccdd. So linker substitutes call printf for goto libs.so file to address 0xaabbccdd kind-of instruction.
After all "symbols" (ie. function names, variables names) are "resolved" (the linker has found them somewhere), then you can run your program. The call printf will jump into the file libc.so at specified location.
What I have written above is only for illustration purposes.
Why the linker knows to link my object file to printf.o
Because the complier notes this inside what it produces, typically called object files (.o).
why does it declare this function in the first step ...
To know about it.
... and link the concrete implementation in the last step
Because there is no need to do this earlier.
All the C and C++ standards tell you is that you need to #include a given header file in order to introduce some functionality (on some platforms that might not even be necessary although inclusion is a good idea since then you're writing portable code).
That affords compilers a lot of flexibility.
The linking, if any, will be done automatically. Note that some functions might even be hardcoded into the compiler itself.
By default the library ( containing the implementation of printf ) is linked everytime in your C program.
By including headers you just specify (for the time being) at compile time that the implementations of the declared functions (inside the header) are somewhere else. And later in the linking phase, those function implementations are 'added' in your code.
Why the linker knows to link my object file to printf.o?
LD knows how to search and find them. You can see the with man ld.so:
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not
exist. Use of DT_RPATH is deprecated.
Using the environment variable LD_LIBRARY_PATH, unless the executable is being run in secure-execution mode (see below), in which
case this variable is ignored.
Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present. Such directories are searched only
to find those objects required by DT_NEEDED (direct dependencies)
entries and do not apply to those objects' children, which must
themselves have their own DT_RUNPATH entries. This is unlike DT_RPATH,
which is applied to searches for all children in the dependency tree.
From the cache file /etc/ld.so.cache, which contains a compiled list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z nodeflib
linker option, shared objects in the default paths are skipped. Shared
objects installed in hardware capability directories (see below) are
preferred to other shared objects.
In the default path /lib, and then /usr/lib. (On some 64-bit architectures, the default paths for 64-bit shared objects are /lib64,
and then /usr/lib64.) If the binary was linked with the -z nodeflib
linker option, this step is skipped.
In a compilation system, why does it declare this function in the first step(Pre-processor phase) and link the concrete implementation in the last step(linking phase)?
In the compilation stage, you need to know what you're going to link to and compile accordingly, so it needs to read the .h files with the definition. In the linking stage, only .o files are needed.
The question is in the context of the following question: Why are libraries not found, even though they are clearly on the -L path? . It seems that my libraries and my object file have a different ELF format, which might cause the linker to not "find" the libraries.
Now, this leads to a couple of questions:
It seems that my compiler normally generates SYSV ELF files. (Checked with file). However, for that particular C++ source, it generates a Linux/GNU ELF object file. I wonder why, so I reduced that source to an empty main method - and suddenly I get a SYSV object file. What inside a C++ source file can cause the compiler to switch the ELF format?
Is it true or false that I can not link Linux/GNU together with SYSV ELF?
Is there an option to force the compiler to create a certain ELF format?
I'm working with a Cray g++ (GCC) 5.3.0 20151204.
Regarding question 1: one of the answer seems to be that functions of type STT_GNU_IFUNC will cause the compiler/linker to switch the ELF file format of the corresponding object file from SYSV to GNU/Linux.
However, I still couldn't find out how I can identify functions in an object file which have type STT_GNU_IFUNC. Looking at object files which are GNU/Linux with objdump, readelf and nm still don't show a single function of type STT_GNU_IFUNC.
Some more information on STT_GNU_IFUNC can be found at https://www.airs.com/blog/archives/403 and in the following related question: How do I compile on linux to share with all distributions?
Motivation
I have 2 static libraries, libStatic1.a and libStatic2.a. In addition, I have many SOs (Shared Objects) that compile with libStatic1.a.
Up until now, libStatic1.a and libStatic2.a were independent and everything was okay. But now I added to the code that generates libStatic1.a a dependency on the code that generates libStatic2.a. Therefore, any SO that depends on libStatic1.a now needs to be compiled with libStatic2.a. This is undesirable, because it adds a dependency on libStatic2.a to every build targets that depends on libStatic1.a.
only on libStatic1.a now need to compile their code with libStatic2.a in order for the compilation/runtime to succeed/not crash. This creates an unnecessary coupling and I would like to avoid it.
Therefore, I need to somehow "embed" the object code of libStatic2.a in libStatic1.a. If I would just compile libStatic1.a with all the object files of libStatic2.a (in addition to its own), it will basically contain it but this creates another problem- If some user of libStatic1.a will decide to use libStatic2.a and will link it, he will get a weird "multiple definitions" error. If I could somehow tell the compiler to generate the object files of libStatic2.a with weak symbols (only for the use in libStatic1.a) this would solve the problem- no one will get multiple definitions, and no makefile of all the many SOs that use libStatic1.a will need to change.
My thinking: I know that it is possible (using GCC/g++ extensions to the C language) to declare a function with the keyword __attribute__ and the weak attribute as following:
void __attribute__((weak)) foo(int j);
Is there a way to tell the compiler (g++) to compile an entire compilation unit as "weak", meaning all its global symbols in the symbol table will be considered weak when linking?
Alternatively, is there a way to tell the linker (ld) to consider all the symbols of some object file/library as if they are weak?
If your library is small, the simplest way is still to change the declarations by adding manually the __attribute__((weak)).
Another possibility might be to ask g++ to spill the assembly code (with -S) and have some (perhaps awkor ed) script work on it.
You could also code a GCC plugin (assuming your g++ is a 4.6 version) or a GCC MELT extension for that.
Compile it normally and then objcopy the object file with --weaken.
No, there doesn't appear to be; are there so many weak external functions that it's not practical to set their attributes individually?
Description :
a. Class X contains a static private data member ptr and static public function member getptr()/setptr().
In X.cpp, the ptr is set to NULL.
b. libXYZ.so (shared object) contains the object of class X (i.e libXYZ.so contains X.o).
c. libVWX.so (shared object) contains the object of class X (i.e libVWX.so contains X.o).
d. Executable a.exe contains X.cpp as part of translation units and finally is linked to libXYZ.so, libVWX.so
PS:
1. There are no user namespaces involved in any of the classes.
2. The libraries and executable contain many other classes also.
3. no dlopen() has been done. All libraries are linked during compile time using -L and -l flags.
Problem Statement:
When compiling and linking a.exe with other libraries (i.e libXYZ.so and libVWX.so), I expected a linker error (conflict/occurance of same symbol multiple times) but did not get one.
When the program was executed - the behavior was strange in SUSE 10 Linux and HP-UX 11 IA64.
In Linux, when execution flow was pushed across all the objects in different libraries, the effect was registered in only one copy of X.
In HPUX, when execution flow was pushed across all the objects in different libraries, the effect was registered in 3 differnt copies of X (2 belonging to each libraries and 1 for executable)
PS : I mean during running the program, the flow did passed thourgh multiple objects belonging to a.exe, libXYZ.so and libVWX.so) which interacted with static pointer belonging to X.
Question:
Is Expecting linker error not correct? Since two compilers passed through compilation silently, May be there is a standard rule in case of this type of scenario which I am missing. If so, Please let me know the same.
How does the compiler (gcc in Linux and aCC in HPUX) decide how many copies of X to keep in the final executable and refer them in such scenarios.
Is there any flag supported by gcc and aCC which will warn/stop compilation to the users in these kind of scenarios?
Thanks for your help in advance.
I'm not too sure that I've completely understood the scenario. However,
the default behavior on loading dynamic objects under Linux (and other
Unices) is to make all symbols in the library available, and to only use
the first encountered. Thus, if you both libXYZ.so and libVWX.so
contain a symbol X::ourData, it is not an error; if you load them in
that order, libVWX.so will use the X::ourData from libXYZ.so,
instead of its own. Logically, this is a lot like a template definition
in a header: the compiler chooses one, more or less by chance, and if
any of the definitions is not the same as all of the others, it's
undefined behavior. This behavior can be
overridden by passing the flag RTLD_LOCAL to dlopen.
With regards to your questions:
The linker is simply implementing the default behavior of dlopen (that which you get when the system loads the library implicitely). Thus, no error (but the logical equivalent of undefined behavior if any of the definitions isn't the same).
The compiler doesn't decide. The decision is made when the .so is loaded, depending on whether you specify RTLD_GLOBAL or RTLD_LOCAL when calling dlopen. When the runtime calls dlopen implicitly, to resolve a dependency, it will use RTLD_GLOBAL if this occurs when loading the main executable, and what ever was used to load the library when the dependency comes from a library. (This means, of course, that RTLD_GLOBAL will propagate until you invoke dlopen explicitly.)
The function is "public static", so I assume it's OOP-meaning of "static" (does not need instance), not C meaning of static (file-static; local to compilation unit). Therefore the functions are extern.
Now in Linux you have explicit right to override library symbols, both using another library or in the executable. All extern symbols in libraries are resolved using the global offset table, even the one the library actually defines itself. And while functions defined in the executable are normally not resolved like this, but the linker notices the symbols will get to the symbol table from the libraries and puts the reference to the executable-defined one there. So the libraries will see the symbol defined in the executable, if you generated it.
This is explicit feature, designed so you can do things like replace memory allocation functions or wrap filesystem operations. HP-UX probably does not have the feature, so each library ends up calling it's own implementation, while any other object that would have the symbol undefined will see one of them.
There is a difference beetween "extern" symbols (which is the default in c++) and "shared libary extern". By default symbols are only "extern" which means the scope of one "link unit" e.g. an executable or a library.
So the expected behaviour would be: no compiler error and every module works with its own copy.
That leads to problems of course in case of inline compiling etc...etc...
To declare a symbol "shared library extern" you have to use a ".def" file or an compiler declaration.
e.g. in visual c++ that would be "_declspec(dllexport)" and "_declspec(dllimport)".
I do not know the declarations for gcc at the moment but I am sure someone does :-)
I have an ARM project that I'm building with make. I'm creating the list of object files to link based on the names of all of the .c and .cpp files in my source directory. However, I would like to exclude objects from being linked if they are never used. Will the linker exclude these objects from the .elf file automatically even if I include them in the list of objects to link? If not, is there a way to generate a list of only the objects that need to be linked?
You have to compile your code differently to strip out function and data that isn't used. Usually all the objects are compiled into the same symbol, so they can't be individually omitted if they're not used.
Add the two following switches to your compiler line:
-ffunction-sections -fdata-sections
When you compile, the compiler will now put individual functions and data into their own sections instead of lumping them all in one module section.
Then, in your linker, specify the following:
--gc-sections
This instructs the linker to remove unused sections ("gc" is for garbage collection). It will garbage collect parts of files and entire files. For example, if you're compiling an object, but only use 1 function of 100 in the object, it will toss out the other 99 you're not using.
If you run into issues with functions not found (it happens due to various reasons like externs between libraries), you can use .keep directives in your linker file (*.ld) in order to prevent garbage collection on those individual functions.
If you are using RealView, it seems that it is possible. This section discusses it:
3.3.3 Unused section elimination
Unused section elimination removes code that is never executed, or data that is not
referred to by the code, from the final image. This optimization can be controlled by the
--remove, --no_remove, --first, --last, and --keep linker options. Use the --info unused
linker option to instruct the linker to generate a list of the unused sections that have been
eliminated.
Like many people said, the answer is "depends". In my experience, RVCT is very good about dead code stripping. Unused code and data will almost always be removed in the final link stage. GCC, on the other hand (at least without the LLVM back end), is rather poor at whole image static analysis and will not do a very good job at removing unused code (and woe be it to you if your code is in different sections requiring long jumps). You can take some steps to mitigate it, such as using function-sections, which creates a separate section for each function and enables some better dead code stripping.
Have your linker generate a map file of your binary so you can see what made it in there and what got stripped out.
Depending on the sophistication of the compiler/linker and optimization level, the linker will not link in code that isn't being called.
What compiler/linker are you using? Some linkers do this automatically, and some provide the feature as a command-line option.
In my experience, many compilers will not include unused code on an object file basis. Some may not have this resolution and will include entire libraries ("because this makes the build process faster").
For example, given a file junk.c and it has three functions: Func1, Func2 and Func3. The build process creates an object file, junk.o, which has all three functions in it. If function Func2 is not used, it will be included anyway because the linker can't exclude one function out of an object file.
On the other hand, given files: Func1.c, Func2.c, and Func3.c, with the functions above, one per file. If Func2 in Func2.c is not used, the linker will not include it.
Some linkers are intelligent enough to exclude files out of libraries. However, each linker is different on its granularity of file inclusion (and thus file exclusion). Read your linker's manual or contact their customer support for exact information.
I suggest moving the suspect functions into a separate file (one function per file) and rebuild. Measure the code size before and after. Also, there may be a difference between Debug and Release linking. The Debug linking could be lazy and just throw everything in while the Release linking puts more effort into removing unused code.
Just my thoughts and experience, Your Mileage May Vary (YMMV).
Traditionally linkers link in all object files that are explicity specified in the command line, even if they could be left out and the program would not have any unresolved symbols. This means that you can deliberately change the behaviour of a program by including an object file that does something triggered from static initialization but is not called directly or indirectly from main.
Typically if you place most of your object files in a static library and link this library with a single object file containing your entry point the linker will only pick out members of the library (iteratively) that help resolve an unresolved symbol reference in the original object file or one included subsequently because it resolved a previous unresolved symbol.
In short, place most of your object files in a library and just link this with one object containing your entry point.