Recently I changed (for packaging) my own libraries to STATIC. Now i receive error messages that a variable defined in the library is multiply defined. It is OK, the libraries use each other. But why was this not noticed by the linker until I changed to STATIC?
In one of my files, I set the variable declared as 'extern' and the linker also flags it as 'multiply defined'. Is it OK?
Basically the compiler has 4 stages:
pre-processing: macro and symbol edition
compiling : generate assembling code to be executed by the processor
assembling : generate binary code that the machine can understand(0/1 binary code)
Linking: the three previous operations are done separately for each file, however we need to edit the address mapping of each variable, pointer, function for the whole project here when we will have some problems when we have some multiple definition of variable because linking will check all file and generate an output for the whole project.
if a library is declared as static then the declared and defined function inside this library can't be used until the run time but in compile time it is not allowed to use this library in other files so if it is the case we will get errors during linking stage because the compiler will figure out the use of the function inside this static library by another file which is not allowed.
If you want it static , then use runtime concept to use this library (for example in C you can use pointer to function .
But why was this not noticed by the linker until I changed to STATIC?
Runtime linker allows duplicate symbol definitions (only one will be used at runtime, this is symbol interposition).
Related
I recently read the CSAPP and had some doubts about the compilation system part of it.
Now we have a sample using HelloWorld.c(just print hello world). The book said in Pre-processor phase, they replace the "#include " line with the content of this header file. But when I open the stdio.h, I find that there is only a declaration for printf() and there is no concrete implementation. So in the compilation system, when will the specific implementation of printf() be introduced?
And the book also said, in linking phase, the linker(ld) linked helloworld.o and printf.o . Why the linker knows to link my object file to printf.o? In a compilation system, why does it declare this function in the first step(Pre-processor phase) and link the concrete implementation in the last step(linking phase)?
Practically, over-simplified:
You can compile a function into a library (ex. .a or .so file on unix).
The library has a function body (assembly instructions) and a function name. Ex. the library libc.so has printf function that starts at character number 0xaabbccdd in the library file libc.so.
You want to compile your program.
You need to know what arguments printf takes. Does it take int ? Does it take char *? Does it take uint_least64_t? It's in the header file - int printf(const char *, ...);. The header tells the compiler how to call the function (what parameters does the function take and what type it returns). Note that each .c file is compiled separately.
The function declaration (what arguments the function takes and what does it return) is not stored in the library file. It is stored in the header (only). The library has function name (only printf) and compiled function body. The header has int printf(const char *, ...); without function body.
You compile your program. The compiler generates the code, so that arguments with proper size are pushed onto the stack. And from the stack your code takes variable returned from the function. Now your program is compiled into assembly that looks like push pointer to "%d\n" on the stack; push some int on the stack; call printf; pop from the stack the returned "int"; rest of the instructions;.
Linker searches through your compiled program and it sees call printf. It then says: "Och, there is no printf body in your code". So then it searches printf in the libraries, to see where it is. The linker goes through all the libraries you link your program with and it finds printf in the standard library - it's in libc.so at address 0xaabbccdd. So linker substitutes call printf for goto libs.so file to address 0xaabbccdd kind-of instruction.
After all "symbols" (ie. function names, variables names) are "resolved" (the linker has found them somewhere), then you can run your program. The call printf will jump into the file libc.so at specified location.
What I have written above is only for illustration purposes.
Why the linker knows to link my object file to printf.o
Because the complier notes this inside what it produces, typically called object files (.o).
why does it declare this function in the first step ...
To know about it.
... and link the concrete implementation in the last step
Because there is no need to do this earlier.
All the C and C++ standards tell you is that you need to #include a given header file in order to introduce some functionality (on some platforms that might not even be necessary although inclusion is a good idea since then you're writing portable code).
That affords compilers a lot of flexibility.
The linking, if any, will be done automatically. Note that some functions might even be hardcoded into the compiler itself.
By default the library ( containing the implementation of printf ) is linked everytime in your C program.
By including headers you just specify (for the time being) at compile time that the implementations of the declared functions (inside the header) are somewhere else. And later in the linking phase, those function implementations are 'added' in your code.
Why the linker knows to link my object file to printf.o?
LD knows how to search and find them. You can see the with man ld.so:
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not
exist. Use of DT_RPATH is deprecated.
Using the environment variable LD_LIBRARY_PATH, unless the executable is being run in secure-execution mode (see below), in which
case this variable is ignored.
Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present. Such directories are searched only
to find those objects required by DT_NEEDED (direct dependencies)
entries and do not apply to those objects' children, which must
themselves have their own DT_RUNPATH entries. This is unlike DT_RPATH,
which is applied to searches for all children in the dependency tree.
From the cache file /etc/ld.so.cache, which contains a compiled list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z nodeflib
linker option, shared objects in the default paths are skipped. Shared
objects installed in hardware capability directories (see below) are
preferred to other shared objects.
In the default path /lib, and then /usr/lib. (On some 64-bit architectures, the default paths for 64-bit shared objects are /lib64,
and then /usr/lib64.) If the binary was linked with the -z nodeflib
linker option, this step is skipped.
In a compilation system, why does it declare this function in the first step(Pre-processor phase) and link the concrete implementation in the last step(linking phase)?
In the compilation stage, you need to know what you're going to link to and compile accordingly, so it needs to read the .h files with the definition. In the linking stage, only .o files are needed.
If I defined a function twice, I'll get a redefinition error message, but
I'm confused that redefinition happened in compile or link time?
and why you can override malloc in libc without a redefinition error?
You get a function redefinition error when you have two function with the same prototype or signature (function signature is made of the function name number of parameters and parameter types, does NOT include the return type).
This is a compile time error if the compiler see two functions with the same signature:
int foo(int a);
double foo(int b);
Why you can override function calls in libraries? Let's look at how the code is build into an executable:
the compiler is called for each source file and outputs an object file: any function call which cannot be resolved (i.e. calling a function in a different file) is an external symbol which the linker will have to resolve.
the linker take all the object files and tries to resolve all the symbols; but it does this on a first come first served manner. For a external symbol it will consider the first definition it finds and not worry about the fact that there may be more definitions of the same symbol available.
So, the linker actually allows you to override a function's behavior. And it all depends on the order the files are linked - the first function definition it finds is the one used to resolve the symbol.
Hope this sheds some light on the matter.
Either or both. It can also result from the programmer editing source code or modifying build scripts.
"Redefinition" errors are emitted when the linker find two things (symbols) with the same name.
There are many reasons the linker might find two symbols with the same name. Some possibilities (there are many permutations) include;
An object file is specified twice in the link command. This usually results from an error in a build script.
Two object files that contain the same function definition. This results from code duplication - for example, a function definition being copied into different source files, which are then compiled and linked. It can also result from monkey business with the preprocessor (e.g. #includeing a file that contains the definition of a global variable by two source files).
The causes of things like the above are generally programmer error (e.g. supplying a bad linker command in a build script, misuse of the preprocessor, copying and pasting code between projects.
The reason functions in libraries like libc can often be "overridden" is that the linker typically only looks for symbols in libraries if it can't find them in object files. So, if an object file defines malloc() the linker will resolve all calls to that, and not attempt to resolve using the version in a library. This sort of thing is quite dangerous, because some other functions within libraries (e.g. even within libc) may resolve directly to the original malloc() (e.g. some calls may be inlined) which can unpredictable behaviours. This sort of behaviour is also linker specific: although this sort of thing is common with unix/linux variants, there are systems where the linkers do things differently.
For Example
#include <iostream>
int add(int x, int y);
int main()
{
cout << add(5, 5) << endl;
}
This would compile but not link. I understand the problem, I just don't understand why it compiles fine but doesn't link.
Because the compiler doesn't know whether that function is provided by a library (or other translation unit). There's nothing in your prototype that tells the compiler the function is defined locally (you could use static for that).
The input to a C or C++ compiler is one translation unit - more or less one source code file. Once the compiler is finished with that one source code, it has done its job.
If you call/use a symbol, such as a function, which is not part of that translation unit, the compiler assumes it's defined somewhere else.
Later on, you link together all the object files and possibly the libraries you want to use, all references are tied together - it's only at this point, when pulling together everything that's supposed to create an executable, one can know that something is missing.
When a compiler compiles, it generates the output (object file) with the table of defined symbols (T) and undefined symbols (U) (see man page of nm). Hence there is no requirement that all the references are defined in every translation unit. When all the object files are linked (with any libraries etc), the final binary should have all the symbols defined (unless the target in itself is a library). This is the job of the linker. Hence based on the requested target type (library or not), the linker might not or might give an error for undefined functions. Even if the target is a non-library, if it is not statically linked, it still might refer to shared libraries (.so or .dll), hence if on the target machine while the binary is run, if the shared libraries are missing or if any symbols missing, you might even get a linker error. Hence between compiler, linker and loader, every one is trying to best provide you with the definition of every symbol needed. Here by giving declaring add, you are pacifying the compiler, which hopes that the linker or loader would do the required job. Since you didnt pacify the linker (by say providing it with a shared library reference), it stops and cribs. If you have even pacified the linker, you would have got the error in the loader.
The #include directive will result in placing the content of the header file in the source code before compilation : for example if I included stdio.h , the preprocessor will work on placing all the content of stdio.h in the source code then compile isn't it ?
So let's pretend that I'm just using the printf() function in my code. So there must be something that will happen after the compilation and between the linking that will delete all function implementations that were included from the header file and only insert the printf() function implementation in the Import table of the executable , knowing that all other functions were compiled with the source. Can you explain that to me please ?
The printf function is not actually in the header file, it's in a library that is automatically linked to the executable by the linker.
Header files should contain only function prototypes, i.e. the declaration of the functions and not the definitions. And as there's no function definitions in the header file, no code will actually be generated for them, and the compiler will make sure that only the functions you actually call will get an entry in the generated object file so the linker knows about it.
A function prototype declared in a C header file acts as a compile-time constraint. It identifies the correct number of parameters and correct types of those parameters. The verification of such constraints happens during the compilation phase (i.e. *.c -> *.o).
The (static) linking phase eliminates unused binary objects from programming libraries on a per-object-file basis. A programming library is an archive of binary object files (*.o), each containing the implementation of a collection of functions, constants, etc. As for your question, the binary object file containing the implementation of printf (and everything else in the same object file) will be linked into your program. Other unused object files will be eliminated as a link-time optimization.
The header file only contains a declaration, e.g. of the form
size_t printf(char const *format, ...);
This tells the compiler that if it encounters the word printf and it is used as a function, then it can generate a function call.
The call instruction contains a placeholder for the actual address of the function, which is then inserted by the linker when building the final executable.
The linker generally just keeps a running list of yet unresolved symbols, and adds to this list when it encounters one of these placeholders, and fills in the actual address when it finds a definition for the symbol. In the case of printf, that definition is found in the C standard library, so the linker ensures that the library is loaded at runtime and the call instructions point at the right address.
Description :
a. Class X contains a static private data member ptr and static public function member getptr()/setptr().
In X.cpp, the ptr is set to NULL.
b. libXYZ.so (shared object) contains the object of class X (i.e libXYZ.so contains X.o).
c. libVWX.so (shared object) contains the object of class X (i.e libVWX.so contains X.o).
d. Executable a.exe contains X.cpp as part of translation units and finally is linked to libXYZ.so, libVWX.so
PS:
1. There are no user namespaces involved in any of the classes.
2. The libraries and executable contain many other classes also.
3. no dlopen() has been done. All libraries are linked during compile time using -L and -l flags.
Problem Statement:
When compiling and linking a.exe with other libraries (i.e libXYZ.so and libVWX.so), I expected a linker error (conflict/occurance of same symbol multiple times) but did not get one.
When the program was executed - the behavior was strange in SUSE 10 Linux and HP-UX 11 IA64.
In Linux, when execution flow was pushed across all the objects in different libraries, the effect was registered in only one copy of X.
In HPUX, when execution flow was pushed across all the objects in different libraries, the effect was registered in 3 differnt copies of X (2 belonging to each libraries and 1 for executable)
PS : I mean during running the program, the flow did passed thourgh multiple objects belonging to a.exe, libXYZ.so and libVWX.so) which interacted with static pointer belonging to X.
Question:
Is Expecting linker error not correct? Since two compilers passed through compilation silently, May be there is a standard rule in case of this type of scenario which I am missing. If so, Please let me know the same.
How does the compiler (gcc in Linux and aCC in HPUX) decide how many copies of X to keep in the final executable and refer them in such scenarios.
Is there any flag supported by gcc and aCC which will warn/stop compilation to the users in these kind of scenarios?
Thanks for your help in advance.
I'm not too sure that I've completely understood the scenario. However,
the default behavior on loading dynamic objects under Linux (and other
Unices) is to make all symbols in the library available, and to only use
the first encountered. Thus, if you both libXYZ.so and libVWX.so
contain a symbol X::ourData, it is not an error; if you load them in
that order, libVWX.so will use the X::ourData from libXYZ.so,
instead of its own. Logically, this is a lot like a template definition
in a header: the compiler chooses one, more or less by chance, and if
any of the definitions is not the same as all of the others, it's
undefined behavior. This behavior can be
overridden by passing the flag RTLD_LOCAL to dlopen.
With regards to your questions:
The linker is simply implementing the default behavior of dlopen (that which you get when the system loads the library implicitely). Thus, no error (but the logical equivalent of undefined behavior if any of the definitions isn't the same).
The compiler doesn't decide. The decision is made when the .so is loaded, depending on whether you specify RTLD_GLOBAL or RTLD_LOCAL when calling dlopen. When the runtime calls dlopen implicitly, to resolve a dependency, it will use RTLD_GLOBAL if this occurs when loading the main executable, and what ever was used to load the library when the dependency comes from a library. (This means, of course, that RTLD_GLOBAL will propagate until you invoke dlopen explicitly.)
The function is "public static", so I assume it's OOP-meaning of "static" (does not need instance), not C meaning of static (file-static; local to compilation unit). Therefore the functions are extern.
Now in Linux you have explicit right to override library symbols, both using another library or in the executable. All extern symbols in libraries are resolved using the global offset table, even the one the library actually defines itself. And while functions defined in the executable are normally not resolved like this, but the linker notices the symbols will get to the symbol table from the libraries and puts the reference to the executable-defined one there. So the libraries will see the symbol defined in the executable, if you generated it.
This is explicit feature, designed so you can do things like replace memory allocation functions or wrap filesystem operations. HP-UX probably does not have the feature, so each library ends up calling it's own implementation, while any other object that would have the symbol undefined will see one of them.
There is a difference beetween "extern" symbols (which is the default in c++) and "shared libary extern". By default symbols are only "extern" which means the scope of one "link unit" e.g. an executable or a library.
So the expected behaviour would be: no compiler error and every module works with its own copy.
That leads to problems of course in case of inline compiling etc...etc...
To declare a symbol "shared library extern" you have to use a ".def" file or an compiler declaration.
e.g. in visual c++ that would be "_declspec(dllexport)" and "_declspec(dllimport)".
I do not know the declarations for gcc at the moment but I am sure someone does :-)