Motivation
I have 2 static libraries, libStatic1.a and libStatic2.a. In addition, I have many SOs (Shared Objects) that compile with libStatic1.a.
Up until now, libStatic1.a and libStatic2.a were independent and everything was okay. But now I added to the code that generates libStatic1.a a dependency on the code that generates libStatic2.a. Therefore, any SO that depends on libStatic1.a now needs to be compiled with libStatic2.a. This is undesirable, because it adds a dependency on libStatic2.a to every build targets that depends on libStatic1.a.
only on libStatic1.a now need to compile their code with libStatic2.a in order for the compilation/runtime to succeed/not crash. This creates an unnecessary coupling and I would like to avoid it.
Therefore, I need to somehow "embed" the object code of libStatic2.a in libStatic1.a. If I would just compile libStatic1.a with all the object files of libStatic2.a (in addition to its own), it will basically contain it but this creates another problem- If some user of libStatic1.a will decide to use libStatic2.a and will link it, he will get a weird "multiple definitions" error. If I could somehow tell the compiler to generate the object files of libStatic2.a with weak symbols (only for the use in libStatic1.a) this would solve the problem- no one will get multiple definitions, and no makefile of all the many SOs that use libStatic1.a will need to change.
My thinking: I know that it is possible (using GCC/g++ extensions to the C language) to declare a function with the keyword __attribute__ and the weak attribute as following:
void __attribute__((weak)) foo(int j);
Is there a way to tell the compiler (g++) to compile an entire compilation unit as "weak", meaning all its global symbols in the symbol table will be considered weak when linking?
Alternatively, is there a way to tell the linker (ld) to consider all the symbols of some object file/library as if they are weak?
If your library is small, the simplest way is still to change the declarations by adding manually the __attribute__((weak)).
Another possibility might be to ask g++ to spill the assembly code (with -S) and have some (perhaps awkor ed) script work on it.
You could also code a GCC plugin (assuming your g++ is a 4.6 version) or a GCC MELT extension for that.
Compile it normally and then objcopy the object file with --weaken.
No, there doesn't appear to be; are there so many weak external functions that it's not practical to set their attributes individually?
Related
I found one question about compiling and linking in C++ and I don't know which answer is correct. It was discussed with my friends and opinions are divided. Here is a question:
In order to run program written in C++ language its source code is:
(A) compiled to machine code,
(B) compiled and linked to machine code
In my opinion the correct answer is A but I don't have any source to prove it.
Google, first hit.
Linkage is needed as well to create a standalone executable.
You need to link the code you have produced to make it into an executable file. For simple programs, the compiler does this for you, by calling the linker at the end of the compilation process.
The compiler proper simply translates C code to either assembler (classic C compiler) which is then assembled with an assembler or directly to machine code (many modern compilers). The machine code is usually produced as "object files", which are not "executable", because they refer to external units - such as when you call printf(). It is possible to write C code that is completely standalone, but you still typically need to combine more than one object file, and it certainly needs to be "formatted" to the right way to make an executable file - which is a different file-format than an object file [although typically fairly SIMILAR].
Compilation does nothing except creation of object files which means converting C/C++ source code to machine codes.
Linking process is the creation of executable file from multiple obj files. So for running an application/executable you have to also link it.
During compilation, compiler doesn't complain about non existing functions or broken functions, because it will assume it might be defined in another object (source code file). Linker verifies all functions and their existance, so if you have a broken function, you'll get error in linking process
Compiling: Takes input C/C++-code and produces machinecode (object file)
gcc –c MyProgram.c
Note that the object file does not contain all external references!
Linking: Combines object file with external references into an executable file
gcc MyProgram.o –o MyProgram
Note that no unresolved references!
Illustration:
Where libc.a is the standard C library and it's automatically linked into your programs by the gcc.
I've just noticed that your question was about c++, the same concept is in c++ too, if you understand this, you'll understand how it works in c++ too
strictly speaking. Answer A.
But for you to see the whole picture, lets say you have defined some function. Then the compiler writes the machine code code of that function at some address, and puts that address and the name of the function in the object ".o" file where the linker can find it. The linker then take this "machine code" and resolve the symbols as you might heard in some previous error.
I asked myself the following question, when I was discussing this topic .
Are there cases when some unused code from translation units will link to final executable code (in release mode of course) for popular compilers like GCC and VC++?
For example suppose we have 2 compilation units:
//A.hpp
//Here are declarations of some classes, functions, extern variables etc.
And source file
//A.cpp
//defination of A.hpp declarations
And finally main
//main.cpp
//including A.hpp library
#include "A.hpp"
//here we will use some stuff from A.hpp library, but not everything
My question is. What if in main.cpp not all the stuff from A.hpp is used? Will the linker remove all unused code, or there are some cases, when some unused code can link with executable file?
Edit: I'm interested in G++ and VC++ linkers.
Edit: Of course I mean in release mode.
Edit: I'm starting bounty for this question to get good and full answer. I'm expecting answer, which will explain in which cases g++ and VC++ linkers are linking junk and what kind of code they are able to remove from executable file(unneeded functions, unneeded global variables, unneeded class definitions, etc...) and why aren't they able to remove some kind of unneeded stuff.
As other posters have indicated, the linker typically does not remove dead code before building the final executable. However, there are often Optimization settings you can use to force the linker to try extra hard to do this.
For GCC, this is accomplished in two stages:
First compile the data but tell the compiler to separate the code into separate sections within the translation unit. This will be done for functions, classes, and external variables by using the following two compiler flags:
-fdata-sections -ffunction-sections
Link the translation units together using the linker optimization flag (this causes the linker to discard unreferenced sections):
-Wl,--gc-sections
So if you had one file called test.cpp that had two functions declared in it, but one of them was unused, you could omit the unused one with the following command to gcc(g++):
gcc -Os -fdata-sections -ffunction-sections test.cpp -o test.o -Wl,--gc-sections
(Note that -Os is an additional linker flag that tells GCC to optimize for size)
I have also read somewhere that linking static libraries is different though. That GCC automatically omits unused symbols in this case. Perhaps another poster can confirm/disprove this.
As for MSVC, as others have mentioned, function level linking accomplishes the same thing.
I believe the compiler flag for this is (to sort things into sections):
/Gy
And then the linker flag (to discard unused sections):
/OPT:REF
EDIT: After further research, I think that bit about GCC automatically doing this for static libraries is false.
The linker will not remove code.
You can still access it via dlsym dynamically in your code.
In general, linkers tend to include everything from the object files explicitly passed on the command line, but only pull in those object files from a static library that contain symbols needed to resolve external references from object files already linked.
However, a linker may decide to discard functions that are never called, or data which is never referenced. The precise details will depend on the compiler and linker switches.
In C++ code, if a source file is explicitly compiled and linked in to your application then I would expect that the objects with static storage duration that have constructors and/or destructors will be included, and their constructors/destructors run at the appropriate times. Consequently, any code called from those constructors or destructors must be in the final executable. However, if the code is not called from anywhere then you cannot write a program to tell whether or not the code is included without using things like dlsym, so the linker may well omit to include it in the final executable.
I would also expect that any symbols defined with global visibility such that they could be found via dlsym (as opposed to "hidden" symbols which are only visible within the executable) would be present in the final executable. However, this is an expectation rather than something I have confirmed by testing or reading the docs.
If you wanted to ensure code was in your executable even if it isn't called by inside it, you could load it in as a statically aware dynamic link library (a statically aware library is one which is loaded automatically into memory as the program is loaded, as opposed to the functionality where you can pass a string to a function that loads a library and then you manually search for hooks)
I have a static library (lib.a) and a program that links to it. The library doesn't have any entry point that would always be called before using it, but I need to execute a piece of code very early in the program (preferably before main() starts). Therefore I thought I would use static variable of my own class. I added new source file that contains something like:
#include <MyClass.h>
static MyClass myVar;
The constructor of MyClass would then execute my code. When I link lib.a and try executing "nm" on it I get information that myVar is there. However, when I link my program and try "nm" on it I do not see myVar. When I put this piece of code into an existing file then the symbol is visible in the final executable. Why is that? Can linker omit object file from lib.a library in this case? I know that the variable is not referenced from outside (it cannot be as it is static) but it should execute code on it's own and therefore I don't get why should it be removed.
In case it makes a difference I'm using some old SunPro compiler.
Technically speaking, the linker should be forced to include that object file while compiling your program. However, support for this is buggy in many compilers, such as MSVC++. Adding an external reference somewhere in your main program should force that object file to be included.
Also note that in the case of nm, it's possible that your static initializer was inlined, and therefore the symbol need not exist in your final binary. Try something with side effects (such as a std::cout statement) in your static, and make sure it doesn't run before blaming the compiler :)
It turns out that what the linker does is pretty standard (I don't mean C++ standard, just generally observer behaviour) and you can work around it. In GNU ld it is --whole-archive option, in my case of Sun tools it is -z allextract. Which didn't actually work as expected for my project, so I used some magic with weak symbols an -z weakextract to achieve what I wanted.
I have an ARM project that I'm building with make. I'm creating the list of object files to link based on the names of all of the .c and .cpp files in my source directory. However, I would like to exclude objects from being linked if they are never used. Will the linker exclude these objects from the .elf file automatically even if I include them in the list of objects to link? If not, is there a way to generate a list of only the objects that need to be linked?
You have to compile your code differently to strip out function and data that isn't used. Usually all the objects are compiled into the same symbol, so they can't be individually omitted if they're not used.
Add the two following switches to your compiler line:
-ffunction-sections -fdata-sections
When you compile, the compiler will now put individual functions and data into their own sections instead of lumping them all in one module section.
Then, in your linker, specify the following:
--gc-sections
This instructs the linker to remove unused sections ("gc" is for garbage collection). It will garbage collect parts of files and entire files. For example, if you're compiling an object, but only use 1 function of 100 in the object, it will toss out the other 99 you're not using.
If you run into issues with functions not found (it happens due to various reasons like externs between libraries), you can use .keep directives in your linker file (*.ld) in order to prevent garbage collection on those individual functions.
If you are using RealView, it seems that it is possible. This section discusses it:
3.3.3 Unused section elimination
Unused section elimination removes code that is never executed, or data that is not
referred to by the code, from the final image. This optimization can be controlled by the
--remove, --no_remove, --first, --last, and --keep linker options. Use the --info unused
linker option to instruct the linker to generate a list of the unused sections that have been
eliminated.
Like many people said, the answer is "depends". In my experience, RVCT is very good about dead code stripping. Unused code and data will almost always be removed in the final link stage. GCC, on the other hand (at least without the LLVM back end), is rather poor at whole image static analysis and will not do a very good job at removing unused code (and woe be it to you if your code is in different sections requiring long jumps). You can take some steps to mitigate it, such as using function-sections, which creates a separate section for each function and enables some better dead code stripping.
Have your linker generate a map file of your binary so you can see what made it in there and what got stripped out.
Depending on the sophistication of the compiler/linker and optimization level, the linker will not link in code that isn't being called.
What compiler/linker are you using? Some linkers do this automatically, and some provide the feature as a command-line option.
In my experience, many compilers will not include unused code on an object file basis. Some may not have this resolution and will include entire libraries ("because this makes the build process faster").
For example, given a file junk.c and it has three functions: Func1, Func2 and Func3. The build process creates an object file, junk.o, which has all three functions in it. If function Func2 is not used, it will be included anyway because the linker can't exclude one function out of an object file.
On the other hand, given files: Func1.c, Func2.c, and Func3.c, with the functions above, one per file. If Func2 in Func2.c is not used, the linker will not include it.
Some linkers are intelligent enough to exclude files out of libraries. However, each linker is different on its granularity of file inclusion (and thus file exclusion). Read your linker's manual or contact their customer support for exact information.
I suggest moving the suspect functions into a separate file (one function per file) and rebuild. Measure the code size before and after. Also, there may be a difference between Debug and Release linking. The Debug linking could be lazy and just throw everything in while the Release linking puts more effort into removing unused code.
Just my thoughts and experience, Your Mileage May Vary (YMMV).
Traditionally linkers link in all object files that are explicity specified in the command line, even if they could be left out and the program would not have any unresolved symbols. This means that you can deliberately change the behaviour of a program by including an object file that does something triggered from static initialization but is not called directly or indirectly from main.
Typically if you place most of your object files in a static library and link this library with a single object file containing your entry point the linker will only pick out members of the library (iteratively) that help resolve an unresolved symbol reference in the original object file or one included subsequently because it resolved a previous unresolved symbol.
In short, place most of your object files in a library and just link this with one object containing your entry point.
I have some doubt about how do programs use shared library.
When I build a shared library ( with -shared -fPIC switches) I make some functions available from an external program.
Usually I do a dlopen() to load the library and then dlsym() to link the said functions to some function pointers.
This approach does not involve including any .h file.
Is there a way to avoid doing dlopen() & dlsym() and just including the .h of the shared library?
I guess this may be how c++ programs uses code stored in system shared library. ie just including stdlib.h etc.
Nick, I think all the other answers are actually answering your question, which is how you link libraries, but the way you phrase your question suggests you have a misunderstanding of the difference between headers files and libraries. They are not the same. You need both, and they are not doing the same thing.
Building an executable has two main phases, compilation (which turns your source into an intermediate form, containing executable binary instructions, but is not a runnable program), and linking (which combines these intermediate files into a single running executable or library).
When you do gcc -c program.c, you are compiling, and you generate program.o. This step is where headers matter. You need to #include <stdlib.h> in program.c to (for example) use malloc and free. (Similarly you need #include <dlfcn.h> for dlopen and dlsym.) If you don't do that the compiler will complain that it doesn't know what those names are, and halt with an error. But if you do #include the header the compiler does not insert the code for the function you call into program.o. It merely inserts a reference to them. The reason is to avoid duplication of code: The code is only going to need to be accessed once by every part of your program, so if you needed further files (module1.c, module2.c and so on), even if they all used malloc you would merely end up with many references to a single copy of malloc. That single copy is present in the standard library in either it's shared or static form (libc.so or libc.a) but these are not referenced in your source, and the compiler is not aware of them.
The linker is. In the linking phase you do gcc -o program program.o. The linker will then search all libraries you pass it on the command line and find the single definition of all functions you've called which are not defined in your own code. That is what the -l does (as the others have explained): tell the linker the list of libraries you need to use. Their names often have little to do with the headers you used in the previous step. For example to get use of dlsym you need libdl.so or libdl.a, so your command-line would be gcc -o program program.o -ldl. To use malloc or most of the functions in the std*.h headers you need libc, but because that library is used by every C program it is automatically linked (as if you had done -lc).
Sorry if I'm going into a lot of detail but if you don't know the difference you will want to. It's very hard to make sense of how C compilation works if you don't.
One last thing: dlopen and dlsym are not the normal method of linking. They are used for special cases where you want to dynamically determine what behavior you want based on information that is, for whatever reason, only available at runtime. If you know what functions you want to call at compile time (true in 99% of the cases) you do not need to use the dl* functions.
You can link shared libraries like static one. They are then searched for when launching the program. As a matter of fact, by default -lXXX will prefer libXXX.so to libXXX.a.
You need to give the linker the proper instructions to link your shared library.
The shared library names are like libNAME.so, so for linking you should use -lNAME
Call it libmysharedlib.so and then link your main program as:
gcc -o myprogram myprogram.c -lmysharedlib
If you use CMake to build your project, you can use
TARGET_LINK_LIBRARIES(targetname libraryname)
As in:
TARGET_LINK_LIBRARIES(myprogram mylibrary)
To create the library "mylibrary", you can use
ADD_LIBRARY(targetname sourceslist)
As in:
ADD_LIBRARY(mylibrary ${mylibrary_SRCS})
Additionally, this method is cross-platform (whereas simply passing flags to gcc is not).
Shared libraries (.so) are object files where the actual source code of function/class/... are stored (in binary)
Header files (.h) are files indicating (the reference) where the compiler can find function/class/... (in .so) that are required by the main code
Therefore, you need both of them.