I made one SO file and compiled it with a compile option called "-Xlinker --strip-all " to counter any reverse engineering (use clang).
Thanks to this, most of the symbols of functions other than functions directly exposed to the outside do not appear (objdump -TC test.so). The question is, if a symbol is deleted like this, it should not be used inside the program, so I think it is normal. What am I missing?
You're right, debugging symbols aren't needed by the program itself to execute; the linker computes (and therefore knows at link-time) what the memory-address of each function/global-variable/etc will be at run-time, so it can just place that memory-address directly into the executable where necessary.
The symbols are there for a debugger to use, to make the debugging output easier for a human (or a debugging tool) to use and understand.
Related
This question gets cumbersome, lets try short version:
Usually when you fail with unresolved symbol reference it is quite strait forward, here you call something that linker cant find. You just feed your linker with library and it just works. Sometimes, there are cases when you banging your head on the wall and dont see why the linker wants this symbol here and there, it is not called, at least not directly. Is there a tool/linker switch that may explain why it thinks the symbol is needed "here"?
The original question:
It is all about static linkage. I have small utility, couple of lines of code, a couple of includes. The utility linked statically with library named lib1. Lets say lib1 has a dependency on another library, lib2, since lib1 uses symbol sym1 from lib2. However nothing that uses sym1 from lib2 is used/called from the utility nor anything from lib1 that may be dependent on lib2. The aforementioned tiny utility however fails with unresolved symbol for sym1. The first question is why? Since, in the utility the sym1 is not required anywhere and even no symbol from lib1 that uses sym1 used in the utility, why linker bothers with looking for this symbol in the first place? The second question, there is a chance that inclusion chain introduces the symbol sym1 to my utility, then it answers the "why" but it should not introduce it (at least there is no obvious reason for that), so the second question is how do I find why linker thinks the utility needs sym1 from lib2?
What/whenre/why: Linux, C/C++, GCC-9/Clang-9
Well, apparently I managed to answer the question not seeing the code as well as error message. Time to open my psi-consultancy.
Concerning linking on Linux/ELF target, it is important to remember that linker, while trying to satisfy/resolve symbols, is merging (and copying to final executable) sections (aka segments). Typically app has .text (code segment), .rodata (read-only data) segment, .data (r/w initialized data) segment, .bss (uninitialized data) etc. So if needed symbol is among, say, three functions in one compiled file, whole .text section of the file will be picked. And if unused but present in the section functions calls something else, linker will start searching for that "something else" to satisfy, even if it's irrelevant to the application.
Plus, there is some C++ specific thingy: for class with virtual functions compiler generates vtable, with pointers to each virtual function, and move this table to .rodata section. Note, that what
we think as code is actually ended up in (read-only) DATA section.
If you have all but one virtual functions defined, linker most likely will complain with error message like
/tmp/cc5YTcBb.o:(.rodata._ZTV3CL1[_ZTV3CL1]+0x18): undefined reference to `CL1::fnc2()
where you could see that problem is with .rodata, not .text.
Moral of the story: chop your code and data into large number of smallest possible sections/segments, your atoms of linking. Ideally, each functions goes into its own section, as well as piece of initialized or r/o data.
Final step is to instruct linker (via -Wl option) to discard (garbage-collect) all unused sections.
In general, one should expect more RAM used by linker, probably slower link stage, but smaller and faster app.
Command line to use, take a look at GCC manual wrt options meaning.
g++ -fdata-sections -ffunction-sections -fipa-pta main.cpp -Wl,--gc-sections -Wl,-O1 -Wl,--as-needed
i'm trying to cross -ompile an application to another system. I created all dependencies and started compiling. This then stops with one of my dependency libraries, namely Qt3, causing compiler errors:
Error: expected class-name before "{" token
and
Error: "QMutex" does not name a type
I'm suspecting the Q_EXPORT symbol to be defined wrong because i forgot to simulate some environment settings. But because it's definition depends on symbols which depend on symbols which depend on symbols, and so on, it's hard to check.
Just outputing it in an test program isn't working either because the value of Q_EXPORT is not always convertable to string.
My question is:
How do i check the value of a preprocessor symbol (while compiling/preprocessing) with GNU Compiler.
I thought there would be an option for this but i havn't found anything while searching on the web.
Debugging of macro symbols can be tricky, because it happens before the actual compilation [1]. Running the build system in such a way that you print the actual compilation command is a good starting point.
Then you can grab the actual compile command, and substitute the -c with -E or something similar, to inspect the actual generated preprocessor output. Then locate the actual place in the source that you are compiling - expect the output from -E to be HUGE - a million lines of output is not unusual. Use the #file and #line preprocessor symbols to track which file you are in, and what line you're at.
[1] Not strictly true in all compilers, as to help with precisely the problem that macros are making it hard to follow what the code is actually doing, modern compilers expand macros during the proper parsing of the code. However, that's not helping in this particular case, apparently.
I asked myself the following question, when I was discussing this topic .
Are there cases when some unused code from translation units will link to final executable code (in release mode of course) for popular compilers like GCC and VC++?
For example suppose we have 2 compilation units:
//A.hpp
//Here are declarations of some classes, functions, extern variables etc.
And source file
//A.cpp
//defination of A.hpp declarations
And finally main
//main.cpp
//including A.hpp library
#include "A.hpp"
//here we will use some stuff from A.hpp library, but not everything
My question is. What if in main.cpp not all the stuff from A.hpp is used? Will the linker remove all unused code, or there are some cases, when some unused code can link with executable file?
Edit: I'm interested in G++ and VC++ linkers.
Edit: Of course I mean in release mode.
Edit: I'm starting bounty for this question to get good and full answer. I'm expecting answer, which will explain in which cases g++ and VC++ linkers are linking junk and what kind of code they are able to remove from executable file(unneeded functions, unneeded global variables, unneeded class definitions, etc...) and why aren't they able to remove some kind of unneeded stuff.
As other posters have indicated, the linker typically does not remove dead code before building the final executable. However, there are often Optimization settings you can use to force the linker to try extra hard to do this.
For GCC, this is accomplished in two stages:
First compile the data but tell the compiler to separate the code into separate sections within the translation unit. This will be done for functions, classes, and external variables by using the following two compiler flags:
-fdata-sections -ffunction-sections
Link the translation units together using the linker optimization flag (this causes the linker to discard unreferenced sections):
-Wl,--gc-sections
So if you had one file called test.cpp that had two functions declared in it, but one of them was unused, you could omit the unused one with the following command to gcc(g++):
gcc -Os -fdata-sections -ffunction-sections test.cpp -o test.o -Wl,--gc-sections
(Note that -Os is an additional linker flag that tells GCC to optimize for size)
I have also read somewhere that linking static libraries is different though. That GCC automatically omits unused symbols in this case. Perhaps another poster can confirm/disprove this.
As for MSVC, as others have mentioned, function level linking accomplishes the same thing.
I believe the compiler flag for this is (to sort things into sections):
/Gy
And then the linker flag (to discard unused sections):
/OPT:REF
EDIT: After further research, I think that bit about GCC automatically doing this for static libraries is false.
The linker will not remove code.
You can still access it via dlsym dynamically in your code.
In general, linkers tend to include everything from the object files explicitly passed on the command line, but only pull in those object files from a static library that contain symbols needed to resolve external references from object files already linked.
However, a linker may decide to discard functions that are never called, or data which is never referenced. The precise details will depend on the compiler and linker switches.
In C++ code, if a source file is explicitly compiled and linked in to your application then I would expect that the objects with static storage duration that have constructors and/or destructors will be included, and their constructors/destructors run at the appropriate times. Consequently, any code called from those constructors or destructors must be in the final executable. However, if the code is not called from anywhere then you cannot write a program to tell whether or not the code is included without using things like dlsym, so the linker may well omit to include it in the final executable.
I would also expect that any symbols defined with global visibility such that they could be found via dlsym (as opposed to "hidden" symbols which are only visible within the executable) would be present in the final executable. However, this is an expectation rather than something I have confirmed by testing or reading the docs.
If you wanted to ensure code was in your executable even if it isn't called by inside it, you could load it in as a statically aware dynamic link library (a statically aware library is one which is loaded automatically into memory as the program is loaded, as opposed to the functionality where you can pass a string to a function that loads a library and then you manually search for hooks)
I have a static library (lib.a) and a program that links to it. The library doesn't have any entry point that would always be called before using it, but I need to execute a piece of code very early in the program (preferably before main() starts). Therefore I thought I would use static variable of my own class. I added new source file that contains something like:
#include <MyClass.h>
static MyClass myVar;
The constructor of MyClass would then execute my code. When I link lib.a and try executing "nm" on it I get information that myVar is there. However, when I link my program and try "nm" on it I do not see myVar. When I put this piece of code into an existing file then the symbol is visible in the final executable. Why is that? Can linker omit object file from lib.a library in this case? I know that the variable is not referenced from outside (it cannot be as it is static) but it should execute code on it's own and therefore I don't get why should it be removed.
In case it makes a difference I'm using some old SunPro compiler.
Technically speaking, the linker should be forced to include that object file while compiling your program. However, support for this is buggy in many compilers, such as MSVC++. Adding an external reference somewhere in your main program should force that object file to be included.
Also note that in the case of nm, it's possible that your static initializer was inlined, and therefore the symbol need not exist in your final binary. Try something with side effects (such as a std::cout statement) in your static, and make sure it doesn't run before blaming the compiler :)
It turns out that what the linker does is pretty standard (I don't mean C++ standard, just generally observer behaviour) and you can work around it. In GNU ld it is --whole-archive option, in my case of Sun tools it is -z allextract. Which didn't actually work as expected for my project, so I used some magic with weak symbols an -z weakextract to achieve what I wanted.
Linux/Gcc/LD - Toolchain.
I would like to remove STL/Boost debug symbols from libraries and executable, for two reasons:
Linking gets very slow for big programs
Debugging jumps into stl/boost code, which is annoying
For 1. incremental linking would be a big improvement, but AFAIK ld does not support incremental linking. There is a workaround "pseudo incremental linking" in an 1999 dr.dobb's journal (not in the web any more, but at archive.org (the idea is to put everything in a dynamic library and all updated object files in an second one that is loaded first) but this is not really a general solution.
For 2. there is a script here, but a) it did not work for me (it did not remove symbols), b) it is very slow as it works at the end of the pipe, while it would be more efficient to remove the symbols earlier.
Obviously, the other debug symbols should stay in place.
GNU strip accepts regex arguments to --strip-symbols=
The STL and boost symbols are name-mangled because of the namespaces they're in. I don't have GCC binutils handy at this moment, but just peek at the name mangling used for namespaces and construct the regex for 'symbols from namespace X' and pass this to --strip-symbols=
As far as I know there's no real option to do what you want in gcc. The main problem being that all the code you want to strip debug symbols for is defined in headers.
Otherwhise it would be possible to build a library separatly, strip that, and link with the stripped version.
But only getting debug symbols from certain parts of a compilation unit, while building and linking (for your desired link time speedup) is not possible in gcc as far as I know.
You probably don't want to strip the debug symbols from the shared libraries, as you may need that at some point.
If you are using GDB or DDD to debug, you may be able to get away with removing the Boost source files from the Source Path so it can't trace into the functions. (Or just don't trace into them, trace over!)
You can remove the option to compile the program with debug symbols, which will speed the link time.
Like the script you link to, you can consult the strip program ("man strip") to remove all or certain symbols.
You may want to use strip.
strip --strip-unneeded --strip-debug libfoo.so
Why don't you just build without debugging in the first place though?
This answer provides some specifics that I needed to make MSalters' answer work for removing STL symbols.
The STL symbol names are mangled. The trick is to find a regular expression that covers these names. I looked these symbols up with GNU's Binutils:
> nm --debug-syms <objectfile>
I basically searched on STL functions, like resize. If this is difficult, the output becomes readable when using the following command:
> nm --debug-syms --demangle <objectfile>
Look up a line number containing an STL function call, then look up it's mangled name on that same line number using the first provided command. This allowed me to see that all STL symbol names began with _ZNSt[0-9]+ or _ZSt[0-9]+, etc.
To allow GNU Strip to remove these symbols I used:
> strip --wildcard \
--strip-symbol='_ZNKSt*' \
--strip-symbol='_ZNSt*' \
--strip-symbol='_ZSt*' \
--strip-symbol='_ZNSa*' \
<objectfile>
I used these commands directly on the compiled/linked binary. I verified the removal of these symbols by comparing the output of nm before and after the removal (I wrote the output to files and used vimdiff). The --wildcard option allows the use of regular expressions. Although I would expect [0-9]* to mean 0 to an infinite amount of numbers, here it actually means 1 number followed by an infinite amount of anything (until the end of the line).
If you are looking to not step into STL code this can be achieved by gdb's skip file command, as done here.
Hope it helps
Which compiler are you using? For example, if I understand your question correctly, this is a trivial matter in MS Visual Studio.