I am having trouble with very large object files being produced. We are working with VxWorks 5.5.1, but we have a GCC 4.1.2 available.
Our modules are roughly 6.2MB in size, and we are looking for ways to reduce that. The problem seems to be mainly caused by excessive use of templates. When dumping the symbols in the file using nm I get a text-file of 1.8MB. This tells me that almost ⅓ of the file is just the names. Is there any way to reduce the file size?
The following approaches have not worked:
--strip-all seems to have no effect - the output is the same as using --strip-debug
I cannot use --gc-sections, because it is not supported for that platform (the option is simply ignored)
I understand that VxWorks links the code at load time, but all it has to link is the C++ runtime library, and I don't want any symbols to be added to the global symbol table, so there should be a way to strip that information, right?
For reference, here is my linker version:
i386-wrs-vxworks-ld.exe --version
>>> GNU ld (Wind River VxWorks G++ DWARF-EH 4.1-131) 2.17.50.20070509
>>> SPR fixes: cq103489 cq111170 cq116027 cq116652 cq118878 cq125145
and my compiler version:
i386-wrs-vxworks-g++.exe --version
>>> i386-wrs-vxworks-g++.exe (GCC) 4.1.2
I see what you mean by wanting to strip the symbols out of the object. But if you were to strip all the symbols, you wouldn't have any symbol to use as the entry point to start your application. But I believe you still have options. Unfortunately VxWorks 5 is known for having a not very effecient C++ compiler.
If you compiled your application into a *.a (archive - aka static library), you would be able to link this into your operating system at build time, and call this from within usrAppInit.c. This should allow for striping out symbols - or at least moving them to an optional downloaded symbol table. From your application build properties, select the Macros tab and add your archive to the LIBS macro.
To make an archive, goto the build properties and select the Rules tab, then select archive from the dropdown box.
To trim the size of your OS (with your application linked in), modify your vxworks settings. Disable as many components as you can. Also be sure to use a downloaded symbol table (development tool components -> symbol table components -> symbol table initialization componts -> selecte symbol table initialization -> downloaded symbol table).
This will strip all the symbols out of the OS, and make a downloadable symbol table, that can be downloaded after boot time to debug.
Good luck!
PS. Make sure you have turned off debug (-g) in your compiler. Maybe we could help more if you post your compiler switches.
The way we have normally handled this is to compress the image. You will also need to build the bootrom so that it will decompress to RAM before running. I believe that there are standard makefile options that will allow you to do this.
Related
I am trying to build a C++ software with debug symbols enabled (-g). The problem is that the additional debug symbols make the included library that large that the final linking step fails with
../../lib/libutil.a: could not read symbols: Malformed archive
(at least I think it's failing because of it's size, which is slightly over 6 GB on disk)
I looked around and found hints that there might be maximum size for static libraries of 4 GB. Not sure if this applies to my system which is a 64 bit CentOS:
$ uname -a
Linux host 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
I am using the g++ (GCC) 4.8.2 compiler and the binutils version is GNU ld version 2.20.51.0.2-5.36.el6.
If there is indeed a maximum limit of 4GB that the archive tool or linker can handle what are my options without tempering too much with the interna of the build system (which is autotools by the way)?
It looks like the maximum size of an archive is, indeed four gigabytes.
Wikipedia has a nice write-up of archive file format. The limit that you appear to be hitting is:
A set of 32-bit big endian integers. One for each symbol,
recording the position within the archive of the header for the
file containing this symbol.
The way I parse this, is that all individual files in the .a file must start before the 4 gigabyte cutoff. Would be nice to get a more meaningful error message in this situation, though.
No easy way around this. Your only realistic option is to chop up your source code so that it gets linked into multiple .a archives, with each one under 4 gigs in size.
I do see a hard way around this. The limit in question is part of the symbol table, which I believe gets created by ranlib. If you hack your Makefile so that a symbol table does not get generated (perhaps by setting RANLIB=/bin/true), no symbol file would get generated, so you won't hit this limit. Your link time would suffer greatly, thought, and this would only allow you to create archives up to 9,999,999,999 bytes (not much more than what you already are creating), due to the 10 character limit of the file size in the ar header itself.
Just create multiple .a files.
I'm compiling Linux libraries (for Android, using NDK's g++, but I bet my question makes sense for any Linux system). When delivering those libraries to partners, I need to mark them with a version number. I must also be able to access the version number programatically (to show it in an "About" dialog or a GetVersion function for instance).
I first compile the libraries with an unversioned flag (version 0.0) and need to change this version to a real one when I'm done testing just before sending it to the partner. I know it would be easier to modify the source and recompile, but we don't want to do that (because we should then test everything again if we recompile the code, we feel like it would be less error prone, see comments to this post and finally because our development environment works this way: we do this process for Windows binaries: we set a 0.0 resources version string (.rc) and we later change it by using verpatch...we'd like to work with the same kind of process when shipping Linux binaries).
What would be the best strategy here?
To summarize, requirements are:
Compile binaries with "unset" version (0.0 or anything else)
Be able to modify this "unset" version to a specific one without having to recompile the binary (ideally, run a 3rd party tool command, as we do with verpatch under Windows)
Be able to have the library code retrieve it's version information at runtime
If your answer is "rename the .so", then please provide a solution for 3.: how to retrieve version name (i.e.: file name) at runtime.
I was thinking of some solutions but have no idea if they could work and how to achieve them.
Have a version variable (one string or 3 int) in the code and have a way to change it in the binary file later? Using a binary sed...?
Have a version variable within a resource and have a way to change it in the binary file later? (as we do for win32/win64)
Use a field of the .so (like SONAME) dedicated to this and have a tool allowing to change it...and make it accessible from C++ code.
Rename the lib + change SONAME (did not find how this can be achieved)...and find a way to retrieve it from C++ code.
...
Note that we use QtCreator to compile the Android .so files, but they may not rely on Qt. So using Qt resources is not an ideal solution.
I am afraid you started to solve your problem from the end. First of all SONAME is provided at link time as a parameter of linker, so in the beginning you need to find a way to get version from source and pass to the linker. One of the possible solutions - use ident utility and supply a version string in your binary, for example:
const char version[] = "$Revision:1.2$"
this string should appear in binary and ident utility will detect it. Or you can parse source file directly with grep or something alike instead. If there is possibility of conflicts put additional marker, that you can use later to detect this string, for example:
const char version[] = "VERSION_1.2_VERSION"
So you detect version number either from source file or from .o file and just pass it to linker. This should work.
As for debug version to have version 0.0 it is easy - just avoid detection when you build debug and just use 0.0 as version unconditionally.
For 3rd party build system I would recommend to use cmake, but this is just my personal preference. Solution can be easily implemented in standard Makefile as well. I am not sure about qmake though.
Discussion with Slava made me realize that any const char* was actually visible in the binary file and could then be easily patched to anything else.
So here is a nice way to fix my own problem:
Create a library with:
a definition of const char version[] = "VERSIONSTRING:00000.00000.00000.00000"; (we need it long enough as we can later safely modify the binary file content but not extend it...)
a GetVersion function that would clean the version variable above (remove VERSIONSTRING: and useless 0). It would return:
0.0 if version is VERSIONSTRING:00000.00000.00000.00000
2.3 if version is VERSIONSTRING:00002.00003.00000.00000
2.3.40 if version is VERSIONSTRING:00002.00003.00040.00000
...
Compile the library, let's name it mylib.so
Load it from a program, ask its version (call GetVersion), it returns 0.0, no surprise
Create a little program (did it in C++, but could be done in Python or any other languauge) that will:
load a whole binary file content in memory (using std::fstream with std::ios_base::binary)
find VERSIONSTRING:00000.00000.00000.00000 in it
confirms it appears once only (to be sure we don't modify something we did not mean to, that's why I prefix the string with VERSIONSTRING, to make it more unic...)
patch it to VERSIONSTRING:00002.00003.00040.00000 if expected binary number is 2.3.40
save the binary file back from patched content
Patch mylib.so using the above tool (requesting version 2.3 for instance)
Run the same program as step 3., it now reports 2.3!
No recompilation nor linking, you patched the binary version!
When I compile the release version of my iOS app (based on standard Apple supplied iOS app template), look into the resulting executable binary, I see all sorts of symbols and even local cpp source and header paths in there. I'm really stumped why this is (I haven't enabled RTTI*). Especially the source file paths make me feel uncomfortable sending this app across the globe (why should everyone be able to see the directory layout of my development machine?).
Here's are two (randomly picked, moderated) excerpts:
TS/../ACTORS/CActorCanvasCharPart.cpplastMeshcapVerticesOFF BOUNDSupload VERTICES: %d
20CActorCanvasCharPartgrassscrub/Volumes/Data/iOS_projects/code/MyAppName_proj/MyAppName/source/STATES/GAMES/2/CStateGame2_grass.cppbaseShadowmowerstartmowerloopmowermowerCharcutGrassChargrassStuffgrassParticles/Volumes/Data/iOS_projects/code/MyAppName_proj/MyAppName/source/STATES/GAMES/2/CStateGame2_grass.h17CStateGame2_grasssinwriteStroke/Volumes/Data/iOS_projects/code/MyAppName_proj/MyAppName/source/STATES/GAMES/2/CStateGame2_flowers.hflowerBedsandTrailclickstart3inplace2sandDrag/Volumes/Data/iOS_projects/code/MyAppName_proj/MyAppName/source/STATES/GAMES/2/CStateGame
And here are a lot of symbols for self-defined types and structs:
CAssetMgr="_vptr$CMgrBase"^^?"pMain"^{CMain}"inited"B"curveCount"S"curveSpecs"^{CCurveSpec}"gameSpecs"[23{CGameStateSpec="header"{SpecDiskHeader="type"i"version"S}"gameID"C"backgroundColor"{CRGBAcolorf="r"f"g"f"b"f"a"f}"clickPointColor"{CRGBAcolorf="r"f"g"f"b"f"a"f}"clickPointIconColor"{CRGBAcolorf="r"f"g"f"b"f"a"f}"hintColor"{CRGBAcolorf="r"f"g"f"b"f"a"f}}]"currentFont"^{CCharset}"userCharParts"^^{CCharPart}"words"{CDataSet<CName4,CCharArray>="_vptr$CObjectBase"^^?"pMain"^{CMain}"count"i"data"*"dataSize"l}"sets"{CDataSet<CName16,CCharArray>="_vptr$CObjectBase"^^?"pMain"^{CMain}"count"i"data"*"dataSize"l
Can this be avoided, how?
*UPDATE: I just found out that RTTI is on by default. So I cleaned the target, disabled RTTI (GCC_ENABLE_CPP_RTTI = NO) and recompiled. I still see a lot of symbols and source paths in the binary.
UPDATE 2: I checked a few other apps from the app store, and many of them also have their source file paths show up. Pretty scary, if you ask me:
Joined Up Lite
/Users/lloydy/Documents/Development/iPhone/ABC Joined Up/main.m
/Users/lloydy/Documents/Development/iPhone/ABC Joined Up/Classes/SettingsView.m
Crayon Physics
/Users/smproot/Desktop/unzip/CrayonPhysics/v104/Classes/crayon/src/ceng/gameutils/killspriteslowly/killspriteslowly.cpp
/Users/smproot/Desktop/unzip/CrayonPhysics/v104/Classes/crayon/src/ceng/tasks/task/sdl/mixer/ctaskaudiosdlmixer.cpp
Wall Times
/Users/fred/_WORK/ZDNDRP/WallTimes/main.m
/Users/fred/_WORK/ZDNDRP/WallTimes/Classes/SystemCategories.m
Jumbo Calculator
/Users/Christopher/Documents/Development/JumboCalculator 1.0.3/main.m
/Users/Christopher/Documents/Development/JumboCalculator 1.0.3/Classes/CalculatorFaceViewController.m
The file paths are most likely from assert macros which stringify __FILE__ as part of their failure message. iOS's implementation of assert(3) does this, as do the NSAssert macros.
You can remove asserts in release builds by defining NDEBUG (for the C asserts) and NS_BLOCK_ASSERTIONS (for NSAsserts).
In Xcode set Deployment Prostprocessing to Yes in order to trigger Xcode to call the strip command during build process. Then you don't see any source path via nm -a.
However, I still see the source paths of some m files via the strings command :/
What worked for me was setting Generate Debug Symbols to No for release builds. This is under the Apple LLVM 7.0 - Code Generation in Xcode 7.2.
Have ticked the strip debug symbols in the build settings? You can do this (or not) depending on the configuration (build/release). Also you can look into Objective-C Code Obfuscation (which is long winded). From what I gather, you cannot completely remove objective-c information as all method calls are done dynamically, so the library has to have information about your classes/method names in order to function. A useful tip here.
If you have c++ code then you can use the gcc strip utility, although I'm not sure how it like Objetive-C++, if it doesn't you could compile all you cpp into a lib, strip that and link against it in your iOS project.
I have used vtune several times in the past, usually without too much trouble. Unfortunately the gaps between each use are often so long that I forget some aspects of how to use it each time. I know that the line number and symbols information needs to be stored somehow. I thought that all that was required was to compile your exe with "Program Database" (/Zi), but I have just done a sampling and found that vtune reports there are no symbols available.
Is there anything I missed?
There are two options for debugging (check $> cl /?):
/Zi enable debugging information
/ZI enable Edit and Continue debug info
Make sure that you have .pdb and manifest file (if generated).
It's not related but maybe turn off optimizations as well.
Like Bua mentioned, you definitely need to be compiling with debugging information enabled. If the pdb files are in the same directory as the exe that you're profiling, then it should be able to find them. If not, you can also try explicitly adding the path to the pdbs in config -> options -> directories. alt text http://software.intel.com/file/21331 Add an item with your symbols directory. You might also want to add a symbol server and symbol cache, because then you'll get symbols for all of Microsoft's public binaries. The image above shows how to add a symbol server with a symbol cache at c:\websymbols. Generally, the format for a "symbol server" is a string of the form:
an example:
SRV*C:\MySymbolCache\*http://msdl.microsoft.com/download/symbol
of the form:
SRV * [CACHE] * [SYM SERVER PATH]
Hope this helps!
The problem has been solved: It turned out that it was a mistake in setting the working directory; "/Zi" appears to be all that is required after all. I don't need to switch off optimization.
When I tried to debug an executable:
(gdb) break +1
No symbol table is loaded. Use the "file" command.
What does that mean exactly?
Is the symbol table appended to the executable?
There are two sets of symbols that gdb uses.
The -g set are debugging symbols, which make things a lot easier as they allow you to see your code and look at variables while debugging.
Another set of symbols is included by default when you compile. These are the linking symbols and live in the ELF (executable linkable format) symbol table. This contains a lot less info than the debug symbols, but contain the most important stuff, such as the addresses of the things in your executable (or library or object file). Without this information gdb won't even know where main is, so (gdb) break main would fail.
If you don't have the debugging symbols ( -g ) then you will still be able to (gdb) break main but you gdb will not have any concept of the lines of code in your source file. When you try to step through the code you will only advance 1 machine instruction at a time, rather than a line at a time.
The strip command is often used to strip off symbols from an executable (or other object file).
This is often used if you don't want someone to be able to see the symbols or if you want to save space in the file. Symbol tables can get big. Strip removes both the debug symbols and the linker symbols, but it has several command line switches which can limit what it removes.
If you run the file command on your program one of the things it will tell you is weather or not the executable is has been stripped.
$ gcc my_prog.c -o my_prog
$ file my_prog
my_prog: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped
$ strip my_prog
my_prog: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped
$
It's because you didn't compile with debugging turned on. Try gcc -g file.c
The symbol table contains debugging information that tells a debugger what memory locations correspond to which symbols (like function names and variable names) in the original source code file. The symbol table is usually stored inside the executable, yes.
gdb is telling you that it can't find that table. If you compiled with gcc, unless you used the -g flag, it will not include the symbol table in the file. The easiest method is probably to recompile your file with -g. gdb should then automatically find the symbol table information.
Either add the -g flag to the command line arguments of gcc or to the Makefile that you used to compile the program. (A lot of times, there will be a variable called CFLAGS or similar inside the Makefile).
If you are trying to debug an arbitrary third-party program, a lot of times the information will have been "stripped" out of it. This is done to make reverse engineering harder and to make the size of the executable file smaller. Unless you have access to the source code and can compile the program yourself, you will have a very hard time using gdb on it.
Find the entry point of the application.
objdump -f main
main: file format elf32-i386
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x08048054
Put a breakpoint there using the gnu debugger
gdb
exec-file main
break *0x8048054
set disassemble-next-line on
run
Then step through the code
gdb
stepi
Special Notes
If you are using the latest version of Ubuntu you would not be affected by this, but you may run into this bug if you are running Ubuntu 10.04 or older.
https://bugs.launchpad.net/ubuntu/+source/gdb/+bug/151518G
The solution would be to start debugging at the entry point address plus one.