I have a binary that is 10s of MB large without debugging symbols, but with debugging symbols its 100s of MB large. In the normal development cycle I copy the several 100 MB binary (with debug symbols) over a very slow link repeatedly. I am trying to minimize the amount of information I need to send to speed up the transfer.
I have researched binary diff tools like bsdiff and courgette, but the time and memory they take is prohibitive for me given the size of the binary and the frequency I'd like to be able to transfer it. As pointed out in responses, there are ways to mitigate the problem of needing to send the debug info to the remote host. gdbserver won't work in my use case because we'd also like the application to be able to log backtrace information with symbols. Using PC values and addr2line was considered, but keeping the source binary around can be confusing if trying to make forward progress while also testing on a remote machine. Additionally, with multiple developers, having access to debug info on some other developer's machine isn't always easy.
strip can separate out the binary from the debug info, so I was wondering if there were tools to compare and "diff" two debug info files, since that's where 95% of my space is anyway? And between iterations, a lot of the raw content in the debug info file is the same (i.e. the names and relationships, which is painfully verbose in C++).
Using the suggestion from user657267 I have also investigated using -gsplit-dwarf to separate out the .dwo files. This may work, but my concern is that over time core headers will change and cause minor changes to every .dwo file, so I'll end up transferring everything anyway assuming my "base" stays the same, even though most of the content of the .dwo file is unchanged. This could possibly be worked around in interesting ways (e.g. repository of .dwo files), but I'd like to avoid it if possible.
So, assuming I have a DWARF debug info file from a previous compilation, is there a way to compare it to the DWARF debug info file from the current compilation and get something smaller to transfer?
As an absolute last resort, I can write some type of lookup and translation code. But are there convenient tools for viewing, modifying, and then "unmodifying" a DWARF debug info file? I have found tools like pyelftools and DWARF utilities. The former only reads the DIEs, and too slowly in my case, while the latter doesn't work well with C++ and I'm still investigating building it from the latest source.
Along these lines, I have investigated what the dwz tool announced here is doing to see if can be tweaked to borrow DIEs from an already existing (but stale) debug info file. Any tips, documents, or pseudo-code in this direction would also be helpful.
In the normal development cycle I have to copy my several 100 MB binary (with debug symbols) over a very slow link over and over again.
Have to?
Your use case screams for using remote debugging, where all the debug info stays on the development system, and you only have to transfer the stripped binary to the target.
Info about using gdbserver is here.
If, for some reason you can't use gdbserver ...
From this link gcc.gnu.org/wiki/DebugFission, I still don't understand how having a separate dwarf file is going to help me diff easier?
With separate debug info, in a usual compile/debug cycle, you'll be recompiling only a few files, and relinking the final binary.
That means that most of the .o and .dwo files will not be rebuilt, which means that you wouldn't have to re-send the unchanged .dwo files to the target, i.e. you get incremental debug info updates "for free".
Update:
We also use the debug symbols to generate backtraces for exceptions in the running application. I think having the symbols locally is the only option for that use case.
Only if you insist on the backtrace being fully-symbolized with file/line info.
The usual way to deal with this is to have the backtrace contain only PC values (and perhaps function names), and use addr2line on the development system to recover file/line info when necessary.
Related
I'm currently developing an application (Windows) that needs internal modifications upon download time.
Also, I'm delivering it using a Linux host, so, can't compile on demand as proposed.
How does Ninite deal with it?
In Ninite.com, each time you select different options, you get the same .exe, however, with minor modifications inside.
Option 1
Compile the program with predefined data (in Windows).
Use PHP to fseek the file and replace my custom strings.
Option 2
Append the original .EXE with a different resource file
Other?
Has someone developed something like this? What would be the best approach?
Thank you.
You can just append data to the back of your original executable. The Windows PE file format is robust enough that this does not invalidate the executable itself. (It will however invalidate any existing digital signatures.)
Finding the start of this data can be a challenge if its size isn't known up front. In that case, it may be necessary to append the variable-length data, and then append the data length (itself a fixed length field - 4 bytes should do). To read the extra data, read the last 4 bytes to get the data length. Get the file length, subtract 4 for the length field, then subtract the variable length to get the start of the data.
The most portable way could be to have a plugin (whose path in wired inside your main program) inside your application. That plugin would be modified (e.g. on Linux by generating C++ code gencod.cc, forking a g++ -Wall -shared -fPIC -O gencod.cc -o gencod.so compilation, then dlopen-ing the ./gencod.so) and your application could have something to generate the C++ source code of that plugin and to compile it.
I guess that the same might be doable on Windows (which I don't know). Probably the issue is to compile it (the compilation command would be different on Windows and on Linux). Beware that AFAIK on Windows a process cannot modify its own executable (but you should check).
Qt has a portable layer for plugins. See QPluginLoader & Qt Plugins HowTo
Alternatively, don't modify the application, but use some persistent file or data (at a well defined place, -whose location or filepath is wired in the executable- preferably in textual format like JSON, or maybe using sqlite, or a real database) keeping the changing information. Read also about application checkpointing.
If you need to implement your specific application checkpointing, you'll better design your application very early with this concern. Study garbage collection algorithms (a checkpointing procedure is similar to a precise copying GC) and read more about continuations. See also this answer to a very similar question.
Is there any consistency when generating a .so file between two builds? When we perform clean and build?
Basically, I wanted the .so file for an app for a previous state of the code (C++), the files changed were very few, which i reverted back, if i build now will the so file be same as the one i got before?
I can replicate the code state to exactly what it was before, I needed that to map the stack trace to the code using this file as we can map hex values to function names.
Thanks
Assuming you are building from the same source code with the same build options, your output product should be the same (with the possible exception of some timestamps embedded into the code). Any compiler/kernel/library upgrades might break this guarantee.
This is exactly what version control (especially tagged snapshots) are for.
I get crash reports from users with some Dr. Watson info, including the EIP. I want to start debugging, then set the EIP and see which line this takes me to. I also have Line Info enabled in the Release builds.
I have a label in source control so I can pull the sources I used to build it, but I have no idea if the linker will produce an EXE with the same offsets, otherwise the EIP would not be useful.
There is no garuntee that the offsets will be the same, unless you can ensure that all libraries used, all source, and the linker have not changed since you checked the code into version control. This is why many companies will actualy go to great lengths to keep the executables under configuration management.
From experience, if the offsets are not the same, then it becomes apparent very quickly so you can at least check, and if things are not making sense, ignore the EIP and other executable address offsets.
If you build a map file and saved that from the version that the crash report came from, and a map file from the rebuilt source, then you can do some comparative analysis (Function by function really) to get some usefullness back from executable addresses and offsets.
One of my co-workers shipped a hot fix build to a customer, and subsequently deleted the pdb file. The build in question is crashing (intermittently) and we have a couple of crash dumps. We have all the source code in version control, and can compile it to an equivalent .exe and get symbols for that one. However, those symbols don't match the crash dump exactly. It seems like several of the functions are off by some constant offset, but we've only looked at a handful.
I'd love to be able to do the following (I can fake parts of this manually, but it's a huge amount of work): get a stack trace for each thread in the dump and cast pointers in the dump to the appropriate type and have them show up in the Visual Studio debugger. I'm using 2005, if that matters.
Is there a tool to let us recreate a pdb given the source code, all the .obj files, and the original .exe? Or is there a setting when we compile/link to say "make it exactly like this other exe you just did" or something like that?
Quick update, based on answers so far: I have the exe file that we sent to the customer, just not the pdb that corresponds to it, if that helps. I'd just as soon not send them a new build (if possible), because it takes about a week of running to get the crash dumps, and the customer is already at the "why isn't this already fixed?" stage. (If we do send another build, I'd prefer it to be one that either fixes the problem or has additional debugging in the area of interest, not just the same code.) I know it's possible to do some of this manually with a lot of guesswork; that's what we're currently doing. But it's a pain, so I'm hoping there's a way to automate it.
You cannot recreate a PDB to match a pre-existing executable. The PDB contains a "finger print" that is unique for each compilation. Unless you can make the old PDB magically reappear, you should whack your cow-orker in the back of the head (Gibbs-style, if you watch NCIS), recompile the whole thing, store the PDB somewhere safe, and ship a new executable to your customer, and let the crashes come.
If your build system enables you to recreate any binary from any revision you have in your history, then you should be able to get the build ID from the customer, and regenerate that same exact build ID, along with all the binaries and so forth. That will take a while if you have a large project, of course, but it will also yield the debugging file that you need.
If you have no way to perform an exact reproduction of a build, then look at this situation, think hard about some others that might crop up, and start moving to make it possible to regenerate all successful builds and associated files in the project's history. This will make it much easier to be able to work problems like this in the future.
When you have the sources, it's quite easy to find the correspondence between them and the exe file. Just ask them to send you the exe file along with the crash log and use IDA.
What you are asking is much more difficult than that, considering also that you need it for "one use only".
I have frequently encounter the following debugging scenario:
Tester provide some reproduce steps for a bug. And to find out where the problem is, I try to play with these reproduce steps to get the minimum necessary reproduce steps. Sometimes, luckily I found that when do a minor change to the steps, the problem is gone.
Then the job turns to find the difference in code workflow between these two reproduce steps. This job is tedious and painful especially when you are working on a large code base and it go through a lot code and involve lots of state changes which you are not familiar with.
So I was wondering is there any tools available to compare "code workflow". As I've learned the "wt" command in WinDbg, I thought it might be possible to do it. For example, I can run the "wt" command on some out most functions with 2 different reproduce steps and then compare the difference between outputs. Then it should be easy to found where the code flow starts to diverge.
But the problem with WinDBG is "wt" is quite slow (maybe I should use a log file instead of output to screen) and not very user-friendly (compared with visual studio debugger) ... So I want to ask you guys is there any existing tools available . or is it possible and difficult to develop a "plug-in" for visual studio debugger to support this functionality ?
Thanks
I'd run it under a profiler in "coverage" mode, then use diff on the results to see which parts of the code were executed in one run by not the other.
Sorry, I don't know of a tool which can do what you want, but even if it existed it doesn't sound like the quickest approach to finding out where the lower layer code is failing.
I would recommend to instrument your layer's code with high-level logs so you can know which module fails, stalls, etc. In debug, your logger can write to file, to output debug window, etc.
In general, failing fast and using exceptions are good ways to find out easily where things go bad.
Doing something after the fact is not going to cut it, since your problem is reproducing it.
The issue with bugs is seldom some interal wackiness but usually what the user's actually doing. If you log all the commands that the user enters then they can simply send you the log. You can substitute button clicks, mouse selects, etc. This will have some cost but certainly much less than something that keeps track of every method visited.
I am assuming that if you have a large application that you have good logging or tracing.
I work on a large server product with over 40 processes and over one million lines of code. Most of the time the error in the trace file is enough to identify the location of problem. However sometimes the error I see in the trace file is caused by some earlier code and the reason for this can be hard to spot. Then I use a comparative debugging technique:
Reproduce the first scenario, copy the trace to a new file (if the application is multi threaded ensure you only have the trace for the thread that does the work).
Reproduce the second scenario, copy the trace to a new file.
Remove the timestamps from the log files (I use awk or sed for this).
Compare the log files with winmerge or similar, to see where and how they diverge.
This technique can be a little time consuming, but is much quicker than stepping through thousand of lines in the debugger.
Another useful technique is producing uml sequence diagrams from trace files. For this you need the function entry and exit positions logged consistently. Then write a small script to parse your trace files and use sequence.jar to produce uml diagrams as png files. This is a great way to understand the logic of code you haven't touched in a while. I wrapped a small awk script in a batch file, I just provide trace file and line number to start then it untangles the threads and generates the input text to sequence.jar then runs its to create the uml diagram.