I've written a small test program which throws a C++ exception. I've setup Breakpad to write a minidump on this thrown exception. I now have a .dmp file I'd like to see the stack trace for. Several references indicate I should.
Generate a .sym file using Breakpad's 'dump_syms' utility program, which I've done. I'm running dump_syms on the debug binary (which should have the debug symbols built in?). ./dump_syms breakpad_testing > breakpad_testing.sym
At this point I have the .dmp file and the .sym file
Inspect the first line of the .sym file to view to get a binary version hash? This will look something like this -
MODULE mac x86_64 ED3C7C3C3C283C749036117557E0A8500 breakpad_testing.
Use this to create an expected folder structure mkdir -p
./symbols/breakpad_testing/ED3C7C3C3C283C749036117557E0A8500 and
move the .sym file there. mv breakpad_testing.sym
./symbols/breakpad_testing/ED3C7C3C3C283C749036117557E0A8500
Show stack using 'minidump_stackwalk' tool. minidump_stackwalk
breakpad_testing.dmp ./symbols
However these steps don't seem to have any effect on the output of minidump_stackwalk, I still see output lines such as minidump.cc:2122: INFO: MinidumpModule could not determine version for /Users/mb/Library/Developer/Xcode/DerivedData/<blah>/Build/Products/Debug/breakpad_testing and an unsymbolicated stack trace.
Is there something I'm misunderstanding or not using properly regarding Breakpad on OSX?
This is one of the references I've been following https://blog.inventic.eu/2012/08/qt-and-google-breakpad/
Related
I have a gdb backtrace of a crashed process, but I can't see the specific line in which the crash occurred because the source code was not in that moment. I don't understand some of the information given by the mentioned backtrace.
The backtrace is made of lines like the following one:
<path_to_binary_file>(_Z12someFunction+0x18)[0x804a378]
Notice that _Z12someFunction is the mangled name of int someFunction(double ).
My questions are:
Does the +0x18 indicate the offset, starting at _Z12someFunction address, of the assembly instruction that produced the crash?
If the previous question is affirmative, and taking into account that I am working with a 32-bit architecture, does the +0x18 indicates 0x18 * 4 bytes?
If the above is affirmative, I assume that the address 0x804a378 is the _Z12someFunction plus 0x18, am I right?
EDIT:
The error has ocurred in a production machine (no cores enabled), and it seems to be a timing-dependant bug, so it is not easy to reproduce it. That is because the information I am asking for is important to me in this occasion.
Most of your assumptions are correct. The +0x18 indeed means offset (in bytes, regardless of architecture) into the executable.
0x804a378 is the actual address in which the error occurred.
With that said, it is important to understand what you can do about it.
First of all, compiling with -g will produce debug symbols. You, rightfully, strip those for your production build, but all is not lost. If you take your original executable (i.e. - before you striped it), you can run:
addr2line -e executable
You can then feed into stdin the addresses gdb is giving you (0x804a378), and addr2line will give you the precise file and line to which this address refers.
If you have a core file, you can also load this core file with the unstriped executable, and get full debug info. It would still be somewhat mangled, as you're probably building with optimizations, but some variables should, still, be accessible.
Building with debug symbols and stripping before shipping is the best option. Even if you did not, however, if you build the same sources again with the same build tools on the same environment and using the same build options, you should get the same binary with the same symbols locations. If the bug is really difficult to reproduce, it might be worthwhile to try.
EDITED to add
Two more important tools are c++filt. You feed it a mangled symbol, and produces the C++ path to the actual source symbol. It works as a filter, so you can just copy the backtrace and paste it into c++filt, and it will give you the same backtrace, only more readable.
The second tool is gdb remote debugging. This allows you to run gdb on a machine that has the executable with debug symbols, but run the actual code on the production machine. This allows live debugging in production (including attaching to already running processes).
You are confused. What you are seeing is backtrace output from glibc's backtrace function, not gdb's backtrace.
but I can't see the specific line in which the crash occurred because
the source code was not in that moment
Now you can load executable in gdb and examine the address 0x804a378 to get line numbers. You can use list *0x804a378 or info symbol 0x804a378. See Convert a libc backtrace to a source line number and How to use addr2line command in linux.
Run man gcc, there you should see -g option that gives you possibility to add debug information to the binary object file, so when crash happens and the core is dumped gdb can detect exact lines where and why the crash happened, or you can run the process using gdb or attach to it and see the trace directly without searching for the core file.
I have met a freaky problem during my internship. My work is to code with VTK in C++ and I worked on OSX 10.8.3.
When I want to debug my program, I ran the gdb and use instruction "file" to load my program, then I used "list" to show the source code to make a breakpoint by line number. Here goes the problem: this freaky gdb showed the source code of a VTK header file which I even hadn't included in my source code!
My program's name is read
I have tried to use gdb read then break read:15 to set a breakpoint but the gdb displayed "no source file named read" that is ridiculous!
I have noticed that gdb works well in my ubuntu 12.04 and when I use file read in linux's gdb, it just displayed
Reading symbols from /Users/apple/Dev/VTKRead/bin/bin/read...done.
but in my OSX 10.8.3's freaky gdb it displayed
Reading symbols for shared libraries ......... done
Reading symbols from /Users/apple/Dev/VTKRead/bin/bin/read...done.
I think that is the reason and I tried to change compiler to solve problem by install gcc4.8 in macport but cmake seems only accept the apple's gcc.
but the gdb displayed "no source file named read" that is ridiculous!
That is not rigiculous at all: you very likely don't have a source file called read. What you do have is probably called read.cc, or read.cpp, so try break read.cc:15.
That is my mistake: I didn't set the build tag to "debug" in ccmake, so the compiler didn't write the information into the file.
After the program I am debugging crashes, I am left with heap dump *.mdmp file & appcompat.txt in my Temp directory. I understand that appcompat.txt is an error report. Is there a description of its format?
My appcompat.txt lists a number of DLLs. Am I correct assuming that the reason for a crash could have only come from one of the listed DLLs? Can I limit my debugging effort to the DLLs listed in appcompat.txt?
Thanks in advance!
The minidump file is far more informative for diagnosing crashes:
Install Debugging Tools for Windows, if you don't already have it.
Set up the symbol path variable _NT_SYMBOL_PATH to point to the Microsoft symbol server
Run Windbg and do File -> Open Crash Dump and locate your .dmp or .mdmp file
Type !analyze -v.
This will try to isolate the location of the crash. Note that just because a crash occurs in a particular dll it doesn't mean that is where the bug resides - it could be because an invalid parameter has been passed in from your application code. The analysis should hopefully show you a meaningful stack and an error code which should help in working out the actual cause of the crash.
I'm working on RHEL WS 4.5.
I've obtained the glibc source rpm matching this system, opened it to get its contents using rpm2cpio.
Working in that tree, I've created a patch to mtrace.c (i want to add more stack backtrace levels) and incorporated it in the spec file and created a new set of RPMs including the debuginfo rpms.
I installed all of these on a test vm (created from the same RH base image) and can confirm that my changes are included.
But with more complex executions, I crash in mtrace.c ... but gdb can't find the debug information so I don't get line number info and I can't actually debug the failure.
Based on dates, I think I can confirm that the debug information is installed on the test system in /usr/src/debug/glibc-2.3.6/
I tried
sharedlibrary libc*
in gdb and it tells me the symbols are already loaded.
My test includes a locally built python and full symbols are found for python.
My sense is that perhaps glibc isn't being built under rpmbuild with debug enabled. I've reviewed the glibc.spec file and even built with
_enable_debug_packages
defined as 1 which looked like it might influence the result. My review of the configure scripts invoked during the rpmbuild build step didn't give me any hints.
Hmmmm .. just found /usr/lib/debug/lib/libc-2.3.4.so.debug
and /usr/lib/debug/lib/tls/i486/libc-2.3.4.so.debug
but both of these are reported as stripped by the file command.
It appears that you are installing non-matching RPMs:
/usr/src/debug/glibc-2.3.6
just found /usr/lib/debug/lib/libc-2.3.4.so.debug
There are not for the same version; there is no way they came from the same -debuginfo RPM.
both of these are reported as stripped by the file command.
These should not show as stripped. Either they were not built correctly, or your strip is busted.
Also note that you don't actually have to get all of this working to debug your problem. In the RPMBUILD directory, you should be able to find the glibc build directory, with full-debug libc.so.6. Just copy that library into your VM, and you wouldn't have to worry about the debuginfo RPM.
Try verifying that debug info for mtrace.c is indeed present. First see if the separate debug info for GLIBC knows about a compilation unit called mtrace.c:
$ eu-readelf -w /usr/lib/debug/lib64/libc-2.15.so.debug > t
$ grep mtrace t
name (strp) "mtrace.c"
name (strp) "mtrace"
1 0 0 0 mtrace.c
[10480] "mtrace.c"
[104bb] "mtrace"
[5052] symbol: mtrace, CUs: 446
Then see if GDB actually finds the source file from the glibc-debuginfo RPM:
(gdb) set pagination off
(gdb) start # pause your test program right after main()
(gdb) set logging on
Copying output to gdb.txt.
(gdb) info sources
Quit GDB then grep for mtrace in gdb.txt and you should find something like /usr/src/debug/glibc-2.15-a316c1f/malloc/mtrace.c
This works with GDB 7.4. I'm not sure the GDB version shipped with RHEL 4.5 supports all the command used above. Building upstream GDB from source is in fact easier than Python though.
When trying to add strack traces to mtrace, make sure you don't call malloc() directly or indirectly in the GLIBC malloc hooks.
In c or c++ While debugging a core dump,i.e., if we are left with a coredump file and try to debug using that core file is there a way we can see the last few lines of code before the dump occured.the stack trace only shows the function call.
for eg:during debugging a running process in dbx or gdb list command
list Display lines of a source file
will give the part of the code currently under execution.in the same way do we have the option while looking into a core dump?
i am working on solaris mdb
The list command in gdb should provide details of source code line numbers and the corresponding source listing from a core file if:
The executable was compiled with debug symbols enabled (eg. -g in gcc and g++)
A non-stripped version of the executable is available (ie. has not had the debug information removed by running strip)
The debugger is able to find the relevant source files
The debugger should still be able to provide file and line numbers even if it is unable to find the source files as the line number information forms a part of the debug symbols. This information should be available through the bt (backtrace) command, as well as the info symbol command.
Note that even if the core file was generated from a stripped executable, as long as you have access to a non-stripped version you can load that in gdb as well as specifying the core file.
Take a look at chapter 13 and 15 of the gdb manual to assist in giving gdb access to your source files.
If you compiled with -g option you can display source lines. In dbx you can use use command to set dbx source directories.
You can use list (l) command to display source lines.
See help command to learn how to use dbx and gdb commands.
If you have set your source path properly using use command in dbx or started it with -I option, then there's hardly any difference between debugging a core dump and a normal process when it comes to reading source lines.