I want to know how does gdb work internally.
e.g. I know a brief idea that it makes use of ptrace() system call to monitor traced program.
But I want to know how it handles signals, how it inserts new code, and other such fabulous things it does.
Check out the GDB Internals Manual, which covers some of the important aspects. There's also an older PDF version of this document.
From the manual:
This document documents the internals of the GNU debugger, gdb. It includes description of gdb's key algorithms and operations, as well as the mechanisms that adapt gdb to specific hosts and targets.
Taken from gdbint.pdf:
It can be done either as hardware breakpoints or as software
breakpoints:
Hardware breakpoints are sometimes available as a builtin debugging features with some chips. Typically these work by having dedicated
register into which the breakpoint address may be stored. If the PC
(shorthand for program counter) ever matches a value in a breakpoint
registers, the CPU raises an exception and reports it to GDB.
Another possibility is when an emulator is in use; many emulators include circuitry that watches the address lines coming out from the
processor, and force it to stop if the address matches a breakpoint's
address.
A third possibility is that the target already has the ability to do breakpoints somehow; for instance, a ROM monitor may do its own
software breakpoints. So although these are not literally hardware
breakpoints, from GDB's point of view they work the same;
Software breakpoints require GDB to do somewhat more work. The basic theory is that GDB will replace a program instruction with a trap,
illegal divide, or some other instruction that will cause an
exception, and then when it's encountered, GDB will take the exception
and stop the program. When the user says to continue, GDB will restore
the original instruction, single-step, re-insert the trap, and
continue on.
The only way you'll find out is by studying the source.
You can also build it and debug it with itself. Step through the code, and you'll know exactly how it does what it does.
Reading GDB source is not for the faint of heart though -- it is chock-full of macros, and heavily uses libbfd, which itself is hard to understand.
It has to, because it is portable (and in particular, builds and works on platforms which do not have ptrace() at all).
Related
In the past, I have been debugging executables loaded in the internal SRAM of my Cortex M3 (STM32F2) without problems. I have recently been loading my executable to Flash (because of size issues).
Ever since, debugging with GDB has not been working. As I understand, when the executable is in Flash, only hardware breakpoint can be used (as opposed to software breakpoints), and I have six hardware breakpoints. However, when setting just one hardware breakpoint GDB yields an error message:
(gdb) break main
Breakpoint 1 at 0x800019a: file src/main.c, line 88.
(gdb) c
Continuing.
Note: automatically using hardware breakpoints for read-only addresses.
(gdb) Warning:
Cannot insert hardware breakpoint 1.
Could not insert hardware breakpoints:
You may have requested too many hardware breakpoints/watchpoints.
What could be going wrong? Have my hardware breakpoints be taken in the background?
Note: I used OpenOCD to load the executable through JTAG.
So, there's basically two ways (plus one really bad way) that breakpoints can be implemented on any given debugger/platform combination:
Use some hardware capabilities ("hardware breakpoints") to cause the processor to trap when it hits a particular address. This is typically restricted to just a couple of breakpoints, if it's available at all.
For each breakpoint that's being set, replace the instruction at the breakpoint with a "trap" instruction of some variety (i.e, an instruction that will break into the debugger). When one of the breakpoints is hit, swap the original instruction back in and single-step once to make it run.
Single-step through the whole program. This one doesn't really count, because it's horrifically slow.
It sounds as though your debugger is only using method #2 ("software breakpoints"). The gotcha with this method is that it requires the program to be writable -- and flash memory isn't writable one instruction at a time, so this technique won't work.
As some of you might know Google provides for free great collection of tools for analyzing c++ code:
http://code.google.com/p/google-perftools/
Problem is that there is apparently some libunwind problem on 64 bits, and authors can't do anything on their side to fix it(
But I don't expect a
fix anytime soon: it depends on the libc folks and the libunwind
folks working out some locking issues. There's unfortunately not
much we ourselves can do.
), so I'm searching for replacement.
Is there any similar tool that provides cool graphical representation of profiling data(for example: )
)
EDIT: paste from README that explains the problem:
2) On x86-64 64-bit systems, while tcmalloc itself works fine, the
cpu-profiler tool is unreliable: it will sometimes work, but sometimes
cause a segfault. I'll explain the problem first, and then some
workarounds.
Note that this only affects the cpu-profiler, which is a
google-perftools feature you must turn on manually by setting the
CPUPROFILE environment variable. If you do not turn on cpu-profiling,
you shouldn't see any crashes due to perftools.
The gory details: The underlying problem is in the backtrace()
function, which is a built-in function in libc. Backtracing is fairly
straightforward in the normal case, but can run into problems when
having to backtrace across a signal frame. Unfortunately, the
cpu-profiler uses signals in order to register a profiling event, so
every backtrace that the profiler does crosses a signal frame.
In our experience, the only time there is trouble is when the signal
fires in the middle of pthread_mutex_lock. pthread_mutex_lock is
called quite a bit from system libraries, particularly at program
startup and when creating a new thread.
The solution: The dwarf debugging format has support for 'cfi
annotations', which make it easy to recognize a signal frame. Some OS
distributions, such as Fedora and gentoo 2007.0, already have added
cfi annotations to their libc. A future version of libunwind should
recognize these annotations; these systems should not see any
crashses.
Workarounds: If you see problems with crashes when running the
cpu-profiler, consider inserting ProfilerStart()/ProfilerStop() into
your code, rather than setting CPUPROFILE. This will profile only
those sections of the codebase. Though we haven't done much testing,
in theory this should reduce the chance of crashes by limiting the
signal generation to only a small part of the codebase. Ideally, you
would not use ProfilerStart()/ProfilerStop() around code that spawns
new threads, or is otherwise likely to cause a call to
pthread_mutex_lock!
--- 17 May 2011
Valgrind has a collection of great tools, including callgrind to profile the code. The gui client for the callgrind and cachegrind is kcachegrind.
How do breakpoints work in C++ code? Are they special instructions inserted in between some assembler instructions when the code is compiled? Or is there something else in place? Also, how are stepping-through-the-code implemented? The same way as breakpoints...?
This is heavly depend on the CPU and debugger.
For example, one of the possible solution on x86 CPU:
Insert one-byte INT3 instruction on the required place
Wait until breakpoint exception hits
Compare exception address to the list of breakpoint to determine which one
Do breakpoint actions
Replace INT3 with original byte and switch the debugged process into trace mode (step-by-step execution of CPU instructions)
Continue debugged process
Immediately you catch trace exception - the instruction was executed
Put INT3 back
Watchpoints can be implemented in the similar way, but instead of INT3 you put the memory page where watched variable is into read only, or into no access mode, and wait for segmentation exception.
Stepping through assembly can also be done by using trace mode. Stepping through source lines can also be done by placing breakpoints onto next instructions, based on debug data.
Also some CPU has hardware breakpoint support, when you just load address into some register.
According to this blog entry on technochakra.com you are correct:
Software breakpoints work by inserting a special instruction in the program being debugged. This special instruction on the Intel platform is “int 3″. When executed it calls the debugger’s exception handler.
I'm not sure how stepping into or over the next instruction is implemented though. However, the article goes on to add:
For practical reasons, it is unwise to ask for a recompilation whenever a breakpoint is added or deleted. Debuggers change the loaded image of the executable in memory and insert the “int 3″ instruction at runtime.
However, this would only be used for the "run to current line option".
Single stepping is implemented at (assembler) code level not at C++ level. The debugger knows how to map the C++ code lines to code addresses.
There are different implementations. There are CPUs that support debugging with breakpoint registers. When the execution reaches the address in the breakpoint register, the CPU executes a breakpoint exception.
A different approach is to patch the code for the time of execution with a special instruction, at best a one-byte instruction. At x86 systems that usually int 3.
The first approach allows breakpoints in ROM, the second allows more breakpoints at the same time.
AFAIK all debuggers (for whatever compiled language) that allow an unlimited number of breakpoints use a variant of replacing the instruction to be breakpointed with a special value (as described above) and keeping a list of places where these values have been placed.
When the processor tries to execute one of these special values, an exception is raised, the debugger catches it and checks if the address of the exception is on its list of breakpoints.
If it is, the debugger is invoked and the user is given an opportunity to interact.
If it is NOT, then the exception is due to something that was in the program from the outset and the debugger lets the exception 'pass' to whatever error handler might be there.
Note also, that debugging self-modifying code can fail precisely because the debugger momentarily modifies the code itself. (Of course, nobody would ever write self-modifying, now would they? >;-)
For these reasons, it is important that the debugger be given the opportunity to remove all the breakpoints it sets before terminating the debugging session.
In my c++ program I'm using a library which will "send?" a Sigtrap on a certain operations when
I'm debugging it (using gdb as a debugger). I can then choose whether I wish to Continue or Stop the program. If I choose to continue the program works as expected, but setting custom breakpoints after a Sigtrap has been caught causes the debugger/program to crash.
So here are my questions:
What causes such a Sigtrap? Is it a leftover line of code that can be removed, or is it caused by the debugger when he "finds something he doesn't like" ?
Is a sigtrap, generally speaking, a bad thing, and if so, why does the program run flawlessly when I compile a Release and not a Debug Version?
What does a Sigtrap indicate?
This is a more general approach to a question I posted yesterday Boost Filesystem: recursive_directory_iterator constructor causes SIGTRAPS and debug problems.
I think my question was far to specific, and I don't want you to solve my problem but help me (and hopefully others) to understand the background.
Thanks a lot.
With processors that support instruction breakpoints or data watchpoints, the debugger will ask the CPU to watch for instruction accesses to a specific address, or data reads/writes to a specific address, and then run full-speed.
When the processor detects the event, it will trap into the kernel, and the kernel will send SIGTRAP to the process being debugged. Normally, SIGTRAP would kill the process, but because it is being debugged, the debugger will be notified of the signal and handle it, mostly by letting you inspect the state of the process before continuing execution.
With processors that don't support breakpoints or watchpoints, the entire debugging environment is probably done through code interpretation and memory emulation, which is immensely slower. (I imagine clever tricks could be done by setting pagetable flags to forbid reading or writing, whichever needs to be trapped, and letting the kernel fix up the pagetables, signaling the debugger, and then restricting the page flags again. This could probably support near-arbitrary number of watchpoints and breakpoints, and run only marginally slower for cases when the watchpoint or breakpoint aren't frequently accessed.)
The question I placed into the comment field looks apropos here, only because Windows isn't actually sending a SIGTRAP, but rather signaling a breakpoint in its own native way. I assume when you're debugging programs, that debug versions of system libraries are used, and ensure that memory accesses appear to make sense. You might have a bug in your program that is papered-over at runtime, but may in fact be causing further problems elsewhere.
I haven't done development on Windows, but perhaps you could get further details by looking through your Windows Event Log?
While working in Eclipse with minGW/gcc compiler, I realized it's reacting very bad with vectors in my code, resulting to an unclear SIGTRAP signal and sometimes even showing abnormal debugger behavior (i.e. jumping somewhere up in the code and continuing execution of the code in reverse order!).
I have copied the files from my project into the VisualStudio and resolved the issues, then copied the changes back to eclipse and voila, worked like a charm. The reasons were like vector initialization differences with reserve() and resize() functions, or trying to access elements out of the bounds of the vector array.
Hope this will help someone else.
I received a SIGTRAP from my debugger and found out that the cause was due to a missing return value.
string getName() { printf("Name!");};
Is there a way for my code to be instrumented to insert a break point or watch on a memory location that will be honored by gdb? (And presumably have no effect when gdb is not attached.)
I know how to do such things as gdb commands within the gdb session, but for certain types of debugging it would be really handy to do it "programmatically", if you know what I mean -- for example, the bug only happens with a particular circumstance, not any of the first 11,024 times the crashing routine is called, or the first 43,028,503 times that memory location is modified, so setting a simple break point on the routine or watch point on the variable is not helpful -- it's all false positives.
I'm concerned mostly about Linux, but curious about if similar solutions exist for OS X (or Windows, though obviously not with gdb).
For breakpoints, on x86 you can break at any location with
asm("int3");
Unfortunately, I don't know how to detect if you're running inside gdb (doing that outside a debugger will kill your program with a SIGTRAP signal)
GDB supports a scripting language that can help in situations like this. For example, you can trigger a bit of custom script on a breakpoint that (for example) may decided to "continue" because some condition hasn't been met.
Not directly related to your question, but may be helpful. Have you looked at backtrace and backtrace_symbol calls in execinfo.h
http://linux.die.net/man/3/backtrace
This can help you log a backtrace whenever your condition is met. It isn't gdb, so you can't break and step through your program, but may be useful as a quick diagnostic.
The commonly used approach is to use a dummy function with non-obvious name. Then, you can augment your .gdbinit or use whatever other technique to always break on that symbol name.
Trivial dummy function:
void my_dummy_breakpoint_loc(void) {}
Code under test (can be an assert-like macro):
if (rare_condition)
my_dummy_breakpoint_loc();
gdb session (obvious, eh?):
b my_dummy_breakpoint_loc
It is important to make sure that "my_dummy_breakpoint_loc" is not optimized away by compiler for this technique to work.
In the fanciest of cases, the actual assembler instruction that calls my_dummy_breakpoint_loc can be replaced by "nops" and enabled on site by site basis by a bit of code self-modification in run-time. This technique is used by Linux kernel development instrumentation, to name a one example.