I have a network software that I need to debug. It forks at multiple places and I need to debug one particular function handling one particular request.
Is there any way to setup a global breakpoint that would be caught even when it is in an inferior process?
I cannot use follow-fork-mode child because this will follow the first request, not the one I need to debug.
One way to do this is to have gdb remain attached to all the processes. Then you would set your breakpoint and run the program as usual; the breakpoint would fire in any sub-process that happened to hit that location. You can use breakpoint conditions to try to reduce the number of hits.
To put gdb into multi-inferior mode, I use this:
set detach-on-fork off
set non-stop on
set pagination off
Depending on your version of gdb, you might also need set target-async on.
This mode can be a bit peculiar to work in. For example, when one thread stops, the other keep going. Also, breakpoint stops are reported, but not always obvious; and I think gdb doesn't immediately switch to the stopping thread (this may have changed in gdb git, I forget).
Related
I know how to create/modify a breakpoint to stop only on a specific thread using breakpoint modify <breakpoint-id> -T <thread-name>, but can I do it the other way around, preventing the breakpoint to stop on a specific thread?
I have a log thread that use the same function I'm trying to debug on other threads and it's annoying to have to let it continue every time it hits.
You can do this sort of thing using the lldb Python API and Python breakpoint callbacks. The callback returns a "should stop" value, i.e. if it returns True, you stop at the breakpoint, and False you continue.
So for instance:
(lldb) breakpoint command add -s python -o 'return frame.thread.name != "MyThread"'
is what you wanted. Note, in this example, I didn't provide a breakpoint ID. In lldb that means "act on the last set breakpoint". But you can also supply the breakpoint ID if the one you want to add the command to doesn't happen to be the last breakpoint you set. And if you want to add it to multiple breakpoints, you can specify a breakpoint ID list.
There's more on the API's and the callback format here:
https://lldb.llvm.org/python_api.html
https://lldb.llvm.org/use/python-reference.html#running-a-python-script-when-a-breakpoint-gets-hit
Note, however, that breakpoints with should-stop callbacks will force a stop every time the breakpoint is hit. lldb auto-continues pretty quickly, so for many applications this isn't a problem. But just getting from the stop over to the debugger and back is not cheap, so if your breakpoint is in a work loop and getting hit hundreds of thousands of times a second, this is going to slow down the run. If that ends up being a problem and you control your source code, tricks like the one Eljay suggested in the comments can be really handy.
I'm trying to set breakpoing (break ) after attaching to process in background mode (attach &). However I got
Cannot insert breakpoint 1.
Cannot access memory at address 0x5560c872b71a
Any reason why it's happening?
Setting breakpoint in foreground mode is fine.
Program was written in C++.
Any reason why it's happening?
The program must be stopped when inserting a breakpoint into it. Inserting a breakpoint is not an atomic operation, and writing to program code (which is what breakpoint insertion amounts to) while that code is executing may result in all kind of badness.
Use interrupt command to stop the process and bring it to foreground, insert your breakpoint, then continue & to put it into background again.
How are SW breakpoints handled (conceptually) by gdb stub or server (I assume client stub and server handle them in pretty much same way)?
I'm interested in a 'bare metal' target where the gdb stub/server runs, and both breakpoints and single stepping use software interrupts.
My actual questions:
When a breakpoint is hit, how is the stored instruction run so that the breakpoint can be 're-installed' and the (saved) machine status (including register contents) is not changed from the moment of hitting the breakpoint?
=>When is the breakpoint re-installed and how? Between breakpoint hit and entering the command interpreter, or during the next single step or coninue?
Also how does single-stepping over breakpoint work such that the original non-breakpoint instruction gets executed, and the breakpoint still remains there after being single-stepped over?
[edit]
Forgot: the document "GDB Internals" seems to be missing that info - and actually the whole subchapter about single stepping in the "Algorithms" chapter.
[edit2]
Ah, I seem to need stronger glasses: The 'Internals'-manual says:
"When the user says to continue, GDB will restore the original instruction, single-step, re-insert the trap, and continue on."
The single stepping over breakpoint, however, is still open question.
The single stepping over breakpoint, however, is still open question.
It's done exactly the same way as continue, except for the last step ("and continue on"). That is:
Process stops. GDB "looks around", discovers that $ip points to one of its breakpoints.
User issues continue, next, step or stepi command.
Restore original instruction (i.e. remove the breakpoint)
Single-step process
Re-insert breakpoint
Continue (this is done for continue but not for next, step or stepi).
For stepi, return control to the user (we are already at the next instruction due to step 4 above). For next, continue single-stepping until we reach a line in source that is not the same line we were on at step 1 above.
I'm trying to debug proftpd in order to understand better this exploit http://www.phrack.org/issues.html?issue=67&id=7. The vulnerable section is in mod_sql.c, I have tried to breakpoint the sql_prepare_where function (that is where the heap overflow is done) and then call the USER ... and PASS ... command but it is never triggered.
To find out why I have breakpoints all the hundreds line of mod_sql.c and then launch the program (with full debugging option), some breakpoints are triggered (sql_setuserinfo, set_sqlauthenticate, get_auth_entry...) but only at the very beginning of the launching process, then when the program goes in it main loop nothing else breakpoint related happens (while the log of proftpd mentions that the USER and PASS commands are dispatched to mod_sql.c)..
Would anyone know what I'm missing?
[ It's possible I am missing something essential of GDB, I am learning on the roll :) ]
Server programs often use a "separate program for each connection" method, where after successful accept, the parent forks a child to handle current connection, and goes back to accepting more connections.
I don't know for sure, but if proftpd used that model, it would explain exactly the symptoms you've described.
You can ask GDB to debug the child instead of the parent, by using (gdb) set follow-fork-mode child.
I'm a little new to GDB. I'm hoping someone can help me with something that should be quite simple, I've used Google/docs but I'm just missing something.
What is the 'normal' way folks debug threaded apps with GDB? I'm using pthreads. I'm wanting to watch only one thread - the two options I see are
a) tell the debugger somehow to attach to a particular thread, such that stepping wont result in jumping threads on each context switch
b) tell the debugger to suspend/free any 'uninteresting' threads
I'd prefer to go route b) - reading the help for GDB I dont see a command for this, tips?
See documentation for set scheduler-locking on.
Beware: if you suspend other threads, and if one of them holds a lock, and if your interesting thread needs that lock at some point while stepping, you'll deadlock.
What is the 'normal' way folks debug threaded apps
You can never debug thread correctness, you can only design it in. In my experience, most of debugging of threaded apps is putting in assertions, and examining state of the world when one of the assertions is violated.
First, you need to enable comfortable for multi-threading debugger behavior with the following commands. No idea why it's disabled by default.
set target-async 1
set non-stop on
I personally put those commands into .gdbinit file. They make your every command to be applied only to the currently focused thread. Note: the thread might be running, so you have to pause it.
To see the focused thread execute the thread.
To switch to another thread append the number of the thread, e.g. thread 2.
To see all threads with their numbers issue info thread.
To apply a command to a particular thread issue something like thread apply threadnum command. E.g. thread apply 4 bt will apply backtrace command to a thread number 4. thread apply all continue continues all paused threads.
There is a small problem though — many commands needs the thread to be paused. I know a few ways of doing that:
interrupt command: interrupts the thread execution, accepts a number of a thread to pause, without an argument breaks the focused one.
Setting a breakpoint somewhere. Note that you may set a breakpoint to a particular thread, so that other threads will ignore it, like break linenum thread threadnum. E.g. break 25 thread 4.
You may also find very useful that you can set a list of commands to be executed when a breakpoint hit through the command commands — so e.g. you may quickly print interesting values, then continue execution.