I am trying to extract the execution sequence of my program (something like a program counter) with gdb on my local computer (windows x86) and gdbserver on a remote target (arm-linux). The idea I had was to insert breakpoints at "important" lines of my source files (i.e.: at the beginning of a specific function, and more in general before and after a conditional statement) with a high ignore count for each breakpoint, and then check if a breakpoint was hit or not. I was actually able to receive the informations with this method, but there is a problem: the application behavior I am debugging depends on real-time, and this specific method slows down the program execution too much. Do you think I could use some other method with gdb? I stumbled upon tracepoints, wich seems the exact thing I am looking for, but I was not able to find some property like a "hit counter" for them. The gdb version I am currently using is 7.5.
Thanks a lot in advance.
If your program execution must not be slowed down, you will probably need some HW tool. See these:
Keil real time trace
Lauterbach PowerDebug
(probably other similar solutions)
Related
I'm working on an implement of gdb server. The server talks with gdb using RSP protocol and drive a CPU model. While developing the line step function (range step mode enabled) i found that my CPU model need to take some time to finish just one instruction step. The reason is the CPU model run in individual process. I have to pass the package (instruction step) through IPC. With thousands of instructions in the source line it spend too much time compared with normal running of that line.
Can I ask gdb using temporary breakpoints ( set on any possible instruction can go out of the range corresponding to the source line) to assist with step functionality? Does gdb really knows where to set the required breakpoints? If the answer is false, is there a good way to deal with this problem?
Thank you for your time!
Ted CH
You say that you are using range-stepping mode, which would imply your gdb server already supports the vCont packet, and the r action. See this page for all the details.
The r action gives a range start and end, and you are free to step until the program counter leaves the specified range.
Your server is free to implement this as you like, so, you could place a temporary breakpoint, then resume execution until the breakpoint is hit. But, you need to think about both control flow, and traps, both of which might cause a thread to leave the range without passing through the end point.
If passing each single step through to your simulator is slow, could you not just pass the vCont range information directly through to the simulator, and have the simulator stop as soon as the program counter leaves the range? Surely this would be quicker than having multiple round trips between simulator and server.
I've built OpenWrt for x86 and I'm using QEMU to run it virtually.I'm trying to debug this system in real time. I need to see things like network traffic flowing etc.
I can attach gdb remotely and execute (mostly) step by step with break points. I really want trace points though. I don't want to pause execution and loose network flow. When I tried setting trace points using tstart, I see the message "Target does not support this command". I did a bit of reading of the gdb documentation and from what I can tell the gdb stub that runs to intercept normal execution in QEMU does not support trace points.
From here I started looking at other tools and ran across PANDA (https://github.com/panda-re/panda). As I understand PANDA will capture a complete system trace in a log and allow for replay. I think this tool is supposed to do what I need, but I cannot seem to replay the results. I see the logs, I just can't replay them.
Now, I'm a bit stuck on what other tools/options I might have to actually trace a running embedded system. Are there any good tools you can recommend? Or perhaps another method I've missed?
If you want to see the system calls and signals use strace.
Strace can also be used with running process and it can put the output in a log file if required.
In OpenWrt it is possible to build with ftrace. Ftrace has much of the functionality I required but not all.
To build with ftrace, the option for ftrace must be selected in the build menu. Additionally there are a variety of tracer options that must also be enabled.
The trace-cmd (ftrace) is located in menuconfig/Development
Tracing support is under menuconfig/Global build settings/Compile the kernel with tracing support and includes: Trace system calls, Trace process context switches and events, and Function tracer (Function graph tracer, Enable/disable function tracing dynamically, and Function profiler)
I'm also planning to build a custom GDB stub to do this a little bit better as I also want to see the data passed to the functions not just the function calls.
I am working on a big code base. It is heavily multithreaded.
After running the linux based application for a few hours, in the end, right before reporting, the application silences. It doesn't die, it doesn't crash, it just waits there. Joins, mutexes, condition variables ... any of these can be the culprit.
If it had crashed, I would at least have a chance to find the source using debugger. But this way, I have no clue how to use what tool to find the bug. I can't even post a code sample for you. The only thing that can possibly help is to tap MANY places with cout to get a visual where the application is.
Have you been in such a situation? What do you recommend?
If you're running under Linux then just use gdb to run the program. When the application 'silences', interrupt it with CTRL+C, then type backtrace to see the call stack. With this you will find out the function where your application was blocked.
Incase of linux, gdb will be great help. Another tool that can be of great help is strace (This can also be used where there are problems with program for with source is not readily available because strace does not need recompilation to trace them.)
strace shall intercept/record system calls that are called by a process and also the signals that are received by a process. It will be able to show the order of events and all the return/resumption paths of calls. This can take you almost closer to the area of problem.
iotop, LTTng and Ftrace are few of other tools that be helpful to you in this scenario.
I know the answer to this, I'm putting it up here for others to see it
If you use eclipse CDT, you probably understand that eclipse isn't a debugger, it's just an application front-end, specifically to GDB. So when debugging C++ programs, you're actually just using GDB in a more comfortable manner. If you ever have to debug a multithreaded program in eclipse CDT, you'll realize that things quickly get hectic because when you hit a breakpoint, all threads stop, and when one tries to execute a single line in a specific thread, it also runs the other threads. In order for it to work properly, the threads have to be able to be run arbitrarily and exlusively-so that when the programmer executes a single line, it only executes the specific thread.
So, by default, gdb's settings by default leave the "scheduler-locking" turned off. If you debug multithreaded applications you'll understand that this must be on in GDB in order for the desired behavior to be achieved. How does one run this command:
set scheduler-locking on
in GDB within eclipse CDT?
At least one way to do it that certainly solves the problem is knowing how to navigate the immense set of features that eclipse offers. Typically, when a program starts, eclipse CDT switches the console window (if you have it open, typically it's on the bottom) to show the input/output of the program.
But you can change this if you didn't know-see this image. That button on the second to last right-the blue one that looks like a monitor-you can select the GDB input console. It was discussed also in this thread.
From there merely type the command.
SOLVED, BUT NEED A BETTER SOLUTION
But now that this has been solved, to solve it in a better way as a matter of convience; having to type set scheduler-locking on every time a program starts is silly. But the problem with loading a gdbinit file is that the gdbinit file gets sourced before eclipse has set the program for gdb to solve. This is a problem, as it causes the debugger view to hang within eclipse, as gdb complains. To understand what is happening, try and fire up gdb, then give the command without loading a binary to execute. It fails-so how does one set this as an option that is sticky?
Maybe if you add the following gdb script which could set the variable when the program stops and turns it off if you continue:
define hook-step
set scheduler-locking on
end
define hookpost-step
set scheduler-locking off
end
define hook-run
set scheduler-locking off
end
define hook-continue
set scheduler-locking off
end
My answer is derived from the one by #user1448557 . Unfortunately, I don't currently have enough reputation to comment on it (or to upvote it by the way). The strategy seems great, but the answer might be a bit outdated because it doesn't involve "set scheduler-locking step". I have put the following in my gdb initialization file (within my Eclipse project) and it does what I want.
#inspired from [link to this thread][1]
define hookpost-run
set scheduler-locking step
end
With regards to the comment by #rbaleksandar, Eclipse CDT launch configurations allow one to specify a "GDB Command File" and the default is usually .gdbinit
I'm writing a software renderer in g++ under mingw32 in Windows 7, using NetBeans 7 as my IDE.
I've been needing to profile it of late, and this need has reached critical mass now that I'm past laying down the structure. I looked around, and to me this answer shows the most promise in being simultaneously cross-platform and keeping things simple.
The gist of that approach is that possibly the most basic (and in many ways, the most accurate) way to profile/optimise is to simply sample the stack directly every now and then by halting execution... Unfortunately, NetBeans won't pause. So I'm trying to find out how to do this sampling with gdb directly.
I don't know a great deal about gdb. What I can tell from the man pages though, is that you set breakpoints before running your executable. That doesn't help me.
Does anyone know of a simple approach to getting gdb (or other gnu tools) to either:
Sample the stack when I say so (preferable)
Take a whole bunch of samples at random intervals over a given period
...give my stated configuration?
Have you tried simply running your executable in gdb, and then just hitting ^C (Ctrl+C) when you want to interrupt it? That should drop you to gdb's prompt, where you can simply run the where command to see where you are, and then carry on execution with continue.
If you find yourself in a irrelevant thread (e.g. a looping UI thread), use thread, info threads and thread n to go to the correct one, then execute where.