I have a large multi-threaded program written in cpp and compiled with gcc.
Every now and then I run into bugs in runtime. Is there a way to attach gdb (or anything else) and try to look what each thread is doing and maybe see some internal class members?
The thing is I do not want gdb to freeze the program. There are timing sensitive parts and freezing the program will change its behavior (and possibly crash it if its long enough).
Is there a way to attach gdb (or anything else) and try to look what each thread is doing and maybe see some internal class members?
Yes: GDB can examine stack trace of each thread, and local and global variables (provided you compiled with debug info).
The thing is I do not want gdb to freeze the program.
That's trickier: GDB can only examine stopped threads.
If you have some threads that should continue to run, you should look into non-stop debugging mode.
Related
So I am trying to use GDB. I compile my code with -g, then gdb ./a.out
GNU gdb starts, but after I type r to start, the program runs like it normally would if I just called ./a.out.
Do you know what could cause this? I don't know much about gdb and I use it lightly , I've been using it the same way for a while and never encountered this type of behavior.
Edit: It works when I set up breakpoints. But I am still confused as to why I was able to use it for months without setting any breakpoints before.
Do you know what could cause this?
This is intended behavior. The run command starts the execution of the inferior (being debugged) program.
That program may encounter a bug (e.g. a crash), in which case GDB will be notified and would stop the execution of the inferior, and let you look around.
The program may also encounter a breakpoint that you have inserted earlier, again allowing you to look around at the current state.
Or the program may run to completion (if it doesn't execute any code in which you have set breakpoints, or if you didn't set any, and if it doesn't have any bugs that manifest in a fatal signal). If that happens, you'll get a 'program exited normally' message.
I am still confused as to why I was able to use it for months without setting any breakpoints before.
Your program was probably crashing, and now doesn't.
I am currently investigating an issue with a multithreaded program on linux. I can use gdb to look what each thread does and why the thing crashed. Nevertheless I do miss a way to simply print an overview of all threads, including data like thread state, mutex a thread is waiting for, etc.
Is there a way to get that data out of the core file? Or is that at least possible when the program runs (/proc?)?
I have a program that depends on an external shared library, but after a function inside the library gets executed I lose the ability to use breakpoints.
I can break and step like normal up until I execute this function, but afterwards it is finished it never breaks. It won't even break on main if I try and use start for the second time executing the program. It's not an inlined function problem, because I've broken on these functions before and when I comment out this particular function everything starts to work again.
Has anyone ever encountered anything like this before? What can I do?
Using gdb 7.1 with gcc 3.2.3
Edit:
After some hints from users I figured out that the process is forking inside the library call. I'm not sure what it's doing (and I really don't care). Can I somehow compensate for this? I've been experimenting with the follow-fork-mode as child, but I'm really confused what happens once it forks and I can't seem to figure out how to continue execution or do anything of use.
Edit:
Further investigation. The nearest I can tell, gdb is losing all its symbol information somewhere. After the 2nd run, all symbols resolve to the #plt address and not to the actual address that they resolved to on the first run. Like somehow the second loading of the process loses all the info it gained the first time around and refuses to reload it. I'm so confused!!
Edit:
So I traced down the problem to the vfork of a popen call. Apparently gdb doesn't play nice with popen? As soon as I detach from the popen'd vforked process, I lose all my symbols. I've read some reports online about this as well. Is there any hope?
It's the combination of vfork(2) and exec(2) that's messing things up. Quoting from gdb manual (debugging forks):
On some systems, when a child process is spawned by vfork, you cannot debug the child or parent until an exec call completes.
...
By default, after an exec call executes, gdb discards the symbols of the previous executable image. You can change this behaviour with the set follow-exec-mode command.
Keep follow-fork-mode set to parent and set follow-exec-mode to same.
Alternatively, if your installation supports multi-process debugging (it should, since your gdb version is 7.1), try using info inferiors to find your original process and switching to it with inferior <num>.
program runs fine. When I put a breakpoint a segmentation fault is generated. Is it me or GDB? At run time this never happens and if I instantiate only one object then no problems.
Im using QtCreator on ubuntu x86_64 karmic koala.
UPDATE1:
I have made a small program containing a simplified version of that class. You can download it at:
example program
simply put a breakpoint on the first line of the function called drawChart() and step into to see the segfault happen
UPDATE2: This is another small program but it is practically the same as the mandlebrot example and it is still happening. You can diff it with mandlebrot to see the small difference.
almost the same as mandlebrot example program
To answer your question: Yes, you should be able to debug multiple threads using GDB. This depends on the concurrent design to be sound.
There is a chance you have a race condition on data that your threads access. It is possible that the problem does not show when you run the program normally, but attaching a debugger changes timing and scheduling. Even so, you should be able to use the debugger to break when the segfault happens. Understanding where this happens can inform you about the race condition or corruption, whatever the case may be.
It is worth looking into because even if it doesn't happen under most 'run time' conditions, it may manifest under different system load conditions.
Are you Calling into Qt's drawing code from multiple threads? (particularly widget methods)
http://doc.qt.nokia.com/4.3/threads.html#reentrancy-and-thread-safety
Seems like Qt is like GTK+ and you should only be touching GUI stuff from one thread (in particular the main one)
I'm not familiar enough with Qt to give you advice on how to change your code, but I'd suggest changing it to be event based (ie rendering starts in response to an event, then triggers an event in the main thread when it's done, every thread has it's own mainloop) that way you can probably completely avoid mutexes and synchronization.
I want to be able to detect when a write to memory address occurs -- for example by setting a callback attached to an interrupt. Does anyone know how?
I'd like to be able to do this at runtime (possibly gdb has this feature, but my particular
application causes gdb to crash).
If you want to intercept writes to a range of addresses, you can use mprotect() to mark the memory in question as non-writeable, and install a signal handler using sigaction() to catch the resulting SIGSEGV, do your logging or whatever and mark the page as writeable again.
What you need is access to the X86 debug registers: http://en.wikipedia.org/wiki/Debug_register
You'll need to set the breakpoint address in one of DR0 to DR3, and then the condition (data write) in DR7. The interrupt will occur and you can run your debug code to read DR6 and find what caused the breakpoint.
If GDB doesn't work, you might try a simpler/smaller debugger such as http://sourceforge.net/projects/minibug/ - if that isn't working, you can at least go through the code and understand how to use the debugging hardware on the processor yourself.
Also, there's a great IBM developer resource on mastering linux debugging techniques which should provide some additional options:
http://www.ibm.com/developerworks/linux/library/l-debug/
A reasonably good article on doing this is windows is here (I know you're running on linux, but others might come along to this question wanting to do it in windows):
http://www.codeproject.com/KB/debug/hardwarebreakpoint.aspx
-Adam
GDB does have that feature: it is called hardware watchpoints, and it is very well supported on Linux/x86:
(gdb) watch *(int *)0x12345678
If your application crashes GDB, build current GDB from CVS Head.
If that GDB still fails, file a GDB bug.
Chances are we can fix GDB faster than you can hack around SIGSEGV handler (provided a good test case), and fixes to GDB help you with future problems as well.
mprotect does have a disadvantage: your memory must be page-boundary aligned. I had my problematic memory on the stack and was not able to use mprotect().
As Adam said, what you want is to manipulate the debug registers. On windows, I used this: http://www.morearty.com/code/breakpoint/ and it worked great. I also ported it to Mach-O (Mac OS X), and it worked great, too. It was also easy, because Mach-O has thread_set_state(), which is equivalent to SetThreadContext().
The Problem with linux is that it doesn't have such equivalents. I found ptrace, but I thought, this can't be it, there must be something simpler. But there isn't. Yet. I think they are working on a hw_breakpoint API for both kernel and user space. (see http://lwn.net/Articles/317153/)
But when I found this: http://blogs.oracle.com/nike/entry/memory_debugger_for_linux I gave it a try and it wasn't that bad. The ptrace method works by some "outside process" acting as a "debugger", attaching to your program, injecting new values for the debug registers, and terminating with your program continuing with a new hw breakpoint set. The thing is, you can create this "outside process" yourself by using fork(), (I had no success with a pthread), and doing these simple steps inline in your code.
The addwatchpoint code must be adapted to work with 64 bit linux, but that's just changing USER_DR7 etc. to offsetof(struct user, u_debugreg[7]). Another thing is that after a PTRACE_ATTACH, you have to wait for the debuggee to actually stop. But instead of retrying a POKEUSER in a busy loop, the correct thing to do would be a waitpid() on your pid.
The only catch with the ptrace method is that your program can have only one "debugger" attached at a time. So a ptrace attach will fail if your program is already running under gdb control. But just like the example code does, you can register a signal handler for SIGTRAP, run without gdb, and when you catch the signal, enter a busy loop waiting for gdb to attach. From there you can see who tried to write your memory.