I have a program that depends on an external shared library, but after a function inside the library gets executed I lose the ability to use breakpoints.
I can break and step like normal up until I execute this function, but afterwards it is finished it never breaks. It won't even break on main if I try and use start for the second time executing the program. It's not an inlined function problem, because I've broken on these functions before and when I comment out this particular function everything starts to work again.
Has anyone ever encountered anything like this before? What can I do?
Using gdb 7.1 with gcc 3.2.3
Edit:
After some hints from users I figured out that the process is forking inside the library call. I'm not sure what it's doing (and I really don't care). Can I somehow compensate for this? I've been experimenting with the follow-fork-mode as child, but I'm really confused what happens once it forks and I can't seem to figure out how to continue execution or do anything of use.
Edit:
Further investigation. The nearest I can tell, gdb is losing all its symbol information somewhere. After the 2nd run, all symbols resolve to the #plt address and not to the actual address that they resolved to on the first run. Like somehow the second loading of the process loses all the info it gained the first time around and refuses to reload it. I'm so confused!!
Edit:
So I traced down the problem to the vfork of a popen call. Apparently gdb doesn't play nice with popen? As soon as I detach from the popen'd vforked process, I lose all my symbols. I've read some reports online about this as well. Is there any hope?
It's the combination of vfork(2) and exec(2) that's messing things up. Quoting from gdb manual (debugging forks):
On some systems, when a child process is spawned by vfork, you cannot debug the child or parent until an exec call completes.
...
By default, after an exec call executes, gdb discards the symbols of the previous executable image. You can change this behaviour with the set follow-exec-mode command.
Keep follow-fork-mode set to parent and set follow-exec-mode to same.
Alternatively, if your installation supports multi-process debugging (it should, since your gdb version is 7.1), try using info inferiors to find your original process and switching to it with inferior <num>.
Related
I wrote a shared library in C++ on Linux, which contains a function f(). This library is used from multiple programs, calling the function.
Now I would like to conduct some debugging with that function, while calling it from program A. When calling it from any other program, it should fail, as long as I debug the function. After I do not have complete control over when the other programs are run, I would like to add an exception, which stops every program except program A when calling function f during Debug sessions.
How could I solve that?
The only way I can think of is by checking the information of the currently running process. You can get the pid by calling the getpid() function. All the information on all processes on a linux system can be found in the /proc/<pid> directory. When calling function f, you can check this information and decide whether to throw an exception or not.
There isn't a way that can't be gotten around. If another process has PTRACE on yours, it can make anything at all happen in your process, so any check you perform to try to make the function not work can be disabled.
So I am trying to use GDB. I compile my code with -g, then gdb ./a.out
GNU gdb starts, but after I type r to start, the program runs like it normally would if I just called ./a.out.
Do you know what could cause this? I don't know much about gdb and I use it lightly , I've been using it the same way for a while and never encountered this type of behavior.
Edit: It works when I set up breakpoints. But I am still confused as to why I was able to use it for months without setting any breakpoints before.
Do you know what could cause this?
This is intended behavior. The run command starts the execution of the inferior (being debugged) program.
That program may encounter a bug (e.g. a crash), in which case GDB will be notified and would stop the execution of the inferior, and let you look around.
The program may also encounter a breakpoint that you have inserted earlier, again allowing you to look around at the current state.
Or the program may run to completion (if it doesn't execute any code in which you have set breakpoints, or if you didn't set any, and if it doesn't have any bugs that manifest in a fatal signal). If that happens, you'll get a 'program exited normally' message.
I am still confused as to why I was able to use it for months without setting any breakpoints before.
Your program was probably crashing, and now doesn't.
When debugging a C++ program emit a SIGSEGV with gdb,it is possible to handle the signal and asked to nostop.
How gdb handles this kind of scenario ??
Have searched gdb source code and couldn't find a starting point.
You cannot automatically ignore SIGSEGV. I also wouldn't recommend doing that anyway. Although you can make gdb ignore the signal and not pass it to the program, the kernel will attempt to re-run the offending instruction once the signal handler returns and results in an infinite loop. See this answer for more information.
One way to work around it is to the skip the instruction or change register values so that it does not segfault. The link shows an example of setting a register. You can also use the jump command to skip over an instruction.
is possible to handle the signal and asked to nostop.
It's unclear whether you want GDB to handle the signal, or the program itself.
If the latter, gdb handle SIGSEGV nostop noprint pass will do exactly that.
This is actually something the OS does. On Windows, if a program has a debugger attached and an exception is thrown, Windows will ask the debugger if it wants to handle it. If/when it declines, it passes it to the program. If the program doesn't handle it, Windows passes it to the debugger again.
program runs fine. When I put a breakpoint a segmentation fault is generated. Is it me or GDB? At run time this never happens and if I instantiate only one object then no problems.
Im using QtCreator on ubuntu x86_64 karmic koala.
UPDATE1:
I have made a small program containing a simplified version of that class. You can download it at:
example program
simply put a breakpoint on the first line of the function called drawChart() and step into to see the segfault happen
UPDATE2: This is another small program but it is practically the same as the mandlebrot example and it is still happening. You can diff it with mandlebrot to see the small difference.
almost the same as mandlebrot example program
To answer your question: Yes, you should be able to debug multiple threads using GDB. This depends on the concurrent design to be sound.
There is a chance you have a race condition on data that your threads access. It is possible that the problem does not show when you run the program normally, but attaching a debugger changes timing and scheduling. Even so, you should be able to use the debugger to break when the segfault happens. Understanding where this happens can inform you about the race condition or corruption, whatever the case may be.
It is worth looking into because even if it doesn't happen under most 'run time' conditions, it may manifest under different system load conditions.
Are you Calling into Qt's drawing code from multiple threads? (particularly widget methods)
http://doc.qt.nokia.com/4.3/threads.html#reentrancy-and-thread-safety
Seems like Qt is like GTK+ and you should only be touching GUI stuff from one thread (in particular the main one)
I'm not familiar enough with Qt to give you advice on how to change your code, but I'd suggest changing it to be event based (ie rendering starts in response to an event, then triggers an event in the main thread when it's done, every thread has it's own mainloop) that way you can probably completely avoid mutexes and synchronization.
I want to be able to detect when a write to memory address occurs -- for example by setting a callback attached to an interrupt. Does anyone know how?
I'd like to be able to do this at runtime (possibly gdb has this feature, but my particular
application causes gdb to crash).
If you want to intercept writes to a range of addresses, you can use mprotect() to mark the memory in question as non-writeable, and install a signal handler using sigaction() to catch the resulting SIGSEGV, do your logging or whatever and mark the page as writeable again.
What you need is access to the X86 debug registers: http://en.wikipedia.org/wiki/Debug_register
You'll need to set the breakpoint address in one of DR0 to DR3, and then the condition (data write) in DR7. The interrupt will occur and you can run your debug code to read DR6 and find what caused the breakpoint.
If GDB doesn't work, you might try a simpler/smaller debugger such as http://sourceforge.net/projects/minibug/ - if that isn't working, you can at least go through the code and understand how to use the debugging hardware on the processor yourself.
Also, there's a great IBM developer resource on mastering linux debugging techniques which should provide some additional options:
http://www.ibm.com/developerworks/linux/library/l-debug/
A reasonably good article on doing this is windows is here (I know you're running on linux, but others might come along to this question wanting to do it in windows):
http://www.codeproject.com/KB/debug/hardwarebreakpoint.aspx
-Adam
GDB does have that feature: it is called hardware watchpoints, and it is very well supported on Linux/x86:
(gdb) watch *(int *)0x12345678
If your application crashes GDB, build current GDB from CVS Head.
If that GDB still fails, file a GDB bug.
Chances are we can fix GDB faster than you can hack around SIGSEGV handler (provided a good test case), and fixes to GDB help you with future problems as well.
mprotect does have a disadvantage: your memory must be page-boundary aligned. I had my problematic memory on the stack and was not able to use mprotect().
As Adam said, what you want is to manipulate the debug registers. On windows, I used this: http://www.morearty.com/code/breakpoint/ and it worked great. I also ported it to Mach-O (Mac OS X), and it worked great, too. It was also easy, because Mach-O has thread_set_state(), which is equivalent to SetThreadContext().
The Problem with linux is that it doesn't have such equivalents. I found ptrace, but I thought, this can't be it, there must be something simpler. But there isn't. Yet. I think they are working on a hw_breakpoint API for both kernel and user space. (see http://lwn.net/Articles/317153/)
But when I found this: http://blogs.oracle.com/nike/entry/memory_debugger_for_linux I gave it a try and it wasn't that bad. The ptrace method works by some "outside process" acting as a "debugger", attaching to your program, injecting new values for the debug registers, and terminating with your program continuing with a new hw breakpoint set. The thing is, you can create this "outside process" yourself by using fork(), (I had no success with a pthread), and doing these simple steps inline in your code.
The addwatchpoint code must be adapted to work with 64 bit linux, but that's just changing USER_DR7 etc. to offsetof(struct user, u_debugreg[7]). Another thing is that after a PTRACE_ATTACH, you have to wait for the debuggee to actually stop. But instead of retrying a POKEUSER in a busy loop, the correct thing to do would be a waitpid() on your pid.
The only catch with the ptrace method is that your program can have only one "debugger" attached at a time. So a ptrace attach will fail if your program is already running under gdb control. But just like the example code does, you can register a signal handler for SIGTRAP, run without gdb, and when you catch the signal, enter a busy loop waiting for gdb to attach. From there you can see who tried to write your memory.