got some big real time project to deal with (multiple processes (IPCs), multi Everything in short).
My working on process is started as service on Linux. I have the root access.
Here is the problem:
I'm trying to attach to a running proc, tried starting it through/with gdb but the result is the same: it stops the executable once I "touched" it with gdb or sometimes it throws:
Program received signal SIGUSR1, User defined signal 1. [Switching to Thread 0x7f9fe869f700 (LWP 2638)]
of course from there nothing can be done.
Tried:
handle all nostop
attach to launched as service (daemon) or launched as regular proc
started from gdb
thought maybe forking/multi-threaded problem - implemented in the very beginning sleep for 10 seconds - attached to it with "continue"
Guys, all I want it is to debug, hit the breakpoints, etc.
Please help! Share ideas.
Editing actual commands:
1) gdb attach myProcId. Then after reading symbols, I hit "c" which results:
Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7f9fe869f700 (LWP 2638)]
0x00007f9fec09bf73 in select () from /lib64/libc.so.6
2) If I make the first line 10 seconds sleep in the code, attaching to the process, hit "c", result: it runs, shows info threads, backtrace of main, but never hits the breakpoint (for sure the code runs there - I get logs and different behaviour if I change code there), meaning the process is stuck.
3) All other combinations like gdb path/to/my/proc args list, then start. Where arg list played with different related options gdb gives us.
Maybe worth to mention: process network packets related, timers driven also.
But for me the important thing is a current snapshot on break, i don't care what will happen to the system after timers expired.
Since you mentioned that you are debugging a multiprocessing program, I think the underlying program you have is to set the breakpoint in the correct subprocess.
Try break fork and set follow-fork-mode child/parent. What you want to achieve is have gdb attached to the process that is running the code you want to debug.
Refer to this link.
Another thought is to generate a crash, since you can compile the programe. For example add a int i = *(int*)NULL and that will generate a core dump. You can then debug the core dump with gdb <program> <core dump>. You can refer to this page for how to configure core dump.
Related
I have web daemon and request that makes it fail with SIGSEGV. So i start daemon, attach with gdb, continuing, send request and getting this:
$ gdb attach -p 630066
(gdb) c
Continuing.
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)
How to make gdb print stacktrace before killing application? Application do not have subprocesses, just threads.
Thanks.
Your GDB session indicates that you have not attached all threads of the multithreaded process, and some other thread (one you didn't attach) ran into SIGSEGV and terminated the entire process.
Another (somewhat unlikely) possibility is that you are using a very old version of GDB, one which still has this bug in it (the bug was fixed in 2009).
When using gdb -p NNNN you need to be careful and specify correct process id. pgrep daemon-name or ps aux | grep daemon-name should give you a good idea which process to attach.
Just enter backtrace or bt right in the gdb shell after getting SIGSEGV.
To explore stack trace for each separate thread, start with info thread, then choose the thread you need, for example thread 3 and then type bt to see the stack trace for that thread.
I'm debugging a C++ application which creates trees of forks. Using GDB defaults, the child processes will be detached on the fork and as a result I see only one inferior shown afterwards.
I tried to attach to one of the child processes and despite it not being listed as an inferior for the other GDB process, in the new GDB session I get an error that the process is already being traced (by the first GDB session).
Is this expected behavior? What steps can I take to debug the forked process in a separate GDB session? What steps can I take to debug the problem further?
I will try to be as specific as I can, but so far I have worded this problem so poorly that Google failed to return any useful results (hence my question here).
I am attaching gdb to a multi-threaded c++ server process. All I can say is that strange things have been happening while trying to do the usual set-breakpoint-break-investigate.
First, while waiting for the breakpoint to be hit (in 'Continuing' mode), I suddenly got back the (gdb) prompt with the message:
Continuing.
[Thread 0x54d5b940 (LWP 28503) exited]
[New Thread 0x54d5b940 (LWP 28726)]
Cannot get thread event message: debugger service failed
Second, also while waiting for the breakpoint to be hit, I'm suddenly told the program has received SIGSEGV and - back to the (gdb) prompt - backtrace tells me the segfault happened in pthread_cancel(). Note the process under investigation does not normally segfault.
I clearly lack enough information about how gdb works to even begin guessing what is happening. Am I doing anything wrong? The steps I take are the same each time:
gdb attach
break 'MyFunction()'
continue
Thoughts? Thanks.
I fought with similar gdb issues for a while. My case was having lots of threads spawned that executed few functions and then exited.
It appears if a thread exits too fast and there's lots of these happening sometimes gdb cannot keep up and when it fails, it fails with style as in crashes :) I think it tries to attach to a thread that is already done as per the error message.
I see this as an issue in gdb 6.5 to 7.6 and still happening. Did not try with older versions.
My advice is look for this use case or similar. Once I changed my design to have a thread serving a queue of requests gdb works flawlessly.
Design wise is healthier to have already created threads that digest actions than always spawning new threads.
Still same code debugs without a problem on Visual Studio so I do have to say that is a small disappointment to me with regards to gdb.
I use Eclipse and looking at the GDB traces (usually enabled by default) will give you a better hint of where GDB fails. One of the buttons on the console shows you the GDB trace.
Without getting into to to much detail, I'm working on a program that consists of several separate processes all running on embedded QNX RTOS. They don't have a parent-child relationship, they are all spawned using spawnlp(P_NOWAIT, ...) and they all communicate with each other using the IPC mechanism provided by the OS.
When I'm debugging with GDB and I hit a breakpoint in the process I'm working in, all of my threads are paused, which is great. But is there a way to also have it pause execution of my other processes? Right now what's happening is all the other processes keep on truckin' while my process is paused and so all the IPC queues get full etc. etc.
Thanks in advance,
HF
You can associate a list of GDB commands with each breakpoint. So when you hit a breakpoint in process A, you can for example send a SIGTRAP to process B, which should drop it into the debugger:
(gdb) b main
Breakpoint 1 at 0x804834a: file testA.c, line 40.
(gdb) command
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>shell kill -s TRAP `pidof testB`
>end
(gdb)
More info at Breakpoint Command Lists
I use ddd as a front-end for both gdb and dbx for C++ programs.
Quite often, without any apparent cause, I will try to next and it will hang with the message "Waiting for gdb to get ready" or "Waiting for dbx to get ready".
Does anybody know what it is that they're doing that takes forever and produces no apparent results? And can I stop it from happening?
Bear in mind that enough stuff has already been loaded that I have quite happily been stepping/nexting a minute earlier in the same process (and in the same function), so whatever they're doing doesn't seem to have been necessary for that. Also the fact that both ddd and dbx have the same pattern of behaviour (in many different executables and on different platforms) makes me think it's something in the data rather than a bug in either debugger.
GDB (and the same applies for DBX) communicate with DDD with the MI protocol, which is a standardize and unambiguous equivalent of the command-line interface.
Remark: the default in my system (Fedora 15) seems to be that they communicate directly using the CLI, but I only noticed the problem you describe with --interpret=mi.
For instance, here are the respective output to get the thread list:
(gdb) info threads
Id Target Id Frame
2 Thread 0x7ffff7fd2700 (LWP 9191) "philosophers" 0x00000037dcc0b4c5 in pthread_cond_wait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
1 Thread 0x7ffff7fd3720 (LWP 9182) "philosophers" 0x00000037dc8df461 in clone () from /lib64/libc.so.6
(gdb) -thread-info
^done,threads=[
{id="2",target-id="Thread 0x7ffff7fd2700 (LWP 9525)",name="philosophers",
frame={level="0",addr="0x0000000000400b31",
func="chopsticks_put",
args=[{name="i",value="0"}],
file="chopsticks.c",fullname="philosphers/chopsticks.c",line="70"},
state="stopped",core="2"},
{id="1",target-id="Thread 0x7ffff7fd3720 (LWP 9522)",name="philosophers",
frame={...},
state="stopped",core="1"
}],current-thread-id="3"
So what you will see in DDD will be quite similar to what is available in the CLI, only the 'presentation layer' is different.
From my experience, most of GDB commands are very fast, at least when they don't depend on the debuggee execution (like a next over a sleep(5)). So there are two possibilities for your problem:
a bug in the communication: for instance a ^done tag is missed by DDD or forgotten by GDB, so DDD waits in vain for the termination of its request
DDD asks GDB for a lot of data, like the definition of structures, function locations or memory contents, etc. (for instance because of the elements you want to watch), so it will take some time for the information to be computed by GDB and transferred to DDD.
At the bottom of DDD you have the GDB console. Try typing some GDB commands in there. If GDB responds correctly (my case) it means that DDD is not synchronized with GDB anymore. (DDD is getting old, 2009/02/11, and MI is extensively used by Eclise, so I think we know who has to be blamed...!)