I have web daemon and request that makes it fail with SIGSEGV. So i start daemon, attach with gdb, continuing, send request and getting this:
$ gdb attach -p 630066
(gdb) c
Continuing.
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)
How to make gdb print stacktrace before killing application? Application do not have subprocesses, just threads.
Thanks.
Your GDB session indicates that you have not attached all threads of the multithreaded process, and some other thread (one you didn't attach) ran into SIGSEGV and terminated the entire process.
Another (somewhat unlikely) possibility is that you are using a very old version of GDB, one which still has this bug in it (the bug was fixed in 2009).
When using gdb -p NNNN you need to be careful and specify correct process id. pgrep daemon-name or ps aux | grep daemon-name should give you a good idea which process to attach.
Just enter backtrace or bt right in the gdb shell after getting SIGSEGV.
To explore stack trace for each separate thread, start with info thread, then choose the thread you need, for example thread 3 and then type bt to see the stack trace for that thread.
Related
I have a double-free bug. I am able to reproduce it using a debug build with Address Sanitizer (AS) detects but when I run under GDB, AS kills the GDB session.
I found this Address Sanitizer page with instructions how to keep GDB:
https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
but when I do:
(gdb) break __asan::ReportGenericError
at the beginning of the session, the GDB state still disappears after the problem is detected:
(gdb) bt
No stack.
the GDB state still disappears after the problem is detected
There are several possible reasons for this:
Somehow you didn't set the breakpoint correctly
It's actually a child process that is dying
Somehow the thread in which the error is detected is not attached by GDB.
To eliminate 1, use catch syscall exit_group (and possibly also catch syscall exit) -- this way GDB is sure to stop before the process disappears.
For 2, AddressSanitizer message should indicate the thread id in which the error is detected, and that id should match one of the threads in GDB info thread output.
For 3, we'd need to understand more about how that thread was created.
got some big real time project to deal with (multiple processes (IPCs), multi Everything in short).
My working on process is started as service on Linux. I have the root access.
Here is the problem:
I'm trying to attach to a running proc, tried starting it through/with gdb but the result is the same: it stops the executable once I "touched" it with gdb or sometimes it throws:
Program received signal SIGUSR1, User defined signal 1. [Switching to Thread 0x7f9fe869f700 (LWP 2638)]
of course from there nothing can be done.
Tried:
handle all nostop
attach to launched as service (daemon) or launched as regular proc
started from gdb
thought maybe forking/multi-threaded problem - implemented in the very beginning sleep for 10 seconds - attached to it with "continue"
Guys, all I want it is to debug, hit the breakpoints, etc.
Please help! Share ideas.
Editing actual commands:
1) gdb attach myProcId. Then after reading symbols, I hit "c" which results:
Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7f9fe869f700 (LWP 2638)]
0x00007f9fec09bf73 in select () from /lib64/libc.so.6
2) If I make the first line 10 seconds sleep in the code, attaching to the process, hit "c", result: it runs, shows info threads, backtrace of main, but never hits the breakpoint (for sure the code runs there - I get logs and different behaviour if I change code there), meaning the process is stuck.
3) All other combinations like gdb path/to/my/proc args list, then start. Where arg list played with different related options gdb gives us.
Maybe worth to mention: process network packets related, timers driven also.
But for me the important thing is a current snapshot on break, i don't care what will happen to the system after timers expired.
Since you mentioned that you are debugging a multiprocessing program, I think the underlying program you have is to set the breakpoint in the correct subprocess.
Try break fork and set follow-fork-mode child/parent. What you want to achieve is have gdb attached to the process that is running the code you want to debug.
Refer to this link.
Another thought is to generate a crash, since you can compile the programe. For example add a int i = *(int*)NULL and that will generate a core dump. You can then debug the core dump with gdb <program> <core dump>. You can refer to this page for how to configure core dump.
I am trying to let my program run continously with GDB.
Currently I have a bash script which starts GDB with my program and when it crashes it prints the backtrace and starts GDB again (endless loop).
Now I added a signal handler for my program which kills specific threads when the handler gets a signal from them. Now I can achieve that GDB does not stop by doing this:
handle SIGSEGV nostop
But this leads me to the problem that I do not get a GDB backtrace which I would like to print automatically without stopping the program (or at least continuing automatically).
Any help would be appreciated!
Continue to use handle to suppress ordinary stops from SEGV. Then set a catchpoint that does what you want:
(gdb) catch signal SIGSEGV
(gdb) commands
> silent # this bit is optional
> bt
> continue
> end
This will print a backtrace on SIGSEGV but not otherwise interfere with normal operation. You may also want handle SIGSEGV noprint.
I attached to my multithread application with gdb and after that type cont to continue execution.
Is there any way to stop execution at any time on cont gdb state and check what every thread do?
How to check state of every thread and get execution line number of each? (commands)
Here's what I do, (taken from here )
Create a little gdb script stackdumper.gdb that dumps the stack trace of all threads:
thread apply all backtrace
Then repeatedly attach gdb and run the dumper:
for i in $(seq 1 10) ; do
gdb -batch -x stackdumper.gdb ./a.out 123456 > stack.$i
sleep 10
done
where ./a.out is the binary you are interested and 123456 is the PID.
Adjust the sleep to match your sampling needs.
thread apply all bt
Or
info threads
t <threadid from above trace >
Followed by
where or bt
To get the backtrace for all of the stopped threads type the
thread apply all bt
command (the output is exactly the same that one might see in the MacOSX crash report box).
Usually the threads are stopped simultaneously in gdb.
Reference: http://www.delorie.com/gnu/docs/gdb/gdb_40.html
And here's about "all-stop" mode, which is default: http://sourceware.org/gdb/onlinedocs/gdb/All_002dStop-Mode.html
Is this any way to stop execution at any time on cont gdb state and check what every thread do
If you ask about ways to check what threads do without gdb the you can just run pstask <pid-of-your application>
Is it "pstack" because i don't think "pstask" is any command in linux, If it is please provide some more info
I have a program which crashes due to a segmentation fault. The core file is produced.
running the core in gdb gives me the following:
HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Core was generated by `gcpf1fwcApp'.
Program terminated with signal 6, Aborted.
I used the command
thread apply all bt
When I check the stack trace I get error in the main thread which is in waiting state.
However when I run the same program in GDB I get a completely different error in stack trace. Which seems more correct than the core dump.
The program has 31 threads.
Why do I get this kind of difference?
It is possible that you are simply looking at the wrong thread.
Try thread apply all where, and see if one of the threads is in fact abort()ing.
When debugging a live process, GDB will stop when a thread receives SIGABRT, and so will likely show you the relevant thread.
When debugging a core (post-mortem), GDB doesn't know which thread is relevant, and so shows them to you in whichever order the OS saved them into the core. Linux kernels save the thread which caused the process to die first, so GDB on Linux shows relevant thread from core. I am guessing that HP-UX does not do that, and so GDB shows you a "random" thread instead.