GDB is killing my inferior process - gdb

Question

GDB is killing my inferior. Inferior is a long-running (20-30 minutes) benchmark. GDB and inferior are both running under my uid. Runs fine for a while then my signal handler is called with a siginfo_t instance with si_signo = 11, si_errno = 0 and si_code = 0; _sifields._kill.si_pid = (gdb-pid), _sifields._kill.si_uid = (my-uid).
I read this as GDB decided to send a kill signal to my inferior process. Under what circumstances would GDB do this?
This is not a SIGSEGV (even though si_signo would suggest that it is) since si_code is 0 and si_pid and si_uid are set). My inferior is a multi-threaded C++ application with a custom signal handler to handle GPFs when the application hits a memory barrier that I set up to protect certain ranges of memory. When I run under GDB I set
handle SIGSEGV noprint
to ensure that GDB passes SIGSEGV signals relating to the memory barrier on to my application for handling. That part seems to be working fine -- SIGSEGV with nonzero si_code in the siginfo_t struct are handled properly (after verifying that the faulting address in siginfo->_sifields.si_addr is within a protected range of memory).
But SIGSEGV with zero si_code indicates that the inferior is being killed, as far as I can tell, and the _sifields._kill fields, which overlays _sifields._sigfault fields, support this interpretation: GDB is killing my inferior process.
I just don't understand what causes GDB to do this.
An update on this: it looks like GDB is sending SIGSTOP to the inferior. If I look at $_siginfo at point of failure I see:
(gdb) p $_siginfo
$2 = {
si_signo = 5,
si_errno = 0,
si_code = 128,
_sifields = {
_pad = {0, 0, -1054653696, 57, 97635496, 0, 5344160, 0, 47838328, 0, -154686444, 32767, 47838328, 0, 4514687, 0, 0, 0, 49642032, 0, 50016832, 0, 49599376, 1, 0, 0, 92410096, 0},
_kill = {
si_pid = 0,
si_uid = 0
},
_timer = {
si_tid = 0,
si_overrun = 0,
si_sigval = {
sival_int = -1054653696,
sival_ptr = 0x39c1234300
}
},
_rt = {
si_pid = 0,
si_uid = 0,
si_sigval = {
sival_int = -1054653696,
sival_ptr = 0x39c1234300
}
},
_sigchld = {
si_pid = 0,
si_uid = 0,
si_status = -1054653696,
si_utime = 419341262248738873,
si_stime = 22952992424591360
},
_sigfault = {
si_addr = 0x0
},
_sigpoll = {
si_band = 0,
si_fd = -1054653696
}
}
}
But my signal handler sees this (somewhat obfuscated * -- I am working in a clean-room environment):
(gdb) bt
#0 ***SignalHandler (signal=11, sigInfo=0x7fff280083f0, contextInfo=0x7fff280082c0) at ***signal.c:***
...
(gdb) setsig 0x7fff280083f0
[signo=11; code=0; addr=0xbb900007022] ((siginfo_t*) 0x7fff280083f0)
...
(gdb) p *((siginfo_t*) 0x7fff280083f0)
$4 = {
si_signo = 11,
si_errno = 0,
si_code = 0,
_sifields = {
_pad = {28706, 3001, -515511096, 32767, -233916640, 32767, -228999566, 32767, 671122824, 32767, -468452105, 1927272, 1, 0, -515510808, 32767, 0, 32767, 37011703, 0, -515511024, 32767, 37011703, 32767, 2, 32767, 1000000000, 0},
_kill = {
si_pid = 28706,
si_uid = 3001
},
_timer = {
si_tid = 28706,
si_overrun = 3001,
si_sigval = {
sival_int = -515511096,
sival_ptr = 0x7fffe145ecc8
}
},
_rt = {
si_pid = 28706,
si_uid = 3001,
si_sigval = {
sival_int = -515511096,
sival_ptr = 0x7fffe145ecc8
}
},
_sigchld = {
si_pid = 28706,
si_uid = 3001,
si_status = -515511096,
si_utime = 140737254438688,
si_stime = 140737259355762
},
_sigfault = {
si_addr = 0xbb900007022
},
_sigpoll = {
si_band = 12889196884002,
si_fd = -515511096
}
}
}
(gdb) shell ps -ef | grep gdb
*** 28706 28704 0 Jun26 pts/17 00:00:02 /usr/bin/gdb -q ***
(gdb) shell echo $UID
3001
So my signal handler sees a siginfo_t struct with si_signo 11 (SIGSEGV), si_code = 0 (kill), si_pid = 28706 (gdb), and si_user = 3001 (me). And GDB reports a siginfo_t with si_signo = 5 (SIGSTOP).
It may be that the inferior process is performing some low-level handling of the original SIGSTOP and sending it up the chain as a kill. But it is the original SIGSTOP that I don't understand/want to eliminate.
I should add that I am setting the following directives before starting the inferior (and it makes no difference whether the handle SIGSTOP directive is set or not):
handle SIGSEGV noprint
handle SIGSTOP nostop print ignore
Does this shed any light on the problem? This is killing me. Also, if no insight here, can anyone suggest other forums that might be helpful to post this to?
(gdb) show version
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6_0.1)
Copyright (C) 2010 Free Software Foundation, Inc.
I am running this on a 1.8GHz 16 Core/32 Thread Xeon, 4x E7520, Nehalem-based server. I get the same result regardless of whether hyperthreading is enabled or disabled.

Answer 1

Under Linux, si_signo = 11 would indicate that this is GDB propagating SIGSEGV. See signal(7) for the signal numbers.
Try:
(gdb) handle SIGSEGV nopass
Signal Stop Print Pass to program Description
SIGSEGV Yes Yes No Segmentation fault
Try casting the third argument of the signal handler function you register with sigaction() to a (ucontext *) and dumping the CPU registers. The instruction pointer in particular could provide some clue:
#include <ucontext.h>
int my_sigsegv_handler(int signo, siginfo_t *info, void *context)
{
ucontext *u = (ucontext *)context;
/* dump u->uc_mtext.gregs[REG_RIP] o REG_EIP */
}
then pass the instruction pointer to info addr in GDB.
To really understand what's happening, I'd try to pin down:
Exactly which signal is seen by your process, is it SIGSEGV as indicated by the si_signo member of siginfo_t? What's the first argument of the signal handler function registered by sigaction()? (Those two things not matching is unlikely but not impossible with PTRACE_SETSIGINFO)
GDB either intercepted a signal that the kernel is sending to your process and then injected the signal again or decided to send the signal itself. Try to determine which. This could be done by running GDB under itself and breaking on kill, tkill, tgkill and ptrace if $rdi == PTRACE_KILL (Sounds time consuming, I know).

Answer 2

I'm running into the exact same problem on a different program with gdb 7.4. I upgraded to gdb 7.9 ---same machine and kernel version--- and the problem disappeared. On a different machine, gdb 7.7 works too. My guess is that the issue was fixed in gdb 7.5-7.7.