I am trying to break out of a read-line loop into lldb, and then continue where I broke out of. When I try using C-C, the program just exits after the "continue" command is given to lldb.
Here is the sample code:
#include<iostream>
#include<string>
using namespace std;
int main(){
string cmd;
while(true){
if (!getline(cin,cmd)) {
cout<<"ending on eof"<<endl;
break;}
else if (cmd=="GO INTO DEBUGGER"){
//??
}
else
cout<<"Got line: "<<cmd<<endl;
}
cout<<"Exiting program"<<endl;
return 0;
};
When this program is executed, it just echoes back the input line. When I interrupt the program using C-C, I bounce back into the debugger. When I then execute "continue" in the debugger, instead of returning to the loop, I just exit with the EOF message.
How can I either return to the loop from when the loop was interrupted, either using C-C or perhaps by using some kind of command in place of the "GO INTO DEBUGGER" clause (returning from "assert(0)" rarely works I find.
This is all compiled with clang++ on Mac Mavericks.
Note: for some reason the lldb backtrace says it received SIGSTOP, I thought C-C was SIGINT, but I guess I'm out of date.
This sort of problem comes because of the interaction between signals and system traps.
When a program that is sitting in a system trap waiting for input (or in any system trap really) gets a signal, the system may need to get the thread sitting on the trap out of the kernel in order to deliver the signal to it. If it has to do that, the trap call will return with an its usual error value, and the thread local "errno" variable will be set to EINTR.
This is happening to you in the debugger because the debugger has to send a signal (lldb uses SIGSTOP rather than SIGINT for uninteresting reasons) to interrupt your program.
However, this isn't specific to the debugger, this could happen because of job control signals or any other signal your program might receive. So to be safe, when you get an error return back from some read (read, select, getline, etc...) type call, you are supposed to check errno and only treat the error as an EOF if errno is not EINTR.
That being said, it seems like getline is buggy w.r.t. signals. If I interrupt the program while it is sitting in getline, I get a 0 return and errno is correctly set to 4. But the next time I call into getline, it again returns with 0, but this time errno has not been reset which makes it kind of hard to use in this context. Interesting...
Rather than using Control-C to stop your program, you should use a breakpoint in lldb. When you attach to your program, before starting execution, you can set a breakpoint by typing:
break foo.c:11
to break in the file foo.c on line 11. See the docs for more information.
Once the debugger stops at the breakpoint, you can examine variables and perform other actions, then type:
continue
to continue the execution of the program.
Related
Suppose we have small program, written in C++, which looks like below.
This program itself intentionally does NOT perform signal handling via WinAPI call SetConsoleCtrlHandler - it's important part of question.
#include <stdio.h>
#include <stdlib.h>
int main() {
while(true) {
int status = system("EXTERNAL COMMAND");
printf("RESULT STATUS = %d\n", status);
}
}
When Ctrl+C key combination have been pressed in terminal, program above have quite different behavior depend on which "EXTERNAL COMMAND" had been invoked
1) If external command is pause, program will stand in infinite loop, invoking pause command step by step, and will print "RESULT STATUS = 0" many times, while not terminated forced via process kill.
2) If external command in choice, program will terminate immediately after Ctrl+C pressure. It will not print anything and does not return from system call.
3) If external command is set /P VAR=, program has much interesting behavior. When Ctrl+C pressed, program prints `"RESULT STATUS = 1" and continue working until first async call is performed.
First and second case can be explained in following manner. Terminal windows is proxt between user input and target program, so when user presses Ctrl+C, terminal window perform dispatch signal itself to target process.
So some subprocesses can take manually terminal handler via hConsole = GetStdHandle(STD_OUTPUT_HANDLE) and perform own signal handling. Another subprocess does not do this, so signal passed into parent process and terminate it.
But third case causes big question. If child process intercepts SIGINT, why parent process perform terminating after first async call. If not, why it does not terminate immediately and why and how it prints `"RESULT STATUS = 1" and continue working.
Thanks
There are no Unix signals in Windows, at least not from the kernel. That said, Windows and the Windows API are fundamentally based on the C programming language, which, having been developed in tandem with Unix, does require six signals. The C runtime in Windows emulates SIGABRT and SIGTERM within process (e.g. for use with C raise). For SIGSEGV, SIGILL, and SIGFPE it uses an OS exception handler. In a console application, standardSIGINT and non-standard SIGBREAK are associated with the C runtime's console control handler, which is usually the first handler registered via SetConsoleCtrlHandler. CTRL_C_EVENT is mapped to the SIGINT signal handler, and all others (CTRL_BREAK_EVENT, CTRL_CLOSE_EVENT) are mapped to the SIGBREAK handler.
Console control events are sent by the console host (conhost.exe), which implements this by having the session server (csrss.exe) create a thread in a client process. This thread begins at the undocumented CtrlRoutine function in kernelbase.dll, which walks the registered control handlers until one of them handles the event by returning true. If none of them handles the event, the default handler calls ExitProcess(STATUS_CONTROL_C_EXIT). Note that SetConsoleCtrlHandler(NULL, TRUE) sets a flag that makes CtrlRoutine ignore CTRL_C_EVENT, and this flag is inherited by child processes and enabled by default when a process is created with the flag CREATE_NEW_PROCESS_GROUP. Also, for CTRL_CLOSE_EVENT, the session server gives each process 5 seconds to handle the event and exit on its own, else it terminates the process forcefully.
To understand what's happening with CMD's internal PAUSE command, see SetConsoleMode, and, in particular, ENABLE_PROCESSED_INPUT. PAUSE calls C _getch, which temporarily sets the console input mode to 0. With the processed-input mode disabled, Ctrl+C is simply read as "\x03" instead of generating a CTRL_C_EVENT.
If I run a program under gdb, how do I break at an arbitrary point? Pressing Ctrl+C kills the program. Ctrl+Break does nothing.
I can't enter gdb commands because my program is itself sitting in a REPL loop so anything I enter goes to my program, not to gdb.
My program uses linenoise to implement the REPL; I assume that this is hiding Ctrl+C, etc., from gdb.
Ctrl+\ results in a 001C square blob thingy in my program, rather than SIGUSR1.
Pressing Ctrl+C kills the program.
That is not the default GDB behavior.
Did you set handle SIGINT nostop pass?
You can examine current signal disposition with:
(gdb) handle SIGINT
Signal Stop Print Pass to program Description
SIGINT Yes Yes No Interrupt
Update:
My program is using linenoise for console input. I assume that it has done something to Ctrl+C
If your program is modifying terminal settings, you are going to have a very bad time debugging it from the same terminal.
For example, suppose the program sets no echo, and then hits a breakpoint. I think you would get a (gdb) prompt, but would not see any commands you are typing into GDB.
It seems that you would be much better off debugging this program from a different terminal. Use gdb -p $PID to attach to it from "outside".
I'm trying to debug a custom thread pool implementation that has rarely deadlocks. So I cannot use a debugger like gdb because I have click like 100 times "launch" debugger before having a deadlock.
Currently, I'm running the threadpool test in an infinite loop in a shell script, but that means I cannot see variables and so on. I'm trying to std::cout data, but that slow down the thread and reduce the risk of deadlocks meaning that I can wait like 1hour with my infinite before getting messages. Then I don't get the error, and I need more messages, which means waiting one more hour...
How to efficiently debug the program so that its restart over and over until it deadlocks ? (Or maybe should I open another question with all the code for some help ?)
Thank you in advance !
Bonus question : how to check everything goes fine with a std::condition_variable ? You cannot really tell which thread are asleep or if a race condition occurs on the wait condition.
There are 2 basic ways:
Automate the running of program under debugger. Using gdb program -ex 'run <args>' -ex 'quit' should run the program under debugger and then quit. If the program is still alive in one form or another (segfault, or you broke it manually) you will be asked for confirmation.
Attach the debugger after reproducing the deadlock. For example gdb can be run as gdb <program> <pid> to attach to running program - just wait for deadlock and attach then. This is especially useful when attached debugger causes timing to be changed and you can no longer repro the bug.
In this way you can just run it in loop and wait for result while you drink coffee. BTW - I find the second option easier.
If this is some kind of homework - restarting again and again with more debug will be a reasonable approach.
If somebody pays money for every hour you wait, they might prefer to invest in a software that supports replay-based debugging, that is, a software that records everything a program does, every instruction, and allows you to replay it again and again, debugging back and forth. Thus instead of adding more debug, you record a session during which a deadlock happens, and then start debugging just before the deadlock happened. You can step back and forth as often as you want, until you finally found the culprit.
The software mentioned in the link actually supports Linux and multithreading.
Mozilla rr open source replay based debugging
https://github.com/mozilla/rr
Hans mentioned replay based debugging, but there is a specific open source implementation that is worth mentioning: Mozilla rr.
First you do a record run, and then you can replay the exact same run as many times as you want, and observe it in GDB, and it preserves everything, including input / output and thread ordering.
The official website mentions:
rr's original motivation was to make debugging of intermittent failures easie
Furthermore, rr enables GDB reverse debugging commands such as reverse-next to go to the previous line, which makes it much easier to find the root cause of the problem.
Here is a minimal example of rr in action: How to go to the previous line in GDB?
You can run your test case under GDB in a loop using the command shown in https://stackoverflow.com/a/8657833/341065: gdb --eval-command=run --eval-command=quit --args ./a.out.
I have used this myself: (while gdb --eval-command=run --eval-command=quit --args ./thread_testU ; do echo . ; done).
Once it deadlocks and does not exit, you can just interrupt it by CTRL+C to enter into the debugger.
An easy quick debug to find deadlocks is to have some global variables that you modify where you want to debug, and then print it in a signal handler. You can use SIGINT (sent when you interrupt with ctrl+c) or SIGTERM (sent when you kill the program):
int dbg;
int multithreaded_function()
{
signal(SIGINT, dbg_sighandler);
...
dbg = someVar;
...
}
void dbg_sighandler(int)
{
std::cout << dbg1 << std::endl;
std::exit(EXIT_FAILURE);
}
Like that you just see the state of all your debug variables when you interrupt the program with ctrl+c.
In addition you can run it in a shell while loop:
$> while [ $? -eq 0 ]
do
./my_program
done
which will run your program forever until it fails ($? is the exit status of your program and you exit with EXIT_FAILURE in your signal handler).
It worked well for me, especially for finding out how many thread passed before and after what locks.
It is quite rustic, but you do not need any extra tool and it is fast to implement.
I am trying to catch floating point exception (SIGFPE) in GDB, not pass it to the process and continue debugging onwards.
I have given gdb this:
handle SIGFPE stop nopass
When a SIGFPE occurs GDB stops at the correct place. The problem is I can't and don't know how can I continue debugging.
I have tried giving GDB
continue
or
signal 0
but it still hangs on the offending line and refuses to continue.
Is there a way to continue debugging after receiving a signal?
I am using GDB 7.5.1, which I have compiled myself and I have also tried with GDB 7.4, which comes with my 12.04 Ubuntu distribution. Both have the same behaviour.
The problem is that when you continue a program after a synchronous signal, it reexecutes the same instruction that caused the signal, which means you'll just get the signal again. If you tell it to ignore the signal (either directly or via gdb) it will go into a tight loop reexecuting that instruction repeatedly.
If you want to actually continue the program somewhere after the instruction that causes the signal, you need to manually set the $pc register to the next (or some other) instruction before issuing the continue command.
I'm running on a Marvell Monahans PXA320 under Green Hills INTEGRITY 5.0.10. I'm using MULTI 4.2.3 for development. I'm using an RTSERV connection for debugging, I've been asked to take over a menu-driven program.
I've noticed that if I halt the program (to modify breakpoints) and then resume it, the task gets into an infinite loop displaying the menu in the debugger I/O tab. After each instance of the menu that gets printed, it says that I have made an illegal selection. So, some input is apparently being fed into the task as if I had typed it in (and this input obviously corresponds to an invalid menu selection). I do not see on the display what this phantom input is.
Is there anything I can do to prevent a halt / resume from screwing up the I/O?
Thanks,
Dave
My first guess is that getc() (or your equivalent) is returning -1. This can happen if your input buffers overflowed as a result of halting the application. I/O keeps flowing while the application is halted...
It is generally not a good idea to halt the program when debugging with INTEGRITY. You're generally better off to attach the debugger to a single thread (something idle or infrequently used), set an "any-task" breakpoint in that thread, then resume the thread. (Don't close the window! Doing so will delete the breakpoint.) You'll see a "DebugBrk" status on the thread that hits the breakpoint -- then you can double-click and attach to that specific thread.
Following that alternate procedure should (hopefully!) prevent the I/O error.