I encounter multiple times where Linux Out of Memory Killer watchdog on Linux is killing my application, this is hard to debug and identify...
Is there anyway in a c/c++ application running under Linux to print a message before the application is killed?
If I'm right the OOM will send your process a SIGTERM signal, so you can handle it as you want.
I was not right, most probably OOM will send you SIGKILL and you can not do anything. But under certain circumstances you will get a SIGTERM before.
(non tested draft)
#include <csignal>
void signal_handler(int signal) {
// Your handling code here
}
int main() {
// Install handler (assign handler to signal)
std::signal(SIGINT, signal_handler);
}
C counterpart:
#include<signal.h>
#include<unistd.h>
void signal_handler(int signo)
{
if (signo == SIGTERM) {
// your handling code
}
}
int main(void)
{
if (signal(SIGTERM, signal_handler) == SIG_ERR) {
printf("\nError installing handler\n");
}
// Rest of your application
}
Be careful when handling signals, as you are overriding the default behavior. Your program should not ignore important signals like SIGTERM or SIGINT: the handling function has to do the work of finishing the program or maybe calling the original handler.
On the other hand, you can play with it: if you are sure the problem is the allocated memory you could try to free unused space and try to continue the work (but you need to ensure the signal reason was that).
Related
I have this simple code that loops the word "SIGNALS ARE COOL" I'm trying to make it take signals like (SIGFPE. SIGABRT, SIGINT, SIGSEGV.) and show the signal type and the time I made this code that takes "SIGINT" signal how do I add more signals and how to control what my program show when the signals are triggered by the user.
// ConsoleApplication3.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
#include <iostream>
#include <csignal>
using namespace std;
void signalHandler(int signum) {
cout << "Interrupt signal (" << signum << ") received.\n";
// cleanup and close up stuff here
// terminate program
exit(signum);
}
int main() {
// register signal SIGINT and signal handler
signal(SIGINT, signalHandler);
while (1) {
cout << "SIGNALS ARE COOL" << endl;
}
return 0;
}
I see that this looks like an assignment; so what I'm saying may not be relevant to you (but might be to someone someday).
--EDIT--
I see you've also got stdafx.h, which I think is a Visual Studio Windows thing, and here I am suggesting a POSIX solution (not pure C++). I didn't read carefully enough, and that invalidates my whole answer (I think). You probably can't use my suggestion, and for that I'm sorry.
However, I'm going to leave it here in case someone one day finds this and needs to work with signals in a Unix system.
--
I've found that it's often a lot more practical to avoid signal handling functions like this altogether, and take signals on your own terms. As noted by others, there's a lot of rules about what you can and can't do within a signal handler, because they can be invoked at any time, in any thread, unless you take extra precautions. I've seen this result in a lot of messy code, things like 'have a global bool got_signal that gets checked by things all over the application to know if they're supposed to shut down'. There's obviously nice ways to do signal handling, but at this point I try to avoid it altogether in favor of other options.
The functions pthread_sigmask and sigwait can be used to invert control here and allow you to accept signals within the defined flow of program execution where you want it, and then you don't need to worry about taking invalid actions when you handle them. Using pthread_sigmask you can tell the OS not to interrupt your program to deliver signals and instead queue them up, and then sigwait can be used to handle them at an appropriate time. You can't do this with all signals (some things like kill -9 and a SEGFAULT can't/shouldn't be ignored), but it works well for most of them.
Using an approach like this, it's really easy to interact with signals in a larger application too. You can block signals at the start of main, and that will propagate to all children threads, and then you can designate one specific child thread to just wait for signals an pass events into the rest of the application in whatever method is appropriate for the framework of your application.
#include <signal.h>
#include <unistd.h>
#include <initializer_list>
#include <functional>
#include <algorithm>
#include <iostream>
sigset_t make_sigset(std::initializer_list<int32_t> signals)
{
sigset_t set;
const int32_t result = sigemptyset(&set);
std::for_each(signals.begin(), signals.end(), std::bind(&sigaddset, &set, std::placeholders::_1));
return set;
}
int main()
{
const auto signal_list = make_sigset({SIGTERM, SIGSEGV, SIGINT, SIGABRT});
pthread_sigmask(SIG_BLOCK, &signal_list, nullptr);
int32_t last_signal;
do
{
sigwait(&signal_list, &last_signal);
std::cout << "Got signal " << last_signal << std::endl;
// Exit on sigint so ctrl+c still works
} while (last_signal != SIGINT);
return 0;
}
As already mentioned by #Eljay in the comments, you have to be careful with the things you do in a signal handler.
I'd also suggest not using namespace std, but that's a story for another time link.
I'd recommend you this page which explains a lot about what signals can and cannot do, according to the c++ standard. Now what they actually do in your compiler (which I assume is MSVC) may be different.
Some of the important bits, as already mentioned, you shouldn't do I/O, you shouldn't throw, etc...
To answer your question, you were on the right track, adding other signals can be done via:
// catch SIGTERM
std::signal(SIGTERM, signalHandler);
std::signal(SIGSEGV, signalHandler);
std::signal(SIGINT, signalHandler);
std::signal(SIGABRT, signalHandler);
// insert others
Then, what I'd suggest is storing the value of your signal into some atomic variable, like: gSignalThatStoppedMe.
std::atomic<int> gSignalThatStoppedMe = -1;
// I also added 'extern "C"' because the standard says so
extern "C" void signalHandler(int signum) {
gSignalThatStoppedMe.store(signum);
}
Then, your while loop would check for != -1, or pick another value for this, I've not checked if some implementations use -1 as a valid value for signals
// ...
while(gSignalThatStoppedMe.load() == -1)
{
// your old code
}
Now, do a switch of sorts, with the values inside and output the signal that stopped it, something like:
switch(gSignalThatStoppedMe.load())
{
case SIGINT:
std::puts("It was SIGINT");
break;
case SIGTERM:
std::puts("It was SIGTERM");
break;
default:
break;
}
I think this has less undefined behavior, which is always a good thing.
EDIT: here's a compiler explorer link
The output with CTRL-C:
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
SIGNALS ARE COOL
Interrupt signal SIGINT (2) received.
What I'm trying to do
I have various things that must run concurrently on a linux, until the program is told to stop through ctrl-C (in which case SIGINT is received) or the service is stopped (in which case SIGTERM is received)
What I've come up with
For each thing that need to be done concurrently, I have a class that launches a thread in the constructor and whose destructor makes the thread stop and joins it. It looks basically like this:
#include <chrono>
#include <condition_variable>
#include <mutex>
#include <thread>
#include <system_error>
class SomeClassToDoStuff
{
public:
// Constructor
SomeClassToDoStuff()
{
_thread = std::thread([this]() {
while (true)
{
// Do some stuff
...
// Before going on to the next iteration
{
std::unique_lock<std::mutex> dataLock(_mutex);
// Wait for 2ms
if (!_shouldStopThreadExecution)
{
_conditionVariable.wait_for(dataLock, std::chrono::milliseconds(2));
}
// End the loop if requested
if (_shouldStopThreadExecution)
{
break;
}
}
}
// Some cleanup
...
});
}
// Destructor
~SomeClassToDoStuff()
{
if (_thread.joinable())
{
{
std::lock_guard<std::mutex> dataLock(_mutex);
_shouldStopThreadExecution = true;
}
_conditionVariable.notify_all();
try
{
_thread.join();
} catch (std::system_error const&) {}
}
}
private:
mutable std::mutex _mutex; // Mutex to keep things thread-safe
std::condition_variable _conditionVariable; // Condition variable used to wait
std::thread _thread; // Thread in which to do the stuff
bool _shouldStopThreadExecution = false; // Whether to stop the thread's execution
};
Then my main() looks like this:
#include <atomic>
#include <chrono>
#include <csignal>
#include <iostream>
#include <thread>
namespace {
std::atomic<int> programReturnValue(-1); // If positive or zero, the program must return with that value
}
static void signalHandler(int sig)
{
std::cout << "Signal received (" << sig << "). This iteration of the main loop will be the last." << std::endl;
programReturnValue.store(0);
}
int main()
{
// Make sure the program stops cleanly when receiving SIGTERM or SIGINT
{
std::signal(SIGTERM, signalHandler);
std::signal(SIGINT, signalHandler);
}
SomeClassToDoStuffA a;
SomeClassToDoStuffB b;
SomeClassToDoStuffC c;
SomeClassToDoStuffD d;
while (programReturnValue.load() < 0)
{
// Check that everything is alright
if (someCondition)
{
programReturnValue.store(1);
}
// Wait for 100us to avoid taking all of the processor's resources
std::this_thread::sleep_for(std::chrono::microseconds(100));
}
return programReturnValue.load();
}
(By the way, if there's an easier way to go about all this I'm interested to know)
The issue
When I hit ctrl+C or end the service, the program prints that signal 2 or 15 has been received (depending on which I used), and the program ends, which is good.
However:
The cleanup involves writing something to a file (in which things are successfully written during execution), but it seems that that doesn't always happen, which means that the cleanup isn't fully performed all the time, and that is a problem
The return code of the program isn't 0 as expected, or even 1, but either 130 or 143 depending on what signal is received
Why is that, and what am I doing wrong?
Edit: From what I understand, 130 and 143 are actually 128 + signal, i.e. what the program would return if I didn't try to handle the signals
Edit2: I'm getting a better idea of what's happening, and only half the issue seems to be coming from my program itself.
The program is actually run by a bash script, which then prints its return value and may relaunch it depending on the situation. Sending SIGINT and SIGTERM to the script is also supposed to send SIGTERM to the program.
It turns out that I suck at bash. I had written something like this:
#!/bin/sh
trap "killall myProgram --quiet --wait" 2 15
/path/to/myProgram&
wait $!
RETURN_VALUE=$?
echo "Code exited with return code ${RETURN_VALUE}"
# Some other stuff
...
ctrl-C while running the script in terminal actually leads to the program receiving both SIGINT then SIGTERM
the return code I'm printing is actually the result of wait+trap, not my program's
I will rework the script, but can the fact that both signals are sent to my program the reason why the cleanup fails sometimes? How? What can I do about it?
I am a bit confused about your signal handling:
To me it seems you use the terminating System-signal only to set the return-value and break the while loop in main; the threads, or rather their wrappers are terminated i.e. destructed only at the time them going out of scope, which is at the end of your main-scope, when you already have returned! Thrown exceptions (in your destructors) cannot be caught anymore.
your threads are therefor not ended yet, while you have already returned from main.
As a solution: I would recommend to set the stopping state _shouldStopThreadExecution at the time the main receives the signal for stopping already. And then remove the try statements for the .join() in your destructor in order to see the correct ending of the threads under quaranty.
Suppose I have loop which awaits for user input. If user presses Ctrl+C the program exits normally. However, I'd like to do a couple of things before exit. Is it possible to run a function once Ctrl+C was pressed and program is about to exit?
You could use core.stdc.signal, which contains bindings to the C header signal.h. Now, if this is for Windows, you might run into some problems:
SIGINT is not supported for any Win32 application. When a
CTRL+Cinterrupt occurs, Win32 operating systems generate a new thread to
specifically handle that interrupt. This can cause a single-thread
application, such as one in UNIX, to become multithreaded and cause
unexpected behavior.
__gshared bool running = true;
extern(C) void handleInterrupt(int) nothrow #nogc
{
running = false;
}
void main()
{
import core.stdc.signal;
signal(SIGINT, &handleInterrupt);
scope(exit)
{
//Cleanup
import std.stdio : writeln;
writeln("Done");
}
while(running)
{
//Do some work
}
}
You could probably get away with something like this:
void main() {
try {
dostuff();
} finally {
printf("bye\n");
};
};
The finally block will run even if the dostuff() throws an Error or Throwable. I'm not very familiar with standard-IO, console signals, or whatever, but give this a try at least.
Furthermore, even if dostuff() calls Runtime.terminate() the finally block will still run. The same cannot be said for abort() though.
Based on man -S 7 signal, when a program which has not defined a signal handler receives SIGTERM, the default action is to Term.
I am registering a custom signal handler for SIGTERM, but I want the final statement (after specific cleanup is done) in my custom handler to be a function call that achieves the exact same effect as Term.
What is the correct function to call here?
I have tried exit(0), but that causes the destructors of statically initialized objects to be called, which does not match the behavior of Term. I have also tried abort(), and that causes a core dump.
In the following example, I am using the lifetime of statically allocated objects as an illustrative example, but my question is more broad than just the lifetime of statically allocated objects.
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
struct Foo{
~Foo() {
fprintf(stderr, "Dying!\n");
}
Foo() {
fprintf(stderr, "Constructing!\n");
}
};
Foo foo;
void sig_handler(int signo) {
// exit(0);
// abort();
}
int main(){
// signal(SIGTERM, sig_handler);
while(1);
}
Here are various behaviors:
The code as-is will print Constructing! but not Dying! if SIGTERM is received.
When the //signal line is commented back in, and exit(0) is commented back in, the code will print both Constructing! and Dying!
When exit(0) is commented out, and abort() is commented in, the program will output Constructing! followed by Aborted (core dumped).
To ask my question in another way, I want to know what I need to put in the body of signal_handler in order to mimic behavior 1 in every way (not just the output I have shown).
signal(SIGTERM, SIG_DFL);
kill(getpid(), SIGTERM);
There is no function call, as termination upon signals is handled by the kernel, not the user process, but this will restore the default handler, then send a signal to yourself, which kills the process just like the signal had never been caught.
You can call _exit instead of exit , which should exit the process without running global destructors.
So here is my code:
void sigHandle(int sig)
{
signal(SIGINT, sigHandle); //Is this line necessairy?
cout<<"Signal: "<<sig<<endl;
}
int main(){
signal(SIGINT, sigHandle);
while(true){ //Supposed to loop until user exits.
//rest of my code
}
}
Now it is my understanding of signal() that when the SIGINT command (Ctrl+C right?) is received my function sigHandle should be called with an integer value of 2 (the SIGINT number), the method should run and the program should NOT exit.
All I would like to do is just print the signal number and move on, however after printing out "Signal: 2" it exits.
(Eventually I'm supposed to handle the first 32 interrupts but I figured Ctrl+C would be the most difficult so I'm starting here.)
In main if I do signal(SIGINT, SIG_IGN); it ignores the signal correctly and doesn't exit but I now have no way of knowing if I recieved the SIGINT interrupt.
Earlier I was playing around with the sigaction struct but I could not find any real comprehensive documentation on it so I decided to go with just "raw" signal handling.
This was my sigaction code (same problem as above):
struct sigaction action;
action.sa_handler = sigHandle;
sigemptyset(&action.sa_mask);
action.sa_flags = 0;
sigaction(SIGINT, &action, 0);
Thanks for your help!
EDIT
OK SO After many many many hours of scowering through man pages and the internet I have happened across a (very) ghetto solution involving saving the stack pre-infinite loop then when the interrupt comes, doing what I need to do, then re-setting the stack back to where it was and calling the sigrelse() command to re-set any states that might have been changed and not re-loaded.
I understand that this is not the most elegant/efficient/or even socially acceptable solution to this problem but it works and as far as I can tell I am not leaking any memory anywhere so it's all good...
I am still looking for a solution to this problem and I view my stack re-setting shenanigins as only a temporary fix...
Thanks!
Also note you should not call stdio (or other non-reentrant functions) in signal handlers.
(your signal handler might be invoked in the middle of a malloc or it's C++ equivalent)
It is not. You just replacing SIGINT's handles with same function. How does you program perform wait?
If you have something like:
int main
{
// ...
int r = read(fd, &buff, read_size); // your program hangs here, waiting for the data.
// but if signal occurred during this period of time
// read will return immediately, and r may != read_size
return 0; // then it will go straight to return.
}