I am creating multiple threads in my program. On pressing Ctrl-C, a signal handler is called. Inside a signal handler, I have put exit(0) at last. The thing is that sometimes the program terminates safely but the other times, I get runtime error stating
abort() has been called
So what would be the possible solution to avoid the error?
The usual way is to set an atomic flag (like std::atomic<bool>) which is checked by all threads (including the main thread). If set, then the sub-threads exit, and the main thread starts to join the sub-threads. Then you can exit cleanly.
If you use std::thread for the threads, that's a possible reason for the crashes you have. You must join the thread before the std::thread object is destructed.
Others have mentioned having the signal-handler set a std::atomic<bool> and having all the other threads periodically check that value to know when to exit.
That approach works well as long as all of your other threads are periodically waking up anyway, at a reasonable frequency.
It's not entirely satisfactory if one or more of your threads is purely event-driven, however -- in an event-driven program, threads are only supposed to wake up when there is some work for them to do, which means that they might well be asleep for days or weeks at a time. If they are forced to wake up every (so many) milliseconds simply to poll an atomic-boolean-flag, that makes an otherwise extremely CPU-efficient program much less CPU-efficient, since now every thread is waking up at short regular intervals, 24/7/365. This can be particularly problematic if you are trying to conserve battery life, as it can prevent the CPU from going into power-saving mode.
An alternative approach that avoids polling would be this one:
On startup, have your main thread create an fd-pipe or socket-pair (by calling pipe() or socketpair())
Have your main thread (or possibly some other responsible thread) include the receiving-socket in its read-ready select() fd_set (or take a similar action for poll() or whatever wait-for-IO function that thread blocks in)
When the signal-handler is executed, have it write a byte (any byte, doesn't matter what) into the sending-socket.
That will cause the main thread's select() call to immediately return, with FD_ISSET(receivingSocket) indicating true because of the received byte
At that point, your main thread knows it is time for the process to exit, so it can start directing all of its child threads to start shutting down (via whatever mechanism is convenient; atomic booleans or pipes or something else)
After telling all the child threads to start shutting down, the main thread should then call join() on each child thread, so that it can be guaranteed that all of the child threads are actually gone before main() returns. (This is necessary because otherwise there is a risk of a race condition -- e.g. the post-main() cleanup code might occasionally free a resource while a still-executing child thread was still using it, leading to a crash)
The first thing you must accept is that threading is hard.
A "program using threading" is about as generic as a "program using memory", and your question is similar to "how do I not corrupt memory in a program using memory?"
The way you handle threading problem is to restrict how you use threads and the behavior of the threads.
If your threading system is a bunch of small operations composed into a data flow network, with an implicit guarantee that if an operation is too big it is broken down into smaller operations and/or does checkpoints with the system, then shutting down looks very different than if you have a thread that loads an external DLL that then runs it for somewhere from 1 second to 10 hours to infinite length.
Like most things in C++, solving your problem is going to be about ownership, control and (at a last resort) hacks.
Like data in C++, every thread should be owned. The owner of a thread should have significant control over that thread, and be able to tell it that the application is shutting down. The shut down mechanism should be robust and tested, and ideally connected to other mechanisms (like early-abort of speculative tasks).
The fact you are calling exit(0) is a bad sign. It implies your main thread of execution doesn't have a clean shutdown path. Start there; the interrupt handler should signal the main thread that shutdown should begin, and then your main thread should shut down gracefully. All stack frames should unwind, data should be cleaned up, etc.
Then the same kind of logic that permits that clean and fast shutdown should also be applied to your threaded off code.
Anyone telling you it is as simple as a condition variable/atomic boolean and polling is selling you a bill of goods. That will only work in simple cases if you are lucky, and determining if it works reliably is going to be quite hard.
Additional to Some programmer dude answer and related to discussion in the comment section, you need to make the flag that controls termination of your threads as atomic type.
Consider following case :
bool done = false;
void pending_thread()
{
while(!done)
{
std::this_thread::sleep(std::milliseconds(1));
}
// do something that depends on working thread results
}
void worker_thread()
{
//do something for pending thread
done = true;
}
Here worker thread can be your main thread also and done is terminating flag of your thread, but pending thread need to do something with given data by working thread, before exiting.
this example has race condition and undefined behaviour along with it, and it's really hard to find what is the actual problem int the real world.
Now the corrected version using std::automic :
std::atomic<bool> done(false);
void pending_thread()
{
while(!done.load())
{
std::this_thread::sleep(std::milliseconds(1));
}
// do something that depends on working thread results
}
void worker_thread()
{
//do something for pending thread
done = true;
}
You can exit thread without being concern of race condition or UB.
Related
I'm thinking about a certain kind of synchronisation primitive, but I don't know what this kind of synchronisation is called or if something like this would be working.
So there is one variable (boolean) which basically signals if one thread is still working on a block of memory or not. At the beginning the bool is set to false, meaning the worker thread is not working on your block of memory. Now the main thread gives the worker thread a "todo-list", describing how it should be working on that block of memory. After that, it changes the state of the boolean to true, so that the worker thread knows it is now allowed to do its work. The main thread can now continue its own work and checks at certain locations if the worker thread is now done working, e.g. if the boolean has been set to false again. If it is stil true, the main thread just continues its own work and doesn't wait for the worker thread. If the boolean is false, the main thread knows the worker thread is done and starts processing the block of memory.
So the boolean just transfers the ownership over a block of memory between two threads. If one thread currently does not have the ownership of that memory, it just continues with its own work, and checks repeatedly if it now has the ownership again. This way, none of the threads is waiting for one another and can continue its own work.
What is this called and how is such a behavior implemented?
EDIT: Basically it's a mutex. But instead of waiting for the mutex to be unlocked again, it continues/skips the critical code.
EDIT: Basically it's a mutex. But instead of waiting for the mutex to
be unlocked again, it continues/skips the critical code.
It's still a mutex, just with "try" methods.
in standard C++, we're talking about std::mutex::try_lock , which tries to lock the mutex, if it fails it returns false and moves on
class unlocker{
std::mutex& m_Parent;
public :
unlocker(std::mutex& parent) : m_Parent(parent){}
~unlocker() {m_Parent.unlock(); }
};
std::mutex mtx;
if (mtx.try_lock()){
unlocker unlock(mtx); // no, you can't use std::lock_guard/unique_lock here
//success, mtx is free
} else{
// do something else
}
on Native OS's code you have similar functions depending on the operating system you are on, like pthread_mutex_trylock on Unix and TryEnterCriticalSection on Windows. needless to say that standard mutex probably does use these functions behind the scenes
What will you do if the main thread runs out of work?
Suppose you keep checking and you keep reading true. Eventually you reach a point where the main thread cannot continue without the result from the worker thread. Since you have no more work to do, the only thing left is now keep checking the value of the flag over and over, wasting CPU resources that other threads could use to do useful work.
In general, this is not what you want. Instead, you would like the operating system to put your main thread to sleep and only wake it up once the worker thread has finished processing. All kinds of locks and semaphores that ship with modern operating systems work this way. Underneath there is some flag in memory that indicates who owns the lock, but there is also a bunch of logic around it that ensure the operating system won't schedule threads that have nothing to do but wait for a lock to become ready.
That being said, there are some situations where this is not what you want. If you are sufficiently sure that you won't run into the situation where one thread just spins on a lock, and you want to save the overhead that comes with the OS locks, just checking a flag like you described might be a viable option.
Note though that low-level stuff like this should be reserved for special circumstances, not be the first tool in your toolbox. It's just too easy to end up with an algorithm that is incorrect or an implementation that is not as efficient as you thought. If you decide to go down this road, be prepared to do some serious work to get it working as expected.
I am recently working with threads in C++11. now I am thinking about how to force stop a thread. I couldn't find it on stackoverflow, and also tried these.
One variable each thread : not so reliable
return in the main thread : I have to force quit only one not all
and I have no more ideas. I have heard about WinAPI, but I want a portable solution. (that also means I wont use fork())
Can you please give me a solution of this? I really want to do it.
One of the biggest problems with force closing a thread in C++ is the RAII violation.
When a function (and subsequently, a thread), gracefully finishes, everything it held is gracefully cleaned up by the destructors of the objects the functions/threads created.
Memory gets freed,
OS resources (handles, file descriptors etc.) are closed and returned to the OS
Locks are getting unlocked so other threads can use the shared resources they protect.
other important tasks are preformed (such as updating counters, logging, etc.).
If you brutally kill a thread (aka by TerminateThread on Windows, for example), non of these actually happen, and the program is left in a very dangerous state.
A (not-so) common pattern that can be used is to register a "cancellation token" on which you can monitor and gracefully shut the thread if other thread asks so (a la TPL/PPL). something like
auto cancellationToken = std::make_shared<std::atomic_bool>();
cancellationToken->store(false);
class ThreadTerminator : public std::exception{/*...*/};
std::thread thread([cancellationToken]{
try{
//... do things
if (cancellationToken->load()){
//somone asked the thred to close
throw ThreadTerminator ();
}
//do other things...
if (cancellationToken->load()){
//somone asked the thred to close
throw ThreadTerminator ();
}
//...
}catch(ThreadTerminator){
return;
}
});
Usually, one doesn't even open a new thread for a small task, it's better to think of a multi threaded application as a collection of concurrent tasks and parallel algorithms. one opens a new thread for some long ongoing background task which is usually performed in some sort of a loop (such as, accepting incoming connections).
So, anyway, the cases for asking a small task to be cancelled are rare anyway.
tldr:
Is there a reliable way to force a thread to stop in C++?
No.
Here is my approach for most of my designs:
Think of 2 kinds of Threads:
1) primary - I call main.
2) subsequent - any thread launched by main or any subsequent thread
When I launch std::thread's in C++ (or posix threads in C++):
a) I provide all subsequent threads access to a boolean "done", initialized to false. This bool can be directly passed from main (or indirectly through other mechanisms).
b) All my threads have a regular 'heartbeat', typically with a posix semaphore or std::mutex, sometimes with just a timer, and sometimes simply during normal thread operation.
Note that a 'heartbeat' is not polling.
Also note that checking a boolean is really cheap.
Thus, whenever main wants to shut down, it merely sets done to true and 'join's with the subsequent threads.
On occasion main will also signal any semaphore (prior to join) that a subsequent thread might be waiting on.
And sometimes, a subsequent thread has to let its own subsequent thread know it is time to end.
Here is an example -
main launching a subsequent thread:
std::thread* thrd =
new std::thread(&MyClass_t::threadStart, this, id);
assert(nullptr != thrd);
Note that I pass the this pointer to this launch ... within this class instance is a boolean m_done.
Main Commanding shutdown:
In main thread, of course, all I do is
m_done = true;
In a subsequent thread (and in this design, all are using the same critical section):
void threadStart(uint id) {
std::cout << id << " " << std::flush; // thread announce
do {
doOnce(id); // the critical section is in this method
}while(!m_done); // exit when done
}
And finally, at an outer scope, main invokes the join.
Perhaps the take away is - when designing a threaded system, you should also design the system shut down, not just add it on.
There are a lot of Qt multi-threading tutorials out there that state that a QThread can be stopped safely using the following two lines.
qthread.quit(); // Cause the thread to cease.
qthread.wait(); // Wait until the thread actually stops to synchronize.
I have a lot of code doing this, and in most cases of stopping thread, I'll always set my own cancel flag and check it often during execution (as is the norm). Until now, I was thinking that calling quit would perhaps cause the thread to simply no longer execute any waiting signals (e.g. signals that are queued will no longer have their slots called) but still wait on the currently executing slot to finish.
But I'm wondering if I was right or if quit() actually stops the execution of the thread where it's at, for instance if something is unfinished, like a file descriptor hasn't been closed, it definitely should be, even though in most cases my worker objects will clean up those resources, I'd feel better if I knew exactly how quit works.
I'm asking this because QThread::quit() documentation says that it's "equivalent to calling QThread::exit(0)". I believe this means that the thread would immediately stop where it's at. But what would happen to the stackframe that quit was called in?
QThread::quit does nothing if the thread does not have an event loop or some code in the thread is blocking the event loop. So it will not necessarily stop the thread.
So QThread::quit tells the thread's event loop to exit. After calling it the thread will get finished as soon as the control returns to the event loop of the thread.
You will have to add some kind of abort flag if you are blocking event loop for example by working in a loop. This can be done by a boolean member variable that is public or at least has a public setter method. Then you can tell the thread to exit ASAP from outside (e.g. from your main thread) by setting the abort flag. Of course this will require your thread code to check the abort flag at regular intervals.
You may also force a thread to terminate right now via QThread::terminate(), but this is a very bad practice, because it may terminate the thread at an undefined position in its code, which means you may end up with resources never getting freed up and other nasty stuff. So use this only if you really can't get around it. From its documentation:
Warning: This function is dangerous and its use is discouraged. The thread can be terminated at any point in its code path. Threads can be terminated while modifying data. There is no chance for the thread to clean up after itself, unlock any held mutexes, etc. In short, use this function only if absolutely necessary.
I think this is a good way to finish a thread when you are using loops in a thread:
myThread->m_abort = true; //Tell the thread to abort
if(!myThread->wait(5000)) //Wait until it actually has terminated (max. 5 sec)
{
myThread->terminate(); //Thread didn't exit in time, probably deadlocked, terminate it!
myThread->wait(); //We have to wait again here!
}
In case, if you want to use Qt's builtin facility then try QThread::requestInterruption().
Main thread
struct X {
QThread m_Thread;
void Quit ()
{
m_Thread.quit();
m_Thread.requestInterruption();
}
};
Some Thread referred by X::m_Thread
while(<condition>) {
if(QThread::currentThread()->isInterruptionRequested())
return;
...
}
As per the documentation:
void QThread::requestInterruption()
Request the interruption of the thread. That request is advisory and it is up to code running on the thread to decide if and how it should act upon such request. This function does not stop any event loop running on the thread and does not terminate it in any way.
I'm working on a C/C++ networking project and am having difficulties synchronizing/signaling my threads. Here is what I am trying to accomplish:
Poll a bunch of sockets using the poll function
If any sockets are ready from the POLLIN event then send a signal to a reader thread and a writer thread to "wake up"
I have a class called MessageHandler that sets the signals mask and spawns the reader and writer threads. Inside them I then wait on the signal(s) that ought to wake them up.
The problem is that I am testing all this functionality by sending a signal to a thread yet it never wakes up.
Here is the problem code with further explanation. Note I just have highlighted how it works with the reader thread as the writer thread is essentially the same.
// Called once if allowedSignalsMask == 0 in constructor
// STATIC
void MessageHandler::setAllowedSignalsMask() {
allowedSignalsMask = (sigset_t*)std::malloc(sizeof(sigset_t));
sigemptyset(allowedSignalsMask);
sigaddset(allowedSignalsMask, SIGCONT);
}
// STATIC
sigset_t *MessageHandler::allowedSignalsMask = 0;
// STATIC
void* MessageHandler::run(void *arg) {
// Apply the signals mask to any new threads created after this point
pthread_sigmask(SIG_BLOCK, allowedSignalsMask, 0);
MessageHandler *mh = (MessageHandler*)arg;
pthread_create(&(mh->readerThread), 0, &runReaderThread, arg);
sleep(1); // Just sleep for testing purposes let reader thread execute first
pthread_kill(mh->readerThread, SIGCONT);
sleep(1); // Just sleep for testing to let reader thread print without the process terminating
return 0;
}
// STATIC
void* MessageHandler::runReaderThread(void *arg) {
int signo;
for (;;) {
sigwait(allowedSignalsMask, &signo);
fprintf(stdout, "Reader thread signaled\n");
}
return 0;
}
I took out all the error handling I had in the code to condense it but do know for a fact that the thread starts properly and gets to the sigwait call.
The error may be obvious (its not a syntax error - the above code is condensed from compilable code and I might of screwed it up while editing it) but I just can't seem to find/see it since I have spent far to much time on this problem and confused myself.
Let me explain what I think I am doing and if it makes sense.
Upon creating an object of type MessageHandler it will set allowedSignalsMask to the set of the one signal (for the time being) that I am interested in using to wake up my threads.
I add the signal to the blocked signals of the current thread with pthread_sigmask. All further threads created after this point ought to have the same signal mask now.
I then create the reader thread with pthread_create where arg is a pointer to an object of type MessageHandler.
I call sleep as a cheap way to ensure that my readerThread executes all the way to sigwait()
I send the signal SIGCONT to the readerThread as I am interested in sigwait to wake up/unblock once receiving it.
Again I call sleep as a cheap way to ensure that my readerThread can execute all the way after it woke up/unblocked from sigwait()
Other helpful notes that may be useful but I don't think affect the problem:
MessageHandler is constructed and then a different thread is created given the function pointer that points to run. This thread will be responsible for creating the reader and writer threads, polling the sockets with the poll function, and then possibly sending signals to both the reader and writer threads.
I know its a long post but do appreciate you reading it and any help you can offer. If I wasn't clear enough or you feel like I didn't provide enough information please let me know and I will correct the post.
Thanks again.
POSIX threads have condition variables for a reason; use them. You're not supposed to need signal hackery to accomplish basic synchronization tasks when programming with threads.
Here is a good pthread tutorial with information on using condition variables:
https://computing.llnl.gov/tutorials/pthreads/
Or, if you're more comfortable with semaphores, you could use POSIX semaphores (sem_init, sem_post, and sem_wait) instead. But once you figure out why the condition variable and mutex pairing makes sense, I think you'll find condition variables are a much more convenient primitive.
Also, note that your current approach incurs several syscalls (user-space/kernel-space transitions) per synchronization. With a good pthreads implementation, using condition variables should drop that to at most one syscall, and possibly none at all if your threads keep up with each other well enough that the waited-for event occurs while they're still spinning in user-space.
This pattern seems a bit odd, and most likely error prone. The pthread library is rich in synchronization methods, the one most likely to serve your need being in the pthread_cond_* family. These methods handle condition variables, which implement the Wait and Signal approach.
Use SIGUSR1 instead of SIGCONT. SIGCONT doesn't work. Maybe a signal expert knows why.
By the way, we use this pattern because condition variables and mutexes are too slow for our particular application. We need to sleep and wake individual threads very rapidly.
R. points out there is extra overhead due to additional kernel space calls. Perhaps if you sleep > N threads, then a single condition variable would beat out multiple sigwaits and pthread_kills. In our application, we only want to wake one thread when work arrives. You have to have a condition variable and mutex for each thread to do this otherwise you get the stampede. In a test where we slept and woke N threads M times, signals beat mutexes and condition variables by a factor of 5 (it could have been a factor of 40 but I cant remember anymore....argh). We didn't test Futexes which can wake 1 thread at a time and specifically are coded to limit trips to kernel space. I suspect futexes would be faster than mutexes.
How can I wait for a detached thread to finish in C++?
I don't care about an exit status, I just want to know whether or not the thread has finished.
I'm trying to provide a synchronous wrapper around an asynchronous thirdarty tool. The problem is a weird race condition crash involving a callback. The progression is:
I call the thirdparty, and register a callback
when the thirdparty finishes, it notifies me using the callback -- in a detached thread I have no real control over.
I want the thread from (1) to wait until (2) is called.
I want to wrap this in a mechanism that provides a blocking call. So far, I have:
class Wait {
public:
void callback() {
pthread_mutex_lock(&m_mutex);
m_done = true;
pthread_cond_broadcast(&m_cond);
pthread_mutex_unlock(&m_mutex);
}
void wait() {
pthread_mutex_lock(&m_mutex);
while (!m_done) {
pthread_cond_wait(&m_cond, &m_mutex);
}
pthread_mutex_unlock(&m_mutex);
}
private:
pthread_mutex_t m_mutex;
pthread_cond_t m_cond;
bool m_done;
};
// elsewhere...
Wait waiter;
thirdparty_utility(&waiter);
waiter.wait();
As far as I can tell, this should work, and it usually does, but sometimes it crashes. As far as I can determine from the corefile, my guess as to the problem is this:
When the callback broadcasts the end of m_done, the wait thread wakes up
The wait thread is now done here, and Wait is destroyed. All of Wait's members are destroyed, including the mutex and cond.
The callback thread tries to continue from the broadcast point, but is now using memory that's been released, which results in memory corruption.
When the callback thread tries to return (above the level of my poor callback method), the program crashes (usually with a SIGSEGV, but I've seen SIGILL a couple of times).
I've tried a lot of different mechanisms to try to fix this, but none of them solve the problem. I still see occasional crashes.
EDIT: More details:
This is part of a massively multithreaded application, so creating a static Wait isn't practical.
I ran a test, creating Wait on the heap, and deliberately leaking the memory (i.e. the Wait objects are never deallocated), and that resulted in no crashes. So I'm sure it's a problem of Wait being deallocated too soon.
I've also tried a test with a sleep(5) after the unlock in wait, and that also produced no crashes. I hate to rely on a kludge like that though.
EDIT: ThirdParty details:
I didn't think this was relevant at first, but the more I think about it, the more I think it's the real problem:
The thirdparty stuff I mentioned, and why I have no control over the thread: this is using CORBA.
So, it's possible that CORBA is holding onto a reference to my object longer than intended.
Yes, I believe that what you're describing is happening (race condition on deallocate). One quick way to fix this is to create a static instance of Wait, one that won't get destroyed. This will work as long as you don't need to have more than one waiter at the same time.
You will also permanently use that memory, it will not deallocate. But it doesn't look like that's too bad.
The main issue is that it's hard to coordinate lifetimes of your thread communication constructs between threads: you will always need at least one leftover communication construct to communicate when it is safe to destroy (at least in languages without garbage collection, like C++).
EDIT:
See comments for some ideas about refcounting with a global mutex.
To the best of my knowledge there's no portable way to directly ask a thread if its done running (i.e. no pthread_ function). What you are doing is the right way to do it, at least as far as having a condition that you signal. If you are seeing crashes that you are sure are due to the Wait object is being deallocated when the thread that creates it quits (and not some other subtle locking issue -- all too common), the issue is that you need to make sure the Wait isn't being deallocated, by managing from a thread other than the one that does the notification. Put it in global memory or dynamically allocate it and share it with that thread. Most simply don't have the thread being waited on own the memory for the Wait, have the thread doing the waiting own it.
Are you initializing and destroying the mutex and condition var properly?
Wait::Wait()
{
pthread_mutex_init(&m_mutex, NULL);
pthread_cond_init(&m_cond, NULL);
m_done = false;
}
Wait::~Wait()
{
assert(m_done);
pthread_mutex_destroy(&m_mutex);
pthread_cond_destroy(&m_cond);
}
Make sure that you aren't prematurely destroying the Wait object -- if it gets destroyed in one thread while the other thread still needs it, you'll get a race condition that will likely result in a segfault. I'd recommend making it a global static variable that gets constructed on program initialization (before main()) and gets destroyed on program exit.
If your assumption is correct then third party module appears to be buggy and you need to come up with some kind of hack to make your application work.
Static Wait is not feasible. How about Wait pool (it even may grow on demand)? Is you application using thread pool to run?
Although there will still be a chance that same Wait will be reused while third party module is still using it. But you can minimize such chance by properly queing vacant Waits in your pool.
Disclaimer: I am in no way an expert in thread safety, so consider this post as a suggestion from a layman.