In C++ using pthreads, what happens to your other threads if one of your threads calls fork?
It appears that the threads do not follow. In my case, I am trying to create a daemon and I use fork() with the parent exiting to deamonize it. However, in a new path through the code, I create some threads before the fork and some after. Is there an easy way to change ownership of the threads over to the new forked process rather than moving all my thread creation after the fork?
Nothing. Only the thread calling fork() gets duplicate. The child process has to start any new threads. The parents threads are left alone.
In POSIX when a multithreaded process forks, the child process looks exactly like a copy of the parent, but in which all the threads stopped dead in their tracks and disappeared.
This is very bad if the threads are holding locks.
For this reason, there is a crude mechanism called pthread_atfork in which you can register handlers for this situation.
Any properly written program module (and especially reusable middleware) which uses mutexes must call pthread_atfork to register some handlers, so that it does not misbehave if the process happens to call fork.
Besides mutex locks, threads could have other resources, such as thread-specific data squirreled away with pthread_setspecific which is only accessible to the thread (and the thread is responsible for cleaning it up via a destructor).
In the child process, no such destructor runs. The address space is copied, but the thread and its thread specific value is not there, so the memory is leaked in the child. This can and should be handled with pthread_atfork handlers also.
Quoting from http://thorstenball.com/blog/2014/10/13/why-threads-cant-fork/
If we call fork(2) in a multi-threaded environment the thread doing the call is now the main-thread in the new process and all the other threads, which ran in the parent process, are dead. And everything they did was left exactly as it was just before the call to fork(2).
So we should think twice before using them
It's usually very bad to fork a thread. The forked process is supposed to be a complete copy of the parent, except with threads it isn't. There is a function pthread_atfork() which helps sometimes. If you must fork a thread, it is best if you call exec() immediately after the fork().
I suggest you read the caveats from the POSIX developers in the documentation of fork() and pthread_atfork() (See http://pubs.opengroup.org/onlinepubs/007904975/functions/fork.html and http://pubs.opengroup.org/onlinepubs/007904975/functions/pthread_atfork.html ).
From the fork() documentation:
The fork() function is thus used only to run new programs, and the effects of calling functions that require certain resources between the call to fork() and the call to an exec function are undefined.
Nothing, unless one happens to be preempted by the thread running the new process.
Related
Using only standard C++ (no platform specific API), I would like to launch an external application that may complete immediately or timeout. If the application halts, my app has a timeout after which it simply terminates the app and relaunches it.
Now, down to the nitty gritty, I tried launching two threads:
first thread launches the app and waits for it to terminate
second thread waits for a few seconds and checks if the first thread terminated. If it did not, then it considers it as stalled.
Question is, how do I terminate the first thread from the second? The way I'm launching the app is using the system() function. It's synchronous so there isn't any way for me to check from that thread if I wish for termination. It has to be forced somehow through an exception, externally.
How is this done properly?7
P.S.: if this is not possible, and I suspect it isn't, then I simply do not wish to wait for that thread anymore. It can simply remain stalled in the background. How do I achieve that? (currently, I'm waiting for that thread with a join())
You cannot forcefully terminate another thread. You can only politely ask it to exit. This holds in C++ and POSIX thread models. Windows has TerminateThread, but it's so dangerous it's practically unusable. POSIX has pthread_cancel. That's cooperative termination which could fit your bill, but there's no standard C++ equivalent.
Even if you terminate a thread somehow, it does nothing to any program it might have launched via system.
To let a thread go free with no obligation to join, use thread::detach().
To answer your question about killing a thread, POSIX offers two functions:
pthread_cancel();
This will stop the thread at a cancellation point.
The other is:
pthread_kill();
This function will send a signal to the thread. Contrary to the cancellation pointer concept, this one can happen at any point in the thread. In other words, if the thread has a mutex locked at that time, you're going to lose that lock... (unless you cleanly handle the signal in that thread).
However, what you are describing is a system() call which you make in a separate thread so that way you are not blocked. I don't think that either of these functions are going to help you because there is another process running, not just a simple thread. What you need is to stop that other process.
In your case, what you need to do is find out the pid of the child (or children) and send a signal to that child process (or all children and grandchildren, etc). In that case, you use the kill() function like so:
kill(child_pid, SIGINT);
Once the child died and cleaned up, the system() call will return and your thread is ready to be joined. So in order, you do:
...
child_pid = find_child_pid(); // Note: there is no such function, you have to write it or use a library which offers such
kill(child_pid, SIGNINT);
pthread_join(thread_id);
If that child process can create children and you want them out of the picture too (like in a shell when you hit Ctrl-C) then you need to find about all the children of your child, and their children, etc. You do so by looking at the PPID (Parent PID) of each process to see if it matches one of the children. That info is available in the /proc/<pid>/stat or /proc/<pid>/status. (the first file is probably best because it's just one line, however, it is tricky to go past the process name since it can include parenthesis... so you have to make sure to search the ')' from the end of the line (otherwise you could find a ) from the program name). Once you've got that, skip the state and there is the PPID. (So ) S <ppid>).
Repeat the search until all the parent/child are found and then start sending a SIGINT or SIGTERM or SIGKILL to each on of them.
As mentioned in the other answer, you can use pthread_detach() to quit your software and leave that other thread behind. This is probably much less desirable if you want that other process to end before your main process ends. It very much depends on what you are trying to accomplish, too.
Another, probably much more complicated way, is to use fork() + execve(). That means you have to re-implement your own system() call, but the advantage is that you do not need a thread and you get the pid of the child for free (i.e. thus you can kill it without searching for the child pid). If the function you need to run is not dynamically defined with a varying set of command line arguments, it's not too complicated. If you need to change stdin, stdout, stderr, and the arguments depend on all sorts of things, it becomes much more involved...
I'm new with multi-threading and I need to get the whole idea about the "join" and do I need to join every thread in my application ?, and how does that work with multi-threading ?
no, you can detach one thread if you want it to leave it alone.
If you start a thread, either you detach it or you join it before the program ends, otherwise this is undefined behaviour.
To know that a thread needs to be detached you need to ask yourself this question: "do I want the the thread to run after the program main function is finished?". Here are some examples:
When you do File/New you create a new thread and you detach it: the thread will be closed when the user closes the document Here you don't need to join the threads
When you do a Monte Carlo simulation, some distributed computing, or any Divide And Conquer type algorithms, you launch all the threads and you need to wait for all the results so that you can combine them. Here you explicitly need to join the thread before combining the results
Not joining a thread is like not deleteing all memory you new. It can be harmless, or it could be a bad habit.
A thread you have not synchronized with is in an unknown state of execution. If it is a file writing thread, it could be half way through writing a file and then the app finishes. If it is a network communications thread, it could be half way through a handshake.
The downside to joining every thread is if one of them has gotten into a bad state and has blocked, your app can hang.
In general you should try to send a message to your outstanding threads to tell them to exit and clean up. Then you should wait a modest amount of time for them to finish or otherwise respond that they are good to die, and then shut down the app. Now prior to this you should signify your program is no longer open for business -- shit down GUI windows, respond to requests from other processes that you are shutting down, etc -- so if this takes longer than anticipated the user is not bothered. Finally if things go imperfectly -- if threads refuse to respond to your request that they shut down and you give up on them -- then you should log errors as well, so you can fix what may be a symptom of a bigger problem.
The last time a worker thread unexpectedly hung I initially thought was a problem with a network outage and a bug in the timeout code. Upon deeper inspection it was because one of the objects in use was deleted prior to the shutdown synchronization: the undefined behaviour that resulted just looked like a hang in my reproduction cases. Had we not carefully joined, that bug would have been harder to track down (now, the right thing to do would have been to use a shared resource that we could not delete: but mistakes happen).
The pthread_join() function suspends execution of the calling thread
until the target thread terminates, unless the target thread has
already terminated. On return from a successful pthread_join() call
with a non-NULL value_ptr argument, the value passed to pthread_exit()
by the terminating thread is made available in the location referenced
by value_ptr. When a pthread_join() returns successfully, the target
thread has been terminated. The results of multiple simultaneous calls
to pthread_join() specifying the same target thread are undefined. If
the thread calling pthread_join() is canceled, then the target thread
will not be detached.
So pthread_join does two things:
Wait for the thread to finish.
Clean up any resources associated
with the thread.
This means that if you exit the process without call to pthread_join, then (2) will be done for you by the OS (although it won't do thread cancellation cleanup), and (1) will not be done.
So whether you need to call pthread_join depends whether you need (1) to happen.
Detached thread
If you don't need the thread to run, then you may as well pthread_detach it. A detached thread cannot be joined (so you can't wait on its completion), but its resources are freed automatically if it does complete.
do I need to join every thread in my application ?
Not necessarily - depends on your design and OS. Join() is actively hazardous in GUI apps - tend If you don't need to know, or don't care, about knowing if one thread has terminated from another thread, you don't need to join it.
I try very hard to not join/WaitFor any threads at all. Pool threads, app-lifetime threads and the like often do not require any explicit termination - depends on OS and whether the thread/s own, or are explicitly bound to, any resources that need explicit termination/close/whatever.
Threads can be either joinable or detached. Detached threads should not be joined. On the other hand, if you didn't join the joinable thread, you app would leak some memory and some thread structures. c++11 std::thread would call std::terminate, if it wasn't marked detached and thread object went out of scope without .join() called. See pthread_detach and pthread_create. This is much alike with processes. When the child exits, it will stay as zombee while it's creater willn't call waitpid. The reson for such behavior is that thread's and process's creater might want to know there exit code.
Update: if pthread_create is called with attribute argument equal to NULL (default attributes are used), joinable thread will be created. To create a detached thread, you can use attributes:
pthread_attr_t attrs;
pthread_attr_init(&attrs);
pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED);
pthread_create(thread, attrs, callback, arg);
Also, you can make a thread to be detached by calling pthread_detach on a created one. If you will try to join with a detached thread, pthread_join will return EINVAL error code. glibc has a non portable extension pthread_getattr_np that allows to get attributes of a running thread. So you can check if thread is detached with pthread_attr_getdetachstate.
I'm designing a thread library. So far I have a method that initializes the library, one that creates threads, and one that yields the current thread to the next one on a queue of ready threads.
Before I move on to implementing semaphores for the threads, I figured I should probably kill the threads as soon as they are done and free up their allocated memory, but I'm having trouble figuring out how to do that. How do I tell when a thread has "finished"?
You don't just kill threads safely or reliably -- let them exit naturally (when their entry returns).
Although the system provides a means to kill the thread, nearly any C++ program out there could expect undefined behavior if it were to continue. You could dream up cases where killing could be accomplished without side effects (to the rest of the program), but that program does not at all resemble idiomatic C++. Such a program would be very exotic, with many unusual and severe restrictions.
When you want to known when a thread has exited or not, you can add some cleanup before it exits in order to track its status.
When you want the ability to request a thread exit (naturally), consider run loops and messages.
You don't explicitly kill the threads when they are finished running their forked procedures as the code which would be doing that would still be in the context of the thread to be killed.
You have a scheduler/interrupt handler which handles the context switching of the threads and maintains a few queues for managing this. You can have it save a reference to to the threads to be killed, something like scheduler->SetThreadToKill( currentThread ); inside probably your finish() method (or similar), which sets a flag for the corresponding threads.
When a context switch occurs, and you have swapped out all data structures of the current thread with that of the next thread, you scheduler can call the destructor for all the threads which have the toBeKilled flag set.
The best policy, by far, for killing threads is to not explicitly do it, (unless you are an OS, ie. on app shutdown). Queue messages and tasks to threads that loop around some queue to perform more work. If you don't write any code to continually new, create, start, terminate, delete, test, check, enlist, delist, enqueue, dequeue and otherwise micro-manage threads, then that code cannot contain bugs.
This question seems like it's probably a duplicate, but I was unable to find one. If I missed a previous question, apologies.
In Java, where I have most of my experience, if your main() forks a thread and immediately returns the process continues to run until all (non-daemon) threads in the process have stopped.
In C++, this appears not to be the case - as soon as the main thread returns the process is terminating with other threads still running. For my current app this is easily solved with the application of pthread_join() but I'm wondering what causes this behavior. Is this compiler (gcc) specific, pthreads specific, or is kind of behavior shared across most/all platforms for which C++ has been implemented? Is this behavior configurable within pthreads (I've looked through the pthread api at the pthread_attr_*() functions and didn't see anything that looked relevant.)?
Completely separate question, but while you're here ... what would one use pthread_detatch() for?
Yes. In modern linux (more importantly newer versions of GNU libc) exit_group is the system call used when main returns, not plain exit. exit_group is described as follows:
This system call is equivalent to
exit(2) except that it terminates not
only the calling thread, but all
threads in the calling process's
thread group.
It is worth noting that current the c++ standard makes no mention of threads, so this behavior is not c++ specific, but instead is specific to your particular implementation. That said, every implementation I've personally seen kills all threads when the main thread terminates.
EDIT: It is also worth noting Jonathan Leffler's answer which points out that the POSIX standard does indeed specify this behavior, so it is certainly normal for an application using pthreads for its threading.
EDIT: To answer the follow up about pthread_detach. Basically it is considered a resource leak if you do not join a non-detached thread. If you have a long running task which you have no need to "wait for", and it just "ends when it ends" then you should detach it which will not have a resource leak when it terminates with no join. The man page says the following:
The pthread_detach() function marks
the thread identified by thread as
detached. When a detached thread
terminates, its resources are
automatically released back to the
system without the need for another
thread to join with the terminated
thread.
So a quick and dirty answer is: "when you don't care when it ends, detach it. If another thread cares when it ends and must wait for it to terminate, then don't."
Yes
The POSIX standard says:
ยง3.297 Process Termination
There are two kinds of process termination:
Normal termination occurs by a return from main(), when requested with the exit(), _exit(), or _Exit() functions; or when the last thread in the process terminates by returning from its start function, by calling the pthread_exit() function, or through cancellation.
Abnormal termination occurs when requested by the abort() function or when some signals are received.
The first normal termination condition applies. (Note that the C++ (1998, 2003) standard says nothing about threads.)
Regarding pthread_detach()
The POSIX standard (again) says:
The pthread_detach() function shall indicate to the implementation that storage for the thread thread can be reclaimed when that thread terminates. If thread has not terminated, pthread_detach() shall not cause it to terminate.
And the rationale says:
The pthread_join() or pthread_detach() functions should eventually be called for every thread that is created so that storage associated with the thread may be reclaimed.
It has been suggested that a "detach" function is not necessary; the detachstate thread creation attribute is sufficient, since a thread need never be dynamically detached. However, need arises in at least two cases:
In a cancellation handler for a pthread_join() it is nearly essential to have a pthread_detach() function in order to detach the thread on which pthread_join() was waiting. Without it, it would be necessary to have the handler do another pthread_join() to attempt to detach the thread, which would both delay the cancellation processing for an unbounded period and introduce a new call to pthread_join(), which might itself need a cancellation handler. A dynamic detach is nearly essential in this case.
In order to detach the "initial thread" (as may be desirable in processes that set up server threads).
This is not compiler specific and is standard behavior; the application terminates when main() exits, so if you want to prevent the application from terminating, you need main() to block until all threads have terminated, which you do by joining those threads. When you invoke pthread_create, it allocates resources for that thread. The resources are not deallocated unless you do a pthread_join (which blocks until the thread terminates) or pthread_detach (which causes the thread to automatically release resources when that thread exits). You should use pthread_detach whenever you launch a background thread that will terminate when its task is completed and for which you do not need to wait.
To make this a little bit more concrete, suppose you have several threads that perform a piece of a computation, and then you aggregate the result in some way. That would be a case where you would use join, because you need the results of the threads to proceed. Now, consider a case where a thread listens on a socket and processes incoming requests, until a flag indicates that the thread should quit. In this case, you would use pthread_detach, since nothing needs the thread to terminate in order to proceed, and so the resources associated with that thread should go away automatically.
HI
I am writing a part of a server which should dispatch some other child processes.
Because I want to wait on some of the processes, and dispatch others without waiting for completion, I use double fork for the second kind of processes (thus avoiding the zombie processes).
Problem is, my server holds a lot of memory, so forking takes a long time (even the copy-on-write fork used in Linux which copies only the paging tables)
I want to replace the fork() with vfork(), and it's easy for the second fork (as it only calls execve() in the child), but I couldn't find any way I can replace the first one.
Does anyone know how I can do that?
Thanks!
The server is a linux (RH5U4), written in C++.
Why not simply have the newly exec'd process do another fork itself? That way only a small simple process will have its page tables copied?
EDIT:
Of course the parent would have to do a short-duration wait() to clean up the zombie from that one, but the grandchild process could then run for as long as it wanted.
vfork() can only be used to fork and then call exec or exit. Also, vfork() will block the parent process until the child calls _exit or exec, which is almost certainly not the behavior that you want.
The reason for this is that vfork() doesn't make any copies of any of the data, including the stack, for the new process. So everything is shared, and it is very easy to accidentally change something that the parent process cannot handle. Since the data is shared without copies, the parent process cannot continue running at the same time as the child, so it must wait for the child to _exit or call exec so it is no longer using the data when the parent starts to modify it.
I think that what you really want to do is to make use of SIGCHLD and maintain a list of child processes. You can then do away with the double fork by having your main process be notified when children change state (mostly, when they die) and perform some action on them based on that. You can also keep track of of any of your child processes take longer than expected to complete (because you stored their creation time in your list) and take action if they go crazy and never complete.
Don't double fork. Handle SIGCHLD to save errno, call wait, restore errno.
I believe you can use the answer to another question I asked, for a similar reason. You can vfork() + exec() to an executable which forks again. See setuid() before calling execv() in vfork() / clone()