Multithreaded program and fork(): alternative or safe implementation - c++

In a multithreaded Linux/C++-program, I want to use fork() with a signal handler for SIGCHLD.
In the child process I use open() to create two new file descriptors, sendfile() and close(), then the child exits.
I planned to use fork() to implement the following requirements:
The threads in the parent process shall be able to
detect the normal termination of the child process, and in that case shall be able to create another fork() doing the open()/sendfile()/close() for a range of files
kill the sendfile()-child process in case of a specific event and detect the intentional termination to clean up
For requirement 1 I could just wait for the result of sendfile().
Requirement 2 is why I think I need to use fork() in the first place.
After reading the following posts
Threads and fork(). How can I deal with that?
fork in multi-threaded program
I think that my solution might not be a good one.
My questions are:
Is there any other solution to implement requirement 2 ?
Or how can I make sure that the library calls open(), close() and sendfile() will be okay?
Update:
The program will run on a Busybox Linux / ARM
I've assumed that I should use sendfile() for having the most efficient file transfer due to several posts I've read regarding this topic.
A safe way to implement my requirement could be using fork() and exec*() with cp, with the disadvantage that the file transfer might be less efficient
Update 2:
it's sufficient to fork() once in case of a specific event (instead of once per file) since I switched to exec*() with rsync in the child process. However the program needs invoke that rsync always in case of a specific event.

You can use threads, but forcefully terminating threads typically leads to memory leaks and other problems.
My linux experience is somewhat limited, but I would probably try to fork the program early, before it gets multithreaded. Now that you have two instances, the single threaded instance can be safely used to manage the starting and stopping of additional instances.

Related

What is the point of the process fork creates being a copy of the parent?

I know the answer to "why is it this way" is because the language was invented so, but it seems like a lot of wasted effort that fork() spawns a copy of the process that called it. Perhaps it is useful sometimes, but surely the majority of time someone wants to start a new process its not to be a duplicate of the calling one? Why does fork create an identical process and not an empty one or one defined by passing an argument?
From yolinux
The fork() system call will spawn a new child process which is an
identical process to the parent except that has a new system process
ID
In other words when is it useful to start with a copy of the parent process?
One big advantage of having the parent process duplicated in the child is that it allows the parent program to make customizations to the child process' environment before executing it. For example, the parent might want to read the child process' stdout, in which case it needs to set up the pipes in order to allow it to read that before execing the new program.
It's also not as bad as it sounds, efficiency wise. The whole thing is implemented on Linux using copy-on-write semantics for the process' memory (except in the special cases noted in the man page):
Under Linux (and in most unices since version 7, parent of all unices alive now), fork() is implemented using copy-on-write pages, so the only
penalty that it incurs is the time and memory required to duplicate the
parent's page tables (which can be also copy-on-write), and to create a unique task structure for the child.
There are some very legitimate uses of the fork system call. Here are a few examples:
Memory saving. Because fork on any modern UNIX/Linux system shares memory between the child and parent (via copy-on-write semantics), a parent process can load some static data which can be instantly shared to a child process. The zygote process on Android does this: it preloads the Java (Dalvik) runtime and many classes, then simply forks to create new application processes on demand (which inherit a copy of the parent's runtime and loaded classes).
Time saving. A process can perform some expensive initialization procedure (such as Apache loading configuration files and modules), then fork off workers to perform tasks which use the preloaded initialization data.
Arbitrary process customization. On systems that have direct process creation methods (e.g. Windows with CreateProcess, QNX with spawn, etc., these direct process creation APIs tend to be very complex since every possible customization of the process has to be specified in the function call itself. By contrast, with fork/exec, a process can just fork, perform customizations via standard system calls (close, signal, dup, etc.) and then exec when it's ready. fork/exec is consequently one of the simplest process creation APIs in existence, yet simultaneously one of the most powerful and flexible.
To be fair, fork also has its fair share of problems. For example, it doesn't play nice with multithreaded programs: only one thread is created in the new process, and locks are not correctly closed (leading to the necessity of atfork handlers to reset lock states across a fork).
Contrary to all expectations, it's mainly fork that makes process creation so incredibly fast on Unices.
AFAIK, on Linux, the actual process memory is not copied upon fork, the child starts with the same virtual memory mapping as the parent, and pages are copied only where and when the child makes changes. The majority of pages are read-only code anyway, so they are never copied. This is called copy-on-write.
Use cases where copying the parent process is useful:
Shells
When you say cat foo >bar, the shell forks, and in the child process (still the shell) prepares the redirection, and then execs cat foo. The executed program runs under the same PID as the child shell and inherits all open file descriptors. You would not believe how easy it is to write a basic Unix shell.
Daemons (services)
Daemons run in the background. Many of them fork after some initial preparation, the parent exits, and the child detaches from the terminal and remains running in the background.
Network servers
Many networking daemons have to handle multiple connections at the same time. Example sshd. The main daemon runs as root and listens for new connections on port 22. When a new connection comes in it forks a child. The child just keeps the new socket representing that connection, authenticates the user, drops privileges and so on.
Etc
Why fork()? It had nothing to do with C. C was itself only coming into existence at the time. It's because of the way the original UNIX memory page and process management worked, it was trivial to cause a process to be paged out, and then paged back in at a different location, without unloading the first copy of the process.
In The Evolution of the Unix Time-sharing System (http://cm.bell-labs.com/cm/cs/who/dmr/hist.html), Dennis Ritchie says "In fact, the PDP-7's fork call required precisely 27 lines of assembly code." See the link for more.
Threads are evil. With threads, you essentially have a number of processes all with access to the same memory space, which can dance all over each others' values. There's no memory protection at all. See The Art of Unix Programming, Chapter 7 (http://www.faqs.org/docs/artu/ch07s03.html#id2923889) for a fuller explanation.

Launch Executable from Secondary thread

I need a way to launch executable (process) from secondary thread. The wxWidgets toolkit I use give me no way to the best of my knowledge and now I have to seek platform dependent way (Linux and Windows only) way to do that.
The executable will read and write to the file. Other than knowing that the process is still running or not (and if possible exit code) there is no much this thread is going to do with process itself. After process terminate, the thread will read the written file, analyze it and send result to GUI thread for displaying. I have no idea how to do it since I have been depending on wxExecute all the times.
Do you really want separate processes, or will threads do.
To get separate processes, you need fork() for Linux and CreateProcess() for Windows. The best approach is often to make a function to encapsulate it.
bool RunProces(const std::string &sCmdLine)
{
#if defined(WIN32)
// launch with CreateProcess() ...
#else
// launch with fork() ...
#endif
return bSuccess;
}
If, however, you can just use threads, you can use the portable std::thread class.
An alternative approach is to use popen() (_popen() for Windows). This launches a new process and redirects its output (stdout) to a pipe. You can then read the pipe to retrieve the sub-process output.
A couple of notes about popen():
Redirecting the standard output makes it impractical for the sub-process to interact with the user.
There would be some overhead setting up the pipe. Probably not important unless you are spawning large numbers of sub-processes.
I've never seen popen() used to launch a non-console Windows program (with no main() function). I wouldn't be surprised if there are problems doing that.
You will have to move quite a few parts of your code from the secondary thread to the main thread. The wxWidgets documentation you refer to says it all:
When a wxProcess object is passed to wxExecute(), its OnTerminate()
virtual method is called when the process terminates. This allows the
program to be (asynchronously) notified about the process termination
and also retrieve its exit status which is unavailable from
wxExecute() in the case of asynchronous execution.
And below:
Currently wxExecute() can only be used from the main thread, calling
this function from another thread will result in an assert failure in
debug build and won't work.
So, you need to eliminate the need to call wxExecute from the secondary thread. How exactly you should do this depends on your application, but it will probably involve the secondary thread sending a custom wxWidgets event to the main thread using wxQueueEvent. The main thread then handles the event by calling wxExecute in asynchronous mode and retrieving its result later on.
Now what to do with the result?
Ideally, you rework your application logic such that the secondary thread does not need the result but instead:
the main thread handles it all by itself, -or-
another secondary thread is started and handles it.
This will minimize the amount of synchronization you have to do, and thus reduce the probability of hard-to-find concurrency programming errors.

Fork and core dump with threads

Similar points to the one in this question have been raised before here and here, and I'm aware of the Google coredump library (which I've appraised and found lacking, though I might try and work on that if I understand the problem better).
I want to obtain a core dump of a running Linux process without interrupting the process. The natural approach is to say:
if (!fork()) { abort(); }
Since the forked process gets a fixed snapshot copy of the original process's memory, I should get a complete core dump, and since the copy uses copy-on-write, it should generally be cheap. However, a critical shortcoming of this approach is that fork() only forks the current thread, and all other threads of the original process won't exist in the forked copy.
My question is whether it is possible to somehow obtain the relevant data of the other, original threads. I'm not entirely sure how to approach this problem, but here are a couple of sub-questions I've come up with:
Is the memory that contains all of the threads' stacks still available and accessible in the forked process?
Is it possible to (quicky) enumerate all the running threads in the original process and store the addresses of the bases of their stacks? As I understand it, the base of a thread stack on Linux contains a pointer to the kernel's thread bookkeeping data, so...
with the stored thread base addresses, could you read out the relevant data for each of the original threads in the forked process?
If that is possible, perhaps it would only be a matter of appending the data of the other threads to the core dump. However, if that data is lost at the point of the fork already, then there doesn't seem to be any hope for this approach.
Are you familiar with process checkpoint-restart? In particular, CRIU? It seems to me it might provide an easy option for you.
I want to obtain a core dump of a running Linux process without interrupting the process [and] to somehow obtain the relevant data of the other, original threads.
Forget about not interrupting the process. If you think about it, a core dump has to interrupt the process for the duration of the dump; your true goal must therefore be to minimize the duration of this interruption. Your original idea of using fork() does interrupt the process, it just does so for a very short time.
Is the memory that contains all of the threads' stacks still available and accessible in the forked process?
No. The fork() only retains the thread that does the actual call, and the stacks for the rest of the threads are lost.
Here is the procedure I'd use, assuming CRIU is unsuitable:
Have a parent process that generates a core dump of the child process whenever the child is stopped. (Note that more than one consecutive stop event may be generated; only the first one until the next continue event should be acted on.)
You can detect the stop/continue events using waitpid(child,,WUNTRACED|WCONTINUED).
Optional: Use sched_setaffinity() to restrict the process to a single CPU, and sched_setscheduler() (and perhaps sched_setparam()) to drop the process priority to IDLE.
You can do this from the parent process, which only needs the CAP_SYS_NICE capability (which you can give it using setcap 'cap_sys_nice=pe' parent-binary to the parent binary, if you have filesystem capabilities enabled like most current Linux distributions do), in both the effective and permitted sets.
The intent is to minimize the progress of the other threads between the moment a thread decides it wants a snapshot/dump, and the moment when all threads have been stopped. I have not tested how long it takes for the changes to take effect -- certainly they only happen at the end of their current timeslices at the very earliest. So, this step should probably be done a bit beforehand.
Personally, I don't bother. On my four-core machine, the following SIGSTOP alone yields similar latencies between threads as a mutex or a semaphore does, so I don't see any need to strive for even better synchronization.
When a thread in the child process decides it wants to take a snapshot of itself, it sends a SIGSTOP to itself (via kill(getpid(), SIGSTOP)). This stops all threads in the process.
The parent process will receive the notification that the child was stopped. It will first examines /proc/PID/task/ to obtain the TIDs for each thread of the child process (and perhaps /proc/PID/task/TID/ pseudofiles for other information), then attaches to each TID using ptrace(PTRACE_ATTACH, TID). Obviously, ptrace(PTRACE_GETREGS, TID, ...) will obtain the per-thread register states, which can be used in conjunction with /proc/PID/task/TID/smaps and /proc/PID/task/TID/mem to obtain the per-thread stack trace, and whatever other information you're interested in. (For example, you could create a debugger-compatible core file for each thread.)
When the parent process is done grabbing the dump, it lets the child process continue. I believe you need to send a separate SIGCONT signal to let the entire child process continue, instead of just relying on ptrace(PTRACE_CONT, TID), but I haven't checked this; do verify this, please.
I do believe that the above will yield a minimal delay in wall clock time between the threads in the process stopping. Quick tests on AMD Athlon II X4 640 on Xubuntu and a 3.8.0-29-generic kernel indicates tight loops incrementing a volatile variable in the other threads only advance the counters by a few thousand, depending on the number of threads (there's too much noise in the few tests I made to say anything more specific).
Limiting the process to a single CPU, and even to IDLE priority, will drastically reduce that delay even further. CAP_SYS_NICE capability allows the parent to not only reduce the priority of the child process, but also lift the priority back to original levels; filesystem capabilities mean the parent process does not even have to be setuid, as CAP_SYS_NICE alone suffices. (I think it'd be safe enough -- with some good checks in the parent program -- to be installed in e.g. university computers, where students are quite active in finding interesting ways to exploit the installed programs.)
It is possible to create a kernel patch (or module) that provides a boosted kill(getpid(), SIGSTOP) that also tries to kick off the other threads from running CPUs, and thus try to make the delay between the threads stopping even smaller. Personally, I wouldn't bother. Even without the CPU/priority manipulation I get sufficient synchronization (small enough delays between the times the threads are stopped).
Do you need some example code to illustrate my ideas above?
When you fork you get a full copy of the running processes memory. This includes all thread's stacks (after all you could have valid pointers into them). But only the calling thread continues to execute in the child.
You can easily test this. Make a multithreaded program and run:
pid_t parent_pid = getpid();
if (!fork()) {
kill(parent_pid, SIGSTOP);
char buffer[0x1000];
pid_t child_pid = getpid();
sprintf(buffer, "diff /proc/%d/maps /proc/%d/maps", parent_pid, child_pid);
system(buffer);
kill(parent_pid, SIGTERM);
return 0;
} else for (;;);
So all your memory is there and when you create a core dump it will contain all the other threads stacks (provided your maximum core file size permits it). The only pieces that will be missing are their register sets. If you need those then you will have to ptrace your parent to obtain them.
You should keep in mind though that core dumps are not designed to contain runtime information of more then one thread - the one that caused the core dump.
To answer some of your other questions:
You can enumerate threads by going through /proc/[pid]/tasks, but you can not identify their stack bases until you ptrace them.
Yes, you have full access to the other threads stacks snapshots (see above) from the forked process. It is not trivial to determine them, but they do get put into a core dump provided the core file size permits it. Your best bet is to save them in some globally accessible structure if you can upon creation.
If you intend to get the core file at non-specific location, and just get core image of the process running without killing, then you can use gcore.
If you intend to get the core file at specific location (condition) and still continue running the process - a crude approach is to execute gcore programmatically from that location.
A more classical, clean approach would be to check the API which gcore uses and embedded it in your application - but would be too much of an effort compared to the need most of the time.
HTH!
If your goal is to snapshot the entire process in order to understand the exact state of all threads at a specific point then I can't see any way to do this that doesn't require some kind of interrupt service routine. You must halt all processors and record off the current state of each thread.
I don't know of any system that provides this kind of full process core dump. The rough outlines of the process would be:
issue an interrupt across all CPUs (both logical and physical cores).
busy wait for all cores to synchronize (this shouldn't take long).
clone the desired process's memory space: duplicate the page tables and mark all pages as copy on write.
have each processor check whether its current thread is in the target process. If so record the current stack pointer for that thread.
for every other thread examine the thread data block for the current stack pointer and record it.
create a kernel thread to save off the copied memory spaces and the thread stack pointers
resume all cores.
This should capture the entire process state, including a snapshot of any processes that were running at the moment the inter-processor interrupt was issued. Because all threads are interrupted (either through standard scheduler suspension process, or via our custom interrupt process) all register states will be on a stack somewhere in the process memory. You then only need to know where the top of each thread stack is. Using the copy on write mechanism to clone the page tables allows transparent save-off while the original process is allowed to resume.
This is a pretty heavyweight option, since it's main functionality requires suspending all processors for a significant amount of time (synchronize, clone, walk all threads). However this should allow you to exactly capture the status of all threads, as well as determine which threads were running (and on which CPUs) when the checkpoint was reached. I would assume some of the framework for doing this process exists (in CRIU for instance). Of course resuming the process will result in a storm of page allocations as the copy on write mechanism protects the check-pointed system state.

double fork using vfork

HI
I am writing a part of a server which should dispatch some other child processes.
Because I want to wait on some of the processes, and dispatch others without waiting for completion, I use double fork for the second kind of processes (thus avoiding the zombie processes).
Problem is, my server holds a lot of memory, so forking takes a long time (even the copy-on-write fork used in Linux which copies only the paging tables)
I want to replace the fork() with vfork(), and it's easy for the second fork (as it only calls execve() in the child), but I couldn't find any way I can replace the first one.
Does anyone know how I can do that?
Thanks!
The server is a linux (RH5U4), written in C++.
Why not simply have the newly exec'd process do another fork itself? That way only a small simple process will have its page tables copied?
EDIT:
Of course the parent would have to do a short-duration wait() to clean up the zombie from that one, but the grandchild process could then run for as long as it wanted.
vfork() can only be used to fork and then call exec or exit. Also, vfork() will block the parent process until the child calls _exit or exec, which is almost certainly not the behavior that you want.
The reason for this is that vfork() doesn't make any copies of any of the data, including the stack, for the new process. So everything is shared, and it is very easy to accidentally change something that the parent process cannot handle. Since the data is shared without copies, the parent process cannot continue running at the same time as the child, so it must wait for the child to _exit or call exec so it is no longer using the data when the parent starts to modify it.
I think that what you really want to do is to make use of SIGCHLD and maintain a list of child processes. You can then do away with the double fork by having your main process be notified when children change state (mostly, when they die) and perform some action on them based on that. You can also keep track of of any of your child processes take longer than expected to complete (because you stored their creation time in your list) and take action if they go crazy and never complete.
Don't double fork. Handle SIGCHLD to save errno, call wait, restore errno.
I believe you can use the answer to another question I asked, for a similar reason. You can vfork() + exec() to an executable which forks again. See setuid() before calling execv() in vfork() / clone()

Mystery pthread problem with fork()

I have a program which:
has a main thread (1) which starts a server thread (2) and another (4).
the server thread (2) does an accept(), then creates a new thread (3) to handle the connection.
At some point, thread (4) does a fork/exec to run another program which should connect to the socket that thread (2) is listening to. Occasionally this fails or takes an unreasonably long time, and it's extremely difficult to diagnose. If I strace the system, it appears that the fork/exec has worked, the accept has happened, the new thread (4) has been created .. but nothing happens in that thread (using strace -ff, the file for the relevant pid is blank).
Any ideas?
I came to the conclusion that it was probably this phenomenon:
http://kerneltrap.org/mailarchive/linux-kernel/2008/8/15/2950234/thread
as the bug is difficult to trigger on our development systems but is generally reported by users running on large shared machines; also the forked application starts a JVM, which itself allocates a lot of threads. The problem is also associated with the machine being loaded, and extensive memory usage (we have a machine with 128Gb of RAM and processes may be 10-100G in size).
I've been reading the O'Reilly pthreads book, which explains pthread_atfork(), and suggests the use of a "surrogate parent" process forked from the main process at startup from which subprocesses are run. It also suggests the use of a pre-created thread pool. Both of these seem like good ideas, so I'm going to implement at least one of them.
It's look like a deadlock condition. Look for blocking functions, like accept(), the problem should be there.
Decrease the code to the smallest possible size that still has the behavior and post it here. Either you will find the answer or we will be able to track it down.
BTW - http://lists.samba.org/archive/linux/2002-February/002171.html it seems that pthread behavior for exec is not well defined and may depend on your OS.
Do you have any code between fork and exec? This may be a problem.
Be very careful with multiple threads and fork. Most of glibc/libstdc++ is thread safe. If a thread, other than the forking thread, is holding a lock when the fork executes the forked process will inherit the mutexes in their current locked state. The new process will never see those mutexes unlocked. For more information see man pthread_atfork.
I've just fallen into same problems, and finally found that fork() duplicates all the threads. Now imagine, what does your program do after a fork() with all the threads running double instance...
The following rules are from "A Mini-guide regarding fork() and Pthreads":
1- You DO NOT WANT to do that.
2- If you needs to fork() then:
whenever possible, fork() all your
childs prior to starting any threads.
Edit: tried, fork() does not duplicate threads.