I am porting Windows application to Linux. I use CreateProcess on Windows to run child processes and redirect all standard streams (in, out, error). Streams redirect is critical, main process sends data to children and receives theirs output and error messages. Main process is very big one with a lot of memory and threads, and child processes are small ones. On Linux I see that fork function has similar functionality as CreateProcess on Windows. However, manual says that fork "creates parent process copy", including code, data and stack. Does it mean that if I create copy of a huge process that uses 1 GB of memory just to run a very simple command line tool that uses 1 MB of memory itself, I will need to fist duplicate 1 GB of memory with fork, and then replace this 1 GB with 1 MB process? So, if I have 100 threads it will be required to have 100 GB of memory to run 100 processes that need just 100 MB of memory to run? Also what about other threads in parent process that "don't know" about fork execution, what will they do? What fork function does "under the hood" and is it really effective way to create a lot of small child processes from huge parent?
When you call fork() then initially only your VM is copied and all pages are marked copy-on write. Your new child process will have a logical copy of your parent processes VM, but it will not consume any additional RAM until you actually start writing to it.
As for threads, fork creates only one new thread in the child process that resembles a copy of the calling thread.
Also as soon as you call any of the exec family of calls (which I assume you want to) then your entire process image is replaced with a new one and only file descriptors are kept.
If your parent process has a lot of open file descriptors then I suggest you go through /proc/self/fd and close all file descriptors in the child that you don't need.
fork basically splits your process into two, with both parent and child processes continuing at the instruction after the fork function call. However, the return value value in the child process is 0, whilst in the parent process it is the process id of the child process.
The creation of the child process is extremly quick since it uses the same pages as the parent. The pages are marker as copy-on-write (COW) so that if either process changes the page then the other won't be affected. Once the child process exists it usually calls one of the exec functions to replace itself with a image. Windows doesn't have an equivilant to fork, instead the CreateProcess call only allows you to start a new process.
There is an alternative to fork called clone which gives you much more control over what happens when the new process is started. For example you can specify a function to call in the new process.
The copies are "copy-on-write", so if your child process does not modify the data, it will not use any memory besides that of the father process. Typically, after a fork(), the child process makes an exec() to replace the program of this process with a different one, then all the memory is dropped anyway.
I haven't used CreateProcess, but fork() is not an exact copy of the process. It creates a child process, but the child starts its execution at the same instruction in which the parent called fork, and continues from there.
I recommend taking a look at Chapter 5 of the Three Easy Pieces OS book. This may get you started and you might find the child spawning call you're looking for.
The forked child process has almost all the parent facility copied: memory, descriptors, text etc. The only exception is parents' threads, they are not copied.
Related
The scenario:
There are several processes running on a machine. Names and handles unknown, but they all have a piece of code running in them that's under our control.
A command line process is run. It signals to the other processes that they need to end (SetEvent), which our code picks up and handles within the other processes.
The goal:
The command line process needs to wait until the other processes have ended. How can this be achieved?
All that's coming to mind is to set up some shared memory or something and have each process write its handle into it so the command line process can wait on them, but this seems like so much effort for what it is. There must be some kernel level reference count that can be waited on?
Edit 1:
I'm thinking maybe assigning the processes to a job object, then the command line processes can wait on that? Not ideal though...
Edit 2:
Can't use job objects as it would interfere with other things using jobs. So now I'm thinking that the processes would obtain a handle to some/any sync object (semaphore, event, etc), and the command line process would poll for its existance. It would have to poll as if it waited it would keep the object alive. The sync object gets cleaned up by windows when the processes die, so the next poll would indicate that there are no processes. Not the niceset, cleanest method, but simple enough for the job it needs to do. Any advance on that?
You can do either of following ways.
Shared Memory (memory mapped object) : CreateFileMapping, then MapViewOfFile --> Proceed the request. UnmapViewFile. Close the file,
Named Pipe : Create a nameed pipe for each application. And keep running a thread to read the file. So, You can write end protocol from your application by connecting to that named pipe. ( U can implement a small database as like same )
WinSock : (Dont use if you have more number of processes. Since you need to send end request to the other process. Either the process should bind to your application or it should be listening in a port.)
Create a file/DB : Share the file between the processes. ( You can have multiple files if u needed ). Make locking before reading or writing.
I would consider a solution using two objects:
a shared semaphore object, created by the main (controller?) app, with an initial count of 0, just before requesting the other processes to terminate (calling SetEvent()) - I assume that the other processes don't create this event object, neither they fail if it has not been created yet.
a mutex object, created by the other (child?) processes, used not for waiting on it, but for allowing the main process to check for its existence (if all child processes terminate it should be destroyed). Mutex objects have the distinction that can be "created" by more than one processes (according to the documentation).
Synchronization would be as follows:
The child processes on initialization should create the Mutex object (set initial ownership to FALSE).
The child processes upon receiving the termination request should increase the semaphore count by one (ReleaseSemaphore()) and then exit normally.
The main process would enter a loop calling WaitForSingleObject() on the semaphore with a reasonably small timeout (eg some 250 msec), and then check not whether the object was granted or a timeout has occurred, but whether the mutex still exists - if not, this means that all child processes terminated.
This setup avoids making an interprocess communication scheme (eg having the child processes communicating their handles back - the number of which is unknown anyway), while it's not strictly speaking "polling" either. Well, there is some timeout involved (and some may argue that this alone is polling), but the check is also performed after each process has reported that it's terminating (you can employ some tracing to see how many times the timeout has actually elapsed).
The simple approach: you already have an event object that every subordinate process has open, so you can use that. After setting the event in the master process, close the handle, and then poll until you discover that the event object no longer exists.
The better approach: named pipes as a synchronization object, as already suggested. That sounds complicated, but it isn't.
The idea is that each of the subordinate processes creates an instance of the named pipe (i.e., all with the same name) when starting up. There's no need for a listening thread, or indeed any I/O logic at all; you just need to create the instance using CreateNamedPipe, then throw away the handle without closing it. When the process exits, the handle is closed automatically, and that's all we need.
To see whether there are any subordinate processes, the master process would attempt to connect to that named pipe using CreateFile. If it gets a file not found error, there are no subordinate processes, so we're done.
If the connection succeeded, there's at least one subordinate process that we need to wait for. (When you attempt to connect to a named pipe with more than one available instance, Windows chooses which instance to connect you to. It doesn't matter to us which one it is.)
The master process would then call ReadFile (just a simple synchronous read, one byte will do) and wait for it to fail. Once you've confirmed that the error code is ERROR_BROKEN_PIPE (it will be, unless something has gone seriously wrong) you know that the subordinate process in question has exited. You can then loop around and attempt another connection, until no more subordinate processes remain.
(I'm assuming here that the user will have to intervene if one or more subordinates have hung. It isn't impossible to keep track of the process IDs and do something programmatically if that is desirable, but it's not entirely trivial and should probably be a separate question.)
I need to execute processes with still being in control of each process.
I want to create a class which stores the threads or pids or what ever is necessary to do so.
I currently have a program which executes one external application with the C function execvp and also loads the environment from a shell script. So my current program is blocking. But I need to be able to keep it freely running and only by time I terminate a currently running or start a new external application.
My current approach would be to create a thread, which uses the execve function. But then the thread would be blocking as far as I can see.
The code which might be in the thread (with variables then):
char *argv[] = { "/bin/bash", "-c", "myApplication", 0 };
execve(argv[0], &argv[0], environment.data());
The applications called are probably not fixed in the code their names will be given by an external setup file, including parameters.
Now my actual question, is there a better way to "manage" external applications like that in c++? Some ready solution (class, library)? And if not how do I terminate the thread if this is the actual way. Using the terminate call is said to be bad practice, that's what I often read.
I hope this is now specific enough for the forum, because I do not know how to get more specific anymore. If you need more hints what I want to create here, feel free to ask in the comments.
Update:
to DBus & others:
Additional information I do not wrote all of the processes I want to start!
So it will be used to start 3rd party applications, which even if I have the code, do not want to change.
You want to fork() before you exec. fork() is a function that creates a new process identical to the original caller of fork() running as a subprocess. The difference is that the parent process gets the child's pid as a return value and the child gets 0. The gist of what you want to do is this:
pid_t pid = fork();
if( pid == 0 )
{
// we're the child process
char *argv[] = { "/bin/bash", "-c", "myApplication", 0 };
int rc = execve(argv[0], &argv[0], environment.data());
// execve only returns if there was an error
// check 'errno' and handle it here
}
else if ( pid < 0 )
{
// pid is less than zero, we didn't successfully fork,
// there is no child process.
throw "error message";
}
// do whatever processing the parent does
More info is here. The kill() function isn't bad practice per se, if you want to quickly and gracefully end the subprocess you can write signal handlers in it, but you should be using something like dbus or zeromq to do proper interprocess communication. You want to tell the program to do something, not just tell it to die (usually what you want it to do if you're killing it).
NEVER USE execv functions in threads because the execve() system call overlays the current process image with a new process image.
The correct pattern if fork-exec or better vfork-exec. Extract from the manpage:
The vfork() system call can be used to create new processes without fully
copying the address space of the old process, which is horrendously inefficient in a paged environment. It is useful when the purpose of fork(2)
would have been to create a new system context for an execve(2). The
vfork() system call differs from fork(2) in that the child borrows the
parent's memory and thread of control until a call to execve(2) or an
exit (either by a call to _exit(2) or abnormally). The parent process is
suspended while the child is using its resources.
Using vfork shortly followed with execve, you avoid the copy of the original process image, and do not erase if with the new process, so the original process has the pid of its child and cat control it, look whether it has ended, send it signals and so on.
Is there a way to kill my app's child process and perform it's cleanup(calling deconstructors and atexit functions), similarly to exit(exit_code), but on another process?
If you are on windows, you probably start your child processes by CreateProcess, which has a PROCESS_INFORMATION as the last parameter.
CreateProcess on MSDN
Process Information on MSDN
Option 1:
This process information contains a handle to the process started in the hProcess member.
You can store this handle and use it to kill your child processes.
Insert
You probably want to send WM_CLOSE and / or WM_QUIT?
to "cleanly" end the process:
Here is a KB Article on what to do KB how to cleanly kill win32 processes
** End Insert**
Option 2:
Here is an discussion on how to properly kill a process tree: Terminate a process tree on windows
There's no simple Win32 API for that kind of thing. The OS doesn't care what language your program's source code was written in, the compiled program appears to it as just a sequence of CPU instructions plus data.
The cleanest way would be to establish some kind of a communication channel between the processes (e.g. via shared memory) and simply request process termination.
You can achieve the same by starting the child process as a debugged process and then using debug APIs to alter the child's behavior, but that's too intrusive and not very straightforward to implement.
I will jump right in, to be brief and descriptive:
C++, Windows API
I am creating child processes using CreateProcess to run external (command-line) applications. I have built in a time-out, and if the child process has not returned normal execution by that time, I wish to force termination on that child process.
Ideally, I would like for that child process to act the same as if it had called ExitProcess, or as if a Ctrl+C was sent to its console (which calls ExitProcess from the default console control handler).
My solution so far has been the use of TerminateProcess to kill the child forcefully. This does force the child to terminate immediately, but unfortunately if that child spawned any children of its own they are left to run until their "natural" completion.
Is there a way to tell the child process to call ExitProcess, or to force all of the child's children to also terminate when TerminateProcess is called?
These external applications are beyond my control, and as such I can not modify them to provide a custom work-around.
Assume no knowledge of grand-child processes (names/pids/etc) that would allow me to manually call TerminateProcess on grand-child processes individually. Although this could be done by manually enumerating all processes, mapping process relationships, and tracking all processes, I do not consider this a valid solution except as the absolute last resort.
Thank you for your time.
You can use Job objects to kill all the processes as a unit. You create a job object via the CreateJobObject API, and assign a process to it with AssignProcessToJobObject. New processes created by a process in a job object belong to the same job object by default. Calling TerminateJobObject will terminate all associated processes in the job object.
I am writing a shell where I need to launch several child processes at once and record the system time and user time.
So far I am able to do it. The only problem is that I am using wait4 to grab the system resources used by the child program and put it in my rusage structure called usage.
How can I launch all the processes at the same time and keep track of the user and system times? I can remove the wait4() system call and use it outside to loop so I can make the parent wait, but if I do that then I can only record the times for the last process and not all of them.
Do you have any idea how I can fix this?
execute(commandPipev,"STANDARD",0);
wait4(pid,&status,0,&usage);
printf("Child process: %s\t PID:%d\n", commandPipev[0], pid);
printf("System time: %ld.%06ld sec\n",usage.ru_stime.tv_sec, usage.ru_stime.tv_usec);
printf("User time: %ld.%06ld sec\n\n",usage.ru_utime.tv_sec, usage.ru_utime.tv_usec);
A convoluted answer.
In a POSIX environment, launch the children, then use waitid() with the WNOWAIT option to tell you that some child has exited. The option leaves the child in a waitable state - that is, you can use another wait-family call to garner the information you need. You can then use the non-POSIX wait4() system call to garner the usage information for the just exited child, and deal with the accounting you need to do. Note that you might find a different process has terminated between the waitid() and wait4() calls; you need to use a loop and appropriate flags and tests to collect all the available corpses (dead child processes) before going back to the waitid() call to find out about the other previously incomplete child processes. You also have to worry about any of the wait-family of functions returning the information for a process that was previously started in the background and has now finished.
The Linux man page for wait4(2) suggests that WNOWAIT might work directly with wait4(2), so you may be able to do it all more cleanly - if, indeed, you need the option at all.
Consider whether you can use process groups to group the child processes together, to make waiting for the members of the process group easier.