I have a program written in C++ intended to run on a Linux OS. Ignoring much of the program, it boils down to this - it starts X number of executables after some amount of time (for simplicity sake, let's use 5 seconds).
Currently, I'm using system(path/to/executable/executable_name) to do the actual starting of the executable(s) and that works just fine for getting the executable(s) to start.
I'm also trying to maintain a status for each executable (for simplicity sake again, let's just say the status is either "UP" or "DOWN" (running or not running)). I have been able to accomplish this...somewhat...
Backing up just a tad, when my program is told to start the executable(s), the logic looks something like this:
pid = fork()
if (pid < 0) exit 0; //fork failed
if (pid == 0) {
system(path/to/executable/executable_name)
set executable's status to DOWN
} else {
verify executable started
set executable's status to UP
}
Herein lies my problem. fork() causes a child process to be spawned, which is what I thought I needed in order for the original process to continue starting additional executables. I don't want to wait for an executable to stop in order to start another.
However, the executable starts in another child process...which is separate from the parent process... and if I try to set the executable's status to DOWN in the child process when system returns, the parent process does not know about it...
I have a few ideas of what I might need to do:
use threads instead of fork: create a new thread to call system, but would the parent/main thread know about the new thread changing the status of the executable?
use fork and exec: but I'm not sure that would be any better than what I already have (I've read the man pages for fork and exec but I guess I'm still a little fuzzy on how to best utilize exec)
Any suggestions?
EDIT 1
I thought I'd better give a little more context for the logic:
void startAll() {
for each 'executable'
call startExecutable(executable_name)
}
...
void startExecutable (executable_name) {
pid = fork()
if (pid < 0) exit 0; //fork failed
if (pid == 0) {
system(path/to/executable/executable_name)
set executable's status to DOWN
exit (1); <-- this is because once the child process's system returns, I don't want it to return to the above loop and start starting executables
} else {
verify executable started
set executable's status to UP
}
}
EDIT 2
As mentioned at the beginning, this is assuming a simplified setup (a first run if you will). The plan is to handle not just an "UP" or "DOWN" state, but also a third state to handle sending a message to the executables my program has started - "STANDBY." I initially left this piece out to avoid complicating the explanation but I now see that it is imperitive to include.
You need to understand what exactly is happening when you fork. What you're doing is creating a subprocess that's an exact clone of the forking process. All variables currently in memory are copied exactly, and the subprocess has access to all of those copies of all of those variables.
But they're copies, so as you've noticed, fork and exec/system does not on its own handle inter-process communication (IPC). Setting a memory value in one of the processes doesn't alter that variable in any other process, including its parent, because the memory spaces are different.
Also, system is very similar to exec, but gives you much less control over the file descriptors and execution environment. You're effectively already doing a fork and exec, which is what you should be doing.
When you fork properly (as you do in your example), you now have two processes, and neither one is waiting for the other - they just run in completely different codepaths. What you basically want is to have the parent do nothing but sit around waiting for new programs to open, and occassionally check the status of the kids, while the kids run and play as long as they want.
There are IPC solutions such as pipes and message FIFO queues, but that's excessive in your case. In your case, you're just looking for process management. The parent is given the pid of the children. Save it and use it. You can call waitpid to wait for the child to end, but you don't want that. You just want the parent to check the status of the child. One way to do that is check if kill(childPid,0) == 0. If not, then the pid has exited, i.e. it's no longer running. You can also check /proc/childPid for all sorts of information.
If your status is less simple than your question implied, you'll want to look into piping after forking and execing. Otherwise, all you need is process monitoring.
Based on your EDIT 2, you're still within the domain of process management, instead of IPC. The kill command sends a signal to a process (if the command is non-0). What you're looking for is to have the parent kill(childPid, SIGTSTP). On the child side, you just need to make a signal handler, using the signal command. Among many other references, see http://www.yolinux.com/TUTORIALS/C++Signals.html. Basically, you want:
void sigTempStopHandler(int signum) { /* ... */ }
signal(SIGTSTP, sigTempStopHandler);
to be executed in the child code. The parent, of course, would know when this state is sent, so can change the status. You can use other signals for resuming when necessary.
When to pipe vs. signal:
Piping is the most robust IPC you could use - it lets you send any amount of data from one process to another, and can be in whichever direction you want. If you want your parent to send "You've been a very bad boy" to the child, it can, and the child can send "But I'll choose your nursing home one day" to the parent. (Less flippantly, you can pass any data, whether text or binary from one process to another - including objects that you serialize, or just the raw data for objects if it doesn't depend on memory, e.g. an int.)
So far, what you've described is sending simple command structures from the parent to the child, and kill is perfect for that. The child could send signals almost as easily - except that it would need to know the parent's pid to do that. (Not hard to do - before forking, save the pid: int pid = getPid();, now the child knows the parent.) Signals have no data, they're just very raw events, but so far, that sounds like all you're looking for.
Related
I am developing a program that is doing various tasks using fork(). I am starting the program, everything works fine. I observed that after some time (1 day) i get flooded with <defunct> processes, over 600 700 ... where max forks is setted to 500. This is the code :
int numforks = 0;
int maxf = 100;
// READ FROM FILE ...
while (fgets(nutt,2048,fp))
{
fflush(stdout);
if (!(fork()))
{
some_time_intensive_function();
exit(0);
}
else
{
numforks++;
if (numforks >= maxf)
{
wait(NULL);
numforks--;
}
}
}
// DON'T EXIT PROGRAM TILL ALL FORKS ARE FINISHED
while(numforks>0)
{
wait(NULL);
numforks--;
}
// CLOSE READ FILE ...
This programs keeps all the time 500 forks oped like a thread pool.
I don't really understand what <defunct> processes are, but i heard that they aren't errors in the child processes like SEG FAULT occurring, but rather parent process is not waiting correctly.
I want to get read of <defunct>s, any ideas to solve this?
I repeat, this happens after some time 1-2 days.
Thank you.
I think you have two problems:
Firstly wait can return for reasons other than a child process has terminated (and if it does, it will leave a defunct process). I think you need to pass in a non-null pointer, and inspect the returned wait status. Only decrement numforks if appropriate.
Secondlynumforks doesn't (effectively) limit the total number of child processes. If the parent process launches two processes, they will each go on to inherit numforks of 0 and 1. Then each of those child processes will launch 500 and 499 more subprocesses.
I think you need exit(0) (or break) after your time_consuming_process().
(I assume you are running on Linux, or some other POSIX system like MacOSX)
Beware of orphan processes.
Read Advanced Linux Programming which has several chapters related to your issue.
You'll better keep the result of fork (in some pid_t variable or field), and handle all three cases (>0: fork was successful; ==0, in child process, <0: fork failed!). And you should probably call waitpid(2) appropriately. In the child process it is reasonable to call exit(3) (or execve(2)...)
Perhaps you should handle SIGCHLD signal. Read carefully signal(7).
(you don't show enough of your program, and an entire book is needed to explain all that)
As a rule of thumb you don't want to have many runnable processes. On a typical laptop or desktop computer, you should not have more than a dozen of runnable processes. Use top(1) or ps(1) to list your processes (and notably to understand how many processes you have). Perhaps use (at least during debugging) bash ulimit builtin (it calls setrlimit(2) from inside your shell) in your terminal e.g. as ulimit -u 50 to limit the number of processes (to 50).
If coding in genuine C++11, you should consider using frameworks like Qt or POCO (both provide support for processes).
You should care about inter-process communication (perhaps with pipe(7)-s or socket(7)-s and some event loop, see poll(2) ...) and synchronization issues. Perhaps look into MPI or 0mq.
(you probably need to read a lot more)
Perhaps strace(1) might be helpful to debug your issues.
Don't forget to check every system call. See syscalls(2) & errno(3).
How can I measure the memory used by a child process after I call fork and exec? Basically I want to be able to write code that corresponds to the following
if (!fork()) {
// run child process
exec();
} else {
while (child active) {
print memory used by child
}
}
There are two things that I do not know here, how can I see if the child process has finished running? Will I have to use some sort of process level mutual exclusion here? If yes then what is a structure I can use? Can I just use the OS filesystem for this purpose?
Also I was looking at the answer at this link Differences between fork and exec, in paragraph 8 the author says copy on write is useful when process calls fork without calling exec. But isn't this true more in the case when the parent calls fork and does not call exec? When the parent calls exec the virtual address space of the child is wiped out and replaced with the one resulting from the new program loaded into memory correct?
Thank you!
Regarding the above comment chain which I evidently can't reply to because I don't have 50 rep:
The return value of fork in the parent if successful is the PID of the child. You should probably save the return value so you can 1. wait on the correct child (if you have more than one), and 2. see if fork fails (in which case you probably don't want to loop until the child exits ).
You could also use signals to figure out when the child dies instead of continuously trying to wait with the WNOHANG option. The process will send SIGCHLD to the parent when it terminates (or stops) and if it died then you can wait on it with waitpid and stop your loop. see:
man 7 signal
man 2 sigaction
for more information on this.
regarding memory usage, it seems you either want /proc/[pid]/statm or /proc/[pid]/stat.
man 5 proc will give you all the information about what is in those files.
I have a program (runs as a background process) in which I installed a hook to capture EVENT_SYSTEM_FOREGROUND events (i.e - when the user switches between windows). The callback which is registered for the hook basically logs what application (process exe filename) the user has switched from and which they have switched to.
I want to add some code to check if the application they have switched from is still active (if not we assume they have closed it and that is what has brought a new window into the foreground). I am testing for it's existence by trying to create a handle to the previous PID using OpenProcess
//Check prev pid still exists - if not, assume the previous app has been closed
HANDLE hPrevProc = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ,false,g_prevPid);
if (hPrevProc==NULL){
prevProcStillRunning=false;
}
else{
CloseHandle(hPrevProc);
}
Assumptions with the code above:
g_prevPid is populated with a PID - I have verified this
prevProcStillRunning has been initialized to true
The problem with the code above is that for some reason, even when the user has exited an app (say notepad.exe for example). For up to 10 seconds after they have exited, this test still passes (i.e - hPrevProc gets initialised). Even though I can see in the task manager that the Notepad.exe process has dissapeared (and yes I only have one instance of it opened), somehow, the OpenProcess line still can get a handle on that PID. I am guessing that somehow the PID actually still exists but it may be in a state where its terminating. I have found that if this code is called a few more times, eventually it will return null.
I would like to find out a better way I can test whether hPrevProc is still acitive.
I tried to test this using the GetExitCodeProcess function but this seems to just give me the PID and I'm not even sure if that's the right approach in any case.
Any help appreciated.
The process subsists in the system after it terminates at least while there is an open handle to it.
The only foolproof method to know whether a process is still active is:
make sure the process cannot exit with code STILL_ACTIVE (259)
try to open the process (OpenProcess)-> if you cannot is is terminated
read the exit process code (GetExitCodeProcess) -> if it is not STILL_ACTIVE the process is terminated.
You code could become:
//Check prev pid still exists - if not, assume the previous app has been closed
HANDLE hPrevProc = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ,false,g_prevPid);
if (hPrevProc==NULL){
prevProcStillRunning=false;
}
else{
DWORD cr;
if ((GetExitCodeProcess(hPrevProc, &cr) == 0) || (cr != STILL_ACTIVE)) {
prevProcStillRunning=false;
}
CloseHandle(hPrevProc);
}
Anyway, closing a GUI application involves different steps:
the GUI elements are destroyed
the message loop ends
eventually the application could do background operations (save state to file, etc.)
the main procedure returns an exit code
the system knows that the application is terminated
The event will be sent as soon as the main window will be closed, which can happen some time before the application actually stops. A good example for that is Firefox. If you close the window and immediately try to start a new process, you will get an error because even if the UI is gone, the process is not still terminated. What is worse, is that you can find applications that simply go into background when you close the UI, and allow user to open UI again through an action on an icon in the status area of the taskbar (Shell_NotifyIcon and its callback). This is common for services of other application working in background (network servers, firewalls, etc.). In that case, the UI is gone but the process will not terminate.
TL/DR: the time between the disparition of the UI and the termination of the process owning it is variable and depends on the system load and the background activity of the process after closing the UI. You can try to use a delay for that but I cannon guarantee anything about it...
Probably some process (maybe yours?) still holds a valid handle to this process. Until CloseHandle was called on all handles, system maintains internal record which allow to access its process data. This is important because as you say it must be possible to call GetExitCodeProcess on closed process, also someone might want to wait for it to stop with WaitForSingleObject.
Also be carefull with PIDs, they can be reused - so in theory you might call OpenProcess on some other newly opened process.
As for checking if given process is not a zombie, you might try enumerating top level windows with EnumWindows, and checking if any of them is associated with given PID (to get window's PID use GetWindowThreadProcessID).
I need to execute processes with still being in control of each process.
I want to create a class which stores the threads or pids or what ever is necessary to do so.
I currently have a program which executes one external application with the C function execvp and also loads the environment from a shell script. So my current program is blocking. But I need to be able to keep it freely running and only by time I terminate a currently running or start a new external application.
My current approach would be to create a thread, which uses the execve function. But then the thread would be blocking as far as I can see.
The code which might be in the thread (with variables then):
char *argv[] = { "/bin/bash", "-c", "myApplication", 0 };
execve(argv[0], &argv[0], environment.data());
The applications called are probably not fixed in the code their names will be given by an external setup file, including parameters.
Now my actual question, is there a better way to "manage" external applications like that in c++? Some ready solution (class, library)? And if not how do I terminate the thread if this is the actual way. Using the terminate call is said to be bad practice, that's what I often read.
I hope this is now specific enough for the forum, because I do not know how to get more specific anymore. If you need more hints what I want to create here, feel free to ask in the comments.
Update:
to DBus & others:
Additional information I do not wrote all of the processes I want to start!
So it will be used to start 3rd party applications, which even if I have the code, do not want to change.
You want to fork() before you exec. fork() is a function that creates a new process identical to the original caller of fork() running as a subprocess. The difference is that the parent process gets the child's pid as a return value and the child gets 0. The gist of what you want to do is this:
pid_t pid = fork();
if( pid == 0 )
{
// we're the child process
char *argv[] = { "/bin/bash", "-c", "myApplication", 0 };
int rc = execve(argv[0], &argv[0], environment.data());
// execve only returns if there was an error
// check 'errno' and handle it here
}
else if ( pid < 0 )
{
// pid is less than zero, we didn't successfully fork,
// there is no child process.
throw "error message";
}
// do whatever processing the parent does
More info is here. The kill() function isn't bad practice per se, if you want to quickly and gracefully end the subprocess you can write signal handlers in it, but you should be using something like dbus or zeromq to do proper interprocess communication. You want to tell the program to do something, not just tell it to die (usually what you want it to do if you're killing it).
NEVER USE execv functions in threads because the execve() system call overlays the current process image with a new process image.
The correct pattern if fork-exec or better vfork-exec. Extract from the manpage:
The vfork() system call can be used to create new processes without fully
copying the address space of the old process, which is horrendously inefficient in a paged environment. It is useful when the purpose of fork(2)
would have been to create a new system context for an execve(2). The
vfork() system call differs from fork(2) in that the child borrows the
parent's memory and thread of control until a call to execve(2) or an
exit (either by a call to _exit(2) or abnormally). The parent process is
suspended while the child is using its resources.
Using vfork shortly followed with execve, you avoid the copy of the original process image, and do not erase if with the new process, so the original process has the pid of its child and cat control it, look whether it has ended, send it signals and so on.
I am writing a shell where I need to launch several child processes at once and record the system time and user time.
So far I am able to do it. The only problem is that I am using wait4 to grab the system resources used by the child program and put it in my rusage structure called usage.
How can I launch all the processes at the same time and keep track of the user and system times? I can remove the wait4() system call and use it outside to loop so I can make the parent wait, but if I do that then I can only record the times for the last process and not all of them.
Do you have any idea how I can fix this?
execute(commandPipev,"STANDARD",0);
wait4(pid,&status,0,&usage);
printf("Child process: %s\t PID:%d\n", commandPipev[0], pid);
printf("System time: %ld.%06ld sec\n",usage.ru_stime.tv_sec, usage.ru_stime.tv_usec);
printf("User time: %ld.%06ld sec\n\n",usage.ru_utime.tv_sec, usage.ru_utime.tv_usec);
A convoluted answer.
In a POSIX environment, launch the children, then use waitid() with the WNOWAIT option to tell you that some child has exited. The option leaves the child in a waitable state - that is, you can use another wait-family call to garner the information you need. You can then use the non-POSIX wait4() system call to garner the usage information for the just exited child, and deal with the accounting you need to do. Note that you might find a different process has terminated between the waitid() and wait4() calls; you need to use a loop and appropriate flags and tests to collect all the available corpses (dead child processes) before going back to the waitid() call to find out about the other previously incomplete child processes. You also have to worry about any of the wait-family of functions returning the information for a process that was previously started in the background and has now finished.
The Linux man page for wait4(2) suggests that WNOWAIT might work directly with wait4(2), so you may be able to do it all more cleanly - if, indeed, you need the option at all.
Consider whether you can use process groups to group the child processes together, to make waiting for the members of the process group easier.