I run some of my C++-programs on a HPC, scheduled using SLURM. Sometimes my programs get killed, either because they are using too many resources, or because they run too long. Usually, if my program is finished running, or encounters an internal error, I get a message telling me that fact, and I can apply appropriate actions.
But if my program is killed by the queue manager, I do not get any messages (and yes, I specified that I would like to get those messages in the job file, but somehow that does not work properly). Thus I was wondering if there is a possibility to call a function within the program when encountering the kill signal, or another way to tell me when my main program is killed?
Maybe you could look the other way around. Stop your program a bit before its end, and do your duty to produce a clean exit. With slurm you can use :
#SBATCH --signal=B:USR1#120
to send a signal to your bash script 120 seconds prior its job limit. Just trap this signal and produce a clean exit.
I use it, and it works very well.
#!/bin/bash -l
# job name
#SBATCH --job-name=example
# replace this by your account
#SBATCH --account=...
# one core only
#SBATCH --ntasks=1
# we give this job 4 minutes
#SBATCH --time=0-00:04:00
# asks SLURM to send the USR1 signal 120 seconds before end of the time limit
#SBATCH --signal=B:USR1#120
# define the handler function
# note that this is not executed here, but rather
# when the associated signal is sent
your_cleanup_function()
{
echo "function your_cleanup_function called at $(date)"
# do whatever cleanup you want here
}
# call your_cleanup_function once we receive USR1 signal
trap 'your_cleanup_function' USR1
echo "starting calculation at $(date)"
# the calculation "computes" (in this case sleeps) for 1000 seconds
# but we asked slurm only for 240 seconds so it will not finish
# the "&" after the compute step and "wait" are important
sleep 1000 &
wait
Those lines were extracted from here
Related
I have a systemd service which runs and does its thing. Periodically I need it to upgrade itself, which requires a shutdown and a restart of the service. For question purposes the upgrade script can be as simple as:
echo "Stopping service..."
systemctl stop myservice
echo "Doing some stuff..."
sleep 10s
echo "Starting service..."
systemctl start myservice
I want to call this within the service itself, preferably using boost::process:
boost::process::child instexe{
boost::process::search_path("bash"),
std::vector<std::string>{"installerscript.sh"},
boost::process::start_dir("/installer/folder"),
boost::process::std_out > "/some/log/file.txt"
};
instexe.detach();
The problem is that as soon as the script calls systemctl stop myservice, the installer script is killed.
Is there a way I can do what I want to do with boost::process? Or how can I do it?
If the upgrades are at predefined period you can think of using crontab.
https://opensource.com/article/17/11/how-use-cron-linux
00 09-17 * * 1-5 /usr/local/bin/installerScript.sh
The above entry in crontab will make the program upgrade every hour between 9 am to 5pm from Monday to Friday. There are many combinations that you can think and configure.
Is there a way I can do what I want to do with boost::process? Or how can I do it?
If you have the child process killing the parent, there's always going to be a race condition by definition.
The quick hack is to put a sleep statement at the start of the installer script, but the correct solution is to explicitly synchronize with the child:
have the installer script detect whether it's running interactively (ie, being run manually from a terminal instead of by your service)
if it is non-interactive (your use case), have it wait for some input in stdin
connect the stdin pipe when you create the child
detach the child and then write something to tell the child it's safe
Other synchronization mechanisms are available, you could use a lockfile or a signal - you just need to make sure the child doesn't do anything until after the parent has detached it.
I turns out (from this question, which leads to the excellent-but-unfindable systemd.kill manpage) that systemd has four different ways of stopping a unit, controlled by the KillMode variable in your unit configuration:
control-group will send SIGTERM (by default, overridable with KillSignal) to every process in the unit's cgroup. That means both parent and child.
mixed will send SIGTERM (or KillSignal) to your main process and SIGKILL to the child.
process will kill only the main process and leave the child alone
none is not recommended, it will just run your ExecStop procedure
You can probably just set KillMode=process, but note that if SendSIGKill or SendSIGUP are true, those signals will still be delivered to your child after TimeoutStopSec.
It seems like it might be simpler to restart your service and have a launch script that can update it at startup, or to perform the update in your ExecStop procedure, than to persuade systemd to leave the child alone until the update is complete, without the risk of a hung child updater hanging around forever.
Either way, your remaining problems are exclusively with systemd rather than with boost.Process.
I'm using perf for profiling on Ubuntu 20.04 (though I can use any other free tool). It allows to pass a delay in CLI, so that event collection starts after a certain time since program launch. However, this time varies a lot (by 20 seconds out of 1000) and there are tail computations which I am not interested in either.
So it would be great to call some API from my program to start perf event collection for the fragment of code I'm interested in, and then stop collection after the code finishes.
It's not really an option to run the code in a loop because there is a ~30 seconds initialization phase and 10 seconds measurement phase and I'm only interested in the latter.
There is an inter-process communication mechanism to achieve this between the program being profiled (or a controlling process) and the perf process: Use the --control option in the format --control=fifo:ctl-fifo[,ack-fifo] or --control=fd:ctl-fd[,ack-fd] as discussed in the perf-stat(1) manpage. This option specifies either a pair of pathnames of FIFO files (named pipes) or a pair of file descriptors. The first file is used for issuing commands to enable or disable all events in any perf process that is listening to the same file. The second file, which is optional, is used to check with perf when it has actually executed the command.
There is an example in the manpage that shows how to use this option to control a perf process from a bash script, which you can easily translate to C/C++:
ctl_dir=/tmp/
ctl_fifo=${ctl_dir}perf_ctl.fifo
test -p ${ctl_fifo} && unlink ${ctl_fifo}
mkfifo ${ctl_fifo}
exec ${ctl_fd}<>${ctl_fifo} # open for read+write as specified FD
This first checks the file /tmp/perf_ctl.fifo, if exists, is a named pipe and only then it deletes it. It's not a problem if the file doesn't exist, but if it exists and it's not a named pipe, the file should not be deleted and mkfifo should fail instead. The mkfifo creates a named pipe with the pathname /tmp/perf_ctl.fifo. The next command then opens the file with read/write permissions and assigns the file descriptor to ctl_fd. The equivalent syscalls are fstat, unlink, mkfifo, and open. Note that the named pipe will be written to by the shell script (controlling process) or the process being profiled and will be read from the perf process. The same commands are repeated for the second named pipe, ctl_fd_ack, which will be used to receive acknowledgements from perf.
perf stat -D -1 -e cpu-cycles -a -I 1000 \
--control fd:${ctl_fd},${ctl_fd_ack} \
-- sleep 30 &
perf_pid=$!
This forks the current process and runs the perf stat program in the child process, which inherits the same file descriptors. The -D -1 option tells perf to start with all events disabled. You probably need to change the perf options as follows:
perf stat -D -1 -e <your event list> --control fd:${ctl_fd},${ctl_fd_ack} -p pid
In this case, the program to be profiled is the the same as the controlling process, so tell perf to profile your already running program using -p. The equivalent syscalls are fork followed by execv in the child process.
sleep 5 && echo 'enable' >&${ctl_fd} && read -u ${ctl_fd_ack} e1 && echo "enabled(${e1})"
sleep 10 && echo 'disable' >&${ctl_fd} && read -u ${ctl_fd_ack} d1 && echo "disabled(${d1})"
The example script sleeps for about 5 seconds, writes 'enable' to the ctl_fd pipe, and then checks the response from perf to ensure that the events have been enabled before proceeding to disable the events after about 10 seconds. The equivalent syscalls are write and read.
The rest of the script deletes the file descriptors and the pipe files.
Putting it all together now, your program should look like this:
/* PART 1
Initialization code.
*/
/* PART 2
Create named pipes and fds.
Fork perf with disabled events.
perf is running now but nothing is being measured.
You can redirect perf output to a file if you wish.
*/
/* PART 3
Enable events.
*/
/* PART 4
The code you want to profile goes here.
*/
/* PART 5
Disable events.
perf is still running but nothing is being measured.
*/
/* PART 6
Cleanup.
Let this process terminate, which would cause the perf process to terminate as well.
Alternatively, use `kill(pid, SIGINT)` to gracefully kill perf.
perf stat outputs the results when it terminates.
*/
In a Linux/C++ library I'm launching a process via the system() call,
system("nohup processName > /dev/null&");
This seems to work fine with a simple test application that exits on it's own, but if I use this from inside of a Nodejs/V8 extension which gets a kill signal, the child process gets killed. I did find that running,
system("sudo nohup processName > /dev/null&");
With the sudoers file set up to not require a password manages to make this run even when the parent process (node) exits. Is there someway to entirely detach the child process so signals sent to the parent and the parent exiting have no effect on the child anymore? Preferably within the system() call and not something that requires getting the process ID and doing something with it.
The procedure to detach from the parent process is simple: Run the command under setsid (so it starts in a new session), redirecting standard input, output and error to /dev/null (or somewhere else, as appropriate), in background of a subshell. Because system() starts a new shell, it is equivalent to such a subshell, so
system("setsid COMMAND </dev/null >/dev/null 2>/dev/null &");
does exactly what is needed. In a shell script, the equivalent is
( setsid COMMAND </dev/null >/dev/null 2>/dev/null & )
(Shell scripts need a subshell, because otherwise the COMMAND would be under job control for the current shell. That is not important when using system(), because it starts a new shell just for the command anyway; the shell will exit when the command exits.)
The redirections are necessary to make sure the COMMAND has no open descriptors to the current terminal. (When the terminal closes, a TERM signal is sent to all such processes.) This means standard input, standard output, and standard error all must be redirected. The above redirections work in both Bash and POSIX shells, but might not work in ancient versions of /bin/sh. In particular, it should work in all Linux distros.
setsid starts a new session; the COMMAND becoming the process group leader for its own process group. Signals can be directed to either a single process, or to all processes in a process group. Termination signals are usually sent to entire process groups (since an application may technically consist of multiple related processes). Starting a new session makes sure COMMAND does not get killed if the process group the parent proces belongs to is killed by a process-group wide signal.
My guess is that the whole process group is being killed. You could try setpgid in the child to start a new process group. The first step should be to get rid of system and use fork and execve or posix_spawn.
I'm currently in the process of building a small shell within C++.
A user may enter a job at the prompt such as exe1 && exe2 &. Similar to the BASH shell, I will only execute exe2 if exe1 exits successfully. In addition, the entire job must be performed in the background (as specified by the trailing & operator).
Right now, I have a jobManager which handles execution of jobs and a job structure which contains the job's executable and their individual arguments / conditions. A job is started by calling fork() and then calling execvp() with the proper arguments. When a job ends, I have a signal handler for SIGCHLD, in which I perform wait() to determine which process has just ended. When exe1 ends, I observe its exit code and make a determination as to whether I should proceed to launch exe2.
My concern is how do I launch exe2. I am concerned that if I use my jobManager start function from the context of my SIGCHLD handler, I could end up with too many SIGCHLD handler functions hanging out on the stack (if there were 10 conditional executions, for instance). In addition, it just doesn't seem like a good idea to be starting the next execution from the signal handler, even if it is occurring indirectly. (I tried doing something similar 1.5 years ago when I was just learning about signal handling -- I seem to recall it failing on me).
All of the above needs to be able to occur in the background and I want to avoid having the jobManager sitting in a busy wait just waiting for exe1 to return. I would also prefer to not have a separate thread sitting around just waiting to start the execution of another process. However, instructing my jobManager to begin execution of the next process from the SIGCHLD handler seems like poor code.
Any feedback is appriciated.
I see two ways:
1)Replace you sighandler with loop that call "sigwait" (see man 3 sigwait)
then in loop
2)before start create pipe, and in mainloop of your program use "select" on pipe handle to wait
events. In signal handler write to pipe, and in mainloop handle situation.
Hmmm that's a good one.
What about forking twice, once per process? The first one runs, and the second one stops. In the parent SIGCHLD handler, send a SIGCONT to the second child, if appropriate, which then goes off and runs the job. Naturally, you SIGKILL the second one if the first one shouldn't run, which should be safe because you won't really have set anything up.
How does that sound? You'll have a process sitting around doing nothing, but it shouldn't be for very long.
I'm trying to test a Bash script which copies files individually and does some stuff to each file. It is meant to be resumable, so I'd like to make sure to test this properly. What is an elegant solution to kill or otherwise abort the script which does the copying from the test script, making sure it does not have time to copy and process all the files?
I have the PID of the child process, I can change the source code of both scripts, and I can create arbitrarily large files to test on.
Clarification: I start the script in the background with &, get the PID as $!, then I have a loop which checks that there is at least one file in the target directory (the test script copies three files). At that point I run kill -9 $PID, but the process is not interrupted - The files are copied successfully. This happens even if the files are big enough that creating them (with dd and /dev/urandom) takes a couple seconds.
Could it be that the files are only visible to the shell when cp has finished? It would be a bit strange, but it would explain why the kill command is too late.
Also, the idea is not to test resuming the same process, but cutting off the first process (simulate a system crash) and resuming with another invocation.
Send a KILL signal to the child process:
kill -KILL $childpid
You can try an play the timing game by using large files and sleeps. You may have an issue with the repeatability of the test.
You can add throttling code to the script your testing and then just throttle it all the way down. You can do throttling code by passing in a value which is:
a sleep value for sleeping in the loop
the number of files to process
the number of seconds after which the script will die
a nice value to execute the script at
Some of these may work better or worse from a testing point of view. nice'ing may get you variable results, as will setting up a background process to kill your script after N seconds. You can also try more than one of these at the same time which may give you the control you want. For example, accepting both a sleep value and the kill seconds could give you fine grained throttling control.