C++ Linux Service - How to use init.d + auto restart? - c++

I've created a linux c++ service (It's basically an app, but it handles requests over TCP/IP quite frequently).
I was wondering if there is any easy way to have it "auto restart" if something goes wrong (like it crashes) or if the server restarts?
I wasn't sure how or even if I should set it up as a service or set up an rc.d script, I'm not 100% familiar w/how to do this on linux (my server is running ubuntu if it matters).
Any advice would be greatly appreciated!
~ Josh

A lot of answers here suggest having a 'parent app' that does it, but you end up with the same problem with the parent app - it's turtles all the way down.
In many unix type systems (especially historically), The init process is the first process that executes, and will execute (and restart automatically) processes as defined in /etc/inittab.
So instead of writing your own watchdog or process to auto-restart - you can use this one that does the job for you automatically, and since it's the init process, if it dies, the system has a lot more to worry about than your service.
#doron suggests another good approach, if your service should spawn a new process for every incoming connection, and only does work when it has an incoming connection.
Finally, these days the init process (and /etc/inittab) has been replaced on Ubuntu type systems with upstart - http://upstart.ubuntu.com/ - a more flexible system for the same thing.

In my product, I've created watchdog process which forks and exec service process in separate process, and waits for its termination. If, for some reason, process terminates, watchdog process will create another thread and it will start process again.
As noted in comments, you should check why it is crashed. For start, you could read program exit value.
Here is simple program to get you started:
#include <stdio.h>
#include <unistd.h>
#include <sys/wait.h>
int main()
{
create_process();
return 0;
}
void create_process()
{
int exit_code;
if(fork() == 0)
{
exec("./your_service");
}
else
{
wait(&exit_code);
if(WIFEXITED(exit_code))
{
/* Program terminated with exit */
/* If you want, you could decode exit code here using
WEXITSTATUS and you can start program again.
*/
return;
}
else
` {
/* Program didn't terminated with exit, restart */
create_process();
}
}
}
In order to start service on system startup, simply edit the /etc/rc.local script and append command for running your watchdog process.

Create a control app which starts and restarts it if necessary.
Do this in your app - fork a child, run the program there, catch stop/crash and fork new child if necessary. Some working code can be found here: monitoring the main app in c .

You might want to take a look at using the Inet Daemon. The inet daemon start a new process every time a new request comes in. So if there is a crash in your server, it just gets restarted when the next request comes in.

The simplest way to write a parent app which auto-restarts a child process, on *NIX, is just using a shell script:
#!/bin/sh
while true;
do
run_my_program;
done
You can optionally have this redirect output, run itself in the background, etc.
This doesn't address starting the process in the first place, but it's less work (for exactly the same result) as writing a parent process in C++.

Related

Using boost::process to completely detach a process

I have a systemd service which runs and does its thing. Periodically I need it to upgrade itself, which requires a shutdown and a restart of the service. For question purposes the upgrade script can be as simple as:
echo "Stopping service..."
systemctl stop myservice
echo "Doing some stuff..."
sleep 10s
echo "Starting service..."
systemctl start myservice
I want to call this within the service itself, preferably using boost::process:
boost::process::child instexe{
boost::process::search_path("bash"),
std::vector<std::string>{"installerscript.sh"},
boost::process::start_dir("/installer/folder"),
boost::process::std_out > "/some/log/file.txt"
};
instexe.detach();
The problem is that as soon as the script calls systemctl stop myservice, the installer script is killed.
Is there a way I can do what I want to do with boost::process? Or how can I do it?
If the upgrades are at predefined period you can think of using crontab.
https://opensource.com/article/17/11/how-use-cron-linux
00 09-17 * * 1-5 /usr/local/bin/installerScript.sh
The above entry in crontab will make the program upgrade every hour between 9 am to 5pm from Monday to Friday. There are many combinations that you can think and configure.
Is there a way I can do what I want to do with boost::process? Or how can I do it?
If you have the child process killing the parent, there's always going to be a race condition by definition.
The quick hack is to put a sleep statement at the start of the installer script, but the correct solution is to explicitly synchronize with the child:
have the installer script detect whether it's running interactively (ie, being run manually from a terminal instead of by your service)
if it is non-interactive (your use case), have it wait for some input in stdin
connect the stdin pipe when you create the child
detach the child and then write something to tell the child it's safe
Other synchronization mechanisms are available, you could use a lockfile or a signal - you just need to make sure the child doesn't do anything until after the parent has detached it.
I turns out (from this question, which leads to the excellent-but-unfindable systemd.kill manpage) that systemd has four different ways of stopping a unit, controlled by the KillMode variable in your unit configuration:
control-group will send SIGTERM (by default, overridable with KillSignal) to every process in the unit's cgroup. That means both parent and child.
mixed will send SIGTERM (or KillSignal) to your main process and SIGKILL to the child.
process will kill only the main process and leave the child alone
none is not recommended, it will just run your ExecStop procedure
You can probably just set KillMode=process, but note that if SendSIGKill or SendSIGUP are true, those signals will still be delivered to your child after TimeoutStopSec.
It seems like it might be simpler to restart your service and have a launch script that can update it at startup, or to perform the update in your ExecStop procedure, than to persuade systemd to leave the child alone until the update is complete, without the risk of a hung child updater hanging around forever.
Either way, your remaining problems are exclusively with systemd rather than with boost.Process.

How to restart QtSingleApplication?

I can restart my application with
QProcess::startDetached(QApplication::applicationFilePath());
exit(12);
But if I have QtSingleApplication new instance will not run. Is there way to trick QtSingleApplication to allow it to run new instance when isRunning() returns true maybe by some command line arguments or another aspects?
I'll be fully fine if I will add Sleep(5000); in the beginning but it doesn't look alike good design, I think I just need an additional mutex to detect if my application is gone :S what do you think?
When you want to start your app as new running instance, pass a specific argument to signal it should close any existing rather the usual opposite.
This can be handled as
//code to send to instances
if(myturn)
{
if (a.isRunning()) {
QString rep("Another instance is running, so I will ask to leave.");
bool sentok = a.sendMessage(message_gtfo,1000);
if(!sentok)
return "some error";
}
}
//code to receive message from instance
connect(&a, SIGNAL(messageReceived(const QString&)),
&this, SLOT(handlMessage(const QString &))
);
...
void instancecomhandler::handlMessage(const QString & msg)
{
if(msg == message_gtfo)
//proceed to exit
}
edit:
You still need to pass the argument to modify the behavior. Default behavior quit when another instance is running. New behavior waits till the other instance terminate.
One way to do such a thing is to use a little helper application (or script). The helper application is started detached and receives an identifier (ie PID) of the currently running program as a command line argument. It then waits until that identifier has disappeared from the process table, then restarts the application and exits.
This is a bit hacky/racy, but it works okay for me in practise. I use this in an update system, where I need to replace the running program with a new version of itself, which on Windows cannot be done while the program is running.
If you are restarting your application from inside itself, it is by definition no more a single-process application (since for a short time at least, two processes running the same application exist).
I would recommend using some shell tricks to restart it.
On Linux with a X11 desktop, you might e.g. popen(3) some at (or batch) command which would restart your application. You'll probably need to explicitly set the DISPLAY environment variable.
On Linux or other POSIX systems, a possible way would be to wrap the application in a shell script (for example mywrapper) like e.g.
#! /bin/sh
# script file mywrapper
aftersource=$(tempfile -p app)
myapp --aftersource $aftersource "$#"
source $aftersource
rm $aftersource
You'll need to improve the above script to handle failures and trap some signals... I guess you might adapt that script to Windows if so needed
Then your application would process the --aftersource argument to write some shell commands in it. If it want to restart itself it would e.g. write some command into that file to do that, etc.
Feel free to improve the script (trapping some signals and exit, looping, etc...)

Restarting a linux daemon

I have Linux daemon that I have written in C++ that should restart itself when given a "restart"-command from a user over the network through its console. Is this possible? I use a /etc/init.d script. How can I program it to restart itself? Should I launch a new process with a very long delay (one minute) that then fires the shell script again ? Problem is that the daemon may take a very long time to close down and it could take even more than a minute in a worst-case scenario.
There are basically three ways for an application to restart itself:
When the application is told to restart, it does proper clean-up, releases all resources it has allocated, and then re-initializes like it was started from scratch.
Fork a new process, where the new child process execs itself and the parent process exits normally.
The daemon is actually just a wrapper application, much like an init-script. It forks a new process which runs the actual application, while the parent process just waits for it to exit. If the child process (and the real application) returns with a special exit-code, it means that it should be restarted so the forks/execs all over again.
Note that points 2 and 3 are basically the same.
Break down the restart as two steps, stop and start. if your program takes time to stop, it should be handled in the stop function, I can't comment on specifics since I don't know your usecase, but I'd imagine monitoring the process to check if it's terminated will be a graceful way to stop
Do whatever shut-down/clean-up you need to do, then call this:
execl( argv[0], argv, reinterpret_cast< char* >( 0 ) );
Just like fork() and exec(), but skipping the fork. The exec will replace the current process with a new copy of itself. cf. http://linux.die.net/man/3/exec
Your init script should just kill your daemon and start it again. Don't try to restart your daemon FROM your daemon.

C++ Having Windows Service Start another Program

Is it possible to create a windows service to create and maintain another process? Like I'm writing a program, and say a virus killed the process, could I have my window service running and basically 'watching' it? I already have the code for a regular application that stays running and executes a program if it's not currently running, to keep it running.
I've never written a service before, but would it be that hard to just write this simple program, which basically runs a check to see if the process is running, if not, it executes it and sleeps for a few minutes?
Thanks.
Yes, it is possible. It is not uncommon to see third-party apps have watchdog services to keep them running in case of crashes. A service can enumerate running processes using EnumProcesses(), and if the desired executable is not running then start a new copy of it using CreateProcessAsUser().
If the service is the one starting the executable process in the first place, or can find it after an enumeration, one optimization would be to keep an open handle to the process (returned by CreateProcess...(), or use OpenProcess() on the process ID an enumeration returns), and then use a wait function, like WaitForSingleObject(), to detect when the process stops running. That way, you don't have to enumerate processes to find out if the intended process is still running or not.

Restart application on unhandled exception

Is it possible to have a program restart automatically if it crashes?
Something like:
An unhandled exception is thrown.
Release all resources allocated by process.
Start over and call main.
I would like this behavior for a server application I'm working on. If clients miss use the server it can get a std::bac_alloc exception, in which case I would like the server to simply restart instead of crashing and shutting down, thus avoiding manual startup.
I've done this before in Windows by running said program from another program via a win32 CreateProcess call. The other program then waits on the "monitored" process to exit, and calls its CreateProcess() again if it does. You wait for a process to exit by performing a WaitForSingleObject on the process' handle, which you get as one of the return values from your CreateProcess() call.
You will of course want to program in some way to make the monitoring process shut itself and its child process down.
Let Windows be your watchdog. You can call ChangeServiceConfig2 to set the failure actions for your service. (If your server isn't a service, then you're doing it wrong.) Specify SERVICE_CONFIG_FAILURE_ACTIONS for the dwInfoLevel parameter, and in the SERVICE_FAILURE_ACTIONS structure, set lpsaActions to an array of one or more SC_ACTION values. The type you want is SC_ACTION_RESTART.
I did something similar by implementing a watchdog. The watchdog ran as a service and would wait for a ping (called petting the dog) from the monitored process. If the monitored process died due to an exception, watchdog would cleanup and relaunch the application.
In case the application was not responding(no ping in a certain time) the watchdog would kill it and then restart it.
Here is a link to an implementation that you might want to use:
http://www.codeproject.com/KB/security/WatchDog.aspx
(PS: I implemented my own version but I cannot post it here. I found this from a quick google search and have no first hand experience with this particular implementation.)
If you just catch the exception, it should be possible to just restart your server by internal programming logic without completely restarting the whole program.
Like #T.E.D., we've done this in an application we built. Our application is a windows service, so the helper program stops the service (eventually kill it, if it hangs) and start the service again.