The azure continuous webjob is running but sometime it was stopped/restarted unexpectedly - azure-webjobs

Question

Our application is using a webjob to generate the data, for a moment we are facing a problem that is sometime it was stopped/restarted unexpectedly when it is processing the messages queue. It leads to our webjob don't know when it is forcing restarting/stopping to mark which data were processed then let the webjob restart/stop afterward.
Is there any idea to get the stopping/restarting notification to synchronize data?
Many thanks!

Answer 1

If you're using queues, a restarting webjob shouldn't cause you to have any data loss. Since the message will not be completed, it will be put back on the queue for (re)processing.
As far as the restarting goes: make sure you don't have any scenario's in code that break the webjob completely.
Add Application Insights and add an alert for the specific case you're looking for.
See Set Alerts in Application Insights

Answer 2

Sometimes webjobs can get killed by scale-in procedures. You can make sure they have a graceful death by listening to the shutdown event by using the class Microsoft.Azure.WebJobs.WebJobsShutdownWatcher in nuget package Microsoft.Azure.WebJobs.
As in version 1.1.2 of the nuget package:
public sealed class WebJobsShutdownWatcher : IDisposable
{
// Begin watching for a shutdown notification from Antares.
public WebJobsShutdownWatcher();
// Get a CancellationToken that is signaled when the shutdown notification is detected.
public CancellationToken Token { get; }
// Stop watching for the shutdown notification
public void Dispose();
}
A way to use this: in your webjob Program.cs class you get a cancellation token and write the code you want to be executed when shutdown happens.
private static void Main()
{
...
var cancellationToken = new WebJobsShutdownWatcher().Token;
...
cancellationToken.Register(() =>
{
//Your data operations here
});
...
}

Answer 3

Thank Diana for your information. I tried this approach but it was not work very well, webjob is just waiting for 5 seconds before restarting/stopping although I set 60 seconds in the settings.job file. Here is my code below
static void Main()
{
var config = new JobHostConfiguration();
var host = new JobHost();
var cancellationToken = new WebJobsShutdownWatcher().Token;
cancellationToken.Register(() =>
{
//Raise the signal
});
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}