We have an Azure Website setup with a "staging" deployment slot, and we use a continuous Azure WebJob to process long-running background jobs. It appears that when you publish to the Website (and include the WebJob binaries), that the WebJob will restart itself if it was previously stopped.
Our current deployment process looks like the following:
Deploy to STAGING
Immediately & Quickly stop the WebJob on
STAGING (try to prevent 2 WebJobs from processing messages)
Smoke test and verify code on STAGING works
Stop the PRODUCTION WebJob (let it drain off queue messages)
Swap between PRODUCTION and STAGING
Start the PRODUCTION WebJob (which was just the STAGING WebJob)
Is there a trick (say a flag file or something) to tell a WebJob NOT to start up after publishing?
To deploy a continuous WebJob in a stopped state simply add a file called disable.job at the root of your WebJob (binaries), this will tell the framework that the WebJob is currently stopped.
To view this behavior you can simply stop a continuous WebJob and see that this file is generated and placed at the WebJob's directory.
You can add Application Setting WEBJOBS_STOPPED to your staging slot and set it to 1. Make this setting 'Slot Setting' so that it is not swapped with production.
AFAIK, there is no out of the box solution for that. However, you can set an app setting and have the webjob code check for it when it starts. Also, you can use the HTTP_HOST variable to figure out the environment.
Related
I have a super small C# .NET Framework web application that I run on AWS Beanstalk. It works fine, except for one peculiar thing:
AWS seems to restart the service with unregular interval. I can see this as I can see log prints that only happens on startup.
In this case I am using ServiceStack AppHostBase, which has a Configure method that should be executed only once, on startup. In this method, a log print is done, and this gets executed between every 15 min to 2 hours approximately, indicating that the service is restarted at those times:
No update of the build has been done at those times seen above.
The Beanstalk configuration looks like this:
I would like to know why it restarts and what I can do to stop it.
I have a AWS pipeline, which:
1) first stage, get template.yaml and build a ec2 windows instance via script
note when this machine boots up, via user data it starts a script to downloads requirements, git etc, code, setups iis and various other stuff.
so this happens once the cloudformation part has completed, and takes about another 5 mins
2) i then want to run external tests on this machine - maybe using blazemeter, as the second part of the pipeline
the problem is that between stage 1 and 2 i need to wait for the website to work on the box, so i need to wait at least 5 mins. i could add a manual approval stage, but this seams cumbersome.
does anyone have a way to add this timed wait? or a pipeline process to check the site is up?
say I have this:
Step 1: A azure webjob triggered by a timer, and this job will create 1000 messages and I will put them in a queue.
Step 2: I have another azure webjob triggered by above message queue, this webjob will process these messages.
Step 3: The final webjob should only be triggered when all messages have been processed by step 2.
Looks like azure Queue doesn't support ordering and the only way is to use ServiceBus. I am wondering is it really the only way?
What I am thinking is this kind of process:
Put all these messages into an azure table, with some guid as primary key and status to be 0.
after finishing step 2, change the status of this message to 1 (i.e. finished) and will trigger step 3 if every messages have been done.
Will it work? Or maybe there are some nuget packages that I can use to achieve what I want?
The simplest way I think is the combination of Azure Logic App and Azure Function.
Logic App is a automated scalable workflow, you could trigger it with Timer, HTTP request and etc. And Azure function is a serverless compute service that enables you to run code on-demand without having to explicitly provision or manage infrastructure.
The Logic App support add code with Function, as for the use of Function, It's similar with WebJob. So I think you could create a Logic App with three Function, the they will run one by one.
As for the WebJob, yes,the QueueTrigger doesn't support ordering. And the Service Bus you mentioned, It did meet your some requirements for its FIFO feature. However you need to make sure your step 3 would be triggered after step 1, because it's already null in your queue before creating queues.
Hope my answer could help you.
I have a WebApp with a WebJob (monitoring a queue with QueueTrigger) in Azure that has been working fine for over a year. I'm trying to re-organize some of my Azure resources and would like the WebJob to monitor a different Queue than it has been.
I've created a new (non-classic) storage account and changed my code to insert new messages into the new Queue. I can see new messages showing up in the new Queue but the WebJob is never triggered. (The old queue is in a classic storage account. I don't think that matters, but it has crossed my mind. I have seen some older posts that make me think this used to be a problem, but some newer ones that make me think it's OK.)
My code is pretty straightforward (almost straight out of tutorial). It wants the Queue connection strings in app.config for both AzureWebJobsDashboard and AzureWebJobsStorage, which I have done.
var host = new JobHost();
host.RunAndBlock();
To verify that the new code is being successfully deployed, I have deleted the old WebJob in the Azure portal and verified the file dates in the App_Data\Jobs... folder are current and I've looked at the value of "AzureWebJobsStorage" in the deployed config file and it is the new Queue's connection string.
I finally thought to manually insert a message into the old queue (that none of my code is pointing at any more). Sure enough - when I do that, the WebJob is triggered and runs.
I think changing the connection string values in App.Config should be all that's needed to have it "watch" a new queue, but that doesn't seem to be enough. Does anyone know what else would need to be changed?
So I figured this out. Not only do you have to have the Connection String entries (pointing to the queue) in the App.Config of the WebJob project, but also for the Web Application. I had configured AzureWebJobsDashboard and AzureWebJobsStorage in the Application Settings section of the Web App in the Azure portal.
Those were still pointing to the old queue, and apparently those were the important ones. Once I updated those settings to point to the new queue, it worked as it should.
What does "Pending Restart" mean? I have stopped and restarted my WebJob numerous times and that doesn't seem to fix it. Does it mean I have to restart my website? What caused my job to get in this state in the first place? Is there any way I can prevent this from happening in the future?
Usually, it means that the job fails to start (an exception?). Look in the jobs dashboard for logs.
Also, make sure that if the job is continuous, you actually have an infinite loop that keeps the process alive.
To add to Victor's answer, the continuous WebJob states are:
Initializing - The site was just started and the WebJob is doing it's initialization process.
Starting - The WebJob is starting up the process/script.
Running - The WebJob's process is running.
PendingRestart - The WebJob's process exited (for any good or bad reason) in less than 2 minutes since it started, for a continuous WebJob it's considered that something was probably not right with it (some exception during start-up probably as mentioned by Victor), at this point the system is waiting for 60 seconds before it'll restart the WebJob process (hence the name "pending restart").
Stopped - The WebJob was stopped (usually from the Azure portal) and is currently not running and will not be running until it is started again, best way to see this is as disabled.
Also, take a look at the webjob log, it should hold cue to what's been happening.
if the JOB is set to run continuously, once the process exits (say you are polling a queue and it's empty) the job shuts down and status changes to "pending restart". Azure Scheduler will typically restart the process in 60 seconds.
Try changing the target framework to .NET 4.5. This same issue was fixed for me when I changed the target framework from 4.6.1 to 4.5.
Had the same problem, found out that i need to keep my webjob alive, so I put a continous loop to keep it alive.
It means that application is failing after start. Check the App Service Application setting might have some problem. .. In my case i am passing date as a configuration and i entered wrong date like 20160431
This is just because Webjobs is failing or giving exception.
Make sure the Webjobs is continuous and you can check that in the log where it failing and can make the changes.
Process went down, waiting for 60 seconds
Status changed to Pending Restart
In my case, we got the deployment package prepared through the Visual Studio folder publish and deployed along with WebApp. Package created through 'Folder publish' lacked 'run.cmd' file in which command to invoke the console application (.exe) is available; this file is automatically created when we directly publish to Azure WebJob from Visual Studio.
After manually adding this to a package folder, the issue got fixed.