I am having a webjob which needs to run at regular intervals but having it as continuous with cron expression rather than triggered type.
Will it make any difference especially in terms of any performance?
Any help in understanding this very much appreciated
Continuous webjob for a schedule run: It will starts immediately when the WebJob is created and run on the schedule.
Triggered webjob for a scheduled run: Starts only when triggered on a schedule.
If your app runs continuous or scheduled WebJobs, enable Always On to ensure that the WebJobs run reliably. This feature is available only in the Basic, Standard, and Premium pricing tiers.
Related
I have a Cloud Scheduler that triggers a Cloud Run Job to run and parallelize the payload into 3 tasks using the CLOUD_RUN_TASK_INDEX environment variable.
I would like to trigger another Cloud Run Job or Cloud Function after the previous Cloud Run Job finishes all the 3 tasks. I looked up in the Google documentation, but I could not find any reference in order to accomplish this. How can this be done?
In your case, the most efficient (IMO) is to use Cloud Workflows with the Cloud Task connectors. The connector run the task and wait for completion. And then you can continue the workflow with different tasks that you want to execute.
Cloud Workflows also support parallel execution
Another idea could be to use Eventarc to catch the audit logs that mention the end of the job processing. But I prefer the first solution.
I am using google cloud dataflow to execute some resource intensive dataflow jobs. And at a given time, my system must execute no more than 2 jobs in parallel.
Since each job is quite resource intensive, I am looking for a way to trigger an alert when more than 2 dataflow jobs are running.
I tried implementing a custom_count which increments after the start of each job. But custom_couter only display after the job has executed. And it might be too late to trigger an alert by then.
You could modify the quota dataflow.googleapis.com/job_count of the project to be limited to 1, and no two jobs could run parallel in that project. The quota is at the project level, it would not affect other projects.
Another option is to use an GCP monitoring system that is observing the running Dataflow jobs. You can e.g. use Elastic Cloud (available via Marketplace) to load all relevant Metrics and Logs. Elastic can visualize and alert on every state you are interested in.
I found this terraform project very helpful in order to get started with that approach.
I was wondering what would be the recommended way to host a long running scheduled task in AWS.
Currently we have an EC2 instance (windows) in charge of triggering our app every few hours. This task will take between 1-3 hours depending on the number of items to process.
Lambda does not seem to be appropriate since my task is too long
Found this topic about Hangfire Scheduled Jobs in .NET Core 2 Web app hosted in AWS. Seems good but outside of AWS.
Any suggestions?
Thx
Seb
I would recommend AWS Step Functions. Very easy to implement. Part of the AWS Serverless Platform.
AWS Step Functions
AWS Step Functions makes it easy to coordinate the components of
distributed applications and microservices using visual workflows.
Building applications from individual components that each perform a
discrete function lets you scale and change applications quickly. Step
Functions is a reliable way to coordinate components and step through
the functions of your application. Step Functions provides a graphical
console to arrange and visualize the components of your application as
a series of steps. This makes it simple to build and run multistep
applications. Step Functions automatically triggers and tracks each
step, and retries when there are errors, so your application executes
in order and as expected. Step Functions logs the state of each step,
so when things do go wrong, you can diagnose and debug problems
quickly. You can change and add steps without even writing code, so
you can easily evolve your application and innovate faster.
Using AWS Lambda with Scheduled Events would let you create an AWS Lambda that will respond to a scheduled event. This Lambda could then trigger your app. Your app doesn't need to be in a Lambda itself.
I am trying to deploy a triggered webjob and schedule it with settings.json file. My webapp runs in 3 instances so I assume the webjob will be deployed in 3 instances.
So if I schedule the webjob with settings.json file with cron expression will the job run in all the instances? I am assuming that setting the "is_singleton" to true will make it run only on one instance? But if the instance in which it is running fails, will Azure failover the webjob in spite of the is_singleton setting?
You shouldn't need to use is_singleton in this case. For each schedule occurence, the runtime will pick an instance at random and run your job. Each schedule occurrence will run on one instance only, as you would expect.
If a schedule occurrence is missed or fails, the runtime will try to get back on schedule by running the function again. You can see that logic in the source here.
I am new to AWS and have a question about an application I'm trying to write. I have a bunch of data that sits within Amazon RDS. On a periodic basis, I would like a small snippet of code to run against this data and in certain situations have notifications sent where appropriate. Of all the AWS services, what is the best architecture for this?
Thanks
You could use a simple cron job running on an EC2 instance. The cron job could run a script (PHP, Perl, whatever) to go fetch the data and then do something with it (notify people, generate reports etc)
Does that help?
See here for details on getting started with a Linux instance: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html
You could achieve the same results using a Windows machine and Scheduled Tasks. Here's the getting started guide for Windows instances: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2Win_GetStarted.html
You can use Scheduled Lambda service driven by CloudWatch events. It acts as a resilient cron.