I am trying to deploy WebJobs on Azure, in such a way it should not collide at same time .
I have 3 types jobs.
Job1 runs continuously / indefinitely
Job2 runs every 30 minutes and it takes 8 minutes to complete
Job3 runs every 60 minutes and it takes 12-15 minutes to complete.
How we can schedule Job2 or Job3 so cycle doesn't overlap or coincide at same time.
Performance thought : Job2 and Job3 both require high memory and cpu utilization, if both start at same time our AppService crashed.
Related
I'm using google cloud tasks with http triggers to invoke cloud functions. I've setup the cloud task queue retry parameters as follows:
max attempts: 2
Max retry duration: 16
Min backoff: 1
Max backoff: 16
Max doublings: 4
I will often have bursts of tasks which will create around 600 tasks within a second or two. There are times when about 15% of these will fail (this is expected and intentional). I'm expecting these failed tasks to retry according to the queue configuration. Thus I would not expect any task retry schedule to be more than 16 seconds beyond its initially scheduled time. However, I'm seeing some failed tasks scheduled several minutes out. Typically, the first few failed tasks will schedule for retry only a few seconds out, but some of the last few failed tasks in this burst will have these retry schedule for many minutes away.
Why are these retry schedules not honoring my retry config?
If it helps, I also have these settings on the queue:
Max dispatches: 40
Max concurrent dispatches: 40
We have a cloud scheduler that runs at an interval of every 6 hours and calls an application deployed in GKE (single Pod) via HTTP. We are observing a strange behaviour where the application is called a second time at 10th min of the initial run. The job runs for nearly an hour.
So, at the end, we finally see two parallel processing of the same event with the diff of 10 mins. Any pointers will be helpful :slightly_smiling_face:
I have a project in google cloud where there are 2 task queues: process-request to receive requests and process them, send-result to send the result of the processed request to another server. They are both running on an instance called remote-processing
My problem is that I see the tasks being enqueued in send-result but they are only executed after the process-request queue is empty and has processed all requests.
This is the instance config:
instance_class: B4
basic_scaling:
max_instances: 8
Here is the queue config:
- name: send-result
max_concurrent_requests: 20
rate: 1/s
retry_parameters:
task_retry_limit: 10
min_backoff_seconds: 5
max_backoff_seconds: 20
target: remote-processing
- name: process-request
bucket_size: 50
max_concurrent_requests: 10
rate: 10/s
target: remote-processing
Clarification : I don't need for the queues to run in an specific order, but I find it very strange that it looks like the insurance only runs one queue at a time, so it will only run the tasks in another queue after its done with the current queue.
over what period of time is this all happening?
how long does a process-request task take to run vs a send-result task
One thing that sticks out is that your rate for process-request is much higher than your rate for send-result. So maybe a couple send-result tasks ARE squeezing off, but it then hits the rate cap and has to run process-request tasks instead.
Same note for bucket_size. The bucket_size for process-request is huge compared to it's rate:
The bucket size limits how fast the queue is processed when many tasks
are in the queue and the rate is high. The maximum value for bucket
size is 500. This allows you to have a high rate so processing starts
shortly after a task is enqueued, but still limit resource usage when
many tasks are enqueued in a short period of time.
If you don't specify bucket_size for a queue, the default value is 5.
We recommend that you set this to a larger value because the default
size might be too small for many use cases: the recommended size is
the processing rate divided by 5 (rate/5).
https://cloud.google.com/appengine/docs/standard/python/config/queueref
Also, by setting max_instances: 8 does a big backlog of work build-up in these queues?
Let's try a two things:
set bucket_size and rate to be the same for both process-request and send-result. If fixes it, then start fiddling with the values to get the desired balance
bump up max_instances: 8 to see if removing that bottleneck fixes it
We set up a schedule to execute a command.
It is scheduled to run every 5 minutes as follows: 20090201T235900|20190201T235900|127|00:05:00
However, from the logs we see it runs only every hour.
Is there a reason for this?
check scheduling frequency in your sitecore.config file
<sitecore>
<scheduling>
<!-- Time between checking for scheduled tasks waiting to execute -->
<frequency>00:05:00</frequency>
</scheduling>
</sitecore>
The scheduling interval is based on the the scheduler interval and the job interval. Every scheduler interval period, all the configured jobs are evaluated. This is logged. During that evaluation, each job checked against the last time it ran, if that interval is greater that the configured job interval, the job is started.
It's fairly simple, but it's important to understand the mechanism. You can also see how it allows no way of inherently running jobs at a specific time, only at approximate intervals.
You can also see that jobs can never run more frequently than the scheduler interval regardless of the job interval. It is not unreasonable to set the scheduler to one-minute intervals to reduce the inaccuracy of job timings to no more than a minute.
In a worse case, with a 5 minute sheduler interval and a 5 minute job interval. The delay to job starting could be up to 9 minutes 59 seconds.
Does anyone know that when a scheduled task in Coldfusion runs it resets the interval timer or does the task run at the set interval time no matter how long that task run for?
For example, i create a task to run every 10 minutes that takes 5 minutes to run starting from 12pm. Will the task run at 12:00, then 12:10, then 12:20 etc etc.
Or would it run at 12:00 which takes 5 minutes, then at 12:15 ten minutes after the task has finished, then another 5 minutes to run so the next one would run at 12:30 etc etc.
Hope that makes sense.
Chris
For example, i create a task to run
every 10 minutes that takes 5 minutes
to run starting from 12pm. Will the
task run at 12:00, then 12:10, then
12:20 etc etc.
Yes.
The task will always run on the interval. So if you set it every 10 minutes it will run every 10 minutes after the first run.
Note: If the task runs over time (i.e. longer than the interval) than it will NOT queue. That particular run will just be skipped and the task will run at the next interval as per usual.
Hope that helps!