This is a question about concurrency/parallelism and processes. I am not sure how to express it, so please forgive my ignorance.
It is not related to any specific language, although I'm using Rust lately.
The question is if it is possible to launch processes in concurrent/parallel mode, without waiting for them to finish, and within a specific time frame, even when the total time of the processes takes more than the given time frame.
For example: lets say I have 100 HTTP requests that I want to launch in one second, separated by 10ms each. Each request will take +/- 50ms. I have a computer with 2 cores to make them.
In parallel that would be 100 tasks / 2 cores, 50 tasks each. The problem is that 50 tasks * 50ms each is 2500ms in total, so two seconds and half to run the 100 tasks in parallel.
Would it be possible to launch all these tasks in 1s?
I'm using the camunda-enginge 7.16.6.
I have a Process with a multi instance loop like this one that repeats parallel a 1000 times.
This loop is execute parallel. My assumption was, that n camunda executors now starts their work so executor #1 executes Task 2, then Task 3, then Task 4, and executor #2 and all others do the same. So after a short while at least some of the 1000 times finished all three Tasks in the loop
However what I observed so far is, that Task 2 gets execute 1000 times and only when that is finished, Task 3 gets executed a 1000 times and so on.
I also noticed, that camunda takes a lot of time by itself, outside of the tasks.
Is my Observation correct and is this behavior documented somewhere? Can you change that behavior?
I've run some tests an can explain the behavior:
The Order of Tasks and the overall time to finish is influenced by whenever or not there are transaction boundaries (async after, the red bars in the Screenshot).
Its a bit described here.
By setting the asyncBefore='true' attribute we introduce an additional save point at which the process state will be persisted and committed to the database. A separate job executor thread will continue the process asynchronously by using a separate database transaction. In case this transaction fails the service task will be retried and eventually marked as failed - in order to be dealt with by a human operator.
repeat 1000 times, parallel, no transaction
One Job Executor rushes trough the process, the Order is 1, [2,3,4|2,3,4|...], 5. Not really parallel. But this is as documented here:
The Job Executor makes sure that jobs from a single process instance are never executed concurrently.
It can be turned off if you are an expert and know what you are doing (and have understood this section).
Overall this took around 5 seconds.
repeat 1000 times, parallel, with transaction
Here, due the transactions, there will be 1000 waiting Jobs for Task 7, and each finish Task 7 creates another Job of Task 8. Since the execution of the Jobs is by the order in the database (see here), the order is 6,[7,7,7...8,8,8...9,9,9...],10.
The transaction handling which includes maintaining the variables has a huge impact on the runtime, with Transactions in parallel mode it runs 06:33 minutes.
If you turn off the exclusive-flag it takes around 4:30 minutes, but at the cost of thousands of OptimisticLockingExceptions.
Afaik the recommended approach to gain true parallelism would be to move Task 7, Task 8 and Task 9 to a seperate process and spawn 1000 instances of that process.
You can influence the order of execution if you tweak the job executor settings & priority, see here, but that seems to require the exclusive flag, too. If you do that, the Order will be 6,[7,7,7|8,9,8,9(in random order),...]10
repeat 1000 times, sequential, no transaction
The Order is 11,[12,13,14|12,13,14,...]15
This takes only 2 seconds.
repeat 1000 times, sequential, with transaction
The order is as expected 16,[17,18,19|17,18,19|...],20
Due the Transactions this takes 02:45 minutes.
I heard from colleges, that one should use parallel only if it involves long running/blocking tasks like a human task - in sequential mode there would only be one human task, and after that one is done, another will be created. in parallel mode, you have 1000 human tasks which is more likely the desired behavior.
Parallel performance seems to be improved in Camunda 8
My streaming flink job has checkpointing time of 2-3s(15-20% of time) and 3-4 mins(8-12% of time) and 2 mins on an average. We have two operators which are stateful. First is kafka consumer as source(FlinkKafkaConsumer010) and another is hdfs sink(CustomBucketingSink). This two makes state of around 1-1.5Gb for savepoints and 800mb-6Gb(3gb average) for checkpoint. We have 30sec of tumbling processing window. Checkpointing duration and minimum pause between two checkpoiting is 3 mins. My job consumes around 3 millions of records per minute on an average and around 20 millions/min records on peak time. There is more than enough cpu and memory for flink.
Now here are my doubts :
1) Even when few checkpointing state sizes are less(70-80% less) as compare to other checkpointing state, it takes minutes(15-20% of time) as compare to other one which takes 5-10 secs.
2) Buffer alignment size sometimes increases to 7-8gb as compare to 800mb-1gb average but checkpointing time is not affected by this. I guess it should take more time as it should wait for checkpoint barrier.
3) Will checkponting time be affected if we increase tumbling window size. I am considering it shouldn't affect neither savepoint time and nor checkpoint time.
4) Few of the sub-tasks which sinks into hdfs takes 2-3 mins (5-10% time). So while 98% of subtasks are completed in 30-50 secs. 1-2(95% of time, it's only one) subtasks takes 2-3 mins. Which delays the whole checkpointing time. Problem is not with the node on which this sub-tasks are running because it happens sometimes to some node and sometimes to another node.
5) We are getting one exception once every 6-8 hour which restarts the job. TimerException{java.nio.channels.ClosedByInterruptException} at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:288)
6) How to minimize the alignment buffer time.
7) Savepoint time increases or decreases with increase and decrease of rate of input or state size but checkpointing time doesn't hold the same. Checkpointing time sometimes shows inverse relation with state size or we can saw it's not affected with the state size.
8) Whenever we restart the job, all sub-tasks take uniform time for 2-3 days on all nodes but afterwards 1-2 sub-tasks takes 2-3 minutes as compare to other which are taking 15-30 secs. I might be wrong on this behaviour but as far i have observed, this is also a case.
Note that windows are stateful, and unless you are doing incremental aggregation, longer windows have more state, which will in turn affect checkpoint sizes and durations.
It would be helpful to know which state backend you are using, and whether or not you are using incremental checkpointing.
I would start by trying to find the cause of the slow sink subtask(s) causing the backpressure, which is in turn causing the painful checkpointing. Could be data skew, or resource starvation, for example. Some common causes include insufficient CPU, network, or disk bandwidth, or AWS (or other API) rate limits. It may seem that you have plenty of CPU, for example, but one hot key can put way too much load on one thread, and thereby hold back the entire cluster.
If you find a way to correct the imbalance at the sink, then the checkpoint alignment problems should calm down. (Note that if you can tolerate duplicate results, you could disable checkpoint barrier alignment by choosing CheckpointingMode.AT_LEAST_ONCE.)
SJF = Shortest Job First, title wouldn't let me fit it
Wouldn't the preemptive SJF scheduling make the average wait time of a process be greater than if it was simply executed in a non-preemptive SJF scheduling algorithm? After all, you are continually context switching and forcing a process to wait longer to be completed.
I can't seem to understand why it is that pre-emptive SJF (aka. Shortest-Time-Remaining-First, or STRF) is better than non-preemptive SJF (in terms of average wait time for a process).
Can someone explain this to me?
Thank you.
Suppose p1(8 ms burst time) arrived at the queue at 0 ms, after executing p1 for 1 ms, another process p2 came into the queue with 4 ms burst time. The processor will stop executing process p1 and will start executing process p2. Why ? because p1 has 7ms remaining to finish execution, while p1 has only 4 ms remaining to finish.
I think it's clear why it is called "shortest-time-remaining-first" scheduling. Because it always chooses a process which has the smallest amount of time remained to execute.
For your other question, why it is better....let's extend the scenario.
Process p1 --> burst time 8 ms, arrival time 0 ms,
Process p2 --> burst time 4 ms, arrival time 1 ms,
Process p3 --> burst time 9 ms, arrival time 2 ms,
Process p4 --> burst time 5 ms, arrival time 3 ms.
for preemptive SJF,
average waiting time =[ (for p1)(10-1) + (for p2)(1-1) + (for p3)(17-2)+(for p4)(5-3)]/4 = 6.5 ms
for non preemptive SJF it would be,
average waiting time =[ (for p1)(0) + (for p2)(8-1) + (for p3)(17-2)+(for p4)(12-3)]/4 = 7.75 ms
you can see why it is said that preemptive is better than non-preemptive, it's because it takes less time to execute all the process using this algorithm.
Reference: Operating System Concepts by Galvin, Silberschatz, Gagne (8th edition).
Hi I am having trouble understanding exactly how wait and turn around times are calculated, I am learning this in preparation for an exam and I do have a past paper for this however the marking schemes do not explain where the values are coming from here is the question:
An operating system has two process priority levels, 1 and 2. Three new
processes, A B and C, arrive at the same time in that order with the
following characteristics
A
----------
Quanta Required 20
Priority 1
----------
B
----------
Quanta Required 30
Priority 1
----------
C
----------
Quanta Required 40
Priority 2
----------
Calculate the average Waiting Time and average Turn-Around Time if:
a)normal Round Robin scheduling is used
b)a modified Round Robin scheduling which gives priority 1 processes a
double quantum and priority 2 processes a single quantum whenever they
are scheduled
c)What advantages or disadvantages has the algorithm described in (c)
compared to a Priority Queue scheduling?
The answers only show calculations with no explanation to the best of my understanding:
Average Wait Time = Sum of Process Finish times / Number of Processes
Average Turn Around = Sum of Process Start Times / Number of Processes
Is this correct or is there a better way to calculate this any help is much appreciated