I want to use Step function Map, and have this logic that for each item in the input array
Lambda: checks a status of something:
CHOICE: true -> move on | false -> wait X hours and check for status again.
Does this mean if we set a max concurrency of 10 & have all 10 concurrently running maps in the wait stage because the status of something is false for all 10, my step function is essentially stuck and won't be able to continue on until one or more status of something comes back true?
It's probably the case that you want to avoid using Max Concurrency with Wait state in it, but I just want to confirm this is the case.
My alternative approach is to simply stage it to be processed again later and not block it.
Thanks.
Related
I have a Map state which iterates my array. Inside the map state, there is a Lambda task and a Wait task. The Wait task is waiting much time, and I need to wait only between iterations. So I would like to skip waiting if this is the last iteration because there is no need for it.
Every time the items are different and their amount is different.
However, the Map context has only $$.Map.Item.Index and $$.Map.Item.Value variables. I couldn't find any mention of any variable with the total amount of steps for example.
How can I achieve that?
My understanding: Each execution has an arbitrary number of Map items that should run serially (i.e. MaxConcurrency:1). You want no delay after the last item. For instance, an execution with 4 items [A, B, C, D] should run in the following order: Lambda(A), Wait, Lambda(B), Wait, Lambda(C), Wait, Lambda(D). A single-item execution [A] should run Lambda(A) only.
Here is one way to do it:
[Update - late 2022: Using the new ArrayLength and MathAdd intrinsic functions we can calculate the last item index without a lambda - see "LastItemIndex.$" below]
Insert a Lambda Task before the Map State. The Lambda counts the items and outputs the last item's index to $.itemsCounter.lastItemIndex.
Add the last item and the item index to the Map iterations' payloads with Parameters on the Map State:
// add to the Map State definition - overrides what each iteration receives
"Parameters": {
"Index.$": "$$.Map.Item.Index",
"Data.$": "$$.Map.Item.Value", // by default, each iteration just gets this
"LastItemIndex.$": "States.MathAdd(States.ArrayLength($$.Execution.Input.Items), -1)", // 3 for {"Items": [A,B,C,D]}
},
Add a ShouldWait? Choice State inside the Map between the Lambda Task and Wait State. Proceed to Wait unless $.Index equals $.LastItemIndex.
I have a code that, when needed, interrupts a Sqlite query using a context with a deadline. My problem is to write a unit test for it: I need to launch a query that I know will run for a long time, ideally in an infinite loop, and check that it is interrupted. I use https://github.com/mattn/go-sqlite3 to access Sqlite 3 from Go.
For example, this:
with recursive rec as
(select 1 as n union all select n + 1 from rec)
select n from rec;
returns 1 immediately instead of looping, as it does in the SQLite console (is there something to do to enable CTE’s?). I also found no sleep function or anything similar.
I need guidence on implementing WHILE loop with Kettle/PDI. The scenario is
(1) I have some (may be thousand or thousands of thousand) data in a table, to be validated with a remote server.
(2) Read them and loopup to the remote server; I use Modified Java Script for this as remote server lookup validation is defined in external Java JAR file (I can use "Change number of copies to start... option on Modified java script and set to 5 or 10)
(3) Update the result on database table. There will be 50 to 60% connection failure cases each session.
(4) Repeat Step 1 to step 3 till all gets updated to success
(5) Stop looping on Nth cycle; this is to avoid very long or infinite looping, N value may be 5 or 10.
How to design such a WhILE loop in Pentaho Kettle?
Have you seen this link? It gives a pretty well detailed explanation of how to implement a while loop.
You need a parent job with a sub-transformation for doing a check on the condition which will return a variable to the job on whether to abort or to continue.
I want to create a timer so that after completing the time(suppose 10 sec) the control should come out of the function..Please note that am starting the timer inside the function.Code is given below..I want to give certain time limit to that function so that after completing the time the control should come out of the function..I don't want to calculate the time..I want to give my own time so that the function should complete its execution within that time period..suppose if function is waiting for an input then also after completing time limit the control should come out indicating that "time has expired"..once it comes out of the function then it should continue with the next function execution...Is this possible in c++...
Begin();
// here I would like to add timer.
v_CallId = v_CallId1;
call_setup_ind();
call_alert_ind();
dir_read_search_cnf();
dir_save_cnf();
END();
If the code is linear and the functions called cannot be chopped into smaller pieces, your stuck to letting an external process/thread do the timing and abort the worker thread when the timeout is exceeded.
When you can chop the worker into smaller pieces you could do something like this
Timeout.Start(5000);
while ((TimeOut.TimeOut() == false) && (completed == false))
{
completed = WorkToDo()
}
This is a pattern we frequently use in our embbeded application. The timeout class was in house develop. It just reads the tick counter and looks if the time has passed. An framework like QT or MFC should have such a class itself.
I have scheduled a coldfusion template to run every 10 minutes how do i prevent it from running when the previous run exceeds 10 minutes.
I've tried using a counter variable in the application scope unfortunately when the template times out or errors out the counter is not decremented.
PS. Is there a coldfuison framework for integrating applications (backend stuff)
Use an exclusive named cflock:
<cflock
timeout = "#createTimeSpan(0,0,0,1)#"
name = "myProcess"
throwOnTimeout = "no"
type = "exclusive">
<cfset start = now()>
<!--- CFML to be synchronized. --->
<cfset interval = 10 * 60> <!--- 10 minutes in seconds --->
<cfif dateDiff("s", start, now()) GT interval>
<cfhttp url="yourtemplate.cfm">
</cfif>
</cflock>
This ensures only thread ever runs the block of code in question. Simultaneous access will fail in one second without error.
To ensure the next run gets kicked off if the prior exceeds the time interval, track the start time inside the lock and then at the end, if its exceeded the interval, have the last statement inside the cflock be a cfhttp call to itself.
One possible route you could explore:
You could set up a database table to track the progress of the task. Maybe table name "task", with columns "taskName" and "inProgress", with the latter being a boolean. When the task starts, set inProgress to true. When it finishes, set inProgress to false.
In the template called by the scheduled task, have it first check the "inProgress" status of the specified task. If it's true, just abort. Otherwise, proceed.
EDIT:
Hmm... that actually wouldn't work any better than your application variables in the case of timeouts or errors. So now thinking that instead of a boolean, you use a timestamp. When the scheduled task fires, update the value with the current time. When it finishes, clear it out.
So when the task starts again, it will see that the previous task either finished (a null value), or it's still in progress. -If- it's still in progress, you can do a dateDiff() on the value to see if it was more than 'x' minutes ago. If it was, you can assume the previous task timed out (or errored out... but in that case I'd think you could put some error handling into the task itself) and run the current instance of the task.