Can AWS step function executes more than 25000 times? - amazon-web-services

Question

I am currently evaluating AWS state machine that can process single document. The state machine would take 5-10 mins to process a single document.
{
"Comment":"Process document",
"StartAt": "InitialState",
"States": {
//the document goes through multiple states here
}
}
The C# code invokes the state machine by passing some json for each document. Something like
// max 100 documents
public Task Process(IEnumerable<Document> documents)
{
var amazonStepFunctionsConfig = new AmazonStepFunctionsConfig { RegionEndpoint = RegionEndpoint.USWest2 };
using (var amazonStepFunctionsClient = new AmazonStepFunctionsClient(awsAccessKeyId, awsSecretAccessKey, amazonStepFunctionsConfig))
{
foreach(var document in documents)
{
var jsonData1 = JsonConvert.SerializeObject(document);
var startExecutionRequest = new StartExecutionRequest
{
Input = jsonData1,
Name = document.Id,
StateMachineArn = "arn:aws:states:us-west-2:<SomeNumber>:stateMachine:ProcessDocument"
};
var taskStartExecutionResponse = await amazonStepFunctionsClient.StartExecutionAsync(startExecutionRequest);
}
}
}
We process the documents in batch of 100. So in above loop the max number of documents will be 100. However we process thousands of documents weekly (25000+).
As per the AWS documentation Maximum execution history size is 25,000 events. If the execution history reaches this limit the execution will fail.
Does that mean we can not execute a single state machine more than 25000 times?
Why execution of state machine should depend on its history, why cant AWS just purge history?
I know there is a way to continue as new execution but I am just trying to understand the history limit and its relation to state machine execution, and is my understanding is correct?
Update 1
I don't think this is duplicate question. I am trying find if my understanding of history limit is correct? Why history has anything to do with number of times state machine can execute? When state machine executes, it creates history record, if history records goes more 25000+, then purge them or archive them. Why would AWS stop execution of state machine. That does not make sense.
So question, Can single state machine (unique arn) execute more than 25000+ times in loop?
if i have to create new state machine (after 25000 executions) wouldn't that state machine will have different arn?
Also if i had to follow linked SO post where would i get current number of executions? Also he is looping with-in the step function, while i am calling step function with-in the loop
Update 2
So just for testing i created the following state machine
{
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Pass",
"Result": "Hello World!",
"End": true
}
}
}
and executed it 26000 times with NO failure
public static async Task Main(string[] args)
{
AmazonStepFunctionsClient client = new AmazonStepFunctionsClient("my key", "my secret key", Amazon.RegionEndpoint.USWest2);
for (int i = 1; i <= 26000; i++)
{
var startExecutionRequest = new StartExecutionRequest
{
Input = JsonConvert.SerializeObject(new { }),
Name = i.ToString(),
StateMachineArn = "arn:aws:states:us-west-2:xxxxx:stateMachine:MySimpleStateMachine"
};
var response = await client.StartExecutionAsync(startExecutionRequest);
}
Console.WriteLine("Press any key to continue");
Console.ReadKey();
}
and on AWS Console i am able to pull the history for all 26000 executions
So i am not sure exactly what does it mean by Maximum execution history size is 25,000 events

Answer 1

I don't think you've got it right. 25,000 limit is for a State Machine execution history. You have tested 26,000 State Machine executions. State Machine executions limit is 1,000,000 open executions.
A State Machine can run for up to 1 year, and during this time its execution history should not reach more than 25,000.
Hope it helps.

Answer 2

The term "Execution History" is used to describe 2 completely different things in the quota docs, which has caused your confusion (and mine until I realized this):
90 day quota on execution history retention: This is the history of all executions, as you'd expect
25,000 quota on execution history size: This is the history of "state events" within 1 execution, NOT across all executions in history. In other words, if your single execution runs through thousands of steps, thereby racking up 25k events (likely because of a looping structure in the workflow), it will suddenly fail and exit.
As long as your executions complete in under 25k steps each, you can run the state machine much more than 25k times sequentially without issue :)