I want to terminate and exit running cloud function. Function was triggered by Firestore event. What are some ways to do this?
There are some reasons why you want a Cloud Function to terminate itself, for example, to avoid an infinite loop or infinite retries.
To avoid infinite retry loops, set an end condition. You can do this by including a well-defined end condition, before the function begins processing.
A simple yet effective approach is to discard events with timestamps older than a certain time. This helps to avoid excessive executions when failures are either persistent or longer-lived than expected.
Events are delivered at least once, but a single event may result in multiple function invocations. Avoid depending on exactly-once mechanics and write idempotent functions.
Note that updating the function-triggering Firestore document may create subsequent update events, which may cascade into an infinite loop within your function. To solve this problem, use trigger types that ignore updates (such as document.create), or configure your function to only write to Firestore if the underlying value has changed.
Also, note the limitations for Firestore triggers for Cloud Functions.
You might also want to check this example about Cloud Function Termination.
Do not manually exit a Function; it can cause unexpected behavior.
Related
I have a cloud function that is triggered 4 times to process a different set of data in each execution. Every time the function runs, it produces a list.
I would like to somehow get these lists together and sort them after the 4th execution is over.
Is this possible to achieve?
As specified in documentation:
There is no guarantee that the state of a Cloud Function will be preserved for future invocations. However, Cloud Functions often recycles the execution environment of a previous invocation. If you declare a variable in global scope, its value can be reused in subsequent invocations without having to be recomputed.
Meaning that, it's possible to preserve a list of items between two or more function executions, but you have no guarantees that this list will keep its state, because cold starts can always happen for scalability reasons. When a cold start happens, everything is recreated.
If there’s a need to share state across function executions, then a secondary service dedicated to persist shared data should be accessed during each execution, such as Firebase.
Good day!
I faced the challenge of writing the function for allocation of the agents into SelectOutputOut blocks. Considering various scenarios of using if...else statements in function I understand that all possibilities must be covered (as suggested here).
However, the problem is that I don't want the agent to leave the function before it gets the appropriate SelectOutputOut block. This situation may occur if there are not enough resources in any Service blocks (Network1, Network2 or Network3). In this case, it is necessary to wait for any Service block will have enough resources for servicing the agent. For this purpose, I tried to use the while loop, but it doesn't help.
The questions are:
How to write the if-else statements to force the agent waits for enough resources in any Service block
Does the Select function monitor the parameters which are outside it? In other words: Does it know about the states of Service blocks during its execution?
Thank you.
What you need to do is have your agents wait in the queue and then have a function to remove them from the queue and then send them to the correct service block. The best way to do this is with an enter block where you can send them to.
See example below
You then need to call this function at the On enter code for the queue as well as the On exit code for the service blocks, to ensure you are always sending new agents when there is space.
I have a lambda function that I'm calling using boto3. There is a high chance that there will be many concurrent executions and I know that Lambda throttles you if you make too many requests. I am doing this in an synchronous manner, so there are no retries. I want to make sure I know when this will happen, so that I can push requests onto a queue, and try them again at a later time.
Boto3 will return an error if there are too many requests, but I would rather not use try and catch for this. From the boto3 docs:
For example, Lambda returns TooManyRequestsException if executing the function would cause you to exceed a concurrency limit at either the account level (ConcurrentInvocationLimitExceeded ) or function level (ReservedFunctionConcurrentInvocationLimitExceeded ).
Does anyone know of a way to check if the function is available for execution before hand?
Thanks.
Does anyone know of a way to check if the function is available for execution before hand?
No, there isn't a way unless you maintain a counter yourself, which would also be a rough estimate.
Use a try catch statement as this is where it is meant to be used at a code level, use asynchronous invocation or retry your synchronous invocation using exponential backoff (increasing the duration between retries every time).
I have a system the receives records from Kinesis stream, Lambda is consuming the stream and invokes one function per shard, this function takes a batch of records and invokes an Async Express Step Function to process each record. The Step Function contains a Task relies on a third party. I have the timeout for this task set but this still can cause high number of concurrent step functions to start executing, when the task is taking longer, as the step functions are not completing quickly enough, causing throttling on Lambda executions further down the line.
To mitigate the issue I am thinking of implementing a "Semaphore" for concurrent Express function executions. There isn't too much out there in terms of similar approach, I found this article but the approach of checking how many active executions there are at a time would only work with Standard Step Function. If it would work with Express I can imagine I could throw error in the function that receives Kinesis record if the arbitrary Step Function execution limit is exceeded, causing Kinesis+Lambda to retry until capacity is available. But as I am using Express workflow, calling ListExecutions is not really an option.
Is there a solution for limiting number of parallel Async Express Step Function executions out there or do you see how I could alternatively implement the "Semaphore" approach?
Have you considered triggering on step function per lambda invoke and using a map state to do the multiple records per batch? The map state allows you to limit the number of concurrent executions. This doesn’t address multiple executions of the step function, and could lead to issues with timeouts if you are pushing the boundary of the five minute limits for express functions.
I think if you find that you need to throttle something across partitions you are going to be in a world of complex solutions. One could imagine a two phase commit system of tracking concurrent executions and handling timeouts, but these solutions are often more complicated than they are worth.
Perhaps the solution is to make adjustments downstream to reduce the concurrency there? If you end up with other lambdas being invoked too many times at once you can put SQS in front of them and enable batching as well as manage throttling there. In general you should use something like SQS to trigger lambdas at the point where high concurrency is a problem, and less so at points that feed into it. In other words if your current step functions can handle the high concurrency you should let them, and anything has issues as a result of it should be managed at that point.
I have a series of lambdas, one of which I want to execute once another lambda has executed and stopped executing. Is it possible, using step functions, to have a choice state which will check if a lambda is executing?
So, basically, Step Functions trigger tasks (such as a Lambda function execution) based on other tasks completion (success or error).
If the first lambda is in your Step Function then, once it has completed, Step Functions will run the next Step, etc... until it reaches completion. Said next Step could be a choice state checking wether the first lambda has successfully completed or errorred (which would be pointless since you can directly handle an error in a better way (see Amazon States Language Errors).
But if what you want is to monitor the execution of a Lambda running 'outside of your Step Function'. This would take some more work, is not implemented 'out of the box', and I would whole-heartedly ask you to think about what you're trying to achieve here before going down that road.