I would like to run one lambda function, that would return a list of parameters. Based on the number of parameters I would like to trigger another lambda functions to finish the process individually (e.g. 100 independent sub-lambda function).
Would like to know how this be done? It would be great if there are some github settings? Thanks a lot.
There are three options: first, call Invoke using the AWS SDK for your language.
Second, use Step Functions.
Third, write each parameter onto an SQS queue, and configure the second Lambda to be triggered by that queue. This is the approach that I'd use.
Related
I'm trying to build a process like this:
In state1, it will trigger 10 lambdas, and only when ALL those 10 lambda respond/ or call callback with taskToken, it will then proceed to next state2.
How to design this process?
This is a perfect scenario for the Map state. You can pass in an array of lambda function names, then add a Lambda task and use the Parameters block to set the function dynamically. And if you want them to run one at a time instead of in parallel, you can set MaxConcurrency.
I have an AWS environment with 3 CodePipelines. Let's call those- P1, P2 and P3. I'd like to run those in "specific" sequence. That sequence will be determined by a Lambda function.
This Lambda function does some calculation, and determine which sequence the pipelines need to be run. So, it could be-
P1 > P2 > P3
P3 > P2 > P1
P2 > P3 > P1
Each codepipeline must finish successfully before the next one is run. How can I achieve this?
At first I tried to do it using that same Lambda function, but it has 15 mins timeout. We don't know how long each pipeline's gonna take. All together they could take even ~30 mins.
Also, since the sequence is dynamic, I couldn't just get one pipeline to export a file to S3, and use that as one of source for another one!
Any suggestion?
CP emits events using EventBridge. Thus you would have to use that.
Basically, you would setup event rules which would trigger executions of subsequent pipeline, based on successful completion of previous pipeline.
Generally EventBridge would do the trick (as mentioned by #Marcin), however, it didn't work with my particular scenario.
I resolved the issue by introducing "Step Function" between the Lambda function and pipelines.
Lambda function passes some parameter to the Step Function. Based on that, it takes different path (different order of pipeline to execute)
Also, it has "wait" mechanism, as well as, GetPipelineExecution to check status. Together they ensure that one pipeline needs to end successfully before the next one is triggered.
Hope this helps someone in future :)
I have nearly 1000 items in my DB. I have to run the same operation on each item. The issue is that this is a third party service that has a 1 second rate limit for each operation. Until now, I was able to do the entire thing inside a lambda function. It is now getting close to the 15 minute (900 second) timeout limit.
I was wondering what the best way for splitting this would be. Can I dump each item (or batches of items) into SQS and have a lambda function process them sequentially? But from what I understood, this isn't the recommended way to do this as I can't delay invocations sufficiently long. Or I would have to call lambda within a lambda, which also sounds weird.
Is AWS Step Functions the way to go here? I haven't used that service yet, so I was wondering if there are other options too. I am also using the serverless framework for doing this if it is of any significance.
Both methods you mentioned are options that would work. Within lambda you could add a delay (sleep) after one item has been processed and then trigger another lambda invocation following the delay. You'll be paying for that dead time, of course, if you use this approach, so step functions may be a more elegant solution. One lambda can certainly invoke another--even invoking itself. If you invoke the next lambda asynchronously, then the initial function will finish while the newly-invoked function starts to run. This article on Asynchronous invocation will be useful for that approach. Essentially, each lambda invocation would be responsible for processing one item, delaying sufficiently to accommodate the service limit, and then invoking the next item.
If anything goes wrong you'd want to build appropriate exception handling so a problem with one item either halts the rest or allows the rest of the chain to continue, depending on what is appropriate for your use case.
Step Functions would also work well to handle this use case. With options like Wait and using a loop you could achieve the same result. For example, your step function flow could invoke one lambda that processes an item and returns the next item, then it could next run a wait step, then process the next item and so on until you reach the end. You could use a Map that runs a lambda task and a wait task:
The Map state ("Type": "Map") can be used to run a set of steps for
each element of an input array. While the Parallel state executes
multiple branches of steps using the same input, a Map state will
execute the same steps for multiple entries of an array in the state
input.
This article on Iterating a Loop Using Lambda is also useful.
If you want the messages to be processed serially and are happy to dump the messages to sqs, set both the concurency of the lambda and the batchsize property of the sqs event that triggers the function to 1
Make it a FIFO queue so that messages dont potentially get processed more than once if that is important.
I have an IoT registry of things, all of them identical and processed in the same way.
I create a rule which triggers whenever some condition holds for one of the things. The rule invokes lambda function, which gets the thing shadow or subset of its fields in the event parameter. (I use Python for lambda)
However it does not seem to be possible for lambda to figure which exactly thing triggered the rule - there are only two parameters, event and context, neither of which contains information about original thing id. Am I missing something?
Hey so some more details about your things would be helpful but to get you started I'll explain how the amazon IoT button works. The event parameter that's passed in is a JSON object with some information regarding the state of the thing. For the IoT button this is:
{
"serialNumber": "0000000000000000",
"batteryVoltage": "xxmV",
"clickType": "SINGLE" | "DOUBLE" | "LONG"
}
When writing code (ahh I'm sorry I've just been assuming node.js) you can refer to what's in the object as event.serialNumber, event.clickType...
This is relevant because if you can get your things to have IDs of some sort, like names or serial numbers, you can access this information through the event parameter and use it when your function is invoked.
I have an AWS Lambda function does operations against Kinesis Firehose.
The function uses backoff mechanism. (which at this time I think wasting my computation time).
But anyway, in some point in my code, I would like to fail the execution.
What command should I use in order to make the execution stop?
P.s.
I found out that there are commands such as:
context.done()
context.succeed()
context.fail()
I've got to tell you, I could not find any documentation about these commands in AWS documentation.
Those methods are available only for backward compatibility, since they were first introduced with Node.js v0.10.42. If you use NodeJS version 4.* or 6.*. Use the callback() function.
Check: Using the Callback Parameter in Lambda for more information how to take advantage of this function.
Here is my solution (probably not perfect, but it works for me)
time.sleep(context.get_remaining_time_in_millis() / 1000)
The code is in Python, but I am sure you can apply the same logic using any other language. The idea is make my lambda function "fall asleep" for the remaining time of "retries".
The full example may loo like this:
...
some code that processes logs from CloudWatch when my ECS container stops(job is finished)
...
# Send an email notification
sns_client = client('sns')
sns_client.publish(
TopicArn=sns_topic,
Subject="ECS task success detected for container",
Message=json.dumps(detail)
)
# Make sure it was send only once, therefore 'sleep'
# untill lambda stops 'retries'
time.sleep(context.get_remaining_time_in_millis() / 1000)
So, the email is send only once. I hope it helps!