I have an IoT registry of things, all of them identical and processed in the same way.
I create a rule which triggers whenever some condition holds for one of the things. The rule invokes lambda function, which gets the thing shadow or subset of its fields in the event parameter. (I use Python for lambda)
However it does not seem to be possible for lambda to figure which exactly thing triggered the rule - there are only two parameters, event and context, neither of which contains information about original thing id. Am I missing something?
Hey so some more details about your things would be helpful but to get you started I'll explain how the amazon IoT button works. The event parameter that's passed in is a JSON object with some information regarding the state of the thing. For the IoT button this is:
{
"serialNumber": "0000000000000000",
"batteryVoltage": "xxmV",
"clickType": "SINGLE" | "DOUBLE" | "LONG"
}
When writing code (ahh I'm sorry I've just been assuming node.js) you can refer to what's in the object as event.serialNumber, event.clickType...
This is relevant because if you can get your things to have IDs of some sort, like names or serial numbers, you can access this information through the event parameter and use it when your function is invoked.
Related
I would like to run one lambda function, that would return a list of parameters. Based on the number of parameters I would like to trigger another lambda functions to finish the process individually (e.g. 100 independent sub-lambda function).
Would like to know how this be done? It would be great if there are some github settings? Thanks a lot.
There are three options: first, call Invoke using the AWS SDK for your language.
Second, use Step Functions.
Third, write each parameter onto an SQS queue, and configure the second Lambda to be triggered by that queue. This is the approach that I'd use.
Im running a workflow using a step function (with SAM), when I needed to send information between lambdas I've used events and everything was perfect! But now, I need that almost every lambda in my workflow have access to a constant received in the invocation input of the step function (it changes on every execution) like a global variable.
I know that I can solve it by returning it in every lambda output but I think that it is a very ugly solution :(
Is there any way to access the context of the execution and add data to it from a lambda in the step function ? Any other solution would be cool too.
Yes, see https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultpath.html#input-output-resultpath-append
You can keep the input of the state machine execution and combine it with the result of the state.
Going through the docs, I see that you can access the context object from each state in the state machine.
You can pass the information that you need to be global as the input to your state machine and then, access the state machine input from the context object.
You can refer the linked doc to see how to access the context object.
I have nearly 1000 items in my DB. I have to run the same operation on each item. The issue is that this is a third party service that has a 1 second rate limit for each operation. Until now, I was able to do the entire thing inside a lambda function. It is now getting close to the 15 minute (900 second) timeout limit.
I was wondering what the best way for splitting this would be. Can I dump each item (or batches of items) into SQS and have a lambda function process them sequentially? But from what I understood, this isn't the recommended way to do this as I can't delay invocations sufficiently long. Or I would have to call lambda within a lambda, which also sounds weird.
Is AWS Step Functions the way to go here? I haven't used that service yet, so I was wondering if there are other options too. I am also using the serverless framework for doing this if it is of any significance.
Both methods you mentioned are options that would work. Within lambda you could add a delay (sleep) after one item has been processed and then trigger another lambda invocation following the delay. You'll be paying for that dead time, of course, if you use this approach, so step functions may be a more elegant solution. One lambda can certainly invoke another--even invoking itself. If you invoke the next lambda asynchronously, then the initial function will finish while the newly-invoked function starts to run. This article on Asynchronous invocation will be useful for that approach. Essentially, each lambda invocation would be responsible for processing one item, delaying sufficiently to accommodate the service limit, and then invoking the next item.
If anything goes wrong you'd want to build appropriate exception handling so a problem with one item either halts the rest or allows the rest of the chain to continue, depending on what is appropriate for your use case.
Step Functions would also work well to handle this use case. With options like Wait and using a loop you could achieve the same result. For example, your step function flow could invoke one lambda that processes an item and returns the next item, then it could next run a wait step, then process the next item and so on until you reach the end. You could use a Map that runs a lambda task and a wait task:
The Map state ("Type": "Map") can be used to run a set of steps for
each element of an input array. While the Parallel state executes
multiple branches of steps using the same input, a Map state will
execute the same steps for multiple entries of an array in the state
input.
This article on Iterating a Loop Using Lambda is also useful.
If you want the messages to be processed serially and are happy to dump the messages to sqs, set both the concurency of the lambda and the batchsize property of the sqs event that triggers the function to 1
Make it a FIFO queue so that messages dont potentially get processed more than once if that is important.
I have an AWS Lambda function does operations against Kinesis Firehose.
The function uses backoff mechanism. (which at this time I think wasting my computation time).
But anyway, in some point in my code, I would like to fail the execution.
What command should I use in order to make the execution stop?
P.s.
I found out that there are commands such as:
context.done()
context.succeed()
context.fail()
I've got to tell you, I could not find any documentation about these commands in AWS documentation.
Those methods are available only for backward compatibility, since they were first introduced with Node.js v0.10.42. If you use NodeJS version 4.* or 6.*. Use the callback() function.
Check: Using the Callback Parameter in Lambda for more information how to take advantage of this function.
Here is my solution (probably not perfect, but it works for me)
time.sleep(context.get_remaining_time_in_millis() / 1000)
The code is in Python, but I am sure you can apply the same logic using any other language. The idea is make my lambda function "fall asleep" for the remaining time of "retries".
The full example may loo like this:
...
some code that processes logs from CloudWatch when my ECS container stops(job is finished)
...
# Send an email notification
sns_client = client('sns')
sns_client.publish(
TopicArn=sns_topic,
Subject="ECS task success detected for container",
Message=json.dumps(detail)
)
# Make sure it was send only once, therefore 'sleep'
# untill lambda stops 'retries'
time.sleep(context.get_remaining_time_in_millis() / 1000)
So, the email is send only once. I hope it helps!
I have a small issue regarding one existing MQ interpretation.
The thing is in each part of the program we have to interrogate the message that is being sent/received to which type it belongs, resulting in a massive switch scenario for each component.
Each type of the message has to be processed accordingly (update GUI progress bar, update a specific file, connect specific signals from where the interrogation happens and so on).
What would be the best approach to move it into a single component?
For now it uses Factory method to create each of the needed objects and like I said before the drawback is that you have to ask what type of object was created to implement the needed logic => big switches.
Instead of a message id, that you process is a switch statement, you can easily send a code chunk to be executed, say, a lambda object. Then, you can merely execute the code chunk in the "slot", without checking and reacting on the message id.