Im running a workflow using a step function (with SAM), when I needed to send information between lambdas I've used events and everything was perfect! But now, I need that almost every lambda in my workflow have access to a constant received in the invocation input of the step function (it changes on every execution) like a global variable.
I know that I can solve it by returning it in every lambda output but I think that it is a very ugly solution :(
Is there any way to access the context of the execution and add data to it from a lambda in the step function ? Any other solution would be cool too.
Yes, see https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultpath.html#input-output-resultpath-append
You can keep the input of the state machine execution and combine it with the result of the state.
Going through the docs, I see that you can access the context object from each state in the state machine.
You can pass the information that you need to be global as the input to your state machine and then, access the state machine input from the context object.
You can refer the linked doc to see how to access the context object.
Related
Colleagues,
Can you please advise me a bit about the following.
I cannot figure out how to pass value through process variable from main flow to its subflow in Camunda. I am putting value to process variable in one task in main flow via execution.setVariable("toolId", toolId);
where execution is an instance of DelegateExecution. I am trying to retrieve in another task of subflow via
Long toolId = (Long) execution.getVariable("toolId");
However I am getting null.
By subflow I assume you mean a call activity (otherwise the data would be available).
A call activity references a technically independent process instance with its own data. Therefore you have to explicitly map the in data, which should be copied from the source (parent) to the target (sub process) and also the out data in the other direction.
Please see: https://docs.camunda.io/docs/components/modeler/bpmn/call-activities/#variable-mappings and https://docs.camunda.io/docs/components/concepts/variables/#inputoutput-variable-mappings
I have nearly 1000 items in my DB. I have to run the same operation on each item. The issue is that this is a third party service that has a 1 second rate limit for each operation. Until now, I was able to do the entire thing inside a lambda function. It is now getting close to the 15 minute (900 second) timeout limit.
I was wondering what the best way for splitting this would be. Can I dump each item (or batches of items) into SQS and have a lambda function process them sequentially? But from what I understood, this isn't the recommended way to do this as I can't delay invocations sufficiently long. Or I would have to call lambda within a lambda, which also sounds weird.
Is AWS Step Functions the way to go here? I haven't used that service yet, so I was wondering if there are other options too. I am also using the serverless framework for doing this if it is of any significance.
Both methods you mentioned are options that would work. Within lambda you could add a delay (sleep) after one item has been processed and then trigger another lambda invocation following the delay. You'll be paying for that dead time, of course, if you use this approach, so step functions may be a more elegant solution. One lambda can certainly invoke another--even invoking itself. If you invoke the next lambda asynchronously, then the initial function will finish while the newly-invoked function starts to run. This article on Asynchronous invocation will be useful for that approach. Essentially, each lambda invocation would be responsible for processing one item, delaying sufficiently to accommodate the service limit, and then invoking the next item.
If anything goes wrong you'd want to build appropriate exception handling so a problem with one item either halts the rest or allows the rest of the chain to continue, depending on what is appropriate for your use case.
Step Functions would also work well to handle this use case. With options like Wait and using a loop you could achieve the same result. For example, your step function flow could invoke one lambda that processes an item and returns the next item, then it could next run a wait step, then process the next item and so on until you reach the end. You could use a Map that runs a lambda task and a wait task:
The Map state ("Type": "Map") can be used to run a set of steps for
each element of an input array. While the Parallel state executes
multiple branches of steps using the same input, a Map state will
execute the same steps for multiple entries of an array in the state
input.
This article on Iterating a Loop Using Lambda is also useful.
If you want the messages to be processed serially and are happy to dump the messages to sqs, set both the concurency of the lambda and the batchsize property of the sqs event that triggers the function to 1
Make it a FIFO queue so that messages dont potentially get processed more than once if that is important.
When my AWS Step Functions' State Machine fails ExecutionsFailed , I'd like to trigger a lambda function in response to it.
Seems that you have to create a rule on CloudWatch; but I couldn't find description on how to do that (in particular how the Event Patterns supposed to look like).
p.s. in my case it happens due to exceeding 25,000 history limit (so not quite so easy to handle within state machine; without having to add loop counters etc.; so I'd prefer for it to fail; and then handle it via lambda)
Current workaround is to create a cron rule for a scheduled event on a cloudwatch; and the check the state machine; and in case it is failed; handle it.
I have an IoT registry of things, all of them identical and processed in the same way.
I create a rule which triggers whenever some condition holds for one of the things. The rule invokes lambda function, which gets the thing shadow or subset of its fields in the event parameter. (I use Python for lambda)
However it does not seem to be possible for lambda to figure which exactly thing triggered the rule - there are only two parameters, event and context, neither of which contains information about original thing id. Am I missing something?
Hey so some more details about your things would be helpful but to get you started I'll explain how the amazon IoT button works. The event parameter that's passed in is a JSON object with some information regarding the state of the thing. For the IoT button this is:
{
"serialNumber": "0000000000000000",
"batteryVoltage": "xxmV",
"clickType": "SINGLE" | "DOUBLE" | "LONG"
}
When writing code (ahh I'm sorry I've just been assuming node.js) you can refer to what's in the object as event.serialNumber, event.clickType...
This is relevant because if you can get your things to have IDs of some sort, like names or serial numbers, you can access this information through the event parameter and use it when your function is invoked.
I have a small issue regarding one existing MQ interpretation.
The thing is in each part of the program we have to interrogate the message that is being sent/received to which type it belongs, resulting in a massive switch scenario for each component.
Each type of the message has to be processed accordingly (update GUI progress bar, update a specific file, connect specific signals from where the interrogation happens and so on).
What would be the best approach to move it into a single component?
For now it uses Factory method to create each of the needed objects and like I said before the drawback is that you have to ask what type of object was created to implement the needed logic => big switches.
Instead of a message id, that you process is a switch statement, you can easily send a code chunk to be executed, say, a lambda object. Then, you can merely execute the code chunk in the "slot", without checking and reacting on the message id.