I have a lambda setup that uses vendia serveless-express as a handler. It has been working well to serve REST APIs from the single lambda function with multiple routes.
I now have a new requirement, where the same lambda needs to be part of a step function's state machine. But that doesn't seem to be allowed by vendia app, as it always throws the error: "Unable to determine event source based on event" as it expects the event to be api gateway / alb only.
So, based on this, it looks like I will need a separate lambda for step, which makes me have duplicate code in multiple lambdas.
Is it possible for the lambda to handle the inputs from step and still be a vendia express app? Please let me know if I am trying something that doesn't make sense at all.
If I were you I would just implement my own converter that transforms a StepFunction event into an API Gateway event and then calling the express-serverless in your lambda.
The package aws-lambda contains definitions in TypeScript of many AWS events including those ones, then try to generate a mock API Gateway event from your own step function event value. From the sources (4.3.9) we have the function:
function getEventSourceNameBasedOnEvent ({
event
}) {
if (event.requestContext && event.requestContext.elb) return 'AWS_ALB'
if (event.Records) return 'AWS_LAMBDA_EDGE'
if (event.requestContext) {
return event.version === '2.0' ? 'AWS_API_GATEWAY_V2' : 'AWS_API_GATEWAY_V1'
}
throw new Error('Unable to determine event source based on event.')
}
So probably to make it work correctly, you have to define a RequestContext mock value in your mock event and it should be enough.
Related
Aws lambdas have 3 steps in their lifecycle:
init
invoke
shutdown
Is there a way to trigger automatically a lambda init without deploying any code?
Update:
I have some actions that are launched during the init: saving the content of a file in a variable. Then during the invoke actions, the content is not downloaded anymore. But I need to launch the action of downloading from time to time. Then, I was wondering if there's a way to trigger the init action.
Why not just call your init() function again?
Take the following Go Lambda for example:
// handleRequest does process every request to the Lambda.
func handleRequest(ctx context.Context) error {
init() // <-- call it whenever you want in your handler again
}
// init does store some data from a file in a variable.
func init() {}
func main() {
init() // <-- called on the first start of the Lambda
lambda.Start(handleRequest)
}
The init function is called in the main() method, which is executed when the Lambda is started for first time (cold start or init). But if you move your "business logic" for the initialisation into a separate function, you can call it whenever you want in your handler function.
Note: I don't think at all that this is a great idea. It is definitely a design "smell" if you need/want to do something like this. So you should rather rethink what you are doing than building workarounds like this.
There is no way you can manually trigger those events as I know however, you can perform some action for those events.
check AWS Extention API if you are interested.
https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html
Perhaps you should take a look at lambda layers. This allows you to place some files in advance, so that your invocations can access it at runtime (without having to download it). You can then easily update your layer whenever you need.
See documentation here
I figured out a way to do this in NodeJs Lambdas. I’m not aware of any previous solutions for JS Lambdas. Normally, the lambda container is paused before async operations complete in the init phase.
The key is to leverage an AWS Lambda custom Extension. Then if you enable provisioned concurrency, the Lambdas execute the init phase before a request. You have to determine how many provisioned concurrency instances to keep online because concurrent requests can trigger on-demand instances. Or you can enable auto-scaling with provisioned concurrency to try to scale out ahead of the load.
This article explains the details and includes a link to the GitHub sample repo.
https://barker.tech/aws-lambda-nodejs-async-init-88b5c24af567?source=friends_link&sk=79dbb80045ad97c62f58fa4b74ade68b
https://github.com/chuckbarkertech/nodejs-example-async-init-extension
You still have to deploy code. But you can trigger the init before requests hit the lambda by combining provisioned concurrency with AWS Lambda Extensions like the example info above.
I currently have a bunch of AWS Lambda functions and would like to spin them each up in containers (preferably Docker) to integrate with the current CICD pipeline. It's totally OK to have these containers spun up in an Amazon service such as Fargate -- the important thing is that it must be spun up in a container. The closest thing I've found is this repo by Amazon that converts images into Lambdas, though this is the opposite of what I'm looking for.
Is this task possible, and if so what's the best way to go about it?
Behind the scenes, a Lambda "function" is a Python script that listens for invocation events and invokes the actual handler function. This means that you can pretty easily invoke the function using the standard Python "script" code:
if __name__ == "__main__":
args = # this is the tricky part
ctx = # some dict that looks like what Lambda expects
handler(args, ctx)
These two parameters are just dicts, so if you can populate them with something that looks right then you can wrap the entire thing in a container and deploy it to ECS or Batch, or your own Docker daemon.
The problem is how you populate them, and that depends on what your lambda functions do -- which you don't say -- and how they're invoked -- which you also don't say.
You could use command-line parameters. For example, if your lambda is intended to respond to an S3 PutObject operation, then all you really need is the bucket and key name. From that you can easily create a dummy S3 invocation event, using fake values for all of the other fields.
If you invoke your Lambda in response to an SQS message it's even easier: your "main" handler enters a loop that polls SQS, and reformats the messages that it pulls off the queue.
Other event sources are going to be much harder. It may make sense (depending on why you're doing this) to use SQS for all invocations, and simply pass a fully-formed event message into the queue.
I have written and saved a lambda function. I see:
Congratulations! Your Lambda function "lambda_name" has been
successfully created. You can now change its code and configuration.
Choose Test to input a test event when you want to test your function.
Now how do I run it? I cannot see a 'run' or 'invoke' button as I would expect
Note
The lambda doesn't accept any arguments (it's extremely simple - for the purposes of this question, please presume it's simply 2 * 2 so when I run it it should not require any inputs and should return 4).
Also note
I can see a tonne of different ways to run the lambda here. I just want the simplest way (preferably a button in the browser)
Sending a test message via the Lambda console will run your Lambda function. The test message that you configure will define what is in the event parameter of your lambda handler function.
Since you are not doing anything with that message, you can send any arbitrary test message and it should work for you. You can just use the default hello world message and give it an arbitrary name.
It should then show you the results: any logs or returned objects right in the AWS Lambda console.
Further reading here
AWS Lambda functions are typically triggered by an event, such as an object being uploaded to Amazon S3 or a message being send to an Amazon SNS topic.
This is because Lambda functions are great at doing a small task very often. Often, Lambda functions only run for a few seconds, or even less than a second! Thus, they are normally triggered in response to something else happening. It's a bit like when somebody rings your phone, which triggers you to answer the phone. You don't normally answer your phone when it isn't ringing.
However, it is also possible to directly invoke an AWS Lambda function using the Invoke() command in the AWS SDK. For convenience, you can also use the AWS Command-Line Interface (CLI) aws lambda invoke command. When directly invoking an AWS Lambda function, you can receive a return value. This is in contrast to situations where a Lambda function is triggered by an event, in which case there is nowhere to 'return' a value since it was not directly invoked.
I followed tutorial on creating and invoking step functions
I'm getting output in my GET request of api as
{
"executionArn": "arn:aws:states:ap-northeast-1:123456789012:execution:HelloWorld:MyExecution",
"startDate": 1.486772644911E9
}
But, instead of above response I want my step functions output, which is given by end state as below.
{
"name":"Hellow World"
}
How to achieve this?
Update: You can now use Express Step Functions for synchronous requests.
AWS Step Functions are asynchronous and do not immediately return their results. API Gateway methods are synchronous and have a maximum timeout of 29 seconds.
To get the function output from a Step Function, you have to add a second method in API Gateway which will call the Step Function with the DescribeExecution action. The API Gateway client will have to call this periodically (poll) until the returned status is no longer "RUNNING".
Here's the DescribeExecution documentation
Use Express Step Functions instead. This type of Step Functions can be called synchronously. Go to your API Gateway and in the Integration Request section make sure you have the StartSyncExecution action:
After that, go a bit lower in the same page to the Mapping Templates:
and include the following template for the application/json Content-Type:
#set($input = $input.json('$'))
{
"input": "$util.escapeJavaScript($input)",
"stateMachineArn": "arn:aws:states:us-east-1:your_aws_account_id:stateMachine:your_step_machine_name"
}
After that, go back to the Method Execution and go to the Integration Response and then to the Mapping Templates section:
And use the following template to have a custom response from your lambda:
#set ($parsedPayload = $util.parseJson($input.json('$.output')))
$parsedPayload
My testing Step Function is like this:
And my Lambda Function code is:
Deploy your API Gateway stage.
Now, if you go to Postman and send a POST request with any json body, now you have a response like this:
New Synchronous Express Workflows for AWS Step Functions is the answer:
https://aws.amazon.com/blogs/compute/new-synchronous-express-workflows-for-aws-step-functions/
Amazon API Gateway now supports integration with Step Functions StartSyncExecution for HTTP APIs:
https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-api-gateway-supports-integration-with-step-functions-startsyncexecution-http-apis/
First of all the step functions executes asynchronously and API Gateway is only capable of invoking the step function (Starting a flow) only.
If you are waiting for the results of a step function invocation from a web application, you can use AWS IOT WebSockets for this. The steps are as follows.
Setup AWS IOT topic with WebSockets.
Configure the API Gateway and Step functions invocation.
From the Web Frontend subscribe to the IOT Topic as a WebSocket listener.
At the last step (And in error steps) in the Step Functions workflow use AWS SDK to trigger the IOT Topic which will broadcast the results to the Web App running in the browser using WebSockets.
For more details on WebSockets with AWS IOT refer the medium article Receiving AWS IoT messages in your browser using websockets.
Expanding on what #MikeD at AWS says, if you're certain that the Step Function won't exceed the 30 second timeout, you could create a lambda that executes the step function and then blocks as it polls for the result. Once it has the result, it can return it.
It is a better idea to have the first call return immediately with the execution id, and then pass that id into a second call to retrieve the result, once it's finished.
I'm attempting to integrate my lambda function, which must run async because it takes too long, with API gateway. I believe I must, instead of choosing the "Lambda" integration type, choose "AWS Service" and specify Lambda. (e.g. this and this seem to imply that.)
However, I get the message "AWS ARN for integration must contain path or action" when I attempt to set the AWS Subdomain to the ARN of my Lambda function. If I set the subdomain to just the name of my Lambda function, when attempting to deploy I get "AWS ARN for integration contains invalid path".
What is the proper AWS Subdomain for this type of integration?
Note that I could also take the advice of this post and set up a Kinesis stream, but that seems excessive for my simple use case. If that's the proper way to resolve my problem, happy to try that.
Edit: Included screen shot
Edit: Please see comment below for an incomplete resolution.
So it's pretty annoying to set up, but here are two ways:
Set up a regular Lambda integration and then add the InvocationType header described here http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html. The value should be 'Event'.
This is annoying because the console won't let you add headers when you have a Lambda function as the Integration type. You'll have to use the SDK or the CLI, or use Swagger where you can add the header easily.
Set the whole thing up as an AWS integration in the console (this is what you're doing in the question), just so you can set the InvocationType header in the console
Leave subdomain blank
"Use path override" and set it to /2015-03-31/functions/<FunctionARN>/invocations where <FunctionARN> is the full ARN of your lambda function
HTTP method is POST
Add a static header X-Amz-Invocation-Type with value 'Event'
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
The other option, which I did, was to still use the Lambda configuration and use two lambdas. The first (code below) runs in under a second and returns immediately. But, what it really does is fire off a second lambda (your primary one) that can be long running (up to the 15 minute limit) as an Event. I found this more straightforward.
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};