I have a multi-endpoint webservice written in Flask and running on API Gateway and Lambda thanks to Zappa.
I have a second, very tiny, lambda, written in Node, that periodically hits one of the webservice endpoints. I do this by configuring the little lambda to have Internet access then use Node's https.request with these options:
const options = {
hostname: 'XXXXXXXXXX.execute-api.us-east-1.amazonaws.com',
port: 443,
path: '/path/to/my/endpoint',
method: 'POST',
headers: {
'Authorization': `Bearer ${s3cretN0tSt0r3d1nTheC0de}`,
}
};
and this works beautifully. But now I am wondering whether I should instead make the little lambda invoke the API endpoint directly using the AWS SDK. I have seen other S.O. questions on invoking lambdas from lambdas but I did not see any examples where the target lambda was a multi-endpoint webservice. All the examples I found used new AWS.Lambda({...}) and then called invokeFunction with params.
Is there a way to pass, say, an event to the target lambda which contained the path of the specific endpoint I want to call? (and the auth headers, etc.) * * * * OR * * * * is this just a really dumb idea, given that I have working code already? My thinking is that a direct SDK lambda invocation might (is this true?) bypass API Gateway and be cheaper, BUT, hitting the endpoint directly via API Gateway is better for logging. And since the periodic lambda runs once a day, it's probably free anyway.
If what I have now is best, that's a fine answer. A lambda invocation answer would be cool too, since I've not been able to find a good example in which the target lambda had multiple https endpoints.
You can invoke the Lambda function directly using the invoke method in AWS SDK.
var params = {
ClientContext: "MyApp",
FunctionName: "MyFunction",
InvocationType: "Event",
LogType: "Tail",
Payload: <Binary String>,
Qualifier: "1"
};
lambda.invoke(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
/*
data = {
FunctionError: "",
LogResult: "",
Payload: <Binary String>,
StatusCode: 123
}
*/
});
Refer the AWS JavaScript SDK lambda.invoke method for more details.
Related
I have a rest API in aws api gateway that access's an external http endpoint and returns some data.
Every time this rest api is accessed and returns data, i need to trigger a lambda function that uses the data returned by the rest api. How can i achieve that ?
Tried looking for answers on this and all of them point to how to trigger a lambda function using a rest api.
You can do this in 2 ways.
When a request is sent to api gateway, invoke a lambda, inside this lambda access your external http endpoint and whatever the result comes, invoke another lambda.
something like this.
const invoke = async (funcName, payload) => {
const client = createClientForDefaultRegion(LambdaClient);
const command = new InvokeCommand({
FunctionName: funcName,
Payload: JSON.stringify(payload),
LogType: LogType.Tail,
});
const { Payload, LogResult } = await client.send(command);
const result = Buffer.from(Payload).toString();
const logs = Buffer.from(LogResult, "base64").toString();
return { logs, result };
};
Use stepfunctions with api gateway ( which would be similar to 1st operation)
I'm trying to get my API Gateway api to:
Run an authorizer
Pass authorizer context to a Step Function execution
Respond to client with Step Function output
I already have #1 and #3 done, but passing the response of the attached authorizer lambda to the step function is proving to be impossible.
I found this page and this page with reference sheets on what interpolation values you can use for your parameter mapping (Create Integration -> Step Function: StartSyncExecution -> Advanced Settings -> Input) but any time I try to use anything related to $context like $context.authorizer.email, API Gateway just responds with an HTTP 400 and gives me this CloudWatch output:
"Unable to resolve property Input from source {\"lambdaName\": \"arn:aws:lambda:us-east-1:xxxxxxx\", \"reqBody\": $request.body.Input, \"authContext\": $context.apiId }. Please make sure that the request to API Gateway contains all the necessary fields specified in request parameters."
These are the JSON objects I've tried using for the Input text box and all of them either give me an errors when trying to save or throw an HTTP 400 and log the above errors when I visit the route:
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context.authorizer.email }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": "$context.authorizer.email" }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context.apiId }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $context }
{"lambdaName": "xxx", "reqBody": $request.body.Input, "authContext": $event.requestContext.authorizer.email }
It seems like the only way to have authorization code to work with step functions is to wrap my step function called by API Gateway in another step function that authorizes the request and then invokes the endpoint step function. I've researched this for hours and I'm not getting anywhere. Any help at all is appreciated.
I ended up solving this by using API Gateway v1 and a REST API instead of a HTTP API. For some reason v2's input field currently doesn't work for anything other than $request.body.Input. From there, I hooked up all of my endpoints to a step function that runs the authorization lambda on their Authorization header in the request.
I have a step function that allows me to chain together step function and lambda actions so for most requests I just chain together the authorizer lambda and the endpoint's action (can be lambda or another step function).
The main takeaway here is that if you're using API Gateway and Step Functions, it looks like passing custom-formatted input into your step function isn't very easy to do without using the v1 of API Gateway in a REST API, not an HTTP api. Hopefully this will be fixed in the future.
Another solution could be to use the Mapping Template in API Gateway Integration Request.
Example: Consider the response from the Lambda Authorizer as this:
{
principalId: 'myuser',
context: {
customKey: 'CustomValue'
},
policyDocument: {
Version: '2012-10-17',
Statement: [
{
Action: ['execute-api:Invoke'],
Effect: 'Allow',
Resource:
'arn:aws:execute-api:_region_:111111111111:0a0a0a0a0a/default/POST/my-endpoint'
}
]
}
}
In the Mapping Template for application/json, map whatever property you want in the input field (quote escaped). Each property sent from the Lambda Authorizer in the context field will be available as $context.authorizer.property.
One possible Template Mapping could be:
{
"input": "{\"origin\":\"$input.json('$.requestvalue').replaceAll('\"','')\", \"customPropertyFromLambdaAuthorizer\" : \"$context.authorizer.customKey\" }",
"name": "DemoStateMachineRequest",
"stateMachineArn": "arn:aws:states:_region_:11111111111:stateMachine:MyStateMachine"
}
Reference $context.authorizer.property
My use case is such that I'll have an AWS Lambda front ended with API Gateway.
My requirement is that once the Lambda is invoked it should return a 200 OK response back to API Gateway which get forwards this to the caller.
And then the Lambda should start its actual processing of the payload.
The reason for this is that the API Gateway caller service expects a response within 10 seconds else it times out. So I want to give the response before I start with the processing.
Is this possible?
With API Gateway's "Lambda Function" integration type, you can't do this with a single Lambda function -- that interface is specifically designed to be synchronous. The workaround, if you want to use the Lambda Function integration type is for the synchronous Lambda function, invoked by the gateway, to invoke a second, asynchronous, Lambda function through the Lambda API.
However, asynchronous invocations are possible without the workaround, using an AWS Service Proxy integration instead of a Lambda Function integration.
If your API makes only synchronous calls to Lambda functions in the back end, you should use the Lambda Function integration type. [...]
If your API makes asynchronous calls to Lambda functions, you must use the AWS Service Proxy integration type described in this section. The instructions apply to requests for synchronous Lambda function invocations as well. For the asynchronous invocation, you must explicitly add the X-Amz-Invocation-Type:Event header to the integration request.
http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html
Yes, simply create two Lambda functions. The first Lambda function will be called by the API Gateway and will simply invoke the second Lambda function and then immediately return successfully so that the API Gateway can respond with an HTTP 200 to the client. The second Lambda function will then take as long as long as it needs to complete.
If anyone is interested, here is the code you can use to do the two lambdas approach. The code below is the first lambda that you should setup which would then call the second, longer running, lambda. It takes well under a second to execute.
const Lambda = new (require('aws-sdk')).Lambda();
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};
Check the answer here on how to set up an Async Invoke to the Lambda function. This will return 200 immediately to the client, but the Lambda will process on it's own asynchronously.
https://stackoverflow.com/a/40982649/5679071
I'm attempting to integrate my lambda function, which must run async because it takes too long, with API gateway. I believe I must, instead of choosing the "Lambda" integration type, choose "AWS Service" and specify Lambda. (e.g. this and this seem to imply that.)
However, I get the message "AWS ARN for integration must contain path or action" when I attempt to set the AWS Subdomain to the ARN of my Lambda function. If I set the subdomain to just the name of my Lambda function, when attempting to deploy I get "AWS ARN for integration contains invalid path".
What is the proper AWS Subdomain for this type of integration?
Note that I could also take the advice of this post and set up a Kinesis stream, but that seems excessive for my simple use case. If that's the proper way to resolve my problem, happy to try that.
Edit: Included screen shot
Edit: Please see comment below for an incomplete resolution.
So it's pretty annoying to set up, but here are two ways:
Set up a regular Lambda integration and then add the InvocationType header described here http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html. The value should be 'Event'.
This is annoying because the console won't let you add headers when you have a Lambda function as the Integration type. You'll have to use the SDK or the CLI, or use Swagger where you can add the header easily.
Set the whole thing up as an AWS integration in the console (this is what you're doing in the question), just so you can set the InvocationType header in the console
Leave subdomain blank
"Use path override" and set it to /2015-03-31/functions/<FunctionARN>/invocations where <FunctionARN> is the full ARN of your lambda function
HTTP method is POST
Add a static header X-Amz-Invocation-Type with value 'Event'
http://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
The other option, which I did, was to still use the Lambda configuration and use two lambdas. The first (code below) runs in under a second and returns immediately. But, what it really does is fire off a second lambda (your primary one) that can be long running (up to the 15 minute limit) as an Event. I found this more straightforward.
/**
* Note: Step Functions, which are called out in many answers online, do NOT actually work in this case. The reason
* being that if you use Sequential or even Parallel steps they both require everything to complete before a response
* is sent. That means that this one will execute quickly but Step Functions will still wait on the other one to
* complete, thus defeating the purpose.
*
* #param {Object} event The Event from Lambda
*/
exports.handler = async (event) => {
let params = {
FunctionName: "<YOUR FUNCTION NAME OR ARN>",
InvocationType: "Event", // <--- This is KEY as it tells Lambda to start execution but immediately return / not wait.
Payload: JSON.stringify( event )
};
// we have to wait for it to at least be submitted. Otherwise Lambda runs too fast and will return before
// the Lambda can be submitted to the backend queue for execution
await new Promise((resolve, reject) => {
Lambda.invoke(params, function(err, data) {
if (err) {
reject(err, err.stack);
}
else {
resolve('Lambda invoked: '+data) ;
}
});
});
// Always return 200 not matter what
return {
statusCode : 200,
body: "Event Handled"
};
};
I have a lambda function which fetches a file from s3 using the input key in event and needs to send the same to client. I am using the following function to get the file from s3
function getObject(key){
var params = {
Bucket: "my_bucket",
Key: key
}
return new Promise(function (resolve, reject){
s3.getObject(params, function (err, data){
if(err){
reject(err);
}
resolve(data.Body)
})
})
}
If I send the response of this promise (buffer) to context.succeed, it is displayed as a JSON array on front end. How can I send it as a file ? The files can be either ZIP or HTTP Archive (HAR) files. The s3 keys contain the appropriate extension. I am guessing it has got something to do with the "Integration Response" in API Gateway. But not able to figure out where to change
Good news, you can now handle binary input and output for API Gateway (announcement and documentation).
Basically, nothing changes in your Lambda Function, but you can now set the contentHandling API Gateway Integration property to CONVERT_TO_BINARY.
Unfortunately, the official AWS examples showcase only the HTTP API Gateway backend, as the AWS Lambda support seems not complete yet. For example, I haven't managed to return gzipped content from AWS Lambda yet, although it should be possible thanks to the new binary support and the $util.base64Decode() mapping utility.