AWS API Gateway Lambda Integration (NOT Proxy) - How to Send Custom Headers and Binary Data - amazon-web-services

I'm using AWS API Gateway integrated with Lambda.
Note: I am NOT using Lambda Proxy.
I need to return a binary response from API Gateway. I have successfully set this up as follows:
Encoded my binary data as base64 UTF-8 string and returning ONLY that from my lambda function return "base64 encoded binary data"
Enabled CONVERT_TO_BINARY on the API Gateway Integration Response
Mapped the Content-Type header on the API Gateway Method Response to the binary media type of my binary content
Added the media type of my binary content to API Gateway's list of Binary Media Types
The issue is that as well as sending the binary data (which I can do successfully from the above steps), I need to include a x-my-header custom header in the API Response.
I know how to set-up header mapping in the API Gateway, but the header has to be calculated from database data, and therefore this value needs to also be returned from lambda.
My understanding of lambda integration (remember, I'm not using lambda proxy here) is that API gateway makes a HTTP request to trigger lambda. Lambda then returns a HTTP response to API Gateway, adding the functions output to the body, and also adding internal aws headers to the response.
Now it is possible to map a header to the Method Response using:
integration.response.header.header-name
My question is...
Can I tell lambda to add my custom header to a binary response, when I'm
using custom lambda integration (not proxy) ?
Note: IF i was using lambda proxy, I know that the return object looks as below, and then I would be able to send custom headers. But for reasons out of my control, I cant use lambda proxy.
Lambda return object solution IF i was using lambda proxy:
return {
'body': "base64 encoded binary data",
'headers': 'x-my-header': 'my-value',
'isBase64Encoded': True
}
For Lambda Integration (not proxy) I have tried modifying my lambda output...
return {
"base64-data": "base64 encoded binary data",
"x-my-header: "some value"
}
And setting up a mapping template in the integration response...
$input.json("$.base64-data")
And setting up a header mapping using...
integration.response.body.x-my-header
But API Gateway returns an error:
Execution failed due to configuration error: Unable to transform response
I believe this error occurs because there cannot be a mapping template when you have CONVERT_TO_BINARY enable. From AWS docs:
When converting a text payload to a binary blob, API Gateway assumes that the text data is a Base64-encoded string and outputs the binary data as a Base64-decoded blob. If the conversion fails, it returns a 500 response indicating an API configuration error. You do not provide a mapping template for such a conversion, although you must enable the passthrough behaviors on the API.

I realize this is an old question, but I ran into a similar header mapping issue recently, even though I'm not using binary data.
To my mind the return from Lambda could look like this
{
"base64-data": "base64 encoded binary data",
"x-my-header: "some value"
}
Based on that Lambda Response, you could apply the following mapping (which I modified based on an AWS example)
$input.json("$.base64-data")
#set($context.responseOverride.header.x-my-header = "$input.json('$.x-my-header')")
$input.json("$") references your response and $.[key] is the right way to reference your sub-keys.
You also have to preconfigure your header "x-my-header" in your method response.
In the integration Response the header mapping can just be an empty value - e.g.
""
The real value will be provided by the override ($context.responseOverride.header) which is set by the mapping template.

Related

LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413

I have node/express + serverless backend api which I deploy to Lambda function.
When I call an api, request goes to API gateway to lambda, lambda connects to S3, reads a large bin file, parses it and generates an output in JSON object.
The response JSON object size is around 8.55 MB (I verified using postman, running node/express code locally). Size can vary as per bin file size.
When I make an api request, it fails with the following msg in cloudwatch,
LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413
I can't/don't want to change this pipeline : HTTP API Gateway + Lambda + S3.
What should I do to resolve the issue ?
the AWS lambda functions have hard limits for the sizes of the request and of the response payloads. These limits cannot be increased.
The limits are:
6MB for Synchronous requests
256KB for Asynchronous requests
You can find additional information in the official documentation here:
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
You might have different solutions:
use EC2, ECS/Fargate
use the lambda to parse and transform the bin file into the desired JSON. Then save this JSON directly in an S3 public bucket. In the lambda response, you might return the client the public URL/URI/FileName of the created JSON.
For the last solution, if you don't want to make the JSON file visible to whole the world, you might consider using AWS Amplify in your client or/and AWS Cognito in order to give only an authorised user access to the file that he has just created.
As noted in other questions, API Gateway/Lambda has limits on on response sizes. From the discussion I read that latency is a concern additionally.
With these two requirements Lambda are mostly out of the question, as they need some time to start up (which can be lowered with provisioned concurrency) and do only have normal network connections (whereas EC2,EKS can have enhanced networking).
With this requirements it would be better (from AWS Point Of View) to move away from Lambda.
Looking further we could also question the application itself:
Large JSON objects need to be generated on demand. Why can't these be pre-generated asynchronously and then downloaded from S3 directly? Which would give you the best latency and speed and can be coupled with CloudFront
Why need the JSON be so large? Large JSONs also need to be parsed on the client side requiring more CPU. Maybe it can be split and/or compressed?

How to send multi part key value via http get

I am trying to learn to use AWS Lambda, DynamoDB, and API Gateway.
I am able to set up a run a successful test of the Lambda that would take querystring parameters and returns the data from the database and I know I have the ability to access the Lambda from my API as I have created one to just return hello world. However, I am stuck as to how to send the parameters I need to the Lambda. These work fine from the Lambda test feature within Lambda.
{
"queryStringParameters": {
"TableName": "Exams",
"Key": {
"ExamName": "MyFirstExam"
}
}
}
However I can not seem to create a html query http://apiurl/route?parameters
as I have no idea how to send the Key.
I have tried everything I can think of here is just a sample of what I have tried
http://apiurl/route?TableName=Exams&Key=ExamName&ExamName=MyFirstExam
http://apiurl/route?TableName=Exams&key=ExamName&Keyvalue=MyFirstExam
http://apiurl/route?TableName=Exams&key=ExamName&value=MyFirstExam
http://apiurl/route?TableName=Exams&key=ExamName:MyFirstExam
but nothing has worked so I am wondering how do you send key = ExamName : MyfirstExam so that it is a part of the querystring parameters?
UPDATE:
I found that the rest api gave me more feed back and it looks like I need to use
````http://apiURL/Route?TableName= Exams&Key= { ExamName: MyFirstExam } ```
as this populates the querystring parameters.
"queryStringParameters":{"TableName":"Exams","Key":"{ ExamName: MyFirstExam }"}
However, I am still getting an error. That leads me to believe that I still may not have the format right. The problem is that when I used the same data in the test from within the lambda console it works and returns the data. When I use the querystring above which appears to be the same data as used in the lambda test it throws an error.
Lambda execution failed with status 200 due to customer function error: The provided key element does not match the schema. So my Key is getting there as I get a status 200 and if I leave the key off of the querystring it throws and error that the key is missing but it appears to not be formatted correctly apparently or is it something else that I am missing. It appears to be the same as the format that is working in the Lambda test.
This works from the lambda test
{ "queryStringParameters": { "TableName": "Exams", "Key": { "ExamName": "MyFirstExam" } } }
this does not work from the API test
"queryStringParameters":{"TableName":"Exams","Key":"{ ExamName: MyFirstExam }"}
This is how I am formatting the querystring
http://apiURL/Route?TableName= Exams&Key= { ExamName: MyFirstExam }
How do I format a querystring so that it works from the API gateway?
The easiest and most commonly used method of joining API Gateway and lambda is through AWS_PROXY integration between your API and your function.
With this type of integration:
API Gateway applies a default mapping template to send the entire request to the Lambda function and transforms the output from the Lambda function to HTTP responses.
This means that every query parameter you provide through your API endpoint is going to be delivered directly into lambda function:
This request data includes the request headers, query string parameters, URL path variables, payload, and API configuration data.
The event format your lambda function gets is described here.
I resolved this by having the key part already in the code and just now only need to pass a normal variable ExamName=Exam.
var queryParameters ={
TableName: request.TableName,
Key:{
"ExamName": request.ExamName
},
};```

Is it possible to access the original request body in a response body mapping template in AWS API Gateway?

Using API Gateway, I am trying to define a POST end point that accepts application/json to do the following:
Trigger a Lambda asynchronously
Respond with a JSON payload composed of elements from the request body
I have #1 working. I think it's by the book.
It's #2 I'm getting tripped up on. It looks like I don't have access to the request body in the context of the response mapping template. I have access to the original query params with $input.params but I cannot find any property that will give me the original request body, and I need it to get the data that I want to respond with. It's either that or I need to figure out how to get the asynchronous launch of a Lambda to somehow provide the original request body.
Does anyone know if this is possible?
My goal is to ensure that my API responds as fast as possible without incurring a cold start of a Lambda to respond AND simultaneously triggering an asynchronous workflow by starting a Lambda. I'd also be willing to integrate with SNS instead of Lambda directly and have Lambda subscribe to the topic but I don't know if that will get me access to the data I need in the response mapping template.
From https://stackoverflow.com/a/61482410/3221253:
Save the original request body in the integration mapping template:
#set($context.requestOverride.path.body = $input.body)
Retrieve it in the integration mapping response:
#set($body = $context.requestOverride.path.body)
{
"statusCode": 200,
"body": $body,
}
You can also access specific attributes:
#set($object = $util.parseJson($body))
{
"id": "$object.id"
}
To access the original request directly, you should use a Proxy Integration for Lambda rather than mapping things via a normal integration. You'll be able to access the entire request context, such as headers, path params, etc.
I have determined that it is not possible to do what I want to do.

Map multiple http response headers in AWS API Gateway with AWS Lambda response body

I could set single http response header through api gateway integration response header mapping.
in that case i was giving context.fail("http://www.google.com") response from aws lambda and use regex (".http.") to identify the response and finally map it to my header like this
Location: integration.response.body.errorMessage.
This is working when i try to map a single header at a time.
But i want to set both Location and Cookie headers in a single response.
to achieve this, i returned a json from my aws lambda like
context.fail(JSON.stringify({Location:"http://www.google.com",Cookie: vid="233kwer34343"}))
but i couldn't map headers via integration.response.body.errorMessage.Location and integration.response.body.errorMessage.Cookie
It is possible to do this by using context.succeed() instead of context.fail() . but this is not checking regex. i want to check regex for categorizing responses.
What is the best way to map multiple response headers from lambda response body?
This is not because of API Gateway or multiple headers. Please note that the Location header will be added to the Response only for redirect (3xx) or successfull process (2xx) of the request. It will not be added when return code is error (4xx, 5xx) etc.
More details here - https://en.wikipedia.org/wiki/HTTP_location
This the reason why it worked for context.succeed() and it did not work for context.fail().
I don't think you can make this working using context.fail. The problem is that the parameter passed to context.fail is a simple string. That string is then encoded into a json string and then passed to API Gateway as a sting value within an attribute named "errorMessage".
The response body coming from Lambda looks something like this: {"errorMessage": "{\"Location\":\"http://www.google.com\",\"Cookie\": \"vid='233kwer34343'\"}"}
Note that the value of errorMessage is a string with the special characters escaped, rather than a json object. There's no way to tell a header mapping to look in integration.response.body.errorMessage, parse that string into json, and grab the location attribute of the resulting json object.
If you were returning the values as a response body, rather than as headers, then you could use velocity to parse the contents of errorMessage and transform them into whatever response body you like. Unfortunately, header mappings don't have the full processing power/flexibility of the velocity templates.
Why do you need to call context.fail in this case?
If you call context.done you can return json as the body like: {"Location":"http://www.google.com","Cookie": "vid='233kwer34343'"}
Then the header mappings are just integration.response.body.Location and integration.response.body.Cookie

AWS Lambda and AWS API Gateway: How to send a binary file?

I have a lambda function which fetches a file from s3 using the input key in event and needs to send the same to client. I am using the following function to get the file from s3
function getObject(key){
var params = {
Bucket: "my_bucket",
Key: key
}
return new Promise(function (resolve, reject){
s3.getObject(params, function (err, data){
if(err){
reject(err);
}
resolve(data.Body)
})
})
}
If I send the response of this promise (buffer) to context.succeed, it is displayed as a JSON array on front end. How can I send it as a file ? The files can be either ZIP or HTTP Archive (HAR) files. The s3 keys contain the appropriate extension. I am guessing it has got something to do with the "Integration Response" in API Gateway. But not able to figure out where to change
Good news, you can now handle binary input and output for API Gateway (announcement and documentation).
Basically, nothing changes in your Lambda Function, but you can now set the contentHandling API Gateway Integration property to CONVERT_TO_BINARY.
Unfortunately, the official AWS examples showcase only the HTTP API Gateway backend, as the AWS Lambda support seems not complete yet. For example, I haven't managed to return gzipped content from AWS Lambda yet, although it should be possible thanks to the new binary support and the $util.base64Decode() mapping utility.