How to use API Gateway and Mapping Templates with SQS Integration? - amazon-web-services

I had an API Gateway endpoint setup that, through a Mapping Template, takes the body of the request and some requester meta, combines them, and sends that payload to a stream. Here is what it looked like-
RequestTemplates:
application/json: >
#set($dataRecord = "{
""body"" : $input.json('$'),
""context"" : {
""client_ip"" : ""$context.identity.sourceIp"",
""user_agent"" : ""$context.identity.userAgent""
}
}")
{
"DeliveryStreamName": "event-firehose",
"Record": { "Data": "$util.base64Encode($dataRecord)" }
}
Now, i'd like to replace the stream with an sqs queue. The problem I'm having is that the Mapping Template is no longer respected. I have to specify the body of the message through the MessageBody query param in the integration, and can pass in method.request.body here, but I can't figure out how to either A) pass the transformed request via the mapping template, or B) construct the MessageBody query param so that it would contain both the request body and context.
How are Mapping Templates used when using SQS in the integration method?
Edit-
Michael and Kannaiyan in the comments both provided working answers

Related

How to correctly formate the response for an AWS Lambda API?

I'm trying to understand how to correctly format the JSON that is sent back from my AWS rest API. I am using Api-Gateway to create an API that return the result of a Lambda.
I am used to API responses in the following format:
{
"name" : "John",
"age" : 34
}
But, in the AWS example for integrating lambda and api-gateway, they said to use the following response format:
// The output from a Lambda proxy integration must be
// in the following JSON object. The 'headers' property
// is for custom response headers in addition to standard
// ones. The 'body' property must be a JSON string. For
// base64-encoded payload, you must also set the 'isBase64Encoded'
// property to 'true'.
let response = {
statusCode: responseCode,
headers: {
"x-custom-header" : "my custom header value"
},
body: JSON.stringify(responseBody)
};
return response;
Is there a right way to do this, or can you just return whatever response style you like?

API GATEWAY not retreiving request body

I'm trying to create a POST endpoint in API gateway with request body in application/json type. Now, in input body mapping template, I want to check if the input is there and if the required fields are present or not. I tried getting input body using $input.body and also tried $input.json('$') and $input.path('$'). Nothing works, input body is always empty, although the $input.body == "" check always returns false. But in the test logs i can see that the body is passed through. I'm using Mock as integration type. What can be an issue?
I found out the answer to this is that, actually the reponse body isn't accessible when we use MOCK integration as provided by AWS, but we can still access body using a hacky method by:
First, in the integration request mapping template you store the body in a path parameter.
#set($context.requestOverride.path.body = $input.body)
{
"statusCode": 200,
}
Then, in the integration response mapping template you fetch it back and return it.
#set($body = $context.requestOverride.path.body)
{
"statusCode": 200,
"body": $body,
}
And then to parse it:
$util.parseJson($body).varName

How to map querystring and header to AWS C# lambda function parameter

I have AWS Gateway REST API that takes 2 querystring parameters
https://xxxxxx.xxxx.us-east-1.amazonaws.com/dev/pets?type=dog&page=1
The caller of the API also include x-api-key in the header. I want API gateway to pass querystring parameters and x-api-key to lambda function. So in AWS API Gateway Console i have configured the Integration Request as below
The lambda function looks like this
namespace AWSLambda1
{
public class Function
{
public string FunctionHandler(LambdaRequest request, ILambdaContext context)
{
return string.Format("{0},{1},{2}", request.Type, request.Page, request.ApiKey);
}
}
}
public class LambdaRequest
{
public string Type { get; set; }
public string Page { get; set; }
public string ApiKey { get; set; }
}
Issues
1> When lambda function receives the request, the Type and Page properties are coming as NULL.
2>As per documentation API Gateway can map the http header using the naming convention method.request.header.{param_name}, however when i try to set map from as method.request.header.x-api-key it throws error
Invalid mapping expression specified: Validation Result: warnings :
[], errors : [Invalid mapping expression parameter specified:
method.request.header.x-api-key]
I am not sure how do i map these query string and header to C# lambda object
(Please note that i have already gone through SO post that suggest to JObject as parameter for lambda function. But it only works for me if i enable Use Lambda Proxy integration in Integration Request. In such case API gateway pass all the information to lambda. This might work for me but i am trying to avoid passing unwanted information to lambda function)
Adding Full answer here.
Header Issue
First thing, you need to make sure header entry is added in Method Request and then you can go map that in Integration Request with mapping method.request.header.x-api-key. The error is happening because you did not add in Method Request section but trying to configure it in Integration Request only.
Lambda Payload Issue
It looks like you are not using Lambda Proxy Integration. If you use Lambda Proxy Integration then you will get full event JSON object event data to Lambda. Similar to answer given in post you have shared. This JSON object will contains headers, queryparameters, path variables, url, request body etc.,. If you want to see some sample on how it looks, just go and create API Gateway Test Event on Lambda.
Now, if you do not want to use Lambda Proxy Integration but want to limit what is being sent to Lambda then you will have to create Integration Mapping Template to send only required info to Lambda such as headers, payload, query params etc., from API Gateway.
Sample Integration Template.
{
"body" : $input.json('$'),
"headers": {
#foreach($header in $input.params().header.keySet())
"$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end
#end
},
"method": "$context.httpMethod",
"params": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
},
"query": {
#foreach($queryParam in $input.params().querystring.keySet())
"$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end
#end
}
}
Reference -
https://kennbrodhagen.net/2015/12/06/how-to-create-a-request-object-for-your-lambda-event-from-api-gateway/
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html

How to get the HTTP method in AWS Lambda?

In an AWS Lambda code, how can I get the HTTP method (e.g. GET, POST...) of an HTTP request coming from the AWS Gateway API?
I understand from the documentation that context.httpMethod is the solution for that.
However, I cannot manage to make it work.
For instance, when I try to add the following 3 lines:
if (context.httpMethod) {
console.log('HTTP method:', context.httpMethod)
}
into the AWS sample code of the "microservice-http-endpoint" blueprint as follows:
exports.handler = function(event, context) {
if (context.httpMethod) {
console.log('HTTP method:', context.httpMethod)
}
console.log('Received event:', JSON.stringify(event, null, 2));
// For clarity, I have removed the remaining part of the sample
// provided by AWS, which works well, for instance when triggered
// with Postman through the API Gateway as an intermediary.
};
I never have anything in the log because httpMethod is always empty.
The context.httpMethod approach works only in templates. So, if you want to have access to the HTTP method in your Lambda function, you need to find the method in the API Gateway (e.g. GET), go to the Integration Request section, click on Mapping Templates, and add a new mapping template for application/json. Then, select the application/json and select Mapping Template and in the edit box enter something like:
{
"http_method": "$context.httpMethod"
}
Then, when your Lambda function is called, you should see a new attribute in the event passed in called http_method which contains the HTTP method used to invoke the function.
API Gateway now has a built-in mapping template that passes along stuff like http method, route, and a lot more. I can't embed because I don't have enough points, but you get the idea.
Here is a screenshot of how you add it in the API Gateway console:
To get there navigate to AWS Console > API Gateway > (select a resource, IE - GET /home) > Integration Request > Mapping Templates > Then click on application/json and select Method Request Passthrough from dropdown shown in the screenshot above
I had this problem when I created a template microservice-http-endpoint-python project from functions.
Since it creates an HTTP API Gateway, and only REST APIs have Mapping template I was not able to put this work. Only changing the code of Lambda.
Basically, the code does the same, but I am not using the event['httpMethod']
Please check this:
import boto3
import json
print('Loading function')
dynamo = boto3.client('dynamodb')
def respond(err, res=None):
return {
'statusCode': '400' if err else '200',
'body': err.message if err else json.dumps(res),
'headers': {
'Content-Type': 'application/json',
},
}
def lambda_handler(event, context):
'''Demonstrates a simple HTTP endpoint using API Gateway. You have full
access to the request and response payload, including headers and
status code.
To scan a DynamoDB table, make a GET request with the TableName as a
query string parameter. To put, update, or delete an item, make a POST,
PUT, or DELETE request respectively, passing in the payload to the
DynamoDB API as a JSON body.
'''
print("Received event: " + json.dumps(event, indent=2))
operations = {
'DELETE': lambda dynamo, x: dynamo.delete_item(**x),
'GET': lambda dynamo, x: dynamo.scan(**x),
'POST': lambda dynamo, x: dynamo.put_item(**x),
'PUT': lambda dynamo, x: dynamo.update_item(**x),
}
operation = event['requestContext']['http']['method']
if operation in operations:
payload = event['queryStringParameters'] if operation == 'GET' else json.loads(event['body'])
return respond(None, operations[operation](dynamo, payload))
else:
return respond(ValueError('Unsupported method "{}"'.format(operation)))
I changed the code from:
operation = event['httpMethod']
to
operation = event['requestContext']['http']['method']
How do I get this solution?
I simply returned the entire event, checked the JSON and put it to work with the correct format.
If event appears an empty object, make sure you enabled proxy integration for the method. Proxy integration for an HTTP method adds request information into event.
See Use Lambda Proxy integration on API Gateway page.
If you are using API gateway, http method will be automatically passed to the event parameter when the lambda is triggered.
export const handler: Handler<APIGatewayProxyEvent> = async (
event: APIGatewayEvent,
context: Context
): Promise<APIGatewayProxyResult> => {
const httpMethod = event.httpMethod;
...
}

How to pass a querystring or route parameter to AWS Lambda from Amazon API Gateway

for instance if we want to use
GET /user?name=bob
or
GET /user/bob
How would you pass both of these examples as a parameter to the Lambda function?
I saw something about setting a "mapped from" in the documentation, but I can't find that setting in the API Gateway console.
method.request.path.parameter-name for a path parameter named parameter-name as defined in the Method Request page.
method.request.querystring.parameter-name for a query string parameter named parameter-name as defined in the Method Request page.
I don't see either of these options even though I defined a query string.
As of September 2017, you no longer have to configure mappings to access the request body.
All you need to do is check, "Use Lambda Proxy integration", under Integration Request, under the resource.
You'll then be able to access query parameters, path parameters and headers like so
event['pathParameters']['param1']
event["queryStringParameters"]['queryparam1']
event['requestContext']['identity']['userAgent']
event['requestContext']['identity']['sourceIP']
The steps to get this working are:
Within the API Gateway Console...
Go to Resources -> Integration Request
Click on the plus or edit icon next to the templates dropdown (odd I know since the template field is already open and the button here looks greyed out)
Explicitly type application/json in the content-type field even though it shows a default (if you don't do this it will not save and will not give you an error message)
put this in the input mapping { "name": "$input.params('name')" }
click on the check box next to the templates dropdown (I'm assuming this is what finally saves it)
I have used this mapping template to provide Body, Headers, Method, Path, and URL Query String Parameters to the Lambda event. I wrote a blog post explaining the template in more detail: http://kennbrodhagen.net/2015/12/06/how-to-create-a-request-object-for-your-lambda-event-from-api-gateway/
Here is the Mapping Template you can use:
{
"method": "$context.httpMethod",
"body" : $input.json('$'),
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
},
"queryParams": {
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
},
"pathParams": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
}
}
These days a drop-down template is included in the API Gateway Console on AWS.
For your API, click on the resource name... then GET
Expand "Body Mapping Templates"
Type in
application/json
for Content-Type (must be explicitly typed out) and click the tick
A new window will open with the words "Generate template" and a dropdown (see image).
Select
Method Request passthrough
Then click save
To access any variables, just use the following syntax (this is Python)
e.g. URL:
https://yourURL.execute-api.us-west-2.amazonaws.com/prod/confirmReg?token=12345&uid=5
You can get variables as follows:
from __future__ import print_function
import boto3
import json
print('Loading function')
def lambda_handler(event, context):
print(event['params']['querystring']['token'])
print(event['params']['querystring']['uid'])
So there is no need to explicitly name or map each variable you desire.
In order to pass parameters to your lambda function you need to create a mapping between the API Gateway request and your lambda function. The mapping is done in the Integration Request -> Mapping templates section of the selected API Gateway resource.
Create a mapping of type application/json, then on the right you will edit (click the pencil) the template.
A mapping template is actually a Velocity template where you can use ifs, loops and of course print variables on it. The template has these variables injected where you can access querystring parameters, request headers, etc. individually. With the following code you can re-create the whole querystring:
{
"querystring" : "#foreach($key in $input.params().querystring.keySet())#if($foreach.index > 0)&#end$util.urlEncode($key)=$util.urlEncode($input.params().querystring.get($key))#end",
"body" : $input.json('$')
}
Note: click on the check symbol to save the template. You can test your changes with the "test" button in your resource. But in order to test querystring parameters in the AWS console you will need to define the parameter names in the Method Request section of your resource.
Note: check the Velocity User Guide for more information about the Velocity templating language.
Then in your lambda template you can do the following to get the querystring parsed:
var query = require('querystring').parse(event.querystring)
// access parameters with query['foo'] or query.foo
The accepted answer worked fine for me, but expanding on gimenete's answer, I wanted a generic template I could use to pass through all query/path/header params (just as strings for now), and I came up the following template. I'm posting it here in case someone finds it useful:
#set($keys = [])
#foreach($key in $input.params().querystring.keySet())
#set($success = $keys.add($key))
#end
#foreach($key in $input.params().headers.keySet())
#if(!$keys.contains($key))
#set($success = $keys.add($key))
#end
#end
#foreach($key in $input.params().path.keySet())
#if(!$keys.contains($key))
#set($success = $keys.add($key))
#end
#end
{
#foreach($key in $keys)
"$key": "$util.escapeJavaScript($input.params($key))"#if($foreach.hasNext),#end
#end
}
As part of trying to answer one of my own questions here, I came across this trick.
In the API Gateway mapping template, use the following to give you the complete query string as sent by the HTTP client:
{
"querystring": "$input.params().querystring"
}
The advantage is that you don't have to limit yourself to a set of predefined mapped keys in your query string. Now you can accept any key-value pairs in the query string, if this is how you want to handle.
Note: According to this, only $input.params(x) is listed as a variable made available for the VTL template. It is possible that the internals might change and querystring may no longer be available.
Now you should be able to use the new proxy integration type for Lambda to automatically get the full request in standard shape, rather than configure mappings.
see: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html#api-gateway-set-up-lambda-proxy-integration-on-proxy-resource
GET /user?name=bob
{
"name": "$input.params().querystring.get('name')"
}
GET /user/bob
{
"name": "$input.params('name')"
}
The query string is straight forward to parse in javascript in the lambda
for GET /user?name=bob
var name = event.queryStringParameters.name;
This doesn't solve the GET user/bob question though.
A lot of the answers here are great. But I wanted something a little simpler.
I wanted something that will work with the "Hello World" sample for free. This means I wanted a simple produces a request body that matches the query string:
{
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
}
I think the top answer produces something more useful when building something real, but for getting a quick hello world running using the template from AWS this works great.
The following parameter-mapping example passes all parameters, including path, querystring and header, through to the integration endpoint via a JSON payload
#set($allParams = $input.params())
{
"params" : {
#foreach($type in $allParams.keySet())
#set($params = $allParams.get($type))
"$type" : {
#foreach($paramName in $params.keySet())
"$paramName" : "$util.escapeJavaScript($params.get($paramName))"
#if($foreach.hasNext),#end
#end
}
#if($foreach.hasNext),#end
#end
}
}
In effect, this mapping template outputs all the request parameters in the payload as outlined as follows:
{
"parameters" : {
"path" : {
"path_name" : "path_value",
...
}
"header" : {
"header_name" : "header_value",
...
}
'querystring" : {
"querystring_name" : "querystring_value",
...
}
}
}
Copied from the Amazon API Gateway Developer Guide
For getting query parameters you get them in queryStringParameters object like this
const name = event.queryStringParameters.name;
The second one is a clean URL. If your path is /user/{name}, to get the value you get it from pathParameters object like this
const name = event.pathParameters.name;
Python 3.8 with boto3 v1.16v - 2020 December
For configuring routes, you have to configure API Gateway to accept routes. otherwise other than the base route everything else will end up in a {missing auth token} or something other...
Once you configured API Gateway to accept routes, make sure that you enabled lambda proxy, so that things will work better,
to access routes,
new_route = event['path'] # /{some_url}
to access query parameter
query_param = event['queryStringParameters'][{query_key}]
As #Jonathan's answer, after mark Use Lambda Proxy integration in Integration Request, in your source code you should implement as below format to by pass 502 Bad Gateway error.
NodeJS 8.10:
exports.handler = async (event, context, callback) => {
// TODO: You could get path, parameter, headers, body value from this
const { path, queryStringParameters, headers, body } = event;
const response = {
"statusCode": 200,
"headers": {
"Content-Type": "application/json"
},
"body": JSON.stringify({
path,
query: queryStringParameters,
headers,
body: JSON.parse(body)
}),
"isBase64Encoded": false
};
return response;
};
Don't forget deploy your resource at API Gateway before re-run your API.
Response JSON just return which set in body is correct.
So, you could get path, parameter, headers, body value from event
const { path, queryStringParameters, headers, body } = event;
The Lambda function expects JSON input, therefore parsing the query string is needed. The solution is to change the query string to JSON using the Mapping Template.I used it for C# .NET Core, so the expected input should be a JSON with "queryStringParameters" parameter. Follow these 4 steps below to achieve that:
Open the mapping template of your API Gateway resource and add new application/json content-tyap:
Copy the template below, which parses the query string into JSON, and paste it into the mapping template:
{
"queryStringParameters": {#foreach($key in $input.params().querystring.keySet())#if($foreach.index > 0),#end"$key":"$input.params().querystring.get($key)"#end}
}
In the API Gateway, call your Lambda function and add the following query string (for the example): param1=111&param2=222&param3=333
The mapping template should create the JSON output below, which is the input for your Lambda function.
{
"queryStringParameters": {"param3":"333","param1":"111","param2":"222"}
}
You're done. From this point, your Lambda function's logic can use the query string parameters.
Good luck!
exports.handler = async (event) => {
let query = event.queryStringParameters;
console.log(`id: ${query.id}`);
const response = {
statusCode: 200,
body: "Hi",
};
return response;
};
You can used Lambda as "Lambda Proxy Integration" ,ref this [https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html#api-gateway-proxy-integration-lambda-function-python] , options avalible to this lambda are
For Nodejs Lambda
'event.headers', 'event.pathParameters', 'event.body', 'event.stageVariables',
and 'event.requestContext'
For Python Lambda
event['headers']['parametername'] and so on
Refer Doc :
https://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html#api-as-lambda-proxy-expose-get-method-with-path-parameters-to-call-lambda-function
You need to modify the Mapping Template
My goal was to pass a query string similar to:
protodb?sql=select * from protodb.prototab
to a Node.js 12 Lambda function via a URL from the API gateway. I tried a number of the ideas from the other answers but really wanted to do something in the most API gateway UI native way possible, so I came up with this that worked for me (as of the UI for API Gateway as of December 2020):
On the API Gateway console for a given API, under resources, select the get method. Then select its Integration Request and fill out the data for the lambda function at the top of the page.
Scroll to the bottom and open up the mapping templates section. Choose Request Body Passthrough when there are no templates defined (recommended).
Click on Add mapping templates and create one with the content-type of application/json and hit the check mark button.
For that mapping template, choose the Method Request passthrough on the drop down list for generate template which will fill the textbox under it with AWS' general way to pass everything.
Hit the save button.
Now when I tested it, I could not get the parameter to come through as event.sql under node JS in the Lambda function. It turns out that when the API gateway sends the URL sql query parameter to the Lambda function, it comes through for Node.js as:
var insql = event.params.querystring.sql;
So the trick that took some time for me was to use JSON.stringify to show the full event stack and then work my way down through the sections to be able to pull out the sql parameter from the query string.
So basically you can use the default passthrough functionality in the API gateway with the trick being how the parameters are passed when you are in the Lambda function.
The way that works for me is to
Go to Integration Request
click URL Query String Parameters
click Add query string
in name field put the query name, which is "name" here
in Mapped From field, put "method.request.querystring.name"
My 2 cents here: Lot of answers suggest to activate the option "Use Lambda Proxy Integration" and get the parameters from $.event.queryStringParameter or $.event.pathParameters. But if you happen to have Access-Control-Allow-Origin (a.k.a. CORS) activated, keep reading.
At the time of this post, Lambda Proxy integration and CORS don't work very well together. My approach was to deactivate the checkbox of Lambda Proxy integration and manually provide a Mapping templates for both request and response as follows:
Request template for application/json:
{
#set($params = $input.params().querystring)
"queryStringParameters" : {
#foreach($param in $params.keySet())
"$param" : "$util.escapeJavaScript($params.get($param))" #if($foreach.hasNext),#end
#end
},
#set($params = $input.params().path)
"pathParameters" : {
#foreach($param in $params.keySet())
"$param" : "$util.escapeJavaScript($params.get($param))" #if($foreach.hasNext),#end
#end
}
}
Mind that I named the properties as queryStringParameters and pathParameters on purpose, to mimic the names that Lambda Proxy integration would have generated. This way my lambdas won't break if one day I activate the Lambda Proxy integration.
Response template for application/json:
#set($payload = $util.parseJson($input.json('$')))
#set($context.responseOverride.status = $payload.statusCode)
$payload.body
How do you read these in your lambda (python)? (assuming parameters are optional)
def handler(event, context):
body = event["queryStringParameters"] or {}
result = myfunction(**body)
return {
"statusCode": code,
"headers": {
"content-type": "application/json",
},
"body": result
}
After reading several of these answers, I used a combination of several in Aug of 2018 to retrieve the query string params through lambda for python 3.6.
First, I went to API Gateway -> My API -> resources (on the left) -> Integration Request. Down at the bottom, select Mapping Templates then for content type enter application/json.
Next, select the Method Request Passthrough template that Amazon provides and select save and deploy your API.
Then in, lambda event['params'] is how you access all of your parameters. For query string: event['params']['querystring']