I setup an API using amazon api gateway and want to put data into Kinesis streams. Amazon API gateway has inbuilt support for it. But when I try to put JSON data it gives "Serialization exception".
var data = {"ua_platform":"iPhone","ua_browsercodename":"Mozilla","ua_browserlanguage":"en-us","ua_header":"Mozilla\/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit\/601.1.46 (KHTML, like Gecko) Version\/9.0 Mobile\/13B143 Safari\/601.1","ua_browsername":"Netscape","key":"livestream_hindi",,"datetime_ut":"1458711871","datetime_dt":"2016-03-23","value":"15","source":"0","browser":"Mobile Safari-9.0.","os":"iOS-9.1.","device_detail":"iPhone Apple iPhone"};
var json = JSON.stringify(data);
var params = {
'ContentType': 'application/json',
'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'
};
var body = {
"Data": json,
"StreamName": "XXXXXX",
"PartitionKey": "XXXX"
};
After this I make a put request
apigClient.functionPut(params, body, additionalParams)
.then(function(result) {
// This is where you would put a success callback
console.log("success");
})
.catch(function(result) {
// This is where you would put an error callback
console.log("catch");
});
The API gives 200 ok along with the serialization exception. It is stated in put request the "Data" key in body variable will accept only "blob" type. Now I also tried converting JSON data to BLOB but no luck at all.
I am not able to figure out what I am doing wrong. Please help.
There's a walkthrough of how to set up API Gateway in front of Kinesis in the official AWS docs at http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-kinesis.html
You must use the $utils variable to convert to base64 on your integration request template, here's my integration request template:
#set($msgBody = $util.parseJson($input.body))
#set($msgId = $msgBody.messageId)
{
"Data": "$util.base64Encode($input.body)",
"PartitionKey": "$msgId",
"StreamName": "arena-hub-dev-ks"
}
See the line with the "Data" I'm using $util.base64Encode. In this case my API Gateway endpoint accepts text/plain.
Related
I'm trying to understand how to correctly format the JSON that is sent back from my AWS rest API. I am using Api-Gateway to create an API that return the result of a Lambda.
I am used to API responses in the following format:
{
"name" : "John",
"age" : 34
}
But, in the AWS example for integrating lambda and api-gateway, they said to use the following response format:
// The output from a Lambda proxy integration must be
// in the following JSON object. The 'headers' property
// is for custom response headers in addition to standard
// ones. The 'body' property must be a JSON string. For
// base64-encoded payload, you must also set the 'isBase64Encoded'
// property to 'true'.
let response = {
statusCode: responseCode,
headers: {
"x-custom-header" : "my custom header value"
},
body: JSON.stringify(responseBody)
};
return response;
Is there a right way to do this, or can you just return whatever response style you like?
I have created an API using AWS api gateway like https://api.mydomain.com/v1/download?id=1234". The download resource has GET method. And the GET method is invoking lambda function using Lambda Proxy Integration.
The Lambda function needs to act as Proxy. It needs to resolve correct backend endpoint based on header x-clientId and then forward the request to that backend endpoint and return response as it is. So it needs to be generic to handle GET request of different content-type.
My lambda function looks like ( .NET Core)
public async Task<APIGatewayProxyResponse> Route(APIGatewayProxyRequest input, ILambdaContext context)
{
var clientId = headers["x-clientId"];
var mappings = new Mappings();
var url = await mappings.GetBackendUrl(clientId, input.Resource);
var httpClient = new HttpClient();
var response = await httpClient.GetAsync(url);
response.EnsureSuccessStatusCode();
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = false,
Body = await response.Content.ReadAsString())
};
}
The handler above works as long as request and response's content-type is application/json or application/xml. However i am not sure how to handle response when backend returns stream.
For download API, the backend returns Content-Disposition: attachment; filename="somefilename and ContentType may be one of the following:
application/pdf
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/x-zip-compressed
application/octet-stream
For these streams, How do i set APIGatewayProxyResponse.Body?
For Excel file I have tried setting body like below
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = true,
Body = Convert.ToBase64String(await response.Content.ReadAsByteArrayAsync())
};
proxyResponse.Headers.Add("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
proxyResponse.Headers.Add("Content-Disposition", "attachment; filename=\"Report.xlsx\"");
When i access the Url from the browser and try to open the file. I get error
Excel cannot open the fileReport.xlsxbecuase the file format or file extension is not valid. Verify that the file has not been corrupted and that the extention matches the format of the file
I think the issue is how i am setting the response body
Update 1
So based on AWS doc Binary Data Now Supported by API Gateway. Now as per the documentation
you can specify if you would like API Gateway to either pass the
Integration Request and Response bodies through, convert them to text
(Base64 encoding), or convert them to binary (Base64 decoding). These
options are available for HTTP, AWS Service, and HTTP Proxy
integrations. In the case of Lambda Function and Lambda Function Proxy
Integrations, which currently only support JSON, the request body is
always converted to JSON.
I am using Lambda Function Proxy, which currently support JSON. However the example here shows how to do it with Lambda Proxy.
I think what i am missing here is Binary Media Types setting and Method Response settings. Below is my setting. Not sure if these settings are correct
Binary Media
Method Response
here how solved it
1>add Binary Media Types. API->Settings->Binary Media Types -> add
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
2>In Method Response Add Content-Disposition and Content-Type headers for thestatus 200
3>In Integration Response map these headers to headers that are coming from the backend. And also set content handling convert to binary. (our backend api is returning file blob in body)
I want to use an AWS API Gateway as a proxy for fetching files from an S3 bucket and returning them to the client. I'm using a Lambda function to talk to S3 and send the file to the client via the AWS API Gateway. I've rad that the best way to do this is to use a "Lambda proxy integration" so the entire request gets piped to Lambda without any modification. But if I do that then I can't setup an Integration Response for the resulting response from my Lambda function. So all the client gets is JSON.
It seems there should be a way for the API Gateway to take the JSON and transform the request to the proper response for the client but I can't seem to figure out how to make that happen. There are lots of examples that point to setting a content-type on the response from the API Gateway manually but I need to set the content-type header to whatever the file type is.
Also for images and binary formats my Lambda function is returning a base64 encoded string and the property isBase64Encoded set to true. When I go to the "Binary Support" section and specify something like image/* as a content type that should be returned as binary, it doesn't work. I only have success by setting the Binary Support content type to */* (aka everything) which won't work for non-binary content types.
What am I missing and why does this seem so difficult?
Turns out API Gateway isn't the problem. My Lambda function wasn't returning proper headers.
For handling binary responses I found you need to set Binary Support content type to */* (aka everything) and then have your Lambda function return the property isBase64Encoded set to true. Responses that are base64 encoded and indicated as such will be decoded and served as binary while other requests will be returned as is.
Here's a simple Gist for a Lambda function that takes a given path and reads the file from S3 and returns it via the API Gateway:
/**
* This is a simple AWS Lambda function that will look for a given file on S3 and return it
* passing along all of the headers of the S3 file. To make this available via a URL use
* API Gateway with an AWS Lambda Proxy Integration.
*
* Set the S3_REGION and S3_BUCKET global parameters in AWS Lambda
* Make sure the Lambda function is passed an object with `{ pathParameters : { proxy: 'path/to/file.jpg' } }` set
*/
var AWS = require('aws-sdk');
exports.handler = function( event, context, callback ) {
var region = process.env.S3_REGION;
var bucket = process.env.S3_BUCKET;
var key = decodeURI( event.pathParameters.proxy );
// Basic server response
/*
var response = {
statusCode: 200,
headers: {
'Content-Type': 'text/plain',
},
body: "Hello world!",
};
callback( null, response );
*/
// Fetch from S3
var s3 = new AWS.S3( Object.assign({ region: region }) );
return s3.makeUnauthenticatedRequest(
'getObject',
{ Bucket: bucket, Key: key },
function(err, data) {
if (err) {
return err;
}
var isBase64Encoded = false;
if ( data.ContentType.indexOf('image/') > -1 ) {
isBase64Encoded = true;
}
var encoding = '';
if ( isBase64Encoded ) {
encoding = 'base64'
}
var resp = {
statusCode: 200,
headers: {
'Content-Type': data.ContentType,
},
body: new Buffer(data.Body).toString(encoding),
isBase64Encoded: isBase64Encoded
};
callback(null, resp);
}
);
};
via https://gist.github.com/kingkool68/26aa7a3641a3851dc70ce7f44f589350
In an AWS Lambda code, how can I get the HTTP method (e.g. GET, POST...) of an HTTP request coming from the AWS Gateway API?
I understand from the documentation that context.httpMethod is the solution for that.
However, I cannot manage to make it work.
For instance, when I try to add the following 3 lines:
if (context.httpMethod) {
console.log('HTTP method:', context.httpMethod)
}
into the AWS sample code of the "microservice-http-endpoint" blueprint as follows:
exports.handler = function(event, context) {
if (context.httpMethod) {
console.log('HTTP method:', context.httpMethod)
}
console.log('Received event:', JSON.stringify(event, null, 2));
// For clarity, I have removed the remaining part of the sample
// provided by AWS, which works well, for instance when triggered
// with Postman through the API Gateway as an intermediary.
};
I never have anything in the log because httpMethod is always empty.
The context.httpMethod approach works only in templates. So, if you want to have access to the HTTP method in your Lambda function, you need to find the method in the API Gateway (e.g. GET), go to the Integration Request section, click on Mapping Templates, and add a new mapping template for application/json. Then, select the application/json and select Mapping Template and in the edit box enter something like:
{
"http_method": "$context.httpMethod"
}
Then, when your Lambda function is called, you should see a new attribute in the event passed in called http_method which contains the HTTP method used to invoke the function.
API Gateway now has a built-in mapping template that passes along stuff like http method, route, and a lot more. I can't embed because I don't have enough points, but you get the idea.
Here is a screenshot of how you add it in the API Gateway console:
To get there navigate to AWS Console > API Gateway > (select a resource, IE - GET /home) > Integration Request > Mapping Templates > Then click on application/json and select Method Request Passthrough from dropdown shown in the screenshot above
I had this problem when I created a template microservice-http-endpoint-python project from functions.
Since it creates an HTTP API Gateway, and only REST APIs have Mapping template I was not able to put this work. Only changing the code of Lambda.
Basically, the code does the same, but I am not using the event['httpMethod']
Please check this:
import boto3
import json
print('Loading function')
dynamo = boto3.client('dynamodb')
def respond(err, res=None):
return {
'statusCode': '400' if err else '200',
'body': err.message if err else json.dumps(res),
'headers': {
'Content-Type': 'application/json',
},
}
def lambda_handler(event, context):
'''Demonstrates a simple HTTP endpoint using API Gateway. You have full
access to the request and response payload, including headers and
status code.
To scan a DynamoDB table, make a GET request with the TableName as a
query string parameter. To put, update, or delete an item, make a POST,
PUT, or DELETE request respectively, passing in the payload to the
DynamoDB API as a JSON body.
'''
print("Received event: " + json.dumps(event, indent=2))
operations = {
'DELETE': lambda dynamo, x: dynamo.delete_item(**x),
'GET': lambda dynamo, x: dynamo.scan(**x),
'POST': lambda dynamo, x: dynamo.put_item(**x),
'PUT': lambda dynamo, x: dynamo.update_item(**x),
}
operation = event['requestContext']['http']['method']
if operation in operations:
payload = event['queryStringParameters'] if operation == 'GET' else json.loads(event['body'])
return respond(None, operations[operation](dynamo, payload))
else:
return respond(ValueError('Unsupported method "{}"'.format(operation)))
I changed the code from:
operation = event['httpMethod']
to
operation = event['requestContext']['http']['method']
How do I get this solution?
I simply returned the entire event, checked the JSON and put it to work with the correct format.
If event appears an empty object, make sure you enabled proxy integration for the method. Proxy integration for an HTTP method adds request information into event.
See Use Lambda Proxy integration on API Gateway page.
If you are using API gateway, http method will be automatically passed to the event parameter when the lambda is triggered.
export const handler: Handler<APIGatewayProxyEvent> = async (
event: APIGatewayEvent,
context: Context
): Promise<APIGatewayProxyResult> => {
const httpMethod = event.httpMethod;
...
}
I see in the API Gateway FAQ that it is possible to access the request headers sent to the API Gateway...
If you already utilize OAuth tokens or any other authorization
mechanism, you can easily setup API Gateway not to require signed API
calls and simply forward the token headers to your backend for
verification.
However, I can find no example of how to do so in the documentation and it is unclear how to access this data using Lambda.
I am able to set up an open API and gain access to the JSON object that is part of a POST (Walkthrough: API Gateway and Lambda Functions), but in order to implement a OAuth 2.0 style API with my own provider I need access to the "Authorization" header.
My preference is to set this up using Lambda and Java 8, but an example using node.js would also be helpful in understanding how to accomplish this.
You can use the following Mapping Template in the Integration Request to generically map all path, query, and header parameters into the Lambda event. You will still need to register them in the Method Request section of the API Gateway but you can at least decouple the Mapping Template from the specific parameters you want to use. This way you don't have to change the Mapping Template code each time you change headers, query, or path parameters.
I wrote a blog post that gives more detail and some explanation of the Mapping Template: http://kennbrodhagen.net/2015/12/06/how-to-create-a-request-object-for-your-lambda-event-from-api-gateway/
Here is the Mapping Template you can use:
{
"method": "$context.httpMethod",
"body" : $input.json('$'),
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
},
"queryParams": {
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
},
"pathParams": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
}
}
First, you need to trap the Authorization header from the HTTP GET request. Then you need to map that value to the Lambda event object.
Go to the API method dashboard and click on Method Request. In there you can add an HTTP Request Header called Authorization as shown below.
This will trap the Authorization header so you can use it later.
Now go back to the method dashboard and click on Integration Request. From here you can pass the value of the header into the Lambda function by using a mapping like this.
{
"Authorization": "$input.params('Authorization')"
}
Now in your Lambda function you can get the value like this.
event.Authorization
You need to create input mapping inside Integration Request panel on the dashboard screen describing your API method.
Following code translates name query input parameter into Lambda Event input object:
{
"name": "$input.params('name')"
}
Screenshot:
You can find more info about this in the original API Gateway to Lambda input thread on AWS Forums.
while this is an old thread, I have found it best to use lambda proxy integration for the purpose. With this you do not have to configure anything in the API gateway and you get all the headers in your lambda function...
https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
As per Prabhat's answer setting up with the lambda proxy integration request is the simplest way to do this, after which you can access the request headers, path parameters and query parameters via
event['pathParameters']['param1']
event["queryStringParameters"]['queryparam1']
event['requestContext']['identity']['userAgent']
event['requestContext']['identity']['sourceIP']
This is an example event object:
{
"requestContext": {
"elb": {
"targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:1234567890123:targetgroup/lambda-279xxxxxxx5rsrxxxxxx/49e9d6xxxxxxxxx"
}
},
"httpMethod": "GET",
"path": "/lambda",
"queryStringParameters": {
"query": "1234ABCD"
},
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "en-US,en;q=0.9",
"connection": "keep-alive",
"host": "lambda-alb-12356789012.us-east-2.elb.amazonaws.com",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
"x-amzn-trace-id": "Root=1-5c5xxxxx-3d683b8xxxxxxxxxx",
"x-forwarded-for": "xx.xx.xxx.xxx",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-imforwards": "20"
},
"body": "",
"isBase64Encoded": false
}
The event object contains "headers" in it, you can access request headers sent to API gateway by using: event.headers.<header key>
The solution by kennbrodhagen worked great for me, see his answer and blog for the details. Since the poster expressed a preference for Java implementation, and it took me a while to figure out how to implement Kenn's handler in java, I'm just sharing the Java code that corresponds:
public class MyHandler implements RequestHandler<Map<String,Object>,String> {
#Override
public String handleRequest(Map<String,Object> eventMap, Context context) {
LambdaLogger logger = context.getLogger();
logger.log("Body:" + eventMap.get("body"));
logger.log("Headers:" + eventMap.get("headers"));
logger.log("Method:" + eventMap.get("method"));
logger.log("Params:" + eventMap.get("params"));
logger.log("Query:" + eventMap.get("query"));
return("{}");
}
}
For .Net Core 3.1+
If you have enabled Lambda Proxy integration you just have to check the Headers collection in your Lambda handler:
public APIGatewayProxyResponse FunctionHandler(APIGatewayProxyRequest data, ILambdaContext context)
{
if (data.Headers.ContainsKey("x-api-key"))
{
...
}