AWS Api gateway write to S3 with timestamp as filename - amazon-web-services

I'm trying to create an endpoint on Api Gateway that writes on AWS s3 bucket using a filename like data/$timestamp where timestamp is the $context.requestTimeEpoch variable that should be available in each request
My api gateway request integration is this
{
"type": "AWS",
"httpMethod": "PUT",
"uri": "arn:aws:apigateway:eu-west-1:bucketname.s3:path/data/{reqTimestamp}",
"credentials": "arn:aws:iam::xxxxx:role/ApiGatewayWriteToDemoS3Bucket",
"requestParameters": {
"integration.request.path.reqTimestamp": "context.requestTime"
},
"passthroughBehavior": "WHEN_NO_MATCH",
"timeoutInMillis": 29000,
"cacheNamespace": "5f92eajcg9",
"cacheKeyParameters": [],
"integrationResponses": {
"200": {
"statusCode": "200",
"responseParameters": {
"method.response.header.Access-Control-Allow-Origin": "'*'"
},
"responseTemplates": {
"application/json": null
}
}
}
}
However I get this error
Execution failed due to configuration error: Illegal character in path at index 49: https://bucketname.s3-eu-west-1.amazonaws.com/data/{reqTimestamp}
it worked only once, then I had an S3 authentication error but that's another story

You need to add a value in the URL Path Parameter configuration of the Integration Request in your API Gateway resource method, setting the name to reqTimestamp, and value to method.request.path.reqTimestamp.
This will map the value received by API Gateway in the URL to the path override parameter you defined.

Related

Is there any official documentation of AWS Gateway events sent to Lambda?

I've been searching for some sort of official documentation for the data types sent to AWS Lambda from integrations such as, AWS API Gateway. I've been able to find some "Examples" in the API Gateway documentation like here and here. It is also relatively easy to create a Lambda that just echoes the input event as output and check the output. For example (using a REST type API with LAMBDA_PROXY integration) you get something like:
{
"resource": "/another/{parameter}",
"path": "/another/some-parameter",
"httpMethod": "GET",
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
...
},
"multiValueHeaders": {
"Accept": [
"*/*"
],
"Accept-Encoding": [
"gzip, deflate, br"
],
...
},
"queryStringParameters": null,
"multiValueQueryStringParameters": null,
"pathParameters": {
"parameter": "some-parameter"
},
"stageVariables": null,
"requestContext": {
"resourceId": "some-id",
"resourcePath": "/another/{parameter}",
"httpMethod": "GET",
...
},
"body": null,
"isBase64Encoded": false
}
But this doesn't tell me what fields I can always, or just sometimes, expect to be in the payload or the types of the field.
The closest I have to a bit of typing of the fields is the aws-lambda TypeScript typings from the Definitely Typed project.
Is there anything more detailed, or perhaps more general, and official source I can go to for the structure and type of Lambda event payloads for API Gateway and other AWS service integrations?
This is the closest I've come to an answer, so far. If anyone has more detailed (or general information) I'd gladly take it. In the Lambda documentation you can read that (my emphasis):
The runtime passes three arguments to the handler method. The first
argument is the event object, which contains information from the
invoker. The invoker passes this information as a JSON-formatted
string when it calls Invoke, and the runtime converts it to an object.
When an AWS service invokes your function, the event structure varies
by service.
Under "Working with other services" in the lambda documentation we only find the examples presented in the question. Instead we can look in the documentation for the service in question, e.g. API Gateway. For API Gateway, under "Working with [HTTP/REST] APIs", you will find descriptions of the event fields for REST and HTTP type APIs.
With that said, the documentation provided still only refer to the described payload structure as "examples". It only provides you with the names of the fields and some basic structure. Data types would have to be inferred from the examples, see the snippets below.
The story seems to be similar for other services you might want to integrate with Lambda like:
S3
DynamoDB
SQS
All that is to be found is examples of payloads, some on the services' own developer's guide other under Lambda's developer's guide.
HTTP v2.0
The following is an example of the payload format for version 2.0 using HTTP type API. For version 1 see the HTTP type integration docs.
{
version: '2.0',
routeKey: '$default',
rawPath: '/my/path',
rawQueryString: 'parameter1=value1&parameter1=value2&parameter2=value',
cookies: [ 'cookie1', 'cookie2' ],
headers: {
'Header1': 'value1',
'Header2': 'value2'
},
queryStringParameters: { parameter1: 'value1,value2', parameter2: 'value' },
requestContext: {
accountId: '123456789012',
apiId: 'api-id',
authorizer: { jwt: {
claims: {'claim1': 'value1', 'claim2': 'value2'},
scopes: ['scope1', 'scope2']
}
},
domainName: 'id.execute-api.us-east-1.amazonaws.com',
domainPrefix: 'id',
http: {
method: 'POST',
path: '/my/path',
protocol: 'HTTP/1.1',
sourceIp: 'IP',
userAgent: 'agent'
},
requestId: 'id',
routeKey: '$default',
stage: '$default',
time: '12/Mar/2020:19:03:58 +0000',
timeEpoch: 1583348638390
},
body: 'Hello from Lambda',
pathParameters: {'parameter1': 'value1'},
isBase64Encoded: false,
stageVariables: {'stageVariable1': 'value1', 'stageVariable2': 'value2'}
}
REST v3.0
The following is an example of the payload format for version 3.0 using REST type API. For version 2.0 see the REST type integration docs.
{
"openapi": "3.0.0",
"info": {
"version": "2016-09-12T17:50:37Z",
"title": "ProxyIntegrationWithLambda"
},
"paths": {
"/{proxy+}": {
"x-amazon-apigateway-any-method": {
"parameters": [
{
"name": "proxy",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {},
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200"
}
},
"uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:123456789012:function:SimpleLambda4ProxyResource/invocations",
"passthroughBehavior": "when_no_match",
"httpMethod": "POST",
"cacheNamespace": "roq9wj",
"cacheKeyParameters": [
"method.request.path.proxy"
],
"type": "aws_proxy"
}
}
}
},
"servers": [
{
"url": "https://gy415nuibc.execute-api.us-east-1.amazonaws.com/{basePath}",
"variables": {
"basePath": {
"default": "/testStage"
}
}
}
]
}

Why is the API Gateway corrupting my binary file?

I have an AWS API Gateway acting as a proxy to a backend service:
{
"apiKeySource": "HEADER",
"name": "-",
"createdDate": 1513820260,
"binaryMediaTypes": [
"application/zip",
"application/octet-stream"
],
"endpointConfiguration": {
"types": [
"EDGE"
]
},
"id": "-"
}
The integration definition is here:
{
"integrationResponses": {
"200": {
"responseTemplates": {
"application/json": null
},
"statusCode": "200"
}
},
"passthroughBehavior": "WHEN_NO_MATCH",
"timeoutInMillis": 29000,
"uri": "http://${stageVariables.backend}:7000/{proxy}",
"connectionType": "INTERNET",
"httpMethod": "ANY",
"cacheNamespace": "iv06s3",
"type": "HTTP_PROXY",
"requestParameters": {
"integration.request.path.proxy": "method.request.path.proxy",
"integration.request.header.X-Source-IP": "context.identity.sourceIp"
},
"cacheKeyParameters": [
"method.request.path.proxy"
]
}
I have an endpoint that generates a Zip file on the fly and returns it to the requester.
When I access the endpoint directly, the file is fine. When I access it via the API Gateway, it gets corrupted.
The corruption takes the form of bytes in the original file being converted to 0xEFBFBD. This is the UTF-8 'replacement character'.
My request has Accept set to application/zip and the response has Content-Type: application/zip.
My expectation is that the API Gateway should recognize this as a binary media type and leave the file alone, but it seems pretty clear that it's processing it as text content.
What am I doing wrong?
Setting the "Binary Media Type" to "multipart/form-data" resolved a similar issue for me.
See here: AWS Api Gateway as a HTTP Proxy is currupting binary uploaded image files

AWS API Gateway CORS from Swagger

I am importing a Swagger file into a Terraform created AWS API Gateway deployment, and everything seems to work fine apart from enabling CORS. Once I have finished deploying, then I always receive the 403 CORS error from any endpoint. I have to manually then go through an enable CORS within the AWS console.
I import the swagger file via the AWS CLI like this;
aws apigateway put-rest-api \
--body file://swagger.json \
--mode overwrite \
--rest-api-id ${REST_API_ID}
The swagger file uses the below to map the correct headers;
"x-amazon-apigateway-integration": {
"responses": {
"2\\d{2}": {
"statusCode": "200",
"responseParameters": {
"method.response.header.Access-Control-Allow-Headers" : "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Correlation-ID'",
"method.response.header.Access-Control-Allow-Methods" : "'*'",
"method.response.header.Access-Control-Allow-Origin": "'*'"
}
}
}
}
This maps to response codes within the top level Swagger file;
"responses": {
"200": {
"description": "Succesfully created Report",
"headers": {
"Access-Control-Allow-Origin": {
"type": "string"
},
"Access-Control-Allow-Headers": {
"type": "string"
},
"Access-Control-Allow-Methods": {
"type": "string"
}
}
}
}
Despite doing this, I still get CORS erros straight after importing the Swagger file.
Is there anything I am missing here?

API Gateway - Lambda proxy integration and asynchronous call

If I set a header X-Amz-Invocation-Type: 'Event', the call is done asynchronously but as the Amazon documentation states (https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html#api-gateway-simple-proxy-for-lambda-output-format), when one uses Proxy Lambda Integration, the lambda function must return a well formatted response of this kind:
callback(null, {"statusCode": 200, "body": "results"})
As the lambda function is called asynchronously, the API Gateway never get an answer and then return a 502 Bad Gateway error instead of a 200 OK status.
Below an extract of the swagger configuration:
"/myFunc": {
"post": {
"parameters": [
{
"name": "myparam",
"in": "query",
"required": true,
"type": "string"
}
],
"responses": {
"200": {
"description": "200 response"
}
},
"x-amazon-apigateway-request-validator": "Validate query string parameters and headers",
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200"
}
},
"uri": "arn:aws:apigateway:ap-northeast-1:lambda:path/2015-03-31/functions/arn:aws:lambda:ap-northeast-1:idAccount:function:myFunc/invocations",
"passthroughBehavior": "when_no_match",
"httpMethod": "POST",
"type": "aws_proxy",
"requestParameters": {
"integration.request.header.X-Amz-Invocation-Type": "'Event'"
}
}
}
}
Is there a way to have it worked?
You can setup a custom lambda integration (without proxy flag). You will need to configure the mapping templates to transform the request/response to your desired format.
http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-lambda-non-proxy-integration.html#getting-started-new-lambda

AWS API Gateway and EC2 Service Proxy

I am trying to POST a json string to API Gateway and in turn have API Gateway send the JSON to an EC2 server.
My issue is I can't find good documentation from Amazon on how to accomplish this.
When I test the setup I get this
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>InvalidHttpRequest</Code><Message>The HTTP request is invalid. Reason: Unable to parse request</Message></Error></Errors><RequestID>1fa47f52-d75c-4ff8-8992-3eac11a79015</RequestID></Response>"
Which means very little to me. I assume it is an issue with API Gateway trying to send the request to EC2 and it can't so it generates this error. So perhaps I am setting up the EC2 AWS Service Proxy in API Gateway incorrectly. Which is likely because I have no idea what I am supposed to set 'Action' to right now I have it pointing to the EC2 instance, only cause i don't see any other place to put that info.
This really shouldn't be that hard I have successfully done this thing connecting to Lambda and have looked through all the documentation and all I can find is this: http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-aws-proxy.html#getting-started-aws-proxy-add-resources
Which is less than helpful for this scenario. Any Ideas?
I think you confused AWS Service Proxy and HTTP Service proxy.
API Gateway can forward API calls to different type of backends:
- a lambda function
- an AWS Service (see http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html for an example)
- an existing API, running on AWS or on premises (your use case)
When defining you API, be sure to define a POST verb and point the Endpoint URL to your EC2 instance URL
I just made a test using the JSON POST service available online at http://gurujsonrpc.appspot.com/ and it works as expected.
Here is the Swagger export of my test API.
{
"swagger": "2.0",
"info": {
"version": "2016-04-11T20:46:13Z",
"title": "test"
},
"host": "c22wfjg4d7.execute-api.eu-west-1.amazonaws.com",
"basePath": "/prod",
"schemes": [
"https"
],
"paths": {
"/": {
"post": {
"produces": [
"application/json"
],
"responses": {
"200": {
"description": "200 response",
"schema": {
"$ref": "#/definitions/Empty"
}
}
},
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200"
}
},
"uri": "http://gurujsonrpc.appspot.com/guru",
"httpMethod": "POST",
"type": "http"
}
}
}
},
"definitions": {
"Empty": {
"type": "object"
}
}
}