I have an AWS API Gateway acting as a proxy to a backend service:
{
"apiKeySource": "HEADER",
"name": "-",
"createdDate": 1513820260,
"binaryMediaTypes": [
"application/zip",
"application/octet-stream"
],
"endpointConfiguration": {
"types": [
"EDGE"
]
},
"id": "-"
}
The integration definition is here:
{
"integrationResponses": {
"200": {
"responseTemplates": {
"application/json": null
},
"statusCode": "200"
}
},
"passthroughBehavior": "WHEN_NO_MATCH",
"timeoutInMillis": 29000,
"uri": "http://${stageVariables.backend}:7000/{proxy}",
"connectionType": "INTERNET",
"httpMethod": "ANY",
"cacheNamespace": "iv06s3",
"type": "HTTP_PROXY",
"requestParameters": {
"integration.request.path.proxy": "method.request.path.proxy",
"integration.request.header.X-Source-IP": "context.identity.sourceIp"
},
"cacheKeyParameters": [
"method.request.path.proxy"
]
}
I have an endpoint that generates a Zip file on the fly and returns it to the requester.
When I access the endpoint directly, the file is fine. When I access it via the API Gateway, it gets corrupted.
The corruption takes the form of bytes in the original file being converted to 0xEFBFBD. This is the UTF-8 'replacement character'.
My request has Accept set to application/zip and the response has Content-Type: application/zip.
My expectation is that the API Gateway should recognize this as a binary media type and leave the file alone, but it seems pretty clear that it's processing it as text content.
What am I doing wrong?
Setting the "Binary Media Type" to "multipart/form-data" resolved a similar issue for me.
See here: AWS Api Gateway as a HTTP Proxy is currupting binary uploaded image files
Related
I'm in the process of learning AWS Lambda. I have created a lambda that will act as a REST API (APIEvent in CloudFormation terms) and want to debug that Lambda locally using an event.
If I understand correctly, running sam local generate-event apigateway aws-proxy generates an event that is suitable for locally running/debugging my Lambda. This produces the following event (some nested values are abbreviated):
{
"body": "eyJ0ZXN0IjoiYm9keSJ9",
"resource": "/{proxy+}",
"path": "/path/to/resource",
"httpMethod": "POST",
"isBase64Encoded": true,
"queryStringParameters": {
"foo": "bar"
},
"multiValueQueryStringParameters": {
"foo": [
"bar"
]
},
"pathParameters": {
"proxy": "/path/to/resource"
},
"stageVariables": {
"baz": "qux"
},
"headers": {
...
},
"multiValueHeaders": {
...
},
"requestContext": {
"accountId": "123456789012",
"resourceId": "123456",
"stage": "prod",
"requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
"requestTime": "09/Apr/2015:12:34:56 +0000",
"requestTimeEpoch": 1428582896000,
"identity": {
...
},
"path": "/prod/path/to/resource",
"resourcePath": "/{proxy+}",
"httpMethod": "POST",
"apiId": "1234567890",
"protocol": "HTTP/1.1"
}
}
However, I do not understand the relation between resource (which is copied in requestContext.resourcePath) and path (which is copied in requestContext.path).
What am I supposed to fill in for these values?
The resource is the API resource you defined in the API gateway. For example in your case /{proxy+}.
The path is the actual path from the request.
So, when you make a request GET https://yourdomain.com/v1/pets :
the path is /v1/pets
the resource is /{proxy+}
I am using flask-restx to build an app with a swagger UI and I trying to upload this swagger file as a documentation part in AWS API Gateway. Through this swagger UI, I am enabling the user to upload a CSV file for further data processing.
I have the following swagger json:
{
"swagger": "2.0",
"basePath": "/",
"paths": {
"/upload_profile/csv": {
"post": {
"responses": {
"200": {
"description": "Profile uploaded"
},
"400": {
"description": "Validation Error"
},
"401": {
"description": "Not authorized"
}
},
"operationId": "Get uploaded profiles from user",
"parameters": [
{
"name": "csv_file",
"in": "formData",
"type": "file",
"required": true,
"description": "CSV file"
}
],
"consumes": [
"multipart/form-data"
],
"tags": [
"upload_profile"
]
}
}
},
"info": {
"title": "Upload Profile",
"version": "0.0.1"
},
"produces": [
"application/json"
],
"consumes": [
"application/json"
],
"tags": [
{
"name": "upload_profile",
"description": "Uploading User Profiles"
}
],
"responses": {
"ParseError": {
"description": "When a mask can't be parsed"
},
"MaskError": {
"description": "When any error occurs on mask"
}
}
}
When I go to API Gateway --> Documentation --> Import Documentation and paste the json, I get the following error:
How can the following issue be solved? If formData isn't supported by API Gateway, is there an alternate for hosting the swagger UI?
The problem is that AWS API Gateway expects swagger/OpenAPI version 3, and your file is version 2. If you only want a way to host swagger UI for documentation/collaboration purposes, take a look at SwaggerHub https://swagger.io/tools/swaggerhub/.
But, if you really have to use AWS API Gateway, then you need to get spec in OpenAPI-3 format. Since the API is rather small, I'd suggest preparing OpenAPI-3 spec yourself (rather than generating it) and testing it locally via swagger UI.
I've been searching for some sort of official documentation for the data types sent to AWS Lambda from integrations such as, AWS API Gateway. I've been able to find some "Examples" in the API Gateway documentation like here and here. It is also relatively easy to create a Lambda that just echoes the input event as output and check the output. For example (using a REST type API with LAMBDA_PROXY integration) you get something like:
{
"resource": "/another/{parameter}",
"path": "/another/some-parameter",
"httpMethod": "GET",
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
...
},
"multiValueHeaders": {
"Accept": [
"*/*"
],
"Accept-Encoding": [
"gzip, deflate, br"
],
...
},
"queryStringParameters": null,
"multiValueQueryStringParameters": null,
"pathParameters": {
"parameter": "some-parameter"
},
"stageVariables": null,
"requestContext": {
"resourceId": "some-id",
"resourcePath": "/another/{parameter}",
"httpMethod": "GET",
...
},
"body": null,
"isBase64Encoded": false
}
But this doesn't tell me what fields I can always, or just sometimes, expect to be in the payload or the types of the field.
The closest I have to a bit of typing of the fields is the aws-lambda TypeScript typings from the Definitely Typed project.
Is there anything more detailed, or perhaps more general, and official source I can go to for the structure and type of Lambda event payloads for API Gateway and other AWS service integrations?
This is the closest I've come to an answer, so far. If anyone has more detailed (or general information) I'd gladly take it. In the Lambda documentation you can read that (my emphasis):
The runtime passes three arguments to the handler method. The first
argument is the event object, which contains information from the
invoker. The invoker passes this information as a JSON-formatted
string when it calls Invoke, and the runtime converts it to an object.
When an AWS service invokes your function, the event structure varies
by service.
Under "Working with other services" in the lambda documentation we only find the examples presented in the question. Instead we can look in the documentation for the service in question, e.g. API Gateway. For API Gateway, under "Working with [HTTP/REST] APIs", you will find descriptions of the event fields for REST and HTTP type APIs.
With that said, the documentation provided still only refer to the described payload structure as "examples". It only provides you with the names of the fields and some basic structure. Data types would have to be inferred from the examples, see the snippets below.
The story seems to be similar for other services you might want to integrate with Lambda like:
S3
DynamoDB
SQS
All that is to be found is examples of payloads, some on the services' own developer's guide other under Lambda's developer's guide.
HTTP v2.0
The following is an example of the payload format for version 2.0 using HTTP type API. For version 1 see the HTTP type integration docs.
{
version: '2.0',
routeKey: '$default',
rawPath: '/my/path',
rawQueryString: 'parameter1=value1¶meter1=value2¶meter2=value',
cookies: [ 'cookie1', 'cookie2' ],
headers: {
'Header1': 'value1',
'Header2': 'value2'
},
queryStringParameters: { parameter1: 'value1,value2', parameter2: 'value' },
requestContext: {
accountId: '123456789012',
apiId: 'api-id',
authorizer: { jwt: {
claims: {'claim1': 'value1', 'claim2': 'value2'},
scopes: ['scope1', 'scope2']
}
},
domainName: 'id.execute-api.us-east-1.amazonaws.com',
domainPrefix: 'id',
http: {
method: 'POST',
path: '/my/path',
protocol: 'HTTP/1.1',
sourceIp: 'IP',
userAgent: 'agent'
},
requestId: 'id',
routeKey: '$default',
stage: '$default',
time: '12/Mar/2020:19:03:58 +0000',
timeEpoch: 1583348638390
},
body: 'Hello from Lambda',
pathParameters: {'parameter1': 'value1'},
isBase64Encoded: false,
stageVariables: {'stageVariable1': 'value1', 'stageVariable2': 'value2'}
}
REST v3.0
The following is an example of the payload format for version 3.0 using REST type API. For version 2.0 see the REST type integration docs.
{
"openapi": "3.0.0",
"info": {
"version": "2016-09-12T17:50:37Z",
"title": "ProxyIntegrationWithLambda"
},
"paths": {
"/{proxy+}": {
"x-amazon-apigateway-any-method": {
"parameters": [
{
"name": "proxy",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {},
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200"
}
},
"uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:123456789012:function:SimpleLambda4ProxyResource/invocations",
"passthroughBehavior": "when_no_match",
"httpMethod": "POST",
"cacheNamespace": "roq9wj",
"cacheKeyParameters": [
"method.request.path.proxy"
],
"type": "aws_proxy"
}
}
}
},
"servers": [
{
"url": "https://gy415nuibc.execute-api.us-east-1.amazonaws.com/{basePath}",
"variables": {
"basePath": {
"default": "/testStage"
}
}
}
]
}
I'm trying to create an endpoint on Api Gateway that writes on AWS s3 bucket using a filename like data/$timestamp where timestamp is the $context.requestTimeEpoch variable that should be available in each request
My api gateway request integration is this
{
"type": "AWS",
"httpMethod": "PUT",
"uri": "arn:aws:apigateway:eu-west-1:bucketname.s3:path/data/{reqTimestamp}",
"credentials": "arn:aws:iam::xxxxx:role/ApiGatewayWriteToDemoS3Bucket",
"requestParameters": {
"integration.request.path.reqTimestamp": "context.requestTime"
},
"passthroughBehavior": "WHEN_NO_MATCH",
"timeoutInMillis": 29000,
"cacheNamespace": "5f92eajcg9",
"cacheKeyParameters": [],
"integrationResponses": {
"200": {
"statusCode": "200",
"responseParameters": {
"method.response.header.Access-Control-Allow-Origin": "'*'"
},
"responseTemplates": {
"application/json": null
}
}
}
}
However I get this error
Execution failed due to configuration error: Illegal character in path at index 49: https://bucketname.s3-eu-west-1.amazonaws.com/data/{reqTimestamp}
it worked only once, then I had an S3 authentication error but that's another story
You need to add a value in the URL Path Parameter configuration of the Integration Request in your API Gateway resource method, setting the name to reqTimestamp, and value to method.request.path.reqTimestamp.
This will map the value received by API Gateway in the URL to the path override parameter you defined.
If I set a header X-Amz-Invocation-Type: 'Event', the call is done asynchronously but as the Amazon documentation states (https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html#api-gateway-simple-proxy-for-lambda-output-format), when one uses Proxy Lambda Integration, the lambda function must return a well formatted response of this kind:
callback(null, {"statusCode": 200, "body": "results"})
As the lambda function is called asynchronously, the API Gateway never get an answer and then return a 502 Bad Gateway error instead of a 200 OK status.
Below an extract of the swagger configuration:
"/myFunc": {
"post": {
"parameters": [
{
"name": "myparam",
"in": "query",
"required": true,
"type": "string"
}
],
"responses": {
"200": {
"description": "200 response"
}
},
"x-amazon-apigateway-request-validator": "Validate query string parameters and headers",
"x-amazon-apigateway-integration": {
"responses": {
"default": {
"statusCode": "200"
}
},
"uri": "arn:aws:apigateway:ap-northeast-1:lambda:path/2015-03-31/functions/arn:aws:lambda:ap-northeast-1:idAccount:function:myFunc/invocations",
"passthroughBehavior": "when_no_match",
"httpMethod": "POST",
"type": "aws_proxy",
"requestParameters": {
"integration.request.header.X-Amz-Invocation-Type": "'Event'"
}
}
}
}
Is there a way to have it worked?
You can setup a custom lambda integration (without proxy flag). You will need to configure the mapping templates to transform the request/response to your desired format.
http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-lambda-non-proxy-integration.html#getting-started-new-lambda