I have created an API using AWS api gateway like https://api.mydomain.com/v1/download?id=1234". The download resource has GET method. And the GET method is invoking lambda function using Lambda Proxy Integration.
The Lambda function needs to act as Proxy. It needs to resolve correct backend endpoint based on header x-clientId and then forward the request to that backend endpoint and return response as it is. So it needs to be generic to handle GET request of different content-type.
My lambda function looks like ( .NET Core)
public async Task<APIGatewayProxyResponse> Route(APIGatewayProxyRequest input, ILambdaContext context)
{
var clientId = headers["x-clientId"];
var mappings = new Mappings();
var url = await mappings.GetBackendUrl(clientId, input.Resource);
var httpClient = new HttpClient();
var response = await httpClient.GetAsync(url);
response.EnsureSuccessStatusCode();
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = false,
Body = await response.Content.ReadAsString())
};
}
The handler above works as long as request and response's content-type is application/json or application/xml. However i am not sure how to handle response when backend returns stream.
For download API, the backend returns Content-Disposition: attachment; filename="somefilename and ContentType may be one of the following:
application/pdf
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/x-zip-compressed
application/octet-stream
For these streams, How do i set APIGatewayProxyResponse.Body?
For Excel file I have tried setting body like below
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = true,
Body = Convert.ToBase64String(await response.Content.ReadAsByteArrayAsync())
};
proxyResponse.Headers.Add("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
proxyResponse.Headers.Add("Content-Disposition", "attachment; filename=\"Report.xlsx\"");
When i access the Url from the browser and try to open the file. I get error
Excel cannot open the fileReport.xlsxbecuase the file format or file extension is not valid. Verify that the file has not been corrupted and that the extention matches the format of the file
I think the issue is how i am setting the response body
Update 1
So based on AWS doc Binary Data Now Supported by API Gateway. Now as per the documentation
you can specify if you would like API Gateway to either pass the
Integration Request and Response bodies through, convert them to text
(Base64 encoding), or convert them to binary (Base64 decoding). These
options are available for HTTP, AWS Service, and HTTP Proxy
integrations. In the case of Lambda Function and Lambda Function Proxy
Integrations, which currently only support JSON, the request body is
always converted to JSON.
I am using Lambda Function Proxy, which currently support JSON. However the example here shows how to do it with Lambda Proxy.
I think what i am missing here is Binary Media Types setting and Method Response settings. Below is my setting. Not sure if these settings are correct
Binary Media
Method Response
here how solved it
1>add Binary Media Types. API->Settings->Binary Media Types -> add
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
2>In Method Response Add Content-Disposition and Content-Type headers for thestatus 200
3>In Integration Response map these headers to headers that are coming from the backend. And also set content handling convert to binary. (our backend api is returning file blob in body)
Related
I have an AWS api gateway and i want to send a post request or any other request if possible when content type is multipart/form-data, the problem is that i want to add a conditional extra header at the api gateway level using template mappings, i was able to do this with GET request and content type is application/json, but the same code i applied doesn't seem to work when request is POST and content type is multipart/form-data, so how can i add an extra header when using different content tyes or different requests at the api gateway level ? my code that worked is this :
#set($header3Value = "service1")
$input.json("$")
#set($context.requestOverride.header.x-service-key = $header3Value)
#elseif
($method.request.header.x-api-key == "*******************nmMBBkbbkBVCXvjhvhkk")
#set($header3Value = "service1")
$input.json("$")
#set($context.requestOverride.header.x-service-key = $header3Value)
#elseif
($method.request.header.x-api-key == "*****IHOHMubbbVYVYUVVVVLkbkbbbmmlb")
#set($header3Value = "service2")
$input.json("$")
#set($context.requestOverride.header.x-service-key = $header3Value)
#end
Edit : When I prepare a template mapping for multipart/form-data with the same code i used for Application/json, i get "internal server error" with no other info, but when i empty the template and leave it blank, it goes back to working but of course, the custom header i want to add won't be added ...
I am attempting to use a pre-signed URL to upload as described in the docs (https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html) I can retrieve the pre-signed URL but when I attempt to do a PUT in Postman, I receive the following error:
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
Obviously, the way my put call is structured doesn't match with the way AWS is calculating the signature. I can't find a lot of information on what this put call requires.
I've attempted to modify the header for Content-Type to multipart/form-data and application/octet-stream. I've also tried to untick the headers section in postman and rely on the body type for both form-data and binary settings where I select the file. The form-data setting results in the following added to the call:
Content-Disposition: form-data; name="thefiletosend.txt"; filename="thefiletosend.txt
In addition, I noticed that postman is including what it calls "temporary headers" as follows:
Host: s3.amazonaws.com
Content-Type: text/plain
User-Agent: PostmanRuntime/7.13.0
Accept: /
Cache-Control: no-cache
Postman-Token: e11d1ef0-8156-4ca7-9317-9f4d22daf6c5,2135bc0e-1285-4438-bb8e-b21d31dc36db
Host: s3.amazonaws.com
accept-encoding: gzip, deflate
content-length: 14
Connection: keep-alive
cache-control: no-cache
The Content-Type header may be one of the issues, but I'm not certain how to exclude these "temporary headers" in postman.
I am generating the pre-signed URL in a lambda as follows:
public string FunctionHandler(Input input, ILambdaContext context)
{
_logger = context.Logger;
_key = input.key;
_bucketname = input.bucketname;
string signedURL = _s3Client.GetPreSignedURL(new GetPreSignedUrlRequest()
{
Verb = HttpVerb.PUT ,
Protocol = Protocol.HTTPS,
BucketName = _bucketname,
Key = _key,
Expires = DateTime.Now.AddMinutes(5)
});
returnObj returnVal = new returnObj() { url = signedURL };
return JsonConvert.SerializeObject(returnVal);
}
Your pre-signed url should be like https://bucket-name.s3.region.amazonaws.com/folder/filename.jpg?AWSAccessKeyId=XXX&Content-Type=image%2Fjpeg&Expires=XXX&Signature=XXX
You can upload to S3 with postman by
Set above url as endpoint
Select PUT request,
Body -> binary -> Select file
I was able to get this working in Postman using a POST request. Here are the details of what worked for me. When I call my lambda to get a presigned URL here is the json that comes back (after I masked sensitive and app-specific information):
{
"attachmentName": "MySecondAttachment.docx",
"url": "https://my-s3-bucket.s3.amazonaws.com/",
"fields": {
"acl": "public-read",
"Content-Type": "multipart/form-data",
"key": "attachment-upload/R271645/65397746_MySecondAttachment.docx",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "WWWWWWWW/20200318/us-east-1/s3/aws4_request",
"x-amz-date": "20200318T133309Z",
"x-amz-security-token": "XXXXXXXX",
"policy": "YYYYYYYY",
"x-amz-signature": "ZZZZZZZZ"
}
}
In Postman, create a POST request, and use “form-data” to enter in all the fields you got back, with exactly the same field names you got back in the signedURL shown above. Do not set the content type, however. Then add one more key named “file”:
To the right of the word file if you click the drop-down you can browse to your file and attach it:
In case it helps, I’m using a lambda written in python to generate a presigned URL so a user can upload an attachment. The code looks like this:
signedURL = self.s3.generate_presigned_post(
Bucket= "my-s3-bucket",
Key=putkey,
Fields = {"acl": "public-read", "Content-Type": "multipart/form-data"},
ExpiresIn = 15,
Conditions = [
{"acl": "public-read"},
["content-length-range", 1, 5120000]
]
)
Hope this helps.
Your pre-signed url should be like https://bucket-name.s3.region.amazonaws.com/folder/filename.jpg?AWSAccessKeyId=XXX&Content-Type=image%2Fjpeg&Expires=XXX&Signature=XXX
You can upload to S3 with postman by
Set above url as endpoint
Select PUT request,
Body -> binary -> Select file
I was facing the same problem and below is how it worked for me.
Note, I am making signed URL by using AWS S3 Java SDK as my backend is in Java. I gave content type as "application/octet-stream" while creating this signed Url so that any type of content can be uploaded. Below is my java code generating signed url.
public String createS3SignedURLUpload(String bucketName, String objectKey) {
try {
PutObjectRequest objectRequest = PutObjectRequest.builder().bucket(bucketName).key(objectKey)
.contentType("**application/octet-stream**").build();
S3Presigner presigner = S3Presigner.builder().region(s3bucketRegions.get(bucketName))
.credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials)).build();
PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(presignedURLTimeoutInMins)).putObjectRequest(objectRequest)
.build();
PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest);
return presignedRequest.url().toString();
} catch (Exception e) {
throw new CustomRuntimeException(e.getMessage());
}
}
## Now to upload file using Postman
Set the generated url as endpoint
Select PUT request,
Body -> binary -> Select file
Set header Content-Type as application/octet-stream (This point I was missing earlier)
It's actually depends in how you generated URL,
If you generated using JAVA,
Set the generated url as endpoint
Select PUT request,
Body -> binary -> Select file
If you generated using PYTHON,
Create a POST request, and use form-data to enter in all the fields you got back, with exactly the same field names you got back in the signedURL shown above.
Do not set the content type, however. Then add one more key named “file”:
Refer the accepted answer picture
I am generating a presigned URL server-side to allow my client application to upload a file directly to the S3 bucket. Everything works fine unless the client application is running on a computer in a timezone that is technically a day ahead of my server clock.
I can recreate the issue locally by setting my system clock ahead to a timezone on the next day.
Here is how I am generating the presigned URL using the .NET SDK (I originally had DateTime.Now instead of UTCNow):
var request = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = objectName,
Verb = HttpVerb.PUT,
Expires = DateTime.UtcNow.AddDays(5),
ContentType = "application/octet-stream"
};
request.Headers["x-amz-acl"] = "bucket-owner-full-control";
request.Metadata.Add("call", JsonConvert.SerializeObject(call).ToString());
return client.GetPreSignedURL(request);
and then I am using that presigned URL in the client application like this:
using (var fileStream = new FileStream(recordingPath, FileMode.Open))
using (var client = new WebClient())
{
HttpContent fileStreamContent = new StreamContent(fileStream);
var bytes = await fileStreamContent.ReadAsByteArrayAsync();
client.Headers.Add("Content-Type", "application/octet-stream");
//include metadata in PUT request
client.Headers.Add("x-amz-meta-call", JsonConvert.SerializeObject(Call));
await client.UploadDataTaskAsync(new Uri(presignedUrl), "PUT", bytes);
}
Here is the error I am receiving from AWS:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>{access}</AWSAccessKeyId><StringToSign>....
The requests appear mostly identical to me in Fiddler.
Works:
PUT https://{bucketname}.s3.amazonaws.com/1c849c76-dd2a-4ff7-aad7-23ec7e9ddd45_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={cred}&X-Amz-Date=20190312T021419Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 28289
Expect: 100-continue
{file}
Does not work:
PUT https://{bucketname}.s3.amazonaws.com/4cca3ec3-9f3f-4ba4-9d81-6336090610c0_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={credentials}&X-Amz-Date=20190312T021541Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 18714
Expect: 100-continue
{file}
In both scenarios, the presigned URL has the same x-amz-date parameter generated. I have even tried parsing out the x-amz-date parameter from the URL and explicitly setting it as a header in my PUT but that did not work either.
What am I missing?
It turned out to me that I was using a different version of the signature.
v4 worked perfectly for me.
In JS, require S3 as
const s3 = new AWS.S3({
signatureVersion: 'v4'
});
So the issue ended up being within the metadata. In our setup, we had the client application posting a JSON string up to our API along with the file to generate the presigned URL. We were using Json.net to deserialize into the C# class:
var call = JsonConvert.DeserializeObject<Call>(request.Params["metadata"]);
Apparently, this call converts any timestamps in the Json to local time. This means that we would sign the URL with metadata timestamps local to the API server, but actually upload the file with metadata timestamps local to the client. This difference is why the calculated signatures are different.
I want to use an AWS API Gateway as a proxy for fetching files from an S3 bucket and returning them to the client. I'm using a Lambda function to talk to S3 and send the file to the client via the AWS API Gateway. I've rad that the best way to do this is to use a "Lambda proxy integration" so the entire request gets piped to Lambda without any modification. But if I do that then I can't setup an Integration Response for the resulting response from my Lambda function. So all the client gets is JSON.
It seems there should be a way for the API Gateway to take the JSON and transform the request to the proper response for the client but I can't seem to figure out how to make that happen. There are lots of examples that point to setting a content-type on the response from the API Gateway manually but I need to set the content-type header to whatever the file type is.
Also for images and binary formats my Lambda function is returning a base64 encoded string and the property isBase64Encoded set to true. When I go to the "Binary Support" section and specify something like image/* as a content type that should be returned as binary, it doesn't work. I only have success by setting the Binary Support content type to */* (aka everything) which won't work for non-binary content types.
What am I missing and why does this seem so difficult?
Turns out API Gateway isn't the problem. My Lambda function wasn't returning proper headers.
For handling binary responses I found you need to set Binary Support content type to */* (aka everything) and then have your Lambda function return the property isBase64Encoded set to true. Responses that are base64 encoded and indicated as such will be decoded and served as binary while other requests will be returned as is.
Here's a simple Gist for a Lambda function that takes a given path and reads the file from S3 and returns it via the API Gateway:
/**
* This is a simple AWS Lambda function that will look for a given file on S3 and return it
* passing along all of the headers of the S3 file. To make this available via a URL use
* API Gateway with an AWS Lambda Proxy Integration.
*
* Set the S3_REGION and S3_BUCKET global parameters in AWS Lambda
* Make sure the Lambda function is passed an object with `{ pathParameters : { proxy: 'path/to/file.jpg' } }` set
*/
var AWS = require('aws-sdk');
exports.handler = function( event, context, callback ) {
var region = process.env.S3_REGION;
var bucket = process.env.S3_BUCKET;
var key = decodeURI( event.pathParameters.proxy );
// Basic server response
/*
var response = {
statusCode: 200,
headers: {
'Content-Type': 'text/plain',
},
body: "Hello world!",
};
callback( null, response );
*/
// Fetch from S3
var s3 = new AWS.S3( Object.assign({ region: region }) );
return s3.makeUnauthenticatedRequest(
'getObject',
{ Bucket: bucket, Key: key },
function(err, data) {
if (err) {
return err;
}
var isBase64Encoded = false;
if ( data.ContentType.indexOf('image/') > -1 ) {
isBase64Encoded = true;
}
var encoding = '';
if ( isBase64Encoded ) {
encoding = 'base64'
}
var resp = {
statusCode: 200,
headers: {
'Content-Type': data.ContentType,
},
body: new Buffer(data.Body).toString(encoding),
isBase64Encoded: isBase64Encoded
};
callback(null, resp);
}
);
};
via https://gist.github.com/kingkool68/26aa7a3641a3851dc70ce7f44f589350
In an AWS Lambda code, how can I get the HTTP method (e.g. GET, POST...) of an HTTP request coming from the AWS Gateway API?
I understand from the documentation that context.httpMethod is the solution for that.
However, I cannot manage to make it work.
For instance, when I try to add the following 3 lines:
if (context.httpMethod) {
console.log('HTTP method:', context.httpMethod)
}
into the AWS sample code of the "microservice-http-endpoint" blueprint as follows:
exports.handler = function(event, context) {
if (context.httpMethod) {
console.log('HTTP method:', context.httpMethod)
}
console.log('Received event:', JSON.stringify(event, null, 2));
// For clarity, I have removed the remaining part of the sample
// provided by AWS, which works well, for instance when triggered
// with Postman through the API Gateway as an intermediary.
};
I never have anything in the log because httpMethod is always empty.
The context.httpMethod approach works only in templates. So, if you want to have access to the HTTP method in your Lambda function, you need to find the method in the API Gateway (e.g. GET), go to the Integration Request section, click on Mapping Templates, and add a new mapping template for application/json. Then, select the application/json and select Mapping Template and in the edit box enter something like:
{
"http_method": "$context.httpMethod"
}
Then, when your Lambda function is called, you should see a new attribute in the event passed in called http_method which contains the HTTP method used to invoke the function.
API Gateway now has a built-in mapping template that passes along stuff like http method, route, and a lot more. I can't embed because I don't have enough points, but you get the idea.
Here is a screenshot of how you add it in the API Gateway console:
To get there navigate to AWS Console > API Gateway > (select a resource, IE - GET /home) > Integration Request > Mapping Templates > Then click on application/json and select Method Request Passthrough from dropdown shown in the screenshot above
I had this problem when I created a template microservice-http-endpoint-python project from functions.
Since it creates an HTTP API Gateway, and only REST APIs have Mapping template I was not able to put this work. Only changing the code of Lambda.
Basically, the code does the same, but I am not using the event['httpMethod']
Please check this:
import boto3
import json
print('Loading function')
dynamo = boto3.client('dynamodb')
def respond(err, res=None):
return {
'statusCode': '400' if err else '200',
'body': err.message if err else json.dumps(res),
'headers': {
'Content-Type': 'application/json',
},
}
def lambda_handler(event, context):
'''Demonstrates a simple HTTP endpoint using API Gateway. You have full
access to the request and response payload, including headers and
status code.
To scan a DynamoDB table, make a GET request with the TableName as a
query string parameter. To put, update, or delete an item, make a POST,
PUT, or DELETE request respectively, passing in the payload to the
DynamoDB API as a JSON body.
'''
print("Received event: " + json.dumps(event, indent=2))
operations = {
'DELETE': lambda dynamo, x: dynamo.delete_item(**x),
'GET': lambda dynamo, x: dynamo.scan(**x),
'POST': lambda dynamo, x: dynamo.put_item(**x),
'PUT': lambda dynamo, x: dynamo.update_item(**x),
}
operation = event['requestContext']['http']['method']
if operation in operations:
payload = event['queryStringParameters'] if operation == 'GET' else json.loads(event['body'])
return respond(None, operations[operation](dynamo, payload))
else:
return respond(ValueError('Unsupported method "{}"'.format(operation)))
I changed the code from:
operation = event['httpMethod']
to
operation = event['requestContext']['http']['method']
How do I get this solution?
I simply returned the entire event, checked the JSON and put it to work with the correct format.
If event appears an empty object, make sure you enabled proxy integration for the method. Proxy integration for an HTTP method adds request information into event.
See Use Lambda Proxy integration on API Gateway page.
If you are using API gateway, http method will be automatically passed to the event parameter when the lambda is triggered.
export const handler: Handler<APIGatewayProxyEvent> = async (
event: APIGatewayEvent,
context: Context
): Promise<APIGatewayProxyResult> => {
const httpMethod = event.httpMethod;
...
}