I'm using API Gateway Lambda proxy integration and trying to return a binary application/protobuf response. No matter what I do, the response body is always a base64 encoded string
I have application/protobuf setup as a binary media types in APIG
My client (javascript) is sending following headers in the POST:
Accept: application/protobuf
Content-Type: application/protobuf
My lambda is responing with content-type: application/protobuf, and correctly setting the IsBase64Encoded Lambda response to true
How do you get APIG to base64 decode the string? I swear I had this working a few months ago when I 1st tried this.
Note: I've also tried */* as a binary media types
Some related posts to add background:
https://github.com/twitchtv/twirp/issues/81
https://github.com/awslabs/aws-serverless-express/issues/39#issuecomment-276019222
Update:
Turns out I can only get it working if binary media type is set to */*. The client Accept header has no impact once it is set to this.
Many bad side effects of using */* because every response is attempted to get decoded (even when IsBase64Encoded is false or not set)
I thought it wasn't decoding because Chrome network inspect tools will always show binary data as base64 encoded in the Preview tab. You can see the protobuf in the Response tab.
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
Your client must send an Accept header who's 1st media type matches what you have set as a binary media types in API Gateway
Your Lambda, if serving a binary media type, must set IsBase64Encoded to true AND the body must be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
Related
I am using AWS API Gateway put method to post an image to S3 bucket. In API Gateway settings when I add / in Binary Media Types I get the following error message
Access to fetch at ‘some_invokation_request/my_s3_bucket/image.png’ from origin ‘http://localhost:3001’ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: It does not have HTTP ok status.
When I remove / from Binary media types in settings I can post an image and when I check the posted image in S3 it's messed up.
I'll post the original image and S3 image
Apparently this is an open issue right now and you can find further details here
I believe you have two different errors present. One is CORS related and the other is the upload of an image getting messed up. I don't think they are necessarily related. I've hit the same issue with the image upload showing the tiny white square but faced no CORS issue.
For me, the problem turned out to be trying to combine the use of Convert to Binary (if needed) and a mapping template on the Integration Request. According to the documentation (and my experience as well), if a mapping template is defined, APIGW doesn't run the Convert to Binary passthrough functionality. What gets passed to the S3 bucket for saving is a base64 encoded string instead of actual binary, thus the messed up image.
Excerpt from the linked documentation:
"On the contentHandling property of the Integration resource, set CONVERT_TO_BINARY. Set WHEN_NO_MATCH as the passthroughBehavior property value without defining a mapping template. This enables API Gateway to invoke the passthrough template."
As you may know AWS Lambdas have a hard request(and response) payload limit of 6M. I’m calling a Java Lambda deployed on API Gateway via a REST API with a gzip Content-Encoding, but API Gateway decompresses the request and calls the lambda with the uncompressed payload, so making a gzip request through API Gateway doesn’t help with the request payload limit. So if I want to increase my request payload limit, how should I make the REST call with a gzipped request body without the API Gateway decompressing it? A couple options:
configure API Gateway to allow gzip Content-Encoding requests through as-is. This is my preferred option, but I haven’t figured out if this is possible. There’s a x-amazon-apigateway-binary-media-types option to treat certain media types as binary, but changing this doesn’t seem to affect the decompression of incoming requests. Ditto with the minimumCompressionSize option.
Change the incoming request to be Content-Type: application/octet-stream instead of Content-Type: application/json and Content-Encoding: gzip, and the in the API if it receives that type we decompress it as gzip in the lambda. This would probably work, but it’s a bit hacky to use the generic octet-stream for a covert gzip request here.
What do you all think about these options(or some other option I haven’t considered)? And yes, I know if I make the request via an async process I can get around the request limit, but async doesn't work well for the application.
I have a service that returns image data as base64 encoded text in the body:e.g.
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASwAAAEsCAYAAAB5fY51AAAgAE...etc
However, when we put the service behind the api-gateway behaving as a proxy, the same request will seemingly convert the response to binary
�PNG
IHDR,,y}�u IDA... etc
and has a response header
Content-Type: application/json
The documentation explains how to convert TO binary automatically, but makes no mention on how to leave the response unmodified. I've tried changing contentHandling to CONVERT_TO_TEXT with no success, so I'm wondering if there's another way of having the api gateway not mess with my service responses.
I believe you are using the same setup as me;
API gateway
V
Proxy (no contentHandling available)
V
Lambda (returns {body: Buffer.toString('base64')})
This solved it for me: https://stackoverflow.com/a/47780921/853015.
I have a lambda that needs to return a binary object and some http headers (e.g. content-type) through an api gateway (using lambda integration) OR redirect to another URL. In the binary support examples (e.g. https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/) the lambda only returns the (base64 of the) binary object (the image). In my case, I also need to return a status code and http headers (or something equivalent). I struggle with how I can make this work with binary support in api gateway.
The lambda returns a json on this form:
{
"statusCode": 200,
"headers": {
"content-type": "image/jpeg"
},
"body": "/9j/4AAQS...gLDAoKCAwZK",
"isBase64Encoded": true
}
In the integration response I add body mappings for image/jpeg (etc) of the form:
$input.json('$.body')
And header mapping for 'content-type' like so:
integration.response.body.headers['content-type']
I've tried many variations of the above, but the result is consistently
Execution failed due to configuration error: Unable to transform response
How do I transform the json from the lambda into a form that can be converted to binary by the api gateway, with http headers and all? Can I get more debug logging out of the api gateway to show more specific what it is unhappy with?
Is there perhaps a way to get more debug logging out of the api gateway?
I recently got this working after facing a similar problem.
In my case, I was missing two things:
First, I needed to change the list of types AWS will send to the upstream in the "Accept" header"
"x-amazon-apigateway-binary-media-types" : [
"image/jpeg"
]
Secondly, I needed to set the Integration Response to "Convert to binary (if needed)":
"contentHandling": "CONVERT_TO_BINARY"
See this answer for the details, and sample config.
I also found that I wasn't being patient enough. Whenever I deployed the API, I was checking immediately, instead of waiting a few minutes for the changes to propagate.
I tired returning binary data with base64 encoding, but I couldn't manage to return it from Lambda function through API Gateway.
Therefore, I decided to redirect the URL. I changed the Method Response to 302 and added "Location" Response header. I also deleted response code 200 from Integration Response, selected 302 as response code and mapped integration.response.body.location value with Location header. My Lambda code returned location of redirect URL in this format:
{"location":"www.google.com/"}
Hope this helps.
I've been struggling to do the same for days, but couldn't find any documentation to back up that it's possible, instead I found this
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-workflow.html
When converting a text payload to a binary blob, API Gateway assumes that the > text data is a Base64-encoded string and outputs the binary data as a
Base64-decoded blob. If the conversion fails, it returns a 500 response
indicating an API configuration error. You do not provide a mapping template
for such a conversion, although you must enable the passthrough behaviors on
the API.
When converting a binary payload to a text string, API Gateway always applies > a Base64 encoding on the binary data. You can define a mapping template for
such a payload, but can only access the Base64-encoded string in the mapping
template through $input.body, as shown in the following excerpt of an example > mapping template.
which sounds to me that mapping like this is only possible the other way around; what I ended up doing is that just return Base64Encoded binary string and hard code the content-type and cache- control in header mapping; for the complete implementation see my blog post https://mesfinmoges.com/dynamic-image-resizing-using-amazon-s3-aws-lambda-mazon-api-gateway-amazon-cloudfront/
Problem Statement:- Not able to stream PDF, using AWS gateway/Lambda setup.
I've following Setup:-
Gateway API -> Lambda Function(java) -> S3
Api should stream a PDF back to the client via Lambda from S3 server.
I am unable to do so.
Things tried
1) On Lambda side
Use RequestStreamHandler, write PDF on OutputStream
outputStream.write();
Tried Setting Base64 encoding using following
AWS Base64 Utility
java.util Base64 Utility
apache Base64 utility
Send Byte[] without encoding
Use non-proxy handler, write PDF as string to one of the output variable.
Tried Setting Base64 encoding using following
AWS Base64 Utility
java.util Base64 Utility
apache Base64 utility
Send Byte[] without encoding
2) On Gateway Side
Set Proxy integration
Remove Proxy integration
Tried Set Content Handling using
Passthrough
Convert to Binary
Convert to String
Set headers
Content-Type = 'application/pdf'
Content-Disposition = 'attachment; filename="nameofpdffile.pdf"'
I've tried all the permutations/combinations of these 2. However, I am not able to get it to work and documentation around this area seems poor.
When I set Base64 encoding in Lambda and on gateway side specify content handling as convert to binary, i get error, fail to conert using Base64 decode.
For other combinations, I get binary data as output when I test it using gateway Test funciton, however my client(Postman, Chrome, Safari) fail to convert it to PDF
Note:- I've a workaround in place, where I create a signedURL to S3 object, and send it as redirect. However, I am trying to make it work by streaming a binary file using Lambda.
Thanks for reading such a long post. Any pointers, links in the right direction are highly appreciated.
- Frustated AWS user :-)
I had a similar problem with trying to offer up gzipped content: Client -> API-Gateway -> S3.
I couldn't figure out why the content I pulled couldn't get recognized by the client (either browser or code) until I inspected the header(the binary header, not http).
It appears that API Gateway by default assumes a string based encoding (utf-8) so what I was seeing is data that API Gateway was transforming on the fly and adding utf-8 headers in various places.
Late last year API Gateway started to support binary payloads. I started reading this article which may help in your case. It did not in mine, but your use case is slightly different and closer to the article so it might.
https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/
As for what I ended up doing... I realized I was just using the API Gateway as a passthrough (to take advantage of the Cognito based authentication I had already setup) so I bypassed API Gateway all together and used the aws js sdk to connect directly to s3 (and set IAM policies on the bucket separately)