As you may know AWS Lambdas have a hard request(and response) payload limit of 6M. I’m calling a Java Lambda deployed on API Gateway via a REST API with a gzip Content-Encoding, but API Gateway decompresses the request and calls the lambda with the uncompressed payload, so making a gzip request through API Gateway doesn’t help with the request payload limit. So if I want to increase my request payload limit, how should I make the REST call with a gzipped request body without the API Gateway decompressing it? A couple options:
configure API Gateway to allow gzip Content-Encoding requests through as-is. This is my preferred option, but I haven’t figured out if this is possible. There’s a x-amazon-apigateway-binary-media-types option to treat certain media types as binary, but changing this doesn’t seem to affect the decompression of incoming requests. Ditto with the minimumCompressionSize option.
Change the incoming request to be Content-Type: application/octet-stream instead of Content-Type: application/json and Content-Encoding: gzip, and the in the API if it receives that type we decompress it as gzip in the lambda. This would probably work, but it’s a bit hacky to use the generic octet-stream for a covert gzip request here.
What do you all think about these options(or some other option I haven’t considered)? And yes, I know if I make the request via an async process I can get around the request limit, but async doesn't work well for the application.
Related
I have a POST request coming in with a header that I need to play back with the response (Example: "Validation: 123").
The integration returns synchronously, immediately.
The header is entirely non-functional and so doesn't need to be passed through to my integration. It just needs to be passed back with the response to the inbound request.
I am trying to do this by mapping the headers from the method request through to the method response, or via the integration request/response.
For example, I'd like to set the header mapping for the integration response to method.request.header.Validation (ie, identically to the integration request mapping). However, this is disallowed.
Is this the right approach?
This is a limitation of AWS API Gateway.
I have a service that returns image data as base64 encoded text in the body:e.g.
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASwAAAEsCAYAAAB5fY51AAAgAE...etc
However, when we put the service behind the api-gateway behaving as a proxy, the same request will seemingly convert the response to binary
�PNG
IHDR,,y}�u IDA... etc
and has a response header
Content-Type: application/json
The documentation explains how to convert TO binary automatically, but makes no mention on how to leave the response unmodified. I've tried changing contentHandling to CONVERT_TO_TEXT with no success, so I'm wondering if there's another way of having the api gateway not mess with my service responses.
I believe you are using the same setup as me;
API gateway
V
Proxy (no contentHandling available)
V
Lambda (returns {body: Buffer.toString('base64')})
This solved it for me: https://stackoverflow.com/a/47780921/853015.
I'm using API Gateway Lambda proxy integration and trying to return a binary application/protobuf response. No matter what I do, the response body is always a base64 encoded string
I have application/protobuf setup as a binary media types in APIG
My client (javascript) is sending following headers in the POST:
Accept: application/protobuf
Content-Type: application/protobuf
My lambda is responing with content-type: application/protobuf, and correctly setting the IsBase64Encoded Lambda response to true
How do you get APIG to base64 decode the string? I swear I had this working a few months ago when I 1st tried this.
Note: I've also tried */* as a binary media types
Some related posts to add background:
https://github.com/twitchtv/twirp/issues/81
https://github.com/awslabs/aws-serverless-express/issues/39#issuecomment-276019222
Update:
Turns out I can only get it working if binary media type is set to */*. The client Accept header has no impact once it is set to this.
Many bad side effects of using */* because every response is attempted to get decoded (even when IsBase64Encoded is false or not set)
I thought it wasn't decoding because Chrome network inspect tools will always show binary data as base64 encoded in the Preview tab. You can see the protobuf in the Response tab.
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
Your client must send an Accept header who's 1st media type matches what you have set as a binary media types in API Gateway
Your Lambda, if serving a binary media type, must set IsBase64Encoded to true AND the body must be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
in my Integration Request I have Body Mapping Tempates for application/xml, application/json (the same template body - reads URL parameters and create JSON body as lambda input).
Client is requesting API with application/json or application/xml. How to make response body format depending on request Content-Type in API Gateway only?
Regards,
Radek
Create ANY integration and handle everything with Lambda. You can avoid all the mappings and handle everything in code and we found that much easier.
More details here on ANY Integration,
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html
Hope it helps.
How should I go about disabling the cache of non 200 OK responses in API Gateway.
For one of our API endpoints we implemented our complementary throttle mechanism and we're sending a 429 HTTP response.
The intention is for the client to retry the request after a short time when the server is ready to fulfil it but what happens now is that the API Gateway caches the initial response and keeps sending that from cache instead.
As per the response to Can AWS API Gateway cache invalidate specific entries based on the response content?, the API Gateway cache doesn't appear to have the functionality to just "sometimes" cache a result. The documentation shows a way to have the client make a request that will ignore the existing cache (by setting Cache-Control: max-age=0), but doesn't show a way for the server to say that "this is a single-use response that shouldn't be cached.
The first thing that I think is worth trying is specifying a header like Cache-Control: max-age=0 in your error response just to try it to see if it works. The AWS API Gateway uses CloudFront under the hood for its distribution, so it may just work.
If that doesn't work, other options include:
Turn off the AWS API Gateway cache. If you need a cache, set up your own caching using CloudFront or another service that allows for more fine-grained control over which responses get cached.
Try to move your throttling earlier in the process (I'm not sure if you're using the built-in API Throttling features), but since you've said you "implemented" your mechanism I'm guessing you are doing it yourself in your back-end handling the requests. If you can do throttling before your caching layer (whether it's the built-in API Gateway caching or some other system), that may end up solving your problem and put less strain on your back-end request handler.
After sending 429 responses to the client, when the service is free to handle further requests, send your own "cache invalidation" request with Cache-Control: max-age=0 to get the "real" value cached. Obviously, this would be a bit tricky as you'd need to know when the service is up and available to handle more requests without getting bogged down again with adding a bunch more requests as soon as it's "free" again.
Depending on your exact caching needs, just have a low-enough TTL in your caching settings. For example, if once throttling kicks in, it's likely to not be available again for at least 60 seconds, then having a 60 second TTL means that the 429 response will get served from the cache for that time. But, since you were just throttling anyway and thus your service is "overloaded", it may be acceptable for your situation to continue serving that 429 until the TTL expires. This would need to be the same short TTL for both "success" and "failure" responses, though.