I am integrating AWS API Gateway with AWS SQS.
What would happen if API Gateway receives a payload > 256KB? Can I divert it to S3 somehow?
In this use case, you probably want to use a lambda proxy integration with API gateway.
So, your API Gateway endpoint sends the payload to a lambda function.
The lambda function works out the size of the payload. If it is <256KB you may want to place his onto the queue in SQS via the lambda.
If it is >256kb then you could write this down to s3 via the lambda, and then place a message into your queue with the s3 details. Your worker can then use the message in the queue to grab the data in s3 and process it from there.
Note there is a 10MB payload limit for API Gateway.
AWS provide the following guide on Lambda Proxy Integration with API Gateway.
https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
Related
I'm trying to build a schedule in AWS EventBridge that is going to invoke an API Gateway endpoint on a specific rate. However, I'm not able to specify the API Gateway endpoint when creating the schedule. I couldn't find examples in the documentation about the JSON body that I need to provide to specify the API Gateway ARN, endpoint and headers.
Follow AWS EventBridge documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-gateway-target.html
I have an API Gateway endpoint that takes the bodies of requests to it and places them in an SQS queue. The API Gateway has the ability to transform the request and add requester meta, like the user agent and ip address, to the message it eventually sends to SQS.
I'd like to change this from API Gateway to SNS so that the requester would publish to an SNS topic that feeds into SQS or directly to SQS.
The issue i'm having with this is that while I can get the useragent from the requester pre-send, i can't get the ip of the user without making a call to an endpoint and having the endpoint return the ip it observed.
Is it possible for the aws SNS/SQS api to append the ip of the request to the messages they receive?
SQS actions SendMessage, SendMessageBatch and SetQueueAttributes will process messages as they are received. There is no SQS/SNS configuration that could be used to modify the message. It would make sense to use SQS directly but in my opinion using SNS instead of API Gateway won't make it any better from cost/performance/implementation point of view. API Gateway appears to be your best option.
I have an API (through API Gateway) where each resource's method is routed to a single lambda function via lambda proxy integration, where each request is processed internally.
How can I automate a scheduled invocation of one resource's method with a CloudWatch event? This preferably will not be done by directly invoking the lambda function, and would allow a specified (constant) input.
Since CloudWatch Events does not support invoking an HTTP endpoint directly, you should be able to accomplish this by using SNS with a HTTP subscription. The workflow would be the following:
Scheduled CloudWatch Event -> SNS endpoint with constant JSON payload -> HTTP subscription, where the HTTP subscription is your API Gateway endpoint.
I originally asked this question here: https://forums.aws.amazon.com/message.jspa?messageID=769734#769734
However I haven't seen any activity on it of any kind I hope I have more success here.
Do my requests going to and coming from Lambda functions get encrypted? Basically my json input that is passed to the Lambda event, should I worry about this information being sent as plain text?
Main Purpose:
EC2 instances are the main ones that will be calling this function. However, this lambda does not reside in the same VPC as the instance. The lambda will be returning data as part of a "RequestResponse" call. The return of the lambda will be stored on the EC2 instance.
If you are triggering AWS Lambda functions via AWS API calls, then the requests are send to a HTTPS endpoint.
See: AWS Lambda Endpoints (Notice that protocol is HTTPS)
HTTPS encrypts your request and the response, so your data is encrypted in transit.
The AWS Command-Line Interface (CLI) is a Python app that uses the AWS SDK for Python (boto3) that in-turn calls the AWS API endpoints, so requests from the AWS CLI are also encrypted.
I have been experimenting with AWS API Gateway and AWS Lambda to try out a serverless architecture. Have been going through blogs and AWS documentation. Have tried out sample GET/POST. But, I have the following requirement w.r.t tracking user events from my custom application
Events are posted from my application to API end point
I wanted the API to respond back with a custom response (Say {'fine'})
(acknowledging that the request has been received)
After the response is sent, hand over the event payload to a AWS Lambda function
As per the documentation, I understand,
a) I can post events to API end point
b) On GET/POST trigger an AWS Lambda Function
- Respond back from AWS Lambda function to API request
I wanted to change the above and modify it to
a) Post events to API end point
a.0) Respond back acknowledging that request is received [Say {'fine'} ]
b) Trigger AWS Lambda function to process the event payload
Please share across suggestions on how to achieve the same.
Another asynchronous model many customers have used:
Set up an API configured to send requests to Amazon Kinesis. This API could acknowledge the request.
Set up AWS Lambda to consume your Kinesis stream.
This setup has some advantages for high workload APIs as fetches from the Kinesis stream can be batched and don't require a 1-to-1 scaling of both your API Gateway limits and Lambda limits.
Update
To answer your questions about scalability:
Kinesis
Kinesis scales by adding what it calls "shards" to the stream. Each shard handles a portion of your traffic, based on a partition key. Each shard scales up to 1000 rps or 1MBps (see limits). Even with the lower default 25 shards, this would support up to 25,000 rps or 25MBps with an evenly distributed partition key.
API Gateway
API Gateway has a default account level limit of 500 rps, but this can easily be extended by requesting a limit increase. We have customers in production that are using the service at limits above your current suggested scale.
If you want a fast response from the API and not have to wait for the processing of data, you could:
post an event to an API Gateway endpoint
trigger an AWS Lambda Function A
call asynchronously a Lambda Function B using the AWS SDK in the Lambda Function A
Call context.succeed() or context.done() or the callback function in the Lambda Function A so it respond back to API Gateway
the Lambda Function B can process the data while API Gateway already received a response
You should first run some tests to see what type of real world response times you are getting from having your lambda function complete all the logic. If the times are above what you feel are acceptable for your use case, here is another asynchronous solution utilizing an SNS Topic to trigger a secondary Lambda function.
Client Request to API Gateway -> Calls Lambda function A
Lambda A verifies payload and then publishes to SNS Topic X
Lambda A returns {fine} success message -> API Gateway -> client
SNS Topic X triggers Lambda function B
Lambda function B implements given logic