We are trying to deploy our API Gateway/Lambda and route it through Route53 in following regions.
ap-south-1
Lambda
API Gateway + Certigicate for API Gateway + Custom Domain
us-east-1
Lambda
API Gateway + Certigicate for API Gateway + Custom Domain
DynamoDB
AWS Elastic Search Service
Our lambda(ap-south-1, us-east-1) connect to DynamoDB(us-east-1) and AWS Elastic search services(us-east-1) to fetch data.
When we test the lambda in us-east-1 it has 200ms of execution time.
But when we test the lambda in ap-south-1 it has around 3 seconds of execution time.
The logic is same in both the lambda. The only thing is it request dynamodb/Elastic Search service in us-east-1 from ap-south-1.
We want to understand why it takes around 3 seconds when lambda is executed from ap-south-1, since it is inter-region request in AWS Network infrastructure only.
What you are observing is a typical latency issue, since data store is too far from application.
And your architechure It is not Truly Multi-region. Even if you are in 2 region, your application is unusable if aws east goes down.
You should
Allow replication of dynamodb tables.
each lambda/applicaiton should hit only regional services and no cross region call.
Elastic search should be replicated using dynamodb streams.
If lambda is using sns and sqs, they should also hookup using dynamodb streams.
It will make sure
You will have low latency reads.
No issues if there is a regional outage.
But it will have issues like
Cost will be higher.
If writes are allowed from both the regions, race conditions might be there.
As others have already said it's probably a latency issue.
If you make multiple synchronous requests to a different region this latencies sums up.
To investigate further, you can try AWS X-Ray. Maybe it can give you some details on where latencies develop.
https://aws.amazon.com/it/xray/
Related
I want the Requests Per Unit Time at route/path level metrics from AWS Lambda.
For example, path = /admin/options, Requests Per Second = 200.
My application supports many routes like this
I have looked into documentation but the Requests per unit time is not at route level.
As already pointed out, you have to do it by analyzing ALB access logs. This process is much simplified as Amazon Athena can query the ALB's logs directly in S3 as explained in Querying Application Load Balancer Logs.
This means that you don't have to download the logs before processing, or write any custom processing application. Instead you can run Amazon Athena queries against the logs.
All my work to access AWS S3 is in the region: us-east2 and the AZ is us-east-2a.
But I saw some throttling complaints from S3, so I am wondering if I move some of my work to another AZ like us-east-2b, could it mitigate the problem? ( Or it will not help since us-east-2a and us-east-2b are actually pointing to same endpoint?)
Thank you.
The throttling is not per AZ, its for a bucket. The below quote is from the AWS documentation.
You can send 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix in an S3 bucket. When you have an increased request rate to your bucket, Amazon S3 might return 503 Slow Down errors while it scales to support the request rate. This scaling process is called partitioning.
To avoid or minimize 503 Slow Down responses, verify that the number of unique prefixes in your bucket supports your required transactions per second (TPS). This helps your bucket leverage the scaling and partitioning capabilities of Amazon S3. Additionally, be sure that the objects and the requests for those objects are distributed evenly across the unique prefixes. For more information, see Best Practices Design Patterns: Optimizing Amazon S3 Performance.
If possible enable exponential backoff to retry in the S3 bucket. If the application uploading to S3 is performance sensitive the suggestion would be to handoff to a background application that can upload to S3 at a later time.
I have my AWS IOT Core setup in AP-South-1 Region, all my devices are present there. But due to the region constraint of alexa skill service endpoint (i.e. only available in North Virginia, Ireland and Oregon), my lambda is in Ireland region (since it is nearest to AP-South-1). All this setup is working fine untill I needed to create a rule in AWS IOt Core. In action option of rule, I can only see lambda functions of same region i.e. Ap-South-1. Can I somehow create a rule in between two regions? Because shifting whole IOT Core seems complex and also I fear that it might increase latency in mqtt messages. Any solution to this?
AWS recommends that you invoke a Lambda in the current region that in turn invokes a Lambda in the other region.
From https://forums.aws.amazon.com/thread.jspa?messageID=914149󟋥:
The Rules Engine does not support cross-region calls at this time. We encourage customers to keep their IoT applications regionalized as a best practice.
As a workaround to achieve cross region calls we suggest you send to a Lambda in the current region and make the cross region call there.
we would like to create serverless architecture for our startup and we would like to support up to 1 million requests per second and 50 millions active users. How can we handle this use case with AWS architecture?
Regarding to AWS documentation API Gateway can handle only 10K requests/s and lamda can process 1K invocations/s and for us this is unacceptable.
How can we overcome this limitation? Can we request this throughput with AWS support or can we connect somehow to another AWS services (queues)?
Thanks!
Those numbers you quoted are the default account limits. Lambda and API Gateway can handle more than that, but you have to send a request to Amazon to raise your account limits. If you are truly going to receive 1 million API requests per second then you should discuss it with an AWS account rep. Are you sure most of those requests won't be handled by a cache like CloudFront?
The gateway is NOT your API Server. Lambda's are the bottleneck.
While the gateway can handle 100000 messages/sec (because it is going through a message queue), Lambdas top out at around 2,200 rps even with scaling (https://amido.com/blog/azure-functions-vs-aws-lambda-vs-google-cloud-functions-javascript-scaling-face-off/)
This differs dramatically from actually API framework implementations wherein the scale goes up to 3,500+ rps...
I think you should go with Application Load Balancer.
It is limitless in terms of RPS and can potentially be even cheaper for a large number of requests. It does have fewer integrations with AWS services though, but in general, it has everything you need for a gateway.
https://dashbird.io/blog/aws-api-gateway-vs-application-load-balancer/
Amazon Cloudwatch provides some very useful metrics for monitoring my EC2s, load balancers, elasticache and RDS databases, etc and allows me to set alarms for a whole range of criteria; but is there any way to configure it to monitor my S3s as well? Or are there any other monitoring tools (besides simply enabling logging) that will help me monitor the numbers of POST/GET requests and data volumes for my S3 resources? And to provide alarms for thresholds of activity or increased datastorage?
AWS S3 is a managed storage service. The only metrics available in AWS CloudWatch for S3 are NumberOfObjects and BucketSizeBytes. In order to understand your S3 usage better you need to do some extra work.
I have recently written an AWS Lambda function to do exactly what you ask for and it's available here:
https://github.com/maginetv/s3logs-cloudwatch
It works by parsing S3 Server side log files and aggregates/exports metrics to AWS Cloudwatch (CloudWatch allows you to publish custom metrics).
Example graphs that you will get in AWS CloudWatch after deploying this function on your AWS account are:
RestGetObject_RequestCount
RestPutObject_RequestCount
RestHeadObject_RequestCount
BatchDeleteObject_RequestCount
RestPostMultiObjectDelete_RequestCount
RestGetObject_HTTP_2XX_RequestCount
RestGetObject_HTTP_4XX_RequestCount
RestGetObject_HTTP_5XX_RequestCount
+ many others
Since metrics are exported to CloudWatch, you can easily set up alarms for them as well.
CloudFormation template is included in GitHub repo and you can deploy this function very quickly to gain visibility into your S3 bucket usage.
EDIT 2016-12-10:
In November 2016 AWS has added extra S3 request metrics in CloudWatch that can be enabled when needed. This includes metrics like AllRequests, GetRequests, PutRequests, DeleteRequests, HeadRequests etc. See Monitoring Metrics with Amazon CloudWatch documentation for more details about this feature.
I was also unable to find any way to do this with CloudWatch. This question from April 2012 was answered by Derek#AWS as not having S3 support in CloudWatch. https://forums.aws.amazon.com/message.jspa?messageID=338089
The only thing I could think of would be to import the S3 access logs to a log service (like Splunk). Then create a custom cloud watch metric where you post the data that you parse from the logs. But then you have to filter out the polling of the access logs and…
And while you were at it, you could just create the alarms in Splunk instead of in S3.
If your use case is to simply alert when you are using it too much, you could set up an account billing alert for your S3 usage.
I think this might depend on where you are looking to track the access from. I.e. if you are trying to measure/watch usage of S3 objects from outside http/https requests then Anthony's suggestion if enabling S3 logging and then importing into splunk (or redshift) for analysis might work. You can also watch billing status on requests every day.
If trying to guage usage from within your own applications, there are some AWS SDK cloudwatch metrics:
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/metrics/package-summary.html
and
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/metrics/S3ServiceMetric.html
S3 is a managed service, meaning that you don't need to take action based on system events in order to keep it up and running (as long as you can afford to pay for the service's usage). The spirit of CloudWatch is to help with monitoring services that require you to take action in order to keep them running.
For example, EC2 instances (which you manage yourself) typically need monitoring to alert when they're overloaded or when they're underused or else when they crash; at some point action needs to be taken in order to spin up new instances to scale out, spin down unused instances to scale back in, or reboot instances that have crashed. CloudWatch is meant to help you do the job of managing these resources more effectively.
To enable Request and Data transfer metrics in your bucket you can run the below command. Be aware that these are paid metrics.
aws s3api put-bucket-metrics-configuration \
--bucket YOUR-BUCKET-NAME \
--metrics-configuration Id=EntireBucket
--id EntireBucket
This tutorial describes how to do it in AWS Console with point and click interface.