I'm running a business API on AWS, through API Gateway and Lambda.
Currently, I handle rate limiting with the built in usage plans and api keys.
Each account tier (think basic, medium, premium) is associated to a usage plan, to which each customer's api key is linked.
I just found out that there is a hard (but increasable) limit of 500 api keys that a single AWS account can have per region (https://docs.aws.amazon.com/fr_fr/apigateway/latest/developerguide/limits.html).
Is it sustainable to rely on api keys to rate limit each customer ? We will get to the 500 limit eventually. Are there other solutions we could use ?
Thanks a lot
If you read the table carefully you will notice that the last column has a header "Can Be Increased" and value "Yes" for "Maximum number of API keys per account per region".
Just contact support once you will be getting close to your limit and ask for an increase. It may take up to 2-3 work days, but otherwise it should be only a matter of asking.
Related
I am uploading photos to Google Photos and placing them in Albums via the Photos Library API.
Every now and then, I get an error "Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:XXXXXXX'.
The documentation states 10,000 requests per project per day. But then it says In addition to these limits there are other quotas that exist to protect the reliability and integrity of our systems. I am assuming it's referring to the "per minute per user" that I am receiving.
So what is the actual limit? How many times can I call the API per minute?
P.S. There is a API quote page for Google Sheets API, which states that the limit is 60 requests per minute. Does that limit also apply to other Google APIs?
P.P.S. There is a similar question from 2018, but the issue was too much bandwidth - doesn't apply to my situation.
Does anywhere officially or unofficially document what the true maximums are for all AWS quotas?
I am new to AWS, and am trying to figure out the maximum values for certain quotas.
For example, the default value for S3 Access Points supports a maximum of 1000 per account.
but in the AWS quota console it says it is Adjustable, and the docs suggest I can request a quota increase.
You can create a maximum of 1,000 access points per AWS account per Region. If you need more than 1,000 access points for a single account in a single Region, you can request a service quota increase. For more information about service quotas and requesting an increase, see AWS Service Quotas in the AWS General Reference.
I'd like to know what the true maximums are across the board for IAM and S3 resources, to ease design of features I'm working on, without having to do a request to increase resources I may not actually use, if appropriate resource limits can't be requested.
After discussing with AWS support, some quota changes aren't reflected in this console at this time (e.g dynamoDb quota changes)
Haven't tried it, but possibly using aws-limit-checker may show the real limits
I have an API Gateway endpoint that I would like to limit access to. For anonymous users, I would like to set both daily and monthly limits (based on IP address).
AWS WAF has the ability to set rate limits, but the interval for them is a fixed 5 minutes, which is not useful in this situation.
API Gateway has the ability to add usage plans with longer term rate quotas that would suit my needs, but unfortunately they seem to be based on API keys, and I don't see a way to do it by IP.
Is there a way to accomplish what I'm trying to do using AWS Services?
Is it maybe possible to use a usage plan and automatically generate an api key for each user who wants to access the api? Or is there some other solution?
Without more context on your specific use-case, or the architecture of your system, it is difficult to give a “best practice” answer.
Like most things tech, there are a few ways you could accomplish this. One way would be to use a combination of CloudWatch API logging, Lambda, DynamoDB (with Streams) and WAF.
At a high level (and regardless of this specific need) I’d protect my API using WAF and the AWS security automations quickstart, found here, and associate it with my API Gateway as guided in the docs here. Once my WAF is setup and associated with my API Gateway, I’d enable CloudWatch API logging for API Gateway, as discussed here. Now that I have things setup, I’d create two Lambdas.
The first will parse the CloudWatch API logs and write the data I’m interested in (IP address and request time) to a DynamoDB table. To avoid unnecessary storage costs, I’d set the TTL on the record I’m writing to my DynamoDB table to be twice whatever my analysis’s temporal metric is... ie If I’m looking to limit it to 1000 requests per 1 month, I’d set the TTL on my DynamoDB record to be 2 months. From there, my CloudWatch API log group will have a subscription filter that sends log data to this Lambda, as described here.
My second Lambda is going to be doing the actual analysis and handling what happens when my metric is exceeded. This Lambda is going to be triggered by the write event to my DynamoDB table, as described here. I can have this Lambda run whatever analysis I want, but I’m going to assume that I want to limit access to 1000 requests per month for a given IP. When the new DynamoDB item triggers my Lambda, the Lambda is going to query the DynamoDB table for all records that were created in the preceding month from that moment, and that contain the IP address. If the number of records returned is less than or equal to 1000, it is going to do nothing. If it exceeds 1000 then the Lambda is going to update the WAF WebACL, and specifically UpdateIPSet to reject traffic for that IP, and that’s it. Pretty simple.
With the above process I have near real-time monitoring of request to my API gateway, in a very efficient, cost-effective, scaleable manner in a way that can be deployed entirely Serverless.
This is just one way to handle this, there are definitely other ways you could accomplish this with say Kinesis and Elastic Search, or instead of logs you could analyze CloudTail events, or by using a third party solution that integrates with AWS, or something else.
selling an AWS API Gateway + lambda solution seems pretty straightforward as the customer is billed based on use.
In my case lambda writes data to an RDS DB which represents an hourly billed center of cost.
What would be a good way to fairly dispatch DB costs between different customers in such an application ?
Thanks
A very open ended question.
Simplest from customer point of view is it course in form of cost to use YOUR service. I.e. you don't wanna show a component/line-item called AWS RDS in your customers' bills.
AWS RDS seems pretty flat rate model (per machine). So unless you're setting up separate instances for each of your customer, I see 2 choices:
Flat tiered subscription. Where subscription gives you N free API calls.
Flat tiered subscription + per API call. Where subscription just gets you on board or gives you N free API calls and you pay a la carte for rest.
E.g your tiers are small, medium & large with a cap on TPS (API calls or sec) of say 5, 10 and 100 for a price of $5, $7 and $30 per month.
Customers who cross TPS for their tier, automatically can be charged for next tier.
Of course you can come up with many other combinations.
Should also add that if you're setting up separate instances for each of your customer then the distribution is pretty straightforward.
I am developing an architecture for push notifications using AWS SNS with APNS and GCM. The model that I am following is
Each user (not device) will have an SNS topic corresponding to it
Each user can have multiple devices
Create an platform application endpoint for each device
Subscribe the platform application endpoint to the topic belonging to the device's user
This way, when we have to send a notification to all the devices of a user, we need to call the publish method using the user's topicArn and all its devices should get the message.
However, by default AWS has a limit of 100,000 topics. If we are anticipating higher number of users like 1 million or say 10 million to be optimistic, is there any workaround for this approach?
Should I ask AWS to raise the limit beforehand? Do they raise limit to a number like 100 million easily? Is there any cost implication of most of them are not used in the beginning few months?
disclosure: I used to work on Amazon SNS
The architecture you propose is a common pattern within Amazon SNS and is sound.
You should ask for a topic limit increase. There is no cost implication for having a higher topic limit, even if you create the topics and don't use them. However, you will pay $0.50/million requests for each CreateTopic call, with the first 1 million requests being free per month.
When you submit the support ticket, please document your use case. It helps the team expedite your limit increase request.
We followed a similar model to OP's with one exception - as the number of users grew, so did the number of dead accounts. Face it, user retention poses a challenge. So you may want to come up with a topic recycling strategy. Hint: Don't permanently bind a topic to a user but allow ARNs to expire, similar to a DHCP lease. Your app may periodically ask your web service what ARN it needs to subscribe to and expect to be re-assigned. If you build this into the client protocol early on, you won't need as many ARNs. After all, 100k active users is entirely different than 100k enrolled users :)
With Amazon SNS, there is no minimum fee and you pay only for what you use. Users pay $0.50 per 1 million Amazon SNS Requests, $0.06 per 100,000 Notification deliveries over HTTP, $0.75 per 100 Notification deliveries over SMS and $2.00 per 100,000 Notification deliveries over Email.
Amazon SNS also includes a Free Tier, where users can get started with Amazon SNS for free. Each month, Amazon SNS customers pay no charges for the first 1 million Amazon SNS Requests, no charges for the first 100,000 Notifications over HTTP, no charges for the first 100 Notifications over SMS and no charges for the first 1,000 Notifications over Email.
So I will suggest increase the limit before hand if that is anticipated by you. you don't get charged for increasing the limits. They will definitely increase to 100 Million.