On my AWS account, I have multiple lambda functions. When lambda costs go up, I'm unable to identify which lambda is to blame. How can I monitor each lambda function by determining the cost and request count for a single lambda function?
AWS Cost Explorer allows you to view your costs and usage trends and you can filter by service, function name, and resource ID.
Choose "Lambda" from list of services and select "Function name" from the "Group by" dropdown menu.
Cost Explorer is more suited if all the Lambda functions are appropriately tagged. This article explores some methods to keep an eye on individual lambda functions's cost.
Related
I am looking for a programmatic way to monitor my lambda serverless environment cost in real time, or x hours retrospective. I am looking at the budget API but it seems like it always goes around a defined budget which is not my use case. The other way I thought might work is the count lambda executions and calculate according to lambda instance type. Any insight or direction how to go about this programmatically would be highly appreciated.
From Using the AWS Cost Explorer API - AWS Billing and Cost Management:
The Cost Explorer API allows you to programmatically query your cost and usage data. You can query for aggregated data such as total monthly costs or total daily usage. You can also query for granular data, such as the number of daily write operations for DynamoDB database tables in your production environment.
Cost Explorer refreshes your cost data at least once every 24 hours, so it isn't "real-time".
Is there a way to allow creation of a resource like a DynamoDB table only if the table to be created was PAY_PER_REQUEST or was provisioned with capacity below a certain amount?
I initially looked at IAM condition keys, but they appear to only be available for interactions with the table data operations (scan, update, put operations etc.) but not creation operations for the table.
Alternatively, are there ways to reduce service quotas for an account?
Ideally, I'm wondering if it is possible to scope down the ability to create DynamoDB table resources beyond a certain capacity and I’m not sure how to do it proactively instead of retroactively processing CloudTrail logs or listing existing table properties.
AWS Config
You can use AWS Config to retrospectively query AWS resources and their properties, and then determine if they are compliant or not. There are rules already available out of the box, but I can't see one which matches your use case. You will need to then write a Lambda function to implement this yourself. Here is an example.
After your rule is working you can either create a remediation action to
Delete the Table
Scale the Table Down
Send a Notification
Adjust Autoscaling (i.e. reduce max)
AWS Budgets
(My Preference)
For determining if an account is using too much DynamoDB, probably the easiest is to setup a budget for the DynamoDB Service. That would have a couple of benefits:
Auto-Scaling: Developers would be free to use high amounts of capacity (such as load tests) for short periods of time.
Potentially Cheaper: what I have found is that if you put restrictions on projects often developers will allocate 100% of the maximum, as opposed to using only what they need, in fear for another developer coming along and taking all the capacity.
Just like before with AWS Config you can setup Billing Alarms to take action and notify developers that they are using too much DynamoDB, also when the Budget is at 50%, 80% ... and so on.
CloudWatch
You could also create CloudWatch Alarms as well for certain DynamoDB metrics, looking at the capacity which has been used and again responding to excessive use.
Conclusion
You have a lot of flexibility how to approach this, so make sure you have gathered up your requirements and then the appropriate response will be easier to see. AWS Config requires a bit more work than budgets so if you can get what you want out of Budgets I would do that.
I am building a simple app in AWS which lets user rent out cars for limited amount of time. I am using AWS Lambda for computation, dynamoDB for storage and API Gateway to handle requests to lambda functions.
My question is if there is any AWS service or dynamoDB feature that allows me to track time for "Car" object in dynamoDB such that when rental time is over, it triggers a lambda function to notify the user and perform other action?
You could consider using DynamoDB Time to Live along with DynamoDB streams and a lambda function.
In this scenario, the items specific to the rental time would be placed in a separate table. They would have TTL values set to the rental time. DynamoDB automatically scans and deletes items based on the TTL. These automatic deletions could be picked up by DynamoDB streams and forwarded to a lambda function. The function would take action based on the expired time.
However, a possible issue could be that sometimes DynamoDB will take 48 hours to delete an item.
DynamoDB Streams and TTL are not good solutions because DynamoDB provides no SLA for TTL deletes (it can even take longer than 48 hours in rare cases) and the item will be deleted so cannot be used by downstream applications or analytics later on.
For this you should use Cloudwatch event rules (or Amazon Eventbridge) with a cron schedule expression. So your code that puts the item into the DynamoDB table can subsequently create a Cloudwatch event rule for the time in the future when the rental time will expire, using a cron schedule expression. This will trigger a lambda that can call your notification service to notify the customer.
A possible solution would be having a Lambda cron job that runs on a timer that scans or queries the DynamoDB table for values that have a date matching the end date of the rental. This lambda could then invoke your NotifyUser lambda using AWS Step Functions, or could emit an event to a SNS Topic where your lambda has subscribed to.
Some links that may be helpful:
CronJob
SNS
I have an API Gateway endpoint that I would like to limit access to. For anonymous users, I would like to set both daily and monthly limits (based on IP address).
AWS WAF has the ability to set rate limits, but the interval for them is a fixed 5 minutes, which is not useful in this situation.
API Gateway has the ability to add usage plans with longer term rate quotas that would suit my needs, but unfortunately they seem to be based on API keys, and I don't see a way to do it by IP.
Is there a way to accomplish what I'm trying to do using AWS Services?
Is it maybe possible to use a usage plan and automatically generate an api key for each user who wants to access the api? Or is there some other solution?
Without more context on your specific use-case, or the architecture of your system, it is difficult to give a “best practice” answer.
Like most things tech, there are a few ways you could accomplish this. One way would be to use a combination of CloudWatch API logging, Lambda, DynamoDB (with Streams) and WAF.
At a high level (and regardless of this specific need) I’d protect my API using WAF and the AWS security automations quickstart, found here, and associate it with my API Gateway as guided in the docs here. Once my WAF is setup and associated with my API Gateway, I’d enable CloudWatch API logging for API Gateway, as discussed here. Now that I have things setup, I’d create two Lambdas.
The first will parse the CloudWatch API logs and write the data I’m interested in (IP address and request time) to a DynamoDB table. To avoid unnecessary storage costs, I’d set the TTL on the record I’m writing to my DynamoDB table to be twice whatever my analysis’s temporal metric is... ie If I’m looking to limit it to 1000 requests per 1 month, I’d set the TTL on my DynamoDB record to be 2 months. From there, my CloudWatch API log group will have a subscription filter that sends log data to this Lambda, as described here.
My second Lambda is going to be doing the actual analysis and handling what happens when my metric is exceeded. This Lambda is going to be triggered by the write event to my DynamoDB table, as described here. I can have this Lambda run whatever analysis I want, but I’m going to assume that I want to limit access to 1000 requests per month for a given IP. When the new DynamoDB item triggers my Lambda, the Lambda is going to query the DynamoDB table for all records that were created in the preceding month from that moment, and that contain the IP address. If the number of records returned is less than or equal to 1000, it is going to do nothing. If it exceeds 1000 then the Lambda is going to update the WAF WebACL, and specifically UpdateIPSet to reject traffic for that IP, and that’s it. Pretty simple.
With the above process I have near real-time monitoring of request to my API gateway, in a very efficient, cost-effective, scaleable manner in a way that can be deployed entirely Serverless.
This is just one way to handle this, there are definitely other ways you could accomplish this with say Kinesis and Elastic Search, or instead of logs you could analyze CloudTail events, or by using a third party solution that integrates with AWS, or something else.
I have AWS lambda function written in c#, and I want to invoke that lambda function for the specific interval. This interval value is not fixed, and the user can customize this interval from the app. Any ideas on how to achieve this?
One method is to programmatically update the schedule of a CloudWatch Scheduled Rule. This works well if you have a relatively low number of schedules, but there are limits to how many schedules you can create. The default limit is 50 rules, though this can be increased to meet your needs by requesting an increase from AWS.
This is an example of creating a rule programmatically in C#, you'll need to permission your Lambda Role to update the rules, also in this doc:
https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/cloudwatch-examples-sending-events.html#create-a-scheduled-rule
try Using AWS Lambda with Amazon SQS
SQS has each URL.
user (or some API) can send request to that url.
The outline of processing is as follows
user (or some API) -> SQS -> Lambda