I am trying to see if there is a way for me to bill and restrict users based on their usage of resources across projects on our GCP instance. I know billing and quotas officially are at the project level, but we will have shared projects which will be used by people in seperate cost codes. I was thinking of building an API per cost code, and having people access the resources only through there, but I'm not sure if this is the best idea.
You can use a grouping system where you use labels to track your resources. Using labels such as cost center, service and environment will allow you to track your GCP resource usage and spending.
Then you can export your billing data to BigQuery where it can be filtered and segmented by labels.
In terms of notifications and restricting access, you may set up budget alerts by following this documentation. Together with Cloud Pub/Sub budget notifications and Cloud Functions, you can have more control on your spending by capping costs and stopping your billing.
Related
Context: As a new intern at a firm, one of my responsibilities is to maintain a clean and ordered QuickSight Analysis and Datasets list.
There are a lot of existing analysis reports and dashboards on the firm's Amazon QuickSight account, dating back to several years. There is a concern of deleting the old reports/supporting datasets which take up a lot of SPICE storage because of the thought that someone is using/accessing it. Is there a way one can see the stats of each report - how many people accessed it, how many times it was used over the last month etc., which could help one decide the analysis reports/datasets that can be deleted. Please help.
This AWS blog post -- Using administrative dashboards for a centralized view of Amazon QuickSight objects -- discussed how BI administrators can use the QuickSight dashboard, Lambda functions, and other AWS services to create a centralized view of groups, users, and objects access permission information and abnormal access auditing.
It is mainly security focused, but you can get the idea, how to find the relevant information about access to QuickSight objects in the AWS CloudTrail events.
Is there any API(preferably python) that can be used to get the resources' usage cost report in GCP? Billing APIs don't seem to return the costs of resources being used.
You can export you cloud billing data to BigQuery:
https://cloud.google.com/billing/docs/how-to/export-data-bigquery
https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup
You select the dataset where the cost metadata goes, once in BigQuery, it's fairly easy to query, which you can do with the python BigQuery client API, it also makes sure you keep a historic in case you change billing provider, of course it'll incur storage cost which will vary based on your usage.
I have newly created an API service that is going to be deployed as a pilot to a customer. It has been built with AWS API Gateway, AWS Lambda, and AWS S3. With a SaaS pricing model, what's the best way for me to monitor this customer's usage and cost? At the moment, I have made a unique API Gateway, Lambda function, and S3 bucket specific to this customer. Is there a good way to create a dashboard that allows me (and perhaps the customer) to detail this monitoring?
Additional question, what's the best way to streamline this process when expanding to multiple different customers? Each customer would have a unique API token — what's the better approach than the naive way of making unique AWS resources per customer?
I am new (a college student), but any insights/resources would help me a long way. Thanks.
Full disclosure: I work for Lumigo, a company that does exactly that.
Regarding your question,
As #gusto2 said, there are many tools that you can use, and the best tool depends on your specific requirements.
The main difference between the tools is the level of configuration that you need to apply.
cloudwatch default metrics - The first tool that you should use. This is an out-of-the-box solution that provides you many metrics on the services, such as: duration, number of invocations and errors, memory. You can configure metrics over different timeslots and aggregators (P99, average, max, etc.)
This tool is great for basic monitoring.
Its limitation is its greatest strength - it provides monitoring which is common to all the services, thus nothing tailored-fit to serverless applications. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
cloudwatch custom metrics - The other side of the scale - getting much more precise metrics, which allows you to upload any metric data and monitor it: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html
This is a great tool if you know exactly what you want to monitor, and you already familiar with your architecture limitations and pain points.
And, of course, you can configure alarms over this data:
Lumigo - 3rd party company (again, as a disclosure, this is my workplace). Provides out-of-the-box monitoring, specifically created for serverless applications, such as an abnormal number of invocations, costs, etc.. This tool also provides troubleshooting capabilities to enable deeper observability.
Of course, there are more 3rd party tools that you can find online. All are great- just find the one that suits your requirement the best.
Is there a good way to create a dashboard
There a are multiple ways and options depending in your scaling, amount of data and requirements. So you could start small and simple, but check if any option is feasible or not.
You can start with the CloudWatch. You can monitor basic metrics, create dashboards and even share with other accounts.
naive way of making unique AWS resources per customer
For the start I would consider creating custom cloudwatch metrics with the customer id as a metric and put the metrics from the Lambda functions.
Looks simple, but you should do the math and a PoC about the number of requested datapoints and the dashboards to prevent a nasty surprise on the billing.
Another option is sending metrics/events to DynamoDB, using atomic functions you could directly build some basic aggregations (kind of naïve stream processing).
When scaling to a lot of events, clients, maybe you will need some serious api analytics, but that may be a different topic.
We are using AWS Organisations service (with Consolidated Billing feature) with a large member of accounts. Can we set a max limit somehow for the usage cost ($) of the Member accounts?
Would you add a little more detailed description about how can I hook an activity on a billing alert?
I am not sure about your exact requirements, you may check the documentation:
Monitor Your AWS Charges with Billing Alerts.
Managing Your Costs with Budgets
In both cases you can setup alerts on cost/budget, where an alert event is sent to an SNS topic. SNS topic can further send a mail, but the same way you can implement any functionality listening on the topic (e.g. a lambda function)
This way you may monitor total cost/budget usage
usage cost ($) of the Member accounts
You can have a look at Cost Explorer where you can create cost reports per tag or as well per account
(I am not sure if you can create an alert for cost per account)
There is no out-of-the-box enforcement to limit the usage of your AWS account. Alerts can only notify you that you're not in your forecasted budget, but that's it mostly, the rest you should implement on your own.
Couple of thoughts:
Enforcing max limit on your production account is not best practice. You don't want a production interruption or user impact because of a needed scale up in infra.
What you can do is to map your needs and usage, and set quota limits to services. https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html - again, this is not really recommended. Just make sure you have visibility on your growth, set alerts and monitor your usage. Optimization is a key in the could, and the financial responsibility is yours, not AWS.
I am currently signed up to the free tier of AWS. I am enjoying experimenting with various services including those not affording by said free tier. Can AWS's enhanced budgets be used to stop services like EC2 instances if I accidentally spend too much? Or do they merely act as alerts?
This is available for EC2, I don't think it is available for all of the AWS resources.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html
Hope it helps.
There are several posts which looks it from different perspectives, such as this and this.
Having a cost cap might be a crucial requirement based on the usage, especially when considering how complex it is to set the things up properly and keeping everything secure on the cloud for an average user. At least we can expect to have a feature to switch on/off a cost-cap service, so a user can decide their own scenario easily.
Closest solution that I found is here:
Serverless Automated Cost Controls
https://aws.amazon.com/blogs/compute/serverless-automated-cost-controls-part1
It explains how to trigger AWS Lambda function to change IAM permission from EC2FullAccess to EC2ReadOnly when the budget exceeds the limit.
There is no built-in way to terminate services based on budgets or billing alarms.
You can get notified automatically, but it is then up to you to determine how to handle it.
Would you really want AWS automatically terminating your production infrastructure because you went $1 over your estimated monthly spending?
Edit: There is now a way to monitor and alert on free tier usage, and when your predicted usage will exceed the free tier. See here for details. You could probably come up with a way to terminate infrastructure based on an alert using SNS & lambda.
Edit 2: In Oct. 2020, AWS released Budget Actions - the ability to trigger an action when a budget thresholds are reached. This should give you the ability to automate a response - you can shut down servers, change IAM permissions to prevent additional infrastructure from being created, etc.
Recently, Amazon has given "budget action" to carry out actions like stop services automatically if the budget has exceeded.
https://aws.amazon.com/about-aws/whats-new/2020/10/announcing-aws-budgets-actions/
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/budgets-controls.html#:~:text=select%20Configure%20thresholds.-,To%20configure%20a%20budget%20action,-Under%20Configure%20thresholds