I am wondering if it is possible to see billing information through the Amazon SDK/APIs. Specifically, I want to see how much EC2 instances are costing the account. Is it feasible to have it break down by user if it is an IAM account, or breakdown by account if you are using consolidated billing?
I have seen ways of doing this by either creating an S3 bucket to have billing info dumped into, or using CloudWatch APIs which require the user to pay. Is there an easier way to get EC2 billing information through just API calls or the SDK?
Billing information can be provided as records in an Amazon S3 bucket. You can then analyze the content of those files to obtain the information you list.
It is not possible to obtain the billing records themselves via an API call.
See: Cost and Usage Report
(The older Detailed Billing Report is scheduled to be retired.)
Related
I'm the administrator of an AWS account that has 4 users. One of the users is racking up higher-than-expected costs.
I checked the Cost Explorer, but could not seem to configure it to view individual users.
As an administrator in AWS, how do I see all of the services this particular user has been using during a given period of time (e.g. the last 12 months)?
Thanks!
AWS resources are associated with an AWS Account, not a specific user.
If a user has the necessary permissions to create resources (eg an Amazon EC2 instance), then the instance is launched in the AWS Account, but there is no link back to the user that requested the resource.
You can, however, use AWS CloudTrail:
AWS CloudTrail is an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.
It will show all API calls made by the user, including the Action ('launch an EC2 instance'), their IP address, timestamp, etc. Operations in the AWS management console will also be shown, since it makes API calls on behalf of the user.
CloudTrail keeps a history of the past 90 days, but you can create specific 'Trails' that retain data permanently. If you have not done this, then it will not be possible to see what they did prior to 90 days ago.
I run a small research group at a large university that manages hundreds of GCP accounts. The university acts as the Billing Administrator, and my research group was assigned a GCP "project" for all of our work. However, for privacy reasons, they cannot give me access to the Billing API because this would allow me to see the billing details for other labs.
Because we have trainees in our lab who WILL make mistakes, I would like to setup an automated system that monitors our current GCP bill, and (1) sends notifications or (2) terminates all VMs, when that bill reaches certain predefined limits. For example, if our monthly budget is $10k, then I would like to receive a notification at $5k, another notification at $10k, and I would like to terminate all VMs at $15k.
My problem is that in order to implement a system like this, I need access to the Billing API. I have already contacted my system administrator and they have said that this is impossible. Instead, they proposed that I write a script that lists all VMs and uses the Cost Calculator to estimate my monthly GCP bill.
However, this seems a little circuitous. When I am using the Google Cloud Console, I can see the total and forecasted costs for my project, so it seems that I should be able to access this information programmatically. However, I cannot find any information on how to do this, since all solutions require me to activate the Billing API. Any ideas?
There is no API to fetch the data you see in the Google Cloud Console. You will need to export the billing data and then process each row of data to generate reports.
There are two options that I can think of:
Option 1) Ask the admin to set up billing data export to BigQuery. Grant you permission to query the billing tables. You can then query BiGQuery to generate your own cost reports.
Set up Cloud Billing data export to BigQuery
Option 2) Create a separate billing account for your project and grant you permission. A GCP ORG can have multiple Billing Accounts tied to the same Payments Account. This option supports creating budget alerts.
I have an AWS account. There are multiple users being managed by IaM service.
Each user has access key and is at liberty to perform various actions such as files upload.
Is there any means to monitor cloud costs and usage by user?
I utilize cost explorer AWS service and intend filtering and grouping costs/usages by user. Unfortunately haven't come up with any way to nail it in the most graceful way.
One common way to do this is to use Cost Allocation Tags.
You can define these tags and enforce them, e.g. using AWS Config and/or tag policies.
AWS does not track costs by user.
When an IAM User makes a request to AWS to create resources (eg an EC2 instance or an RDS database), the user's permissions are checked to confirm that they are permitted to make that API call. If they are permitted, then the API call is allowed and the resources are created.
Resources created in an AWS Account are owned by the AWS Account, not an individual user. Thus, there is no relationship between resources and the credentials used to create the resource.
The closest link between users and resources would be the audit trail of API calls kept by AWS CloudTrail. CloudTrail stores information about the API call and the user that made the call, but it does not directly link to the resources that were created. This would take some effort to back-trace resources to users.
Typically, cost management is done by tagging resources. Such tags would identify cost centers or project codes that can be used to charge-back the cost of systems. Enforcing tagging is difficult. Only some services allow tagging to be enforced when services are launched. For others, it would be a matter of identifying resources that do not meet tagging requirements. See: Using AWS Config Rules to Manage Resource Tag Compliance | Sumo Logic
You can monitor every IAM user action through cloud trails logs. So you could imagine a solution based on those logs to calculate the cost of all actions from one IAM user
I allways recomend to have an account per user type or subscription type in your system (free or premium for exmaple). Depending on the user who use your services, you will login that use with this account. Then, using the AWS Cost Categories, you can to see the cost by users type, and then, knowing your number of users for each tipe or subscription you could know your price per user.
I'm currently under an AWS Organisation Subscription. I want to export cost by CSV of my linked account into a S3 bucket.
I have rights on my account, I can use Cost Explorer, do Budgets, but I'm unable to do the Cost&Usage Report as shown in this screenshot :
The official doc doesn't talk about this case. My question is 'Does the organisation account can enable the billing feature 'Cost and Usage Reports'
for one linked account?' If not, is there a way to automate this export of CSV into a S3 bucket ?
The CUR (Cost and usage report) will be generated only in the billing account if you're under an organization. All you have to do is go to your billing account, and enable CUR from there. It's a simple process and the reports will be sent to the billing account S3. It is also best practice from security and isolation perspective.
Note that the link you sent is the DBR (Detailed billing report) and is considered legacy already.
https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html
If you use the consolidated billing feature in AWS Organizations, the Amazon S3 bucket that you designate to receive the billing reports must be owned by the master account in your organization. You can't receive billing reports in a bucket that is owned by a member account. If you use consolidated billing, you can also have your costs broken down by member account.
In brief
We need to view AWS cost for resources created by a specific IAM user via aws:createdBy tag.
In full
My google search is not helpful - it gives me one idea to use Consolidated Billing for Organizations to have the view of cost-by-user billing but that requires to create separate AWS users, which again requires credit card.
Another solution I found is that we can view AWS cost by tag grouping via Cost Explorer report though we cannot find the mentioned tag createdBy as stated in that AWS guide?
The ideal solution should let us create as many IAM users as the number of projects may grow i.e. one IAM user per project; and then, give us the report of AWS cost for each of those users.
So, how can we get to it?
Costs are not incurred by IAM Users. The AWS Account owning the resource incurs the charge.
When an IAM User makes a request to AWS, their credentials are verified to ensure that they are the authorized person and their permissions are checked to confirm that they have permission to perform the operation. However, there is no further relationship between the person making the request and the resource that is created.
The closest connection is the audit log created by AWS CloudTrail that can show which user made each request. But this is not related to billing information.
If you wish to track costs, you should associate tags with resources. These tags could be based on department, cost center, person, etc. You can then configure your Detailed Billing Report to provide cost breakdowns by tag.
You could use a more concise solution systematically using your own tags by setting up Lambda, CloudWatch, CloudTrail, and SSM to achieve it.
Here are some references:
Automatically tag new AWS resources based on identity or role
GitHub - resource auto tag