I run a small research group at a large university that manages hundreds of GCP accounts. The university acts as the Billing Administrator, and my research group was assigned a GCP "project" for all of our work. However, for privacy reasons, they cannot give me access to the Billing API because this would allow me to see the billing details for other labs.
Because we have trainees in our lab who WILL make mistakes, I would like to setup an automated system that monitors our current GCP bill, and (1) sends notifications or (2) terminates all VMs, when that bill reaches certain predefined limits. For example, if our monthly budget is $10k, then I would like to receive a notification at $5k, another notification at $10k, and I would like to terminate all VMs at $15k.
My problem is that in order to implement a system like this, I need access to the Billing API. I have already contacted my system administrator and they have said that this is impossible. Instead, they proposed that I write a script that lists all VMs and uses the Cost Calculator to estimate my monthly GCP bill.
However, this seems a little circuitous. When I am using the Google Cloud Console, I can see the total and forecasted costs for my project, so it seems that I should be able to access this information programmatically. However, I cannot find any information on how to do this, since all solutions require me to activate the Billing API. Any ideas?
There is no API to fetch the data you see in the Google Cloud Console. You will need to export the billing data and then process each row of data to generate reports.
There are two options that I can think of:
Option 1) Ask the admin to set up billing data export to BigQuery. Grant you permission to query the billing tables. You can then query BiGQuery to generate your own cost reports.
Set up Cloud Billing data export to BigQuery
Option 2) Create a separate billing account for your project and grant you permission. A GCP ORG can have multiple Billing Accounts tied to the same Payments Account. This option supports creating budget alerts.
Related
I have been working on enabling the billing exports to Bigquery. I know the process to set it up based on the documentation but looks like I can set it up for only one project at a time. There are close of 70 projects in GCP and need to load the billing data for all the projects to big query and create a billing report to see the expensive projects and its services. My questions are
how can I configure the billingexport for all projects at a single time?
how can I get access to historical billing data?
Kindly answer my questions. Appreciate your help.
Thanks!
how can I configure the billingexport for all projects at a single
time?
Billing export to BigQuery is based upon the billing account. Enable billing export for each billing account. If you have different billing accounts per project, then you must enable export for each billing account.
how can I get access to historical billing data?
Billing data in BiqQuery is only available after you enable export to BigQuery and after waiting for the export to start. Previous data is not exported to BigQuery. For that reason, it is recommended to enable billing export at the time you create a billing account.
Another recommendation is to create a new project to hold the billing data.
For more details on the individual steps:
Set up Cloud Billing data export to BigQuery
I want write utility which fetches billing information for my project but I am not able to find any specific API from GCP to do the same. I tried a couple of APIs like getBillingInfo but these APIs just give information about billing account, not the pricing. They have the mechanism to export billing data to file but I want to do it programmatically. Is there any API to do the same?
One possible way that I am aware of is to export the Cloud billing to BigQuery. The process (document) to export Cloud Billing to BigQuery can be found here: https://cloud.google.com/billing/docs/how-to/export-data-bigquery
Once the export is done, the billing information including the price and services is available in almost realtime on BigQuery table. Once it's available on BigQuery there are numerous ways of extracting the information. A good solution would be to have Data Studio on Google Cloud to send you a periodic report on your billing information.
I would like to create a GCP cost management platform for resellers/MSP. The platform will pule cost data per each of the reseller's end customers.
Where Can I find information on:
How can I access the reseller's data?
What is the data structure? Does a reseller have one dataset for all of his customers, or one dataset per customer?
What are the needed credentials that the reseller needs to provide in order to fetch his customers data, and what permissions need to be provided?
I created a Video Lab on Google Cloud Billing. The first step is to understand how billing works in Google Cloud. Then understand how Identity and Access Management works so that you know what roles you need for credentials to access billing APIs.
Google Cloud Billing
Review the following documentation links for technical API details:
Get Started with the Cloud Billing API
Cloud Billing API
APIs & Reference
To access reseller information you will need to become a Google Partner first. Then request documentation.
I am wondering if it is possible to see billing information through the Amazon SDK/APIs. Specifically, I want to see how much EC2 instances are costing the account. Is it feasible to have it break down by user if it is an IAM account, or breakdown by account if you are using consolidated billing?
I have seen ways of doing this by either creating an S3 bucket to have billing info dumped into, or using CloudWatch APIs which require the user to pay. Is there an easier way to get EC2 billing information through just API calls or the SDK?
Billing information can be provided as records in an Amazon S3 bucket. You can then analyze the content of those files to obtain the information you list.
It is not possible to obtain the billing records themselves via an API call.
See: Cost and Usage Report
(The older Detailed Billing Report is scheduled to be retired.)
I have domains, instances, and buckets open on my AWS account - some are running databases on them.
I can't pay monthly for all of these services, because some of them aren't my own - I did them as work for others. And rather than going through the hassle of transferring the compute and database to another instance on another account, I'd like an IAM user to pay with his/her credit card for the services he owns.
Can IAM users pay monthly for services on my AWS account? If so, how?
No. There's no mechanism for what you want.
You may be interested in Amazon DevPay, which is sort of like what you're asking for. But I think you're doing contract work for individuals and DevPay is aimed more at providing an AWS-like service atop AWS and selling it to other developers.
The other direction is to make your clients create an AWS account. If you want to be fancy, you could use Consolidated Billing to create a sub-account under your control but billed to the customer. I think this is the best fit for consulting work, but getting there from your current setup will be painful.
There are some services (S3, ..) where you can let the client pay per call, bu for the rest I believe that your account is only yours. What I do is presenting the detailed billing and I let the company reimburse the costs.
Maybe you will have to migrate the environment. If your clients are not proficient aws users, you may want to build a cloud formation script that would enable them to start and access the environment in a few clicks..
G.