I want some apps to use some resources on GCP, all apps must use the same account.
but I want to know how much each app consumes.
in another word, I want to be able to generate a separate bill for each app
Labels can be used to group the resources related to each application and using the same labels we can find out the resource utilization and we can generate separate billing reports for each application. All you need to do is send your billing data to a big query and query the data using the labels which you created for grouping the resources. This document outlines the concept of labeling and getting resource utilization and bills using labels follow it for more information.
Related
I have two EC2 instances in a single AWS account. Both are running for different application services. Now, I want to make billing separate for that particular account. So, I can get the exact spent and charges for the applications and can manage my account as per that for separate accounting purposes.
Is it possible? If not, then can anyone suggest me a better way to achieve this?
I'm planning to handle the different AWS accounts for both services. But, it will be hard to manage so, I can't prefer that option.
For more, In Google Cloud, they're providing to handle different billing accounts within the same google cloud account. So, I think this concept might be available on AWS also.
Thanks in advance for any little bit of help or suggestion.
You can't get separate bills for different sets of resources within the same AWS account. However, you can filter out the costs for different sets of resources using tags. By using a distinct tag (or set of tags) in the resources you allocate to each application, you can get a breakdown of the cost for each application in billing reports and cost explorer. See the documentation for details and steps on how to set it up - https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html
I believe it's the same with Google Cloud as well - although you can have multiple billing accounts within your google account (the hierarchy in GC is different to AWS, and an AWS account is more similar to a GC project than a GC account), a project within your account must have one billing account and does not support multiples.
By this -
Now, I want to make billing separate for that particular account.
do you mean you already have 2 different AWS accounts? If yes, you can get billing details as per account if they become part of same Organization. Check AWS Organizations docs for more info.
With Orgs, you can view bills of different accounts from one account itself if it is part of org. Also your accounts get discounts for services.
We are developing a way of automating the process of running Dataproc on GCP (everything from starting a cluster to submitting a job, and terminating the cluster) for genomic analyses. Ultimately, we want to be aware of how much is being spent for each of these automated “runs” of Dataproc, especially as we open up our program to other users. Therefore, we would also like to automate (if possible) the process of attributing Dataproc costs to a certain GCP user (“run”). That way, at the end of each month, we can see how much was spent per user and specifically for what personal project. What is a good strategy for implementing this?
I have considered creating a separate GCP project for each user/client and tying them to a particular billing account, but is there perhaps a better way of managing this?
The recommended way is to apply labels to your resouces [1]. Billing data can be filtered and segmented by labels [2].
In your example, both user and client would be labels on Cluster and Job resources. Only Cluster resources are billed.
If you automation is really just a one-shot create-cluster/submit-job(s)/delete-cluster flow, then consider Workflow Templates for automating this. Specifically Inline templates are tailor made for this kind of automation.
[1] https://cloudplatform.googleblog.com/2015/10/using-labels-to-organize-Google-Cloud-Platform-resources.html
[2] https://cloud.google.com/billing/docs/how-to/export-data-bigquery
I would like to know a system by which I can keep track of multiple aws accounts, somewhere around 130+ accounts with each account containing around 200+ servers.
I wanna know methods to keep track of machine failure, service failure etc.
I also wanna know methods by which I can automatically turn up a machine if the underlying hardware failed or the machine terminated while on spot.
I'm open to all solutions including chef/terraform automation, healing scripts etc.
You guys will be saving me a lot of sleepless nights :)
Thanks in advance!!
This is purely my take on implementing your problem statement.
1) Well.. for managing and keeping track of multiple aws accounts you can use AWS Organization. This will help you manage centrally with one root account all the other 130+ accounts. You can enable consolidated billing as well.
2) As far as keeping track of failures... you may need to customize this according to your requirements. For example: You can build a micro service on top of docker containers or ecs whose sole purpose is to keep track of failures, generate a report and push to s3 on a daily basis.You can further create a dashboard using AWS quicksight out of this reports in S3.
There can be another micro service which will rectify the failures. It just depends on how exhaustive and fine grained you want your implementation to be.
3) For spawning instances when spot instances are terminated, it can be achieved through you simple autoscaling configurations. Here are some of the articles you may want to go through which will give you some ideas:
Using Spot Instances with On-Demand instances
Optimizing Spot Fleet+Docker with High Availability
AWS Organisations are useful for management. You can also look at multiple account billing strategy and security strategy. A shared services account with your IAM users will make things easier.
Regarding tracking failures you can set up automatic instance recovery using CloudWatch. CloudWatch can also have alerts defined that will email you when something happens you don't expect, though setting them up individually could be time consuming. At your scale I think you should look into third party tools.
I have setup IAM permissions for a certain group to only have read only access to S3, however, the group can still see all the other services in the management console and go into them. As soon as the user tries to do something, a message will read "Not authorised" and so on, however, I would like this group to only see the one service in the management console.
So when a user from this group logs in, all they see is S3.
How is this possible ?
Hiding services from the AWS Management Console is not possible right now, unfortunately. AWS is currently redesigning the console though, and this might include such options down the road as per the respective FAQ Why are you changing the console design?:
Our goal is to improve information display, make interactions more consistent, support devices such as tablets, and deliver a customizable experience. You will see these improvements and visual updates rolled out across our services over the coming months. [...] [emphasis mine]
However, at this point the mentioned customizable experience likely only refers to the recently introduced Resource Groups and Tagging for AWS, which allow you to easily create, maintain, and view a collection of resources that share common tags:
[...] By default, the AWS Management Console is organized by AWS service. But with the Resource Groups tool, you can create a custom console that organizes and consolidates the information you need based on your project and the resources you use. If you manage resources in multiple regions, you can create a resource group to view resources from different regions on the same screen.[emphasis mine]
Based on this new cross region Resource Groups approach, it is indeed possible to create and share a resource group that is constrained to the resource type S3 Buckets (i.e. the initial view would be limited to just S3 buckets) - however, just like with the regular console view, this doesn't prevent your users to roam freely around other areas of the console by themselves, i.e. you cannot enforce the desired limitation, rather only guide in this direction.
We host our clients' infrastructure. We are looking at AWS for some services. However, our business model is to provide the hosting included in our fee so to that end we need to be the account holder. We therefore need to have one account which is billed but clear delineation between different separate client infrastructure. Ideally with a separate console etc. Is this possible? Thanks.
You should take a look at consolidated billing, as it sounds like it will meet your needs:
http://docs.aws.amazon.com/awsaccountbilling/latest/about/consolidatedbilling.html
This would enable you to set up unique "linked accounts" per customer, each fully separate and independent of each other (separate console logins, separate security configurations, etc.).