I have an AWS account. There are multiple users being managed by IaM service.
Each user has access key and is at liberty to perform various actions such as files upload.
Is there any means to monitor cloud costs and usage by user?
I utilize cost explorer AWS service and intend filtering and grouping costs/usages by user. Unfortunately haven't come up with any way to nail it in the most graceful way.
One common way to do this is to use Cost Allocation Tags.
You can define these tags and enforce them, e.g. using AWS Config and/or tag policies.
AWS does not track costs by user.
When an IAM User makes a request to AWS to create resources (eg an EC2 instance or an RDS database), the user's permissions are checked to confirm that they are permitted to make that API call. If they are permitted, then the API call is allowed and the resources are created.
Resources created in an AWS Account are owned by the AWS Account, not an individual user. Thus, there is no relationship between resources and the credentials used to create the resource.
The closest link between users and resources would be the audit trail of API calls kept by AWS CloudTrail. CloudTrail stores information about the API call and the user that made the call, but it does not directly link to the resources that were created. This would take some effort to back-trace resources to users.
Typically, cost management is done by tagging resources. Such tags would identify cost centers or project codes that can be used to charge-back the cost of systems. Enforcing tagging is difficult. Only some services allow tagging to be enforced when services are launched. For others, it would be a matter of identifying resources that do not meet tagging requirements. See: Using AWS Config Rules to Manage Resource Tag Compliance | Sumo Logic
You can monitor every IAM user action through cloud trails logs. So you could imagine a solution based on those logs to calculate the cost of all actions from one IAM user
I allways recomend to have an account per user type or subscription type in your system (free or premium for exmaple). Depending on the user who use your services, you will login that use with this account. Then, using the AWS Cost Categories, you can to see the cost by users type, and then, knowing your number of users for each tipe or subscription you could know your price per user.
Related
I want my website to offer premium services to authenticated Cognito users, and so I want to be able to bill each user for the AWS resources they use. How can I track my monthly AWS spend back to the individual Cognito user? I see that I can assign cost allocation tags to user pools, but I'm looking for something at the user level, not the user pool level. I also see that I can use "Attribute-Based Access Control" in an identity pool, and that I can get tags called "Tag key for Principal" that can be referenced in IAM policies, but I'm not seeing a way to get those tags to map to cost allocation tags in the billing console.
The main resources I'm thinking about tracking initially are S3 storage and Lambda compute.
The only option I know of is the following:
https://aws.amazon.com/es/aws-cost-management/aws-cost-and-usage-reporting/
It may be what you need.
There are several "accounts" in an AWS OU in my organization to isolate service utilization, billing, etc. But, these accounts don't necessarily provide any restrictions w.r.t the number of resources a user ( say in an Active Directory environment ) can use.
I am just wondering if there are any provisions in place for AWS to be able to restrict a specific user to not consume more than the allotted 'quota'?
I am a novice in the AWS area, please excuse me if there is any ambiguity in my question.
Not that I can think of.
If you really want to do this (restrict number of resources per user), you can create one account per user and restrict their rights so that they cannot request quota increase. This may not be a practical solution.
AWS does not associate resources with users.
When a user makes a request to create a resource (eg an Amazon EC2 instance), IAM will confirm that they are entitled to create the resource. If so, then the resource is created in the AWS Account associated with that user. There is no connection made between that resource and the user (aside from an entry in CloudTrail that records the creation request).
In brief
We need to view AWS cost for resources created by a specific IAM user via aws:createdBy tag.
In full
My google search is not helpful - it gives me one idea to use Consolidated Billing for Organizations to have the view of cost-by-user billing but that requires to create separate AWS users, which again requires credit card.
Another solution I found is that we can view AWS cost by tag grouping via Cost Explorer report though we cannot find the mentioned tag createdBy as stated in that AWS guide?
The ideal solution should let us create as many IAM users as the number of projects may grow i.e. one IAM user per project; and then, give us the report of AWS cost for each of those users.
So, how can we get to it?
Costs are not incurred by IAM Users. The AWS Account owning the resource incurs the charge.
When an IAM User makes a request to AWS, their credentials are verified to ensure that they are the authorized person and their permissions are checked to confirm that they have permission to perform the operation. However, there is no further relationship between the person making the request and the resource that is created.
The closest connection is the audit log created by AWS CloudTrail that can show which user made each request. But this is not related to billing information.
If you wish to track costs, you should associate tags with resources. These tags could be based on department, cost center, person, etc. You can then configure your Detailed Billing Report to provide cost breakdowns by tag.
You could use a more concise solution systematically using your own tags by setting up Lambda, CloudWatch, CloudTrail, and SSM to achieve it.
Here are some references:
Automatically tag new AWS resources based on identity or role
GitHub - resource auto tag
I would like to set up different AWS Identity and Access Management (IAM) users so that if an AWS resource is created by that IAM user, the resource is automatically assigned a specific tag.
For example: if IAM user F creates a new EC2 instance, the instance is automatically tagged as User:MrF. Is it possible to build a custom policy that does this?
My company GorillaStack have an open source lambda function that does exactly that.
The function 'listens' for CloudTrail logs to be delivered and tag the created resource with the ARN of the user that created it. It also support cross account tagging, for cases where a central account collects CloudTrail logs for other accounts.
Github: https://github.com/GorillaStack/auto-tag
Blog Post: http://blog.gorillastack.com/gorillastack-presents-auto-tag
It got a shout out at the 2015 re:Invent conference which is pretty cool :)
Hope that helps!
This is not available when using the AWS APIs directly (i.e. there's no way to command all AWS API's to tag new resources automatically on your behalf), however, depending on the specifics of your use case you could work around that limitation by correlating the creating user with the resource via post hoc tagging:
Workaround
You could activate AWS CloudTrail, which records AWS API calls for your account and delivers log files to you and provides exactly the information you are after:
The recorded information includes the identity of the API caller, the
time of the API call, the source IP address of the API caller, the
request parameters, and the response elements returned by the AWS
service.
Based on that information, a dedicated service of yours could analyze the logs and apply post hoc tags to all resources based on the logged user and created resource via the resp. API actions. Please see my answer to Which user launched EC2 instance? for some caveats/constraints to consider when going down this route.
An even better solution (faster plus I believe cheaper than parsing through CloudTrail logs) is to use CloudTrail but in combination with CloudWatch Events.
This is the basic concept described in a diagram
The implementation is detailed in this article:
https://blogs.aws.amazon.com/security/post/Tx150Z810KS4ZEC/How-to-Automatically-Tag-Amazon-EC2-Resources-in-Response-to-API-Events
The article also describes how to setup an IAM policy that only allows the creator of a resource to perform certain actions (like start / stop, describe, edit, terminate) against it.
I would chose AWS Config. Create a rule that automatically tags resources on creation. No cost, works across multiple accounts. Great for enforcing compliance. https://aws.amazon.com/about-aws/whats-new/2019/03/aws-config-now-supports-tagging-of-aws-config-resources/
Currently there is no such feature on IAM. If what you need is allow/deny based on user names, what you could do is use variables on your policy to allow or deny access based on naming conventions, e.g.:
...
"Resource":"arn:aws:dynamodb:us-east-!:123456789:table/ItemsCatalog_${aws:username}"
...
I am in the early stages of writing an AWS app for our users that will run our research algorithms using their AWS resources. For example, our code will need to spin up EC2 instances running our 'worker' app, access RDS databases, and create access SQS queues. The AWS Java SDK examples (we are writing this in Java) use a AwsCredentials.properties file to store the Access Key ID and Secret Access Key, which is fine for examples, but obviously not acceptable for our users, who are would be in essence giving us access to all their resources. What is a clean way to go about running our system on their behalf? I discovered AWS Identity and Access Management (IAM) which seems to be for this purpose (I haven't got my head around it yet), esp. Cross-account access between AWS accounts. This post makes it sound straightforward:
Use the amazon IAM service to create a set of keys that only has
permission to perform the tasks that you require for your script.
http://aws.amazon.com/iam/
However, other posts (e.g., Within IAM, can I restrict a group of users to access/launch/terminate only certain EC2 AMIs or instances?) suggest there are limitations to using IAM with EC2 in particular.
Any advice would be really helpful!
The key limitation with regards to RDS and EC2 is that while you can restrict access to certain API actions there are no resource level constraints. For example with an IAM S3 policy you can restrict a user to only being able to perform certain actions on certain buckets. You can write a policy for EC2 that says that user is allowed to stop instances, but not one that says you can only stop certain instances.
Another option is for them to provide you with temporary credentials via the Security Token Service. Another variant on that is to use the new IAM roles service. With this an instance has a set of policies associated with it. You don't need to provide an AwsCredentials.proprties file because the SDK can fetch credentials from the metadata service.
Finally one last option might be consolidated billing. If the reason you are using their AWS resources is just because of the billing, then setup a new account which is billed from their account. The accounts are isolated from each other so you can't for example delete their instances by accident. Equally you can't access their RDS snapshots and things like that (access to an RDS instance via mysql (as opposed to the AWS api) would depend on the instance's security group). You can of course combine this with the previous options - they could provide you with credentials that only allow you to perform certain actions within that isolated account.