I am trying to see which user was responsible for changes in S3 (at buckets level).
I could not find a audit trail for actions done at S3 bucket level or EC2 who created instances. Beanstalk has a log of the actions the machine performed, but not which user.
Is there a way around AWS that we can see this information in IAM or any other location ?
P.S: I am not interested to know about S3 log buckets which provide access logs
Update
AWS has just announced AWS CloudTrail, finally making auditing API calls available as of today (and for free), see the introductory post AWS CloudTrail - Capture AWS API Activity for details:
Do you have the need to track the API calls for one or more AWS
accounts? If so, the new AWS CloudTrail service is for you.
Once enabled, AWS CloudTrail records the calls made to the AWS APIs
using the AWS Management Console, the AWS Command Line Interface
(CLI), your own applications, and third-party software and publishes
the resulting log files to the Amazon S3 bucket of your choice.
CloudTrail can also issue a notification to an Amazon SNS topic of
your choice each time a file is published. Each call is logged in JSON
format for easy parsing and processing.
Please note the following (temporary) constraints:
Not all services are covered yet, though the most important ones are included in the initial release already and AWS plans to add support for additional services over time.
Update: AWS has recently added Seven New Services, and another one today, see below.
More importantly, not all regions are supported yet (right now the US East (Northern Virginia), and US West (Oregon) Regions only), though AWS will be adding support for additional Regions as quickly as possible.
Update: AWS has just added More Locations and Services, quickly approaching coverage of their entire Global Infrastructure indeed.
Initial Answer
This is a long standing feature request, but unfortunately AWS does not provide (public) audit trails as of today - the most reasonable way to add this feature would probably be a respective extension to AWS Identity and Access Management (IAM), which is the increasingly ubiquitous authentication and authorization layer for access to AWS resources across all existing (and almost certainly future) Products & Services.
Accordingly there are a few respective answers provided within the IAM FAQs along these lines:
Will AWS Identity and Access Management administrative actions be logged to an audit trail?:
No. This is planned for a future release.
Will user actions in AWS services be logged to an audit trail?
No. This is planned for a future release.
Current pricing for a single CloudTrail is free.
1. Enable CloudTrail
Use the CloudTrail dashboard and send all events to an S3 bucket, e.g. my-cloudtrail
2. Go Through the Results
The CloudTrail dashboard let's you do some cursory searches, but if you have many thousands of events, it's a pain to use.
Let's say I want actions for user foo_user, I just use the CLI tool:
mkdir -p /tmp/cloudtrail
cd /tmp/cloudtrail
aws s3 sync s3://mc10-cloudtrail .
cd AWSLogs
zcat `find . -type f` | jq '.Records[] | "\(.eventName) \(.userIdentity.userName)"' | grep food_user | sort | uniq
Example Output:
"CreateGrant foo_user"
"DescribeInstances foo_user"
"GetConsoleOutput foo_user"
"ModifyInstanceAttribute foo_user"
"StartInstances foo_user"
"StopInstances foo_user"
Note: S3 data events are billed differently in CloutTrail, but this is somewhat redundant, because you can just enable logging on your S3 bucket and grep those logs, or point them at Logstash/Kibana.
Related
I'm the administrator of an AWS account that has 4 users. One of the users is racking up higher-than-expected costs.
I checked the Cost Explorer, but could not seem to configure it to view individual users.
As an administrator in AWS, how do I see all of the services this particular user has been using during a given period of time (e.g. the last 12 months)?
Thanks!
AWS resources are associated with an AWS Account, not a specific user.
If a user has the necessary permissions to create resources (eg an Amazon EC2 instance), then the instance is launched in the AWS Account, but there is no link back to the user that requested the resource.
You can, however, use AWS CloudTrail:
AWS CloudTrail is an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.
It will show all API calls made by the user, including the Action ('launch an EC2 instance'), their IP address, timestamp, etc. Operations in the AWS management console will also be shown, since it makes API calls on behalf of the user.
CloudTrail keeps a history of the past 90 days, but you can create specific 'Trails' that retain data permanently. If you have not done this, then it will not be possible to see what they did prior to 90 days ago.
I wanted to know if there was a way to track alerts or audit anything that happens with the AWS account like who changed what and why. I did find this https://docs.aws.amazon.com/opensearch-service/latest/developerguide/audit-logs.html where they use a comand line for enabling audit logs on an existing domain: aws opensearch update-domain-config --domain-name my-domain --log-publishing-options "AUDIT_LOGS={CloudWatchLogsLogGroupArn=arn:aws:logs:us-east-1:123456789012:log-group:my-log-group,Enabled=true}" but this is in regard to Amazon OpenSearch Service which I believe is only free for 12 months if you haven't used already. AWS Audit Manager. I am aware there are services that can do this but require a fee and I wanted to know if there were any free options
From the AWS documentation:
With AWS CloudTrail, you can monitor your AWS deployments in the cloud by getting a history of AWS API calls for your account, including API calls made by using the AWS Management Console, the AWS SDKs, the command line tools, and higher-level AWS services. You can also identify which users and accounts called AWS APIs for services that support CloudTrail, the source IP address from which the calls were made, and when the calls occurred. You can integrate CloudTrail into applications using the API, automate trail creation for your organization, check the status of your trails, and control how administrators turn CloudTrail logging on and off.
AWS Config provides a detailed view of the resources associated with your AWS account, including how they are configured, how they are related to one another, and how the configurations and their relationships have changed over time.
Basically, AWS CloudTrail keeps a log of API calls (requests to AWS to do/change stuff), while AWS Config tracks how individual configurations have changed over time (for a limited range of resources, such as Security Group rule being changed).
I am about to launch a webapp based on subscription. FYI, the web application manages health care data, and my customers are concerned about the security of data in the cloud.
Is there any certificate, or any official information I can give to my customers on the behalf of AWS proving that the data in any storage used by my application will be encrypted?
THANK YOU
From What is AWS Artifact?:
AWS Artifact provides on-demand downloads of AWS security and compliance documents, such as AWS ISO certifications, Payment Card Industry (PCI), and Service Organization Control (SOC) reports. You can submit the security and compliance documents (also known as audit artifacts) to your auditors or regulators to demonstrate the security and compliance of the AWS infrastructure and services that you use. You can also use these documents as guidelines to evaluate your own cloud architecture and assess the effectiveness of your company's internal controls. AWS Artifact provides documents about AWS only. AWS customers are responsible for developing or obtaining documents that demonstrate the security and compliance of their companies.
It explains what AWS does. However, you would also need to prove that you are using the cloud correctly, such as verifying user's identities and not making buckets public.
NO, there is no such a document, you need to apply and obtain this certificate.
AWS is complaint, for there part Security of the cloud, and you are responsible for the Security in the cloud. AWS Artifact is a repository.
AWS Config is the tool you will use to monitor the configuration of
your stack, can repair configurations also.
AWS Cloudwach will monitor the performance, brings you alerts and evoke Lambda
AWS Cloud Trail will monitor the API calls.
AWS Macy to check your buckets for Personal Identifiable information.
Then you are the one who enable encryption and choose the Key management and rotation, AWS KMS.
Just to mention few services to be aware of. Best regards.
Can you get the AWS usage report for subdirectory for buckets? I want to know the amount of traffic of all 'GetObject' requests for all subdirectory of S3.
First, remember that there are no "subdirectories" in S3. Everything within a bucket is in a flat index and identified by an object key. However, in the AWS console, objects that contain a shared prefix are represented together in a "folder" named after the shared prefix.
With that in mind, it should be easier to understand why you cannot get an AWS usage report for a specific "subdirectory". The AWS usage report is meant to be an overview of your AWS services and is not meant to be used for more detailed analytics.
Instead there is another AWS service that allows you insight into more detailed analytics for your other AWS services: AWS CloudWatch. With AWS Cloudwatch you can:
Set up daily storage
metrics
Set up request (GET) metrics on a
bucket
And, for your specific case, you can set up request metrics for specific prefixes (subdirectories) within a bucket.
Using request metrics from AWS CloudWatch is a paid service (and another reason why you cannot get detailed request metrics in the AWS usage report).
What's the difference between the AWS S3 logs and the AWS CloudTrail?
On the doc of CloudTrail I saw this:
CloudTrail adds another dimension to the monitoring capabilities
already offered by AWS. It does not change or replace logging features
you might already be using.
CloudTrail tracks API access for infrastructure-changing events, in S3 this means creating, deleting, and modifying bucket (S3 CloudTrail docs). It is very focused on API methods that modify buckets.
S3 Server Access Logging provides web server-style logging of access to the objects in an S3 bucket. This logging is granular to the object, includes read-only operations, and includes non-API access like static web site browsing.
AWS has added one more functionality since this question was asked, namely CloudTrail Data events
Currently there are 3 features available:
CloudTrail: Which logs almost all API calls at Bucket level Ref
CloudTrail Data Events: Which logs almost all API calls at Object level Ref
S3 server access logs: Which logs almost all (best effort server logs delivery) access calls to S3 objects. Ref
Now, 2 and 3 seem similar functionalities but they have some differences which may prompt users to use one or the other or both(in our case)! Below are the differences which I could find:
Both works at different levels of granularity. e.g. CloudTrail data events can be set for all the S3 buckets for the AWS account or just for some folder in S3 bucket. Whereas, S3 server access logs would be set at individual bucket level
The S3 server access logs seem to give more comprehensive information about the logs like BucketOwner, HTTPStatus, ErrorCode, etc. Full list
Information which is not available in Cloudtrail logs but is available in Server Access logs. Reference:
Fields for Object Size, Total Time, Turn-Around Time, and HTTP Referer for log records
Life cycle transitions, expiration, restores
Logging of keys in a batch delete operation
Authentication failures
CloudTrail does not deliver logs for requests that fail authentication (in which the provided credentials are not valid). However, it does include logs for requests in which authorization fails (AccessDenied) and requests that are made by anonymous users.
If a request is made by a different AWS Account, you will see the CloudTrail log in your account only if the bucket owner owns or has full access to the object in the request. If that is not the case, the logs will only be seen in the requester account. The logs for the same request will however be delivered in the server access logs of your account without any additional requirements.
AWS Support recommends that decisions can be made using CloudTrail logs and if you need that additional information too which is not available in CloudTrail logs, you can then use Server access logs.
There are two reasons to use CloudTrail Logs over S3 Server Access Logs:
You are interested in bucket-level activity logging. CloudTrail has that, S3 logs does not.
You have a log analysis setup that involves CloudWatch log streams. The basic S3 logs just store log events to files on some S3 bucket and from there it's up to you to process them (though most log analytics services can do this for you).
Bottom line: use CloudTrail, which costs extra, if you have a specific scenario that requires it. Otherwise, the "standard" S3 Server Access Logs are good enough.
From the CloudTrail developer guide (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudtrail-logging.html):
Using CloudTrail Logs with Amazon S3 Server Access Logs and CloudWatch Logs
You can use AWS CloudTrail logs together with server access logs for Amazon S3. CloudTrail logs provide you with detailed API tracking for Amazon S3 bucket-level and object-level operations, while server access logs for Amazon S3 provide you visibility into object-level operations on your data in Amazon S3. For more information about server access logs, see Amazon S3 Server Access Logging.
You can also use CloudTrail logs together with CloudWatch for Amazon S3. CloudTrail integration with CloudWatch logs delivers S3 bucket-level API activity captured by CloudTrail to a CloudWatch log stream in the CloudWatch log group you specify. You can create CloudWatch alarms for monitoring specific API activity and receive email notifications when the specific API activity occurs. For more information about CloudWatch alarms for monitoring specific API activity, see the AWS CloudTrail User Guide. For more information about using CloudWatch with Amazon S3, see Monitoring Metrics with Amazon CloudWatch.
AWS CloudTrail is an AWS service for logging all account activities on different AWS resources. It also tracks things like IAM console login etc. Once CloudTrail service is enabled you can just go to CloudTrail console and see all the activity and also apply filters. Also, while enabling you can choose to log these activities and send the data to AWS CloudWatch. In AWS CloudWatch you can apply filters and also create alarms to notify you when a certain kind of activity happens.
S3 logging is enabling logging for basic activity on your S3 buckets/Objects.
CloudTrail logs API calls accessed to your AWS Account.
These CloudTrail logs are stored in Amazon S3 Bucket.
The two offer different services.
The Definition you have shared from CloudTrail Doc:
CloudTrail adds another dimension to the monitoring capabilities already offered by AWS. It does not change or replace logging features you might already be using.
It means you might have already activated some of the other logging features offered in other AWS services like ELB logging etc..
But when you enable CloudTrail monitoring, you need not worry about your previous logging functionalities as they will be still active.
You will recieve logs from all the services.
So By Enabling CloudTrail logging, It does not change or replace logging features you might already be using.
Hope it Helps.. :)