how is the best solution to create a alert with athena data?
the loadbalance logs is saved in S3 and acessed with Athena.
I want alert me based on a athena query. I thinking about populate Cloudwatch metrics with athena result, and create a alert in cloudwatch. Can I create a metric direct with athena result?
I can resolve with lambda, but I'm search a better solution.
Your question is a little bit ambiguous.
Do you want to create an alert when Athena writes to an S3 bucket?
You can achieve this through SNS and the S3 bucket config. Check this tutorial:
Example Walkthrough 1: Configure a Bucket for Notifications (Message Destination: SNS Topic and SQS Queue)
Then you can launch a different AWS service (like a Lambda function) through SNS to react to this event.
Keep in mind that you will have to wait for Athena to finish the query as it will run asynchronously. Here an example in Java to achieve this:
Athena Query execution Handling
Related
Is there a way to find the number of EC2 instances which were launched in last 1/2/3/4/5 or 6 months in all regions? (running and terminated).
From a similar question as below, I can only get the current status (running|stopped|terminated) but not anything from past months.
How to see all running Amazon EC2 instances across all regions?
Please advise. This is purely for audit purpose.
Thanks in advance.
AWS CloudTrail makes it easier to ensure compliance with internal policies and regulatory standards by providing a history of activity in your AWS account.
AWS have an option to view Event History if you have CloudTrail enabled. Please go through this AWS page to view clear instructions.
If you like to use AWS CLI then this documentation provides all the details.
I would recommend a combination of CloudTrail logs stored in S3 and Athena to do the query. The problem with CloudTrail alone is that you have a three month window before logs roll off. Your requirements include as far out as six months.
To deliver log files to an S3 bucket, CloudTrail must have the required permissions, and it cannot be configured as a Requester Pays bucket. CloudTrail automatically attaches the required permissions to a bucket when you create an Amazon S3 bucket as part of creating or updating a trail in the CloudTrail console.
To setup Athena you can configure through the CloudTrail Console:
Open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/
In the navigation pane, choose Event history.
Choose Create Athena table.
For Storage location, use the down arrow to select the Amazon S3 bucket where log files are stored for the trail to query.
Choose Create table. The table is created with a default name that includes the name of the Amazon S3 bucket.
Then you can run a query similar to this in Athena:
SELECT eventname,
useridentity.principalid,
awsregion,
eventtime
FROM cloudtrail_logs
WHERE eventtime >= '2021-02-01T00:00:00Z'
AND eventtime < '2021-08-30T00:00:00Z'
AND (eventname ='RunInstances')
References
Create S3 Bucket Policy for CloudTrail
Query CloudTrail logs with Athena
Athena Search CloudTrail Logs
So I am trying to write AWS Cloudwatch queries to show information of only when sensitive actions occur in a S3 bucket. Therefore I want to filter on the event name. Do you know where I can get the FULL list of all the event names on S3 such as PUTOBJECT and etc. This is so I can select what to filter on
Thank you
You can check S3 API reference for that.
But for some events, such as PutObject, you will need to create CloudTrial trial and log data level operations.
I have a usecase where I want to put data into an S3 bucket, for it to read later, by another account. I only want the other account to be able to read the file in S3, and once they have read it, I will then delete the file myself.
I have been reading the S3 documentation, and cannot see they cover this usecase: of sending a notification when a file in an S3 bucket is read ?
Can anyone help, or suggest an alternative workflow ? I have been looking at AWS SNS and was wondering if that would be a better solution ?
You could use CloudTrail and CloudWatch Events to enable this workflow.
By default S3 API calls are not logged so you'd want to enable that following the instructions here.
Then enable a CloudWatch event rule for the Simple Storage Service where the "GetObject" operation occurs.
Have this event invoke a Lambda function that will remove the object.
More information available here.
I have created some IAM users to my AWS account with permission to launch instances.
Now I want to track and store their instance launch activity like time and instance ID in my MySQL or any other database.
Is there any way to achieve this, any suggestion will be appreciated.
All activities of an IAM user can be monitored using aws cloudtrail. Cloudtrail logs all the events.
The cloudtrail log is stored to a S3 bucket. You can use the storage trigger option in aws lambda functions to watch for a particular log .
In this case the log for new EC2 instance creation.
In the lambda function you need to add the code that takes that log information and stores into a Mysql database that you have setup.
Refer this post https://docs.aws.amazon.com/lambda/latest/dg/with-cloudtrail.html
Also you can try creating a cloudwatch for EC2 instance creation and it can trigger an aws lambda function which will do the data insert to the db.
Here is a sample of cloudwatch based scheduler. You have to setup a specific trigger as per your need though.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
You should use AWS CloudTrail:
CloudTrail is enabled on your AWS account when you create it. When activity occurs in your AWS account, that activity is recorded in a CloudTrail event. You can easily view recent events in the CloudTrail console by going to Event history. For an ongoing record of activity and events in your AWS account, create a trail.
I'm trying to achieve the ask from the title. For example, my architecture design involves trigger a Lambda function whenever a new data land on the open data s3 bucket (say this one: https://registry.opendata.aws/sentinel-2/).
I read https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html and Amazon S3 triggering another a Lambda function in another account but I non of those really help me so far. Had anyone done similar task before? Thanks in advance!
You can configure Amazon S3 events to send a message to an Amazon SNS Topic OR have it trigger an AWS Lambda function.
If you wish to do any further logic (eg checking specific permissions), you would need to do that within the Lambda function.
See: Configuring Amazon S3 Event Notifications - Amazon Simple Storage Service