One of RDS instance running within our account is deleted. We would like to find out who and when it is deleted.
Is there a way to find a log of that somewhere?
How can I see the events on RDS instance which is deleted
The when can be checked form the Events of your RDS dashboard page:
You can retrieve events for your RDS resources through the AWS Management Console, which shows events from the past 24 hours
If it happened earlier, you can use the CLI which retrieves events up to 14 days
[xxx#xxxx ~]# aws rds describe-events --source-identifier test --source-type db-instance
{
"Events": [
{
"Date": "2016-05-20T12:58:26.529Z",
"Message": "DB instance deleted",
"SourceIdentifier": "test",
"EventCategories": [
"deletion"
],
"SourceType": "db-instance"
}
]
}
Related
I am trying to delete orphaned snapshots, but the query I am using keeps giving me snapshots that are deleted. Is there a query I can use to avoid deleted snapshots?
aws ec2 describe-snapshots --snapshot-id snap-00012345cac2b3de1
{
"Snapshots": [
{
"Description": "DescriptionHere",
"Encrypted": false,
"OwnerId": "123456088429",
"Progress": "100%",
"SnapshotId": "snap-00012345cac2b3de1",
"StartTime": "2018-01-24T06:42:50+00:00",
"State": "completed",
"VolumeId": "vol-00123dc456ad5117",
"VolumeSize": 6,
"StorageTier": "standard"
}
]
}
To test your situation, I did the following:
Went to the EC2 Management Console and displayed Amazon EBS Volumes
Created a Snapshot of an EBS Volume: snap-036851d7351b78712
Ran aws ec2 describe-snapshots --snapshot-id snap-036851d7351b78712
It returned a result similar yours
Deleted the Snapshot in the Management Console
Ran the above command again. The result was:
An error occurred (InvalidSnapshot.NotFound) when calling the DescribeSnapshots operation: The snapshot 'snap-036851d7351b78712' does not exist.
So, I was unable to reproduce your situation.
I then wondered whether the Snapshot might be associated with an AMI. I did the following:
Created an AMI of an existing Amazon EC2 instance
Waited until the AMI creation was complete
Listed Snapshots in the console -- a new snapshot appeared snap-047563373ab4c1088
I then tried to delete the snapshot, but received the message:
snap-047563373ab4c1088: The snapshot snap-047563373ab4c1088 is currently in use by ami-0fc62425d087dbbe8
I then 'deregistered' (deleted) the AMI and it told me that the associated Snapshot would not be deleted:
I then manually deleted the Snapshot in the console
I used describe-snapshots and it said that the snapshot did not exist
So, perhaps your Snapshot is associated with an AMI and it was never actually deleted?
I created a CloudWatch Event Rule intended to send notifications when our Aurora clusters failed over. We then had a (manual) failover in our staging environment, but the rule didn't capture any events related to that failover as I expected.
This is the pattern that I used:
{
"source": [
"aws.rds"
],
"detail-type": [
"RDS DB Instance Event"
],
"detail": {
"EventCategories": [
"failover"
]
}
}
This pattern does generate a "sample event" in the rule editor within the web console, so it seems valid, but is there something that I need to do in order to get these particular events shipped to CloudWatch?
It turns out that the correct detail-type for Aurora failovers is "RDS DB Cluster Event" rather than "RDS DB Instance Event". The latter generates sample data in the editor, whereas the former does not, but that seems to be a bug.
We have developed an AWS Serverless Lambda application using dotnetcore to perform operations on EC2 Instances, say start or stop EC2 instance and integrated with Aws API Gateway.
serverless.template in dotnetcore application
"StartInstanceById" : {
"Type" : "AWS::Serverless::Function",
"Properties": {
"Handler": "EC2_Monitoring_Serverless::EC2_Monitoring_Serverless.Functions::StartInstanceById",
"Runtime": "dotnetcore2.1",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": "arn:aws:iam::2808xxxx1013:role/lamda_start_stop",
"Policies": [ "AWSLambdaBasicExecutionRole" ],
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/instances",
"Method": "Get"
}
}
}
}
}
The above Lambda function is working fine for starting ec2 instance when I invoking the API gateway url.
For calling these API's, We have created Angular 6 application and provided authentication using Aws Cognito Userpools.
So the cognito user logins into the website and gets all EC2 informations.
If the user wants to stop / start the EC2 instance, user will click on the particular button which invokes the relevant api gateway url of the lambda functions and It's working fine.
Now the question is who performed that action. After so much of research on stackoverflow and aws community forums for knowing who started or stopped the EC2 instances , I found Aws CloudTrail logs the information when user start or stopped the instance.
So I created a trail and I can see the logs in S3 buckets. But in every log I opened, I saw that the role "arn:aws:iam::2808xxxx1013:role/lamda_start_stop" is captured. I know this is because of the Lambda function. But I want to know who really stopped the instance.
Please advice how to capture user details!
The reason lambda execution role is getting printed in cloudtrail, is because it has initiated the process to stop the ec2 instance. Here the role is assumed (instead of actual user).
To print your actual user, you need to implement logs at your lambda, which will print logs to Cloudwatch. You can get the actual user or any other custom information from those logs.
I have created a workflow like this:
Use requests for an instance creation through a API Gateway endpoint
The gateway invokes a lamda function that executes the following code
Generate a RDP with the public dns to give it to the user so that they can connect.
ec2 = boto3.resource('ec2', region_name='us-east-1')
instances = ec2.create_instances(...)
instance = instances[0]
time.sleep(3)
instance.load()
return instance.public_dns_name
The problem with this approach is that the user has to wait almost 2 minutes before they were able to login successfully. I'm totally okay to let the lamda run for that time by adding the following code:
instance.wait_until_running()
But unfortunately the API gateway has a 29 seconds timeout for lambda integration. So even if I'm willing to spend it wouldn't work. What's the easiest way to overcome this?
My approach to accomplish your scenario could be Cloudwatch Event Rule.
The lambda function after Instance creation must store a kind of relation between the instance and user, something like this:
Proposed table:
The table structure is up to you, but these are the most important columns.
------------------------------
| Instance_id | User_Id |
------------------------------
Creates a CloudWatch Event Rule to execute a Lambda function.
Firstly, pick Event Type: EC2 Instance State-change Notification then select Specific state(s): Running:
Secondly, pick the target: Lambda function:
Lambda Function to send email to the user.
That Lambda function will receive the InstanceId. With that information, you can find the related User_Id and send the necessary information to the user. You can use the SDK to get information of your EC2 instance, for example, its public_dns_name.
This is an example of the payload that will be sent by Cloudwatch Event Rule notification:
{
"version": "0",
"id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
"detail-type": "EC2 Instance State-change Notification",
"source": "aws.ec2",
"account": "111122223333",
"time": "2015-12-22T18:43:48Z",
"region": "us-east-1",
"resources": [
"arn:aws:ec2:us-east-1:123456789012:instance/i-12345678"
],
"detail": {
"instance-id": "i-12345678",
"state": "running"
}
}
That way, you can send the public_dns_name when your instance is totally in service.
Hope it helps!
I'm trying to forward the EC2 Launch logs to cloudwatch from my win 2016-based EC2 instance.
For some reason I can't see the log groups for this specific category.
Here's example of my AWS.EC2.Windows.CloudWatch.json:
{
"IsEnabled": true,
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [
{
"Id": "Ec2Config",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Log",
"TimestampFormat": "yyyy-MM-ddTHH:mm:ss.fffZ:",
"Encoding": "UTF-8",
"Filter": "UserdataExecution.log",
"CultureName": "en-US",
"TimeZoneKind": "UTC"
}
},
{
"Id": "EC2ConfigSink",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"Region": "eu-west-1",
"LogGroup": "/my-customer/deployment/ec2config-userdata",
"LogStream": "ec2config-userdata"
}
}
...
I have a few more definitions in this file
...],
"Flows": {
"Flows":
[
"Ec2Config,EC2ConfigSink",
... other references here
]
}
}
Cloudwatch agent starts and doesn't report any errors, I can see data from other sources (some application log files - I skipped the definitions intentionally)
It means the cloudwatch config file is correct and is applied / placed in a correct directory.
Logs are coming through with no problem except for the EC2 launch logs.
I'm wondering if anybody ran into this problem? It works perfectly on Windows 2012 - based images
Apparently, the SSM Agent starts after the EC2 Launch executes UserDatascript. I can see it from the SSM Agent's log file modification timestamps.
Therefore, there's no log forwarding happening during the EC2 Launch.
When the SSM Agent starts and loads the cloudwatch plugin, the log files are already filled with entries and never change (wallpaper log is the only exception) So they never end up in cloudwatch console.
There's been a lot of changes implemented on AWS side: they switch to .Net core, removed EC2 config service and moved the log forwarding logic to SSM Agent (cloudwatch Plugin) for Windows 2016-based AMIs
It looks like the behavior has changed quite significantly too so there's no way to get the EC2 launch logs in cloudwatch (when using AWS toolset-only)
Basically we have to stick to our Application logs only which is very unfortunate. We rely on EC2 launch logs to see if the instance started & successfully executed user data.