I want email notification if any specific EC2 windows service entered into the stopped state.
I configured CloudWatch, able to receive logs of all windows services.
Created a lambda function to get notify when any service entered into the stopped state, but the problem is I am receiving alert only when I click on the test function.
I am receiving CloudWatch logs like this:
03:43:02 [System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The Background Intelligent Transfer Service service entered the running state.]
03:43:02 [System] [INFORMATION] [7040] [Service Control Manager] [mydomain.com] [The start type of the Background Intelligent Transfer Service service was changed from demand start to auto start.]
03:43:02 [System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The WinHTTP Web Proxy Auto-Discovery Service service entered the running state.]
03:45:02 [System] [INFORMATION] [7040] [Service Control Manager] [mydomain.com] [The start type of the Background Intelligent Transfer Service service was changed from auto start to demand start.]
This is my lambda function:
import boto3
import time
client = boto3.client('logs')
sns = boto3.client('sns')
instance_name = "Development"
a1 = int(round(time.time() * 1000))
def lambda_handler(event, context):
response = client.get_log_events(
logGroupName = 'Eadev',
logStreamName = 'i-01fe1z56y790cq',
startTime = a1,
startFromHead = False
)
event01 = '[System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The DebtManager-Host service entered the stopped state.]'
event02 = '[System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The DebtManager-Controller service entered the stopped state.]'
for i in response['events']:
if event01 == i['message']:
print(event01)
sns.publish( TargetArn = "arn:aws:sns:us-east-1:3913948:testsns",Message = instance_name +" "+ event01)
if event02 == i['message']:
print(event02)
sns.publish( TargetArn = "arn:aws:sns:us-east-1:3913948:testsns",Message = instance_name +" "+ event02)
I expected email notification from any service stopped, but I am receiving alert only when I clicked on test in Lambda function.
It appears that your desired situation is:
The Amazon CloudWatch agent on the Windows instance sends log data to Amazon CloudWatch Logs
Send a notification when a particular entry is detected in the log file
Rather than triggering a Lambda function for every log message, you can use CloudWatch Logs Filter Metrics to trigger a CloudWatch Alarm:
Collecting Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent
Searching and Filtering Log Data to detect the desired messages by Creating Metric Filters
This pushes metrics into Amazon CloudWatch Metrics
You can then create a traditional Amazon CloudWatch Alarm on the metric and have it trigger when a certain number of such messages are received
A CloudWatch Alarm can send a notification an Amazon SNS topic
For an end-to-end example, see: Use Amazon CloudWatch Logs Metric Filters to Send Alerts - The IT Hollow
Alternatively, you can use an AWS Lambda function:
Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent
Use Real-time Processing of Log Data with Subscriptions
It can accept a subscription filter to identify the records of interest
It can then trigger an AWS Lambda function, which you can program to do whatever you wish (eg send a message to an Amazon SNS topic)
Related
We are using cloudwatch metrics filter and setup alarm and send notification through SNS - Email.
We are wondering if it is possible to also see logs that triggers the alarm in the email? or is it possible to do so with the help with a custom lambda function?
Thanks
Locally, you may try to have the following set up.
CloudWatch Alarm
SNS-log-reader with lambda as a subscriber
SNS-send-mails with e-mails as a subscriber
So the logic would be - Alarm trigger SNS-log-reader. Then lambda which is mapped to *SNS-log-reader can read the logs from LogGroup, build the message in needed format and send to e-mail/s via SNS-send-mails (or Simple Email Service (Amazon SES))
I have cloudtrail setup and i am sending trail events to cloudwatch logs group and i setup cloudwatch alarm with metrics filter { ($.eventName = RebootInstances) || ($.eventName = StopInstances) || ($.eventName = TerminateInstances) }
to invoke SNS to send email notifications.
But I am not getting email notification every time instance is terminated. I am getting email notification only sometimes when instance is terminated.
If your instance is in auto-scaling group you can use Lifecycle Hooks to define an action (e.g. run Lambda function or send SNS notification when the instance enters certain state. This way you don't have to wait for 10 mins to receive the notification.
Cloud trail is taking some time to update the event, I will suggest you to wait for at least 10 min after terminating the server. You will get the mail.
You can configure the notification script also under the run level which will send you an email using SES if your server reboot, shutdown or terminate.
You can go to the below blog to setup this.
https://dzone.com/articles/hot-notifyemail-yourself-when
At some point in a day, one of my instances gets stopped and I am trying to send an SNS notification at that time.
I can do it in multiple ways:
1.) Create an alarm on the metric StatusCheckFailed_Instance and when this reaches Insufficient_Data state, i configure it to send an SNS notification.
2.) I can write a lambda script (put_metric_alrm) to create it for me and send it when state changes to Insufficient_Data.
My problem while doing this is the notification is being sent once a day if my instance is stopped for days. Instead, I want a notification to be sent only when it gets stopped.
How can I do this? Please suggest.
Take a look at CloudWatch Events; you can create a new 'rule' such that an event will trigger on EC2 instance state-change notifications for a given instanceid where the state is 'stopped'. When the event is fired, you can then invoke an SNS topic to get notified of the change.
Here is an example configuration:
Service Name: EC2
Event Type: EC2 Instance State-change notification
Specific state(s): stopped
Specific instances: Your Instance ID
Targets: Your SNS Topic Name
The same example, as seen in the CloudWatch Events console:
Simply subscribe your SMS or email to the SNS topic, and you'll get notified when this instance enters the stopped state.
See also:
AWS API Documentation - PutRule
AWS CLI Documentation - aws events put-rule
I need to find a way to report on website uptime (as a percentage) based on AWS Route53 monitoring. This reporting is generally done quarterly.
My initial thought was to have CloudWatch send ALARM and OK states via SNS to SQS, and then process this queue into a database for later reporting. As far as I can tell, however, CloudWatch will only send emails even though an SQS queue is subscribed to the topic.
Any suggestions of how I might achieve this?
Amazon Route 53 can be configured to send healthcheck data to Amazon SQS.
It worked for me -- here's the steps I took:
Create an Amazon SNS notification topic in us-east-1 (where Route 53 performs its health checks)
Create an Amazon SQS queue in us-east-1 (same region as the notification topic)
Subscribe the Amazon SQS queue to the Amazon SNS topic (via the Queue Actions menu option)
Create an Amazon Route 53 Health Check. Set Create Alarm to Yes. Configure it to Send notification to Existing SNS topic and choose the topic created above.
An Amazon CloudWatch alarm will be automatically created by Amazon Route 53.
This will result in health notifications arriving in the SQS queue. However, it will only send an ALARM notification -- there is no notification when it becomes healthy again. To receive a "now healthy" notification, edit the CloudWatch alarm and add a new Notification that triggers when "State is OK".
Here is an example of a failure notification retrieved from the SQS queue:
{
"Type" : "Notification",
"MessageId" : "4768e8e4-0026-51c7-aa6e-a696bf02f808",
"TopicArn" : "arn:aws:sns:us-east-1:123456789012:r53-east",
"Subject" : "ALARM: \"awsroute53--4c2f-9816-a42c50ec8671-High-HealthCheckStatus\" in US - N. Virginia",
"Message" : "{\"AlarmName\":\"awsroute53-4c2f-9816-a42c50ec8671-High-HealthCheckStatus\",\"AlarmDescription\":null,\"AWSAccountId\":\"743112987576\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 datapoint (0.0) was less than the threshold (1.0).\",\"StateChangeTime\":\"2015-09-16T00:50:44.591+0000\",\"Region\":\"US - N. Virginia\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"HealthCheckStatus\",\"Namespace\":\"AWS/Route53\",\"Statistic\":\"MINIMUM\",\"Unit\":null,\"Dimensions\":[{\"name\":\"HealthCheckId\",\"value\":\"4c2f-9816-a42c50ec8671\"}],\"Period\":60,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"LessThanThreshold\",\"Threshold\":1.0}}",
"Timestamp" : "2015-09-16T00:50:44.656Z",
"SignatureVersion" : "1",
"Signature" : "KvCHsBh95q...cw8A==",
"SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-90147a5624348ee.pem",
"UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:r53-east:4b5d-8318-57bd58f0b3a4"
}
One option is to offload Route53 health check statistics to Axibase Time Series Database and enable scheduled reports as discussed in the uptime reports article.
Implementation notes:
Need to create a read-only account to query CloudWatch statistics and Route53 Health Check metadata.
Offload task has a latency of 5-15 minutes (configurable).
Offload task ensures no data gaps in copied CW statistics, i.e. when collection temporarily stops for whatever reason.
Available reports:
Base report: average uptime over the period (day, week, month, quarter).
Enhanced uptime report with additional check configuration details.
Filter results by protocol or custom tag.
Aggregate uptime by custom tag such as GEO, environment.
Filter results by hours of the day or working days.
Aggregate uptime by day of week.
Downtime incident count.
Longest downtime incidents.
The reports can be generated interactively, via a web-based console, delivered via email, or displayed on portals.
Disclaimer: I work for Axibase.
Is it possible to create a CloudWatch alarm that triggers when my EC2 instance is accessed from a remote location (using SSH login with key pairs, for example)?
There is no in-built capability to perform an action when users login to an Amazon EC2 instance. You would have to write a script that activates when users login, checks the situation and then triggers a notification.
Amazon CloudWatch alarms trigger when a metric exceeds a pre-determined value. Therefore, you would need to do the following:
Have your custom script detect the situation and then publish a custom metric to CloudWatch
Create an Alarm in CloudWatch that triggers when the metric exceeds your desired value (for example, when it is greater than zero)
Configure your alarm to respond as desired (eg send a notification via SNS)
If you just wish to receive a notification when the user logs-in, you could bypass CloudWatch and just have your script publish a message to the SNS (Simple Notification Service) topic directly. It would have the same result.