I have a CloudWatch alarm for state change with the following code:
{
"source": [
"aws.ec2"
],
"detail-type": [
"EC2 Instance State-change Notification"
],
"detail": {
"state": [
"shutting-down",
"stopping"
]
}
}
It is working fine, but it also triggers events for AWS Instance Scheduler. Is there any way to prevent scheduled state changes from triggering this alarm? I googled for it, but no success.
Related
I have created cloudwatch event rule for glue job state change. I am getting notifications correctly for all glue job state change. But i need to send notifications for some particular glue jobs. i tried with adding multiple jobs but not working properly.
glue jobs:
glue_job1
glue_job2
glue_job3
event rule: with this rule i can get notifications for all jobs.
{
"source": [
"aws.glue"
],
"detail-type": [
"Glue Job State Change"
],
"detail": {
"state": [
"FAILED",
"TIMEOUT",
"SUCCEEDED"
]
}
}
event rule2: with this i am not getting proper notifications
{
"source": [
"aws.glue"
],
"detail-type": [
"Glue Job State Change"
],
"detail": {
"state": [
"FAILED",
"TIMEOUT",
"SUCCEEDED"
],
"jobName": [
"glue_job1",
"glue_job2",
"glue_job3"
]
}
}
how can we send notifications for only specific glue jobs ? how can we create event rule for this scenario. Thank you
I want to setup Cloudwatch events rule based on the task state changes of my ECS cluster via terraform. I would like to be notified whenever the task goes to STOPPED or RUNNING state.
Will this rule work and match whenever either of the events happen? I will then create resource "aws_cloudwatch_event_target" accordingly:
resource "aws_cloudwatch_event_rule" "ecs-sns-rule" {
name = "ecs task state change"
event_pattern = <<PATTERN
{
"source": [
"aws.ecs"
],
"detail-type": [
"ECS Task State Change"
],
"detail": {
"lastStatus": [
"STOPPED",
"RUNNING"
],
"clusterArn": "arn:aws:ecs:us-west-2:XXXXXXXXX:Cluster/XXXXXXXX"
}
}
PATTERN
}
From the docs:
Match values are always in arrays.
Therefore, I think you could try the following (assuming everything else is fine):
resource "aws_cloudwatch_event_rule" "ecs-sns-rule" {
name = "ecs task state change"
event_pattern = <<PATTERN
{
"source": [
"aws.ecs"
],
"detail-type": [
"ECS Task State Change"
],
"detail": {
"lastStatus": [
"STOPPED",
"RUNNING"
],
"clusterArn": ["arn:aws:ecs:us-west-2:XXXXXXXXX:Cluster/XXXXXXXX"]
}
}
PATTERN
}
I want to receive an email every time I update my ECS service (and once the update finishes or the desired state was reached)
I thought about CloudWatch Events Rules setting an SNS topic as target (which a confirmed email address). However, it doesn't work.
This is my custom Event pattern:
{
"detail-type": [
"ECS Update"
],
"resources": [
"arn:aws:ecs:us-east-1:aws-account:service/myService"
],
"source": [
"aws.ecs"
],
"detail": {
"clusterArn": [
"arn:aws:ecs:us-east-1:aws-account:cluster/myCluster"
],
"eventName": [
"SERVICE_STEADY_STATE"
],
"eventType": [
"INFO"
]
}
}
I also tried:
TASKSET_STEADY_STATE
CAPACITY_PROVIDER_STEADY_STATE
SERVICE_DESIRED_COUNT_UPDATED
I'm updating the service through the cli
aws ecs update-service --cluster myCluster --service myService --task-definition myTaskDef --force-new-deployment --desired-count 2
The status of the event rule is enabled and the target is the SNS topic. The input is matched event.
I don't have any clue. Am I using the wrong event name?
You can also set email notification on Task instead of service, also there is an issue regarding ECS notification.
I was not able to make it base on ECS status change, I controlled notification at lambda level. you can set this rule and its working for me.
{
"source": [
"aws.ecs"
],
"detail-type": [
"ECS Service Action"
]
}
you can expect a bit delay as I already experienced this and also reported in GitHub Issue.
Here is the JSON event that you will receive for above rule.
{
"version": "0",
"id": "c3c27e7b-abcd-efgh-c84e-highgclkl",
"detail-type": "ECS Service Action",
"source": "aws.ecs",
"account": "1234567890",
"time": "2020-06-27T00:00:00.00Z",
"region": "us-west-2",
"resources": [
"arn:aws:ecs:us-west-2:1234567890:service/test"
],
"detail": {
"eventType": "INFO",
"eventName": "SERVICE_STEADY_STATE",
"clusterArn": "arn:aws:ecs:us-west-2:123456789:cluster/mycluster",
"createdAt": "2020-06-27T00:00:00.00Z"
}
}
ecs_cwe_events
or the other option is so you can try task-based changes.
{
"source": [
"aws.ecs"
],
"detail-type": [
"ECS Task State Change"
],
"detail": {
"lastStatus": [
"STOPPED",
"RUNNING"
],
"clusterArn": [
"arn:aws:ecs:us-west-2:123456789:cluster/my_cluster",
]
}
}
I want to create a lambda function that gets triggered whenever a new EC2 instance is created, this Lambda function should configure StatusCheck alarm on this new instance automatically. So that I don't have to manually configure cloudwatch alarm each time a new instance is created. Can someone help with code for lambda function that accomplishes this?
I have something like this:
response = client.put_metric_alarm(
AlarmName='StatusCheckFailed-Alarm-for-i-1234567890abcdef0',
AlarmActions=[
'arn:aws:sns:us-west-2:111122223333:my-sns-topic',
],
MetricName='StatusCheckFailed',
Namespace='AWS/EC2',
Statistic='Maximum',
Dimensions=[
{
'Name': 'InstanceId',
'Value': 'i-1234567890abcdef0'
},
],
Period=300,
Unit='Count',
EvaluationPeriods=2,
Threshold=1,
ComparisonOperator='GreaterThanOrEqualToThreshold')
But I have to map instance ID from cloudwatch rule as an input to Lambda. Because the function would trigger automatically so there is no way to put instance ID manually each time.
You will need two cloud watch rule to handle this as
One for instance launch from auto-scaling group
One for instance launch with EC2
Also, I am going to add Launch and Terminatioin
On Launch (add alarm)
On termination (delete alarm) to avoid reaching max limit
Autoscaling group CW rule:
{
"source": [
"aws.autoscaling"
],
"detail-type": [
"EC2 Instance Launch Successful",
"EC2 Instance Terminate Successful"
]
}
Autoscaling Event:
{
"version": "0",
"id": "3e3c153a-8339-4e30-8c35-687ebef853fe",
"detail-type": "EC2 Instance Launch Successful",
"source": "aws.autoscaling",
"account": "123456789012",
"time": "2015-11-11T21:31:47Z",
"region": "us-east-1",
"resources": [
"arn:aws:autoscaling:us-east-1:123456789012:autoScalingGroup:eb56d16b-bbf0-401d-b893-d5978ed4a025:autoScalingGroupName/sampleLuanchSucASG",
"arn:aws:ec2:us-east-1:123456789012:instance/i-b188560f"
],
"detail": {
"StatusCode": "InProgress",
"AutoScalingGroupName": "sampleLuanchSucASG",
"ActivityId": "9cabb81f-42de-417d-8aa7-ce16bf026590",
"Details": {
"Availability Zone": "us-east-1b",
"Subnet ID": "subnet-95bfcebe"
},
"RequestId": "9cabb81f-42de-417d-8aa7-ce16bf026590",
"EndTime": "2015-11-11T21:31:47.208Z",
"EC2InstanceId": "i-b188560f",
"StartTime": "2015-11-11T21:31:13.671Z",
"Cause": "At 2015-11-11T21:31:10Z a user request created an AutoScalingGroup changing the desired capacity from 0 to 1. At 2015-11-11T21:31:11Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1."
}
}
EC2 CW Rule:
{
"source": [
"aws.ec2"
],
"detail-type": [
"EC2 Instance State-change Notification"
],
"detail": {
"state": [
"running",
"terminated"
]
}
}
EC2 Event:
{
"version": "0",
"id": "ee376907-2647-4179-9203-343cfb3017a4",
"detail-type": "EC2 Instance State-change Notification",
"source": "aws.ec2",
"account": "123456789012",
"time": "2015-11-11T21:30:34Z",
"region": "us-east-1",
"resources": [
"arn:aws:ec2:us-east-1:123456789012:instance/i-abcd1111"
],
"detail": {
"instance-id": "i-abcd1111",
"state": "running"
}
}
So you can do rest of the logic base on the event, below example is base on javascript
If the event from the auto-scaling group
if (event["source"] == "aws.autoscaling") {
if (event["detail-type"] === "EC2 Instance Launch Successful"){
let EC2_ID=event.detail.EC2InstanceId
// Add alarm here
// use EC2 instance ID
}
}
Same logic can be applied for EC2 events, where you can check the status
if (event["source"] == "aws.ec2") {
if (event.detail === "running"){
let EC2_ID=event.detail.EC2InstanceId
// Add alarm here
// use EC2 instance ID
}
// same can be check for termination
if (event.detail === "terminated"){
let EC2_ID=event.detail.EC2InstanceId
// remove alarm for this instance
// use EC2 instance ID here to remove/delete alaram
}
}
What you are looking for is AWS CloudTrail. It's a service that is used to monitor any and every API calls made to AWS for a given account.
Pro Tip: AWS is API driven, everything you do, even on the console (UI) is translated into an API call to get the desired result.
The scenario described a very common one and AWS has addressed it in Automating Amazon EC2 with CloudWatch Events - Amazon Elastic Compute Cloud. You can create a CloudTrail event trail for EC2 and configure it to trigger a lambda function. As you have described, this function can then do the necessary configurations.
I use this setup for a similar use case wherein monitoring for disk utilization and memory is configured for any new instance that any user or system spins up. This is just an additional check that makes sure if the correct/recommended AMI is not used, there is some process that goes in and makes sure the monitoring tools are in place.
A note from my experience:
I prefer using S3 in between CloudTrail and Lambda i.e. CloudTrail would write the events to S3 and then the lambda function would be triggered via S3 events. This has the added benefit of persisting the events for later reference. If the data is not sensitive, you can choose to use S3 Lifecycle hooks to delete the data in some time or even use a cheaper storage option to keep the cost down.
Not sure if you were able to get an answer to your question about getting the instance id of the instance. here is how i did it:
def lambda_handler(event, context):
cloudwatchclient = boto3.client('cloudwatch')
eventdata = json.load(event)
thisInstanceID = eventdata['detail']['instance-id']
I am attempting to get a CloudWatch Event to work for a specific CodeBuild project on Build State Change, but it does not seem to take. It works fine if I remove the resource, but then it also triggers on any CodeBuild project. I have something similar working for a specific CodeCommit repository. Am I doing something wrong or is this not implemented for CodeBuild?
{
"detail-type": [
"CodeBuild Build State Change"
],
"source": [
"aws.codebuild"
],
"resources": [
"arn:aws:codebuild:us-east-2:1234567890:build/project-name:*"
]
}
To create CWE rule for specific CodeBuild project, use 'detail.project-name' filter in your CWE rule
{
"source": [
"aws.codebuild"
],
"detail-type": [
"CodeBuild Build State Change"
],
"detail": {
"project-name": [
"project-name"
]
}
}
https://docs.aws.amazon.com/codebuild/latest/userguide/sample-build-notifications.html