How do I get the InstanceId of all instances of Cloudwatch Alarm, I am trying to create a cloudwatch alarm to send email if the disk reach 90% usage.
Resources:
EC2DiskHealth:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: { "Fn::Join" : ["", [{ "Ref" : "AWSEBEnvironmentName" }, ": Disk Usage" ]]}
Namespace: System/Linux
MetricName: DiskSpaceAvailable
Dimensions:
- Name: InstanceId
Value : { "Ref" : "instance-id" }
- Name: Filesystem
Value: /dev/xvda1
- Name: MountPath
Value: /
Statistic: Average
Period: 60
EvaluationPeriods: 5
Threshold:
Fn::GetOptionSetting:
OptionName: ELBHealth
DefaultValue: "90"
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- arn:aws:sns:awsregion:sns
InsufficientDataActions:
- arn:aws:sns:awsregion:sns
OKActions:
- arn:aws:sns:awsregion:sns
Output:
I should be able to get the instance-id in order for the alarm to work.
Dimensions:
- Name: InstanceId
Value : { "Ref" : "instance-id" }
Error:
Service:AmazonCloudFormation, Message:Template format error: Unresolved resource dependencies [instance-id] in the Resources block of the template
It appears that your situation is:
You have some existing Amazon EC2 instances
You are running some script/code on the instances that send a metric called DiskSpaceAvailable at regular intervals to Amazon CloudWatch
You wish to create a CloudFormation template
The template should create an Alarm for every EC2 instance when DiskSpaceAvailable exceeds a certain metric
This is not possible.
An Amazon CloudWatch template can create resources and can refer to resources, but it cannot go out and discover resources, nor perform loops over discovered resources.
A template could, for example, create an instance and then add an alarm specifically for that instance. However, it won't auto-discover resources.
You can write an AWS Lambda-backed Custom Resource that can do whatever you wish (you'd have to write the code), but your code would need to create the alarms rather than CloudFormation.
Bottom line: Your use-case is best done via your own code (Lambda or just straight code) rather than using CloudFormation.
Your question is to send an email when cloudwatch detect that the disk instance is over 90% used.
It is the basics of cloudwatch task : create the email notification in the cloudwatch alarm itself, set the emails and save.
more details here (it is an example related to CPU, but it is the same principle) :
https://docs.aws.amazon.com/fr_fr/AmazonCloudWatch/latest/monitoring/US_AlarmAtThresholdEC2.html
If you want your instance-id from the instance itself, its instance-id is available through its metadata :
curl http://169.254.169.254/latest/meta-data/instance-id
Related
I am setting up an AWS EC2 template based on a custom image for launching instances for a certain purpose. These instances then also need CloudWatch alarms monitoring their activity and perform some action based on them (e.g. stop instance if inactive for 30 min.).
Is there any way I can include such alarms into the EC2 template? I would like to avoid having to manually add the alarms to the instance after creation. I couldn't find this as an option anywhere in the template creation dialogue.
From management console - could not find a straight forward option.
Using EC2 Tags, Lambda and other services - might be possible - check the link
CloudFormation - you can write a CF template to create EC2 and add an alarm to it. You can continue enhancing it.
This option will make things easier once the template is created as you will not need to select various UI options whenever you launch new EC2 and add alarm.
This template will ask for instance type, will create an alarm for EC2 and publish to an SNS topic.
Verify AMI, AZ if you are logged into a different region.
Parameters:
InstanceType:
Description: EC2 instance type
Type: String
Default: t2.small
AllowedValues:
- t1.micro
- t2.nano
- t2.micro
- t2.small
ConstraintDescription: It must be a valid EC2 instance type.
Resources:
MyInstance1:
Type: AWS::EC2::Instance
Properties:
AvailabilityZone: us-east-1a
ImageId: ami-05912b6333beaa478
InstanceType: !Ref InstanceType
KeyName: KP-EC2-Lambda
SecurityGroups:
- launch-wizard-2
CPUAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: CPU alarm for my instance
AlarmActions:
- Ref: "MyTopic1"
MetricName: CPUUtilization
Namespace: AWS/EC2
Statistic: Average
Period: '60'
EvaluationPeriods: '3'
Threshold: '90'
ComparisonOperator: GreaterThanThreshold
Dimensions:
- Name: InstanceId
Value:
Ref: "MyInstance1"
MyTopic1:
Type: AWS::SNS::Topic
Properties:
DisplayName: MyTopic1
Subscription:
- Endpoint: "xyz#xyz.com"
Protocol: "email"
TopicName: MyTopic1
So i already have an existing redshift cluster running which I created with cloudformation, Now I need to add a new cloudwatch event to this cluster like below code, How do i map the new alarm with existing cluster.
This is for existing AWS Redshift cluster
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: !Join [ " ", [ "Health status alarm for", !Ref RedshiftCluster, "Redshift Cluster"]]
AlarmActions:
- !Ref redshiftClusterSNSTopic
MetricName: HealthStatus
Namespace: AWS/Redshift
Statistic: Average
Period: 300
EvaluationPeriods: 3
Threshold: 1
ComparisonOperator: LessThanThreshold
Dimensions:
- Name: ClusterIdentifier
Value: !Ref CARedshiftCluster
Not sure how to do this, help is appreciated.
You can give cloudkast a try. It is an online cloudformation template generator. It is regularly updated. As of now it does support cloudwatch.
Recently AWS announced that Cloudwatch alarms can use Math Expressions on metrics. I decided to create an alarm that compares the SUM of 2 single metrics with a given threshold. This means that according to AWS documentation my expression should be SUM([m1,m2]), where m1 and m2 are 2 single metrics.
I also decided to implement this idea using a cloudformation template (in yaml). Here's the Cloudwatch alarm definition:
BillingAlarmExpression:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmActions:
- !Ref BillingAlertTopic
AlarmDescription: String
ComparisonOperator: GreaterThanOrEqualToThreshold
EvaluationPeriods: 1
Metrics:
- Id: m1
MetricStat:
Metric:
Dimensions:
- Name: ServiceName
Value: AmazonEC2
- Name: Currency
Value: USD
MetricName: EstimatedCharges
Namespace: AWS/Billing
Period: 86400
Stat: Maximum
ReturnData: False
- Id: m2
MetricStat:
Metric:
Dimensions:
- Name: ServiceName
Value: AmazonCloudwatch
- Name: Currency
Value: USD
MetricName: EstimatedCharges
Namespace: AWS/Billing
Period: 86400
Stat: Maximum
ReturnData: False
- Id: Expr1
Expression: SUM([m1,m2])
Label: Yeap
Threshold: 100
TreatMissingData: ignore
The single metrics , m1 and m2 , have to do with the billing cost of EC2 and Cloudwatch service. What I want to check is if the charging cost for these 2 services has crossed the threshold of 100$. (Note that since the billing costs are exclusively stored in the N.Virginia region, I tried to deploy the above template in N.Virginia).
During the deployment of this template Cloudformation responds with the following error:
"Invalid metrics list (Service: AmazonCloudWatch; Status Code: 400; Error Code: ValidationError; Request ID: c0748047-0378-11e9-ac36-5b1829988d18)"
When Cloudformation says "metrics list" it refers to the definition of m1,m2, Expr1. What is even more strange is that when I use the above metrics-list definition from aws cli the charging data return successfully:
aws cloudwatch get-metric-data --metric-data-queries file://./metric-data.json --start-time 2018-12-03T03:00:00Z --end-time 2018-12-10T04:30:00Z
,where metric-data.json is the above metrics-list.
For creating my template I used the following guides:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-alarm.html
and
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html
Do you have any idea why Cloudformation returns this error? Thanks!
Id must start with a lowercase letter, change Expr1 to expr1.
From docs:
You can change the value of Id. It can include numbers, letters, and underscore, and must start with a lowercase letter.
I also ran into this issue -- in the off chance that your issue is not the accepted answer (id is using lower case), these things are also worth looking out for that can cause the error
Make sure you're following the proper shape of what a MetricDataQuery looks like and take note which properties are actually optional or not
Make sure the math expression you're writing is a String (surrounded by quotes)
Make sure you're using proper MetricStat value (using Max instead of Maximum can throw this error)
I really hope AWS will look to add better error messaging around this. Invalid Metrics list was not very helpful :/
I'm implementing ECS health-check functionality and and I'm thinking about the best way to do that.
For now I have found several solutions:
Using AWS ECS metrics and Dimensions and check whether some metric has insufficient value
Using CloudWatch Alarm:
ECSHealthAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: Alarm for ECS StatusCheckFailed Metric
ComparisonOperator: GreaterThanOrEqualToThreshold
EvaluationPeriods: 2
Statistic: Maximum
MetricName: StatusCheckFailed
Namespace: AWS/ECS
Period: 30
Threshold: 1.0
AlarmActions:
- !Ref AlarmTopic
InsufficientDataActions:
- !Ref AlarmTopic
Dimensions:
- Name: ClusterName
Value: !Ref ClusterName
- Name: ServiceName
Value: !GetAtt service.Name
Using CloudWatch event:
EventRule:
Type: "AWS::Events::Rule"
Properties:
Name: CloudWatchRMExtensionECSStoppedRule
Description: "Notify when ECS container stopped"
EventPattern:
source: ["aws.ecs"]
detail-type: ["ECS Task State Change", "ECS Container Instance State Change"]
detail:
clusterArn: [ 'clusterArn' ]
lastStatus: [ "STOPPED" ]
stoppedReason: [ "Essential container in task exited" ]
group: [ 'service-group' ]
State: "ENABLED"
Targets:
- Arn: !Ref ECSAlarmSNSTopic
Id: "PublishAlarmTopic"
InputTransformer:
InputPathsMap:
stopped-reason: "$.detail.stoppedReason"
InputTemplate: '"This micro-service has been stopped with the following reason: <stopped-reason>"'
Could you please advice whether those variants are correct or there is ant other way to do that more efficient? Thanks for any help!
I am not able to put comment, so here are some thoughts. I am bit unclear of your requirement, whether you are looking for alerts from EC2 server level status check or each ECS service tasks level. I am adding all the possible options here.
I would run ECS cluster EC2 instances under an Auto-Scaling Group and based on ASG CloudWatch metrics, setup a SNS notification when instances are being added/removed.
https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html
We can have AWS ecs-agent docker container logs also sent to CloudWatch and get some SNS notifications based on errors or filtered events.
We can have subscription to CW from ECS event stream as well when each service tasks being started/stopped. References - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch_event_stream.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwet.html
Example event entries are in below link – https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html
Reference for setting alarm based on log events.
https://medium.com/#martatatiana/insufficient-data-cloudwatch-alarm-based-on-custom-metric-filter-4e41c1f82050
Adding healthcheck for each ECS service wise and have containers restarted if they are not doing well.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_healthcheck
Please do let me know your thoughts as well :).
I am looking to create a Spot Fleet in Cloudformation which runs a single game server at a time; if prices spike and the server needs to be terminated it will use the 2 minute heads-up to gracefully shutdown and store anything to be persisted on an EBS Volume. The next instance started by the fleet will then mount the volume and restart the game server from where the previous one left off.
SpotFleet:
Type: "AWS::EC2::SpotFleet"
Properties:
SpotFleetRequestConfigData:
IamFleetRole: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-ec2-spot-fleet-tagging-role
TargetCapacity: 1
LaunchSpecifications:
- InstanceType: "m5.large"
ImageId: "ami-abcd1234"
IamInstanceProfile: !GetAtt InstanceProfile.Arn
WeightedCapacity: 1
Now I'm stuck on defining the persisted volume in the cf template. Initially I would just add it as a resource:
Volume:
Type: "AWS::EC2::Volume"
Properties:
Size: 10
AvailabilityZone: !Ref AWS::Region
But then how do I reference it in the fleet? You can define BlockDeviceMappings on LaunchSpeficiations within a fleet as per
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-spotfleet-spotfleetrequestconfigdata-launchspecifications-blockdevicemappings.html
but from the attributes available I can't seem to reference existing volumes and as such am getting the idea that these volumes are not persisted.
Alternatively I thought of attaching the volume to the spot instance via a VolumeAttachment:
VolumeAttachment:
Type: "AWS::EC2::VolumeAttachment"
Properties:
Device: "dev/server"
InstanceId: !Ref SpotFleet
VolumeId: !Ref Volume
but obviously the SpotFleet reference here returns the fleet name, not the id of any created instances. And neither !Ref nor !GetAtt seem to be able to extract those ids from a fleet.
Am I overlooking anything cruicial as to how to accomplish the above in CloudFormation or should I be looking at adding the EC2:AttachVolume and EC2:DetachVolume permissions to the InstanceProfile and simply attaching the volume manually from within the EC2 instance?
Many thanks,
EC2 Spot instances now support the option of setting the "Interruption behavior" to stop instead of terminate.
When this option is selected, a spot instance retains its instance ID, EBS volumes, its private and elastic IP address, and its EBS volumes, which remain in place and attached.
Some instance types also support a "hybernate" option that writes a snapshot of the entire system state to EBS to allow the instance to "resume" rather than reboot when capacity becomes available again.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
What you are looking for is the BlockDeviceMappings property, found in the SpotFleet SpotFleetRequestConfigData LaunchSpecifications, which is a property of SpotFleetRequestConfigData, which is a property of the AWS::EC2::SpotFleet resource type.
The BlockDeviceMappings property will allow you to define additional EBS volumes to attach to your launch specification. This is the specification that controls device mappings at launch time.
For exmample:
"BlockDeviceMappings" : [{
"DeviceName" : "/dev/sdf",
"Ebs" : {"VolumeSize": "10", "VolumeType" : "gp2", "DeleteOnTermination" : "true"}
}],
will specify a 10GB volume on the /dev/sdf device of your spot fleet instance.