I need to create an Alarm in AWS which notifies me when my Storage used >= 80%
AWS has no visibility inside your Amazon EC2 instance. This is because the instance is run by the Operating System and AWS does not have a login to the instance.
However, you can Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent - Amazon CloudWatch, which is a piece of software you install on the instance. It then runs inside the instance and sends metrics (such as available disk space) to Amazon CloudWatch. You can then create an Alarm on that metric to receive notification when the disk spaces metric passes a threshold.
Related
I am trying to build an analytics Dashboard using the below Metrics/KPIs for all the EC2 Instance.
Total CPU vs CPUUtilized
Total RAM vs RAMUtilized
Total EBS Volume vs EBSUtilized.
For example, I have lunch an EC2 instance with 4 CPU, 16GiB RAM and 50GB SSD, I would like to know the above KPIs in a time series trend. I am not getting any clue on where to get the data from EC2. Tried the EC2 instance metrics through CloudWatch using boto3 client, however did not get the above Metrics. I would like to know :
Where to find the data with above Metrics ?
Need the above metrics data in s3 on an daily basis.
Similarly is there a way to get similar metrics for AWS RDS and AWS EKS Cluster ?
Thanks!
The Amazon EC2 service collects information about the virtual machine (instance) and sends it to Amazon CloudWatch Logs.
See: List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud
Note that it only collects metrics that can be observed from the virtual machine itself -- CPU Utilization, network traffic and Amazon EBS traffic. The EC2 service cannot see what is happening 'inside' the instance, since it is the Operating System that controls memory and manages the contents of the disks.
If you wish to collect metrics from the Operating System, then you would need to Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent - Amazon CloudWatch. This agent runs in the instance and sends metrics out to CloudWatch.
You can write code that calls the CloudWatch Metrics APIs to retrieve metrics. Note that the metrics returned are calculated over a time period (eg average CPU Utilization over a 5-minute period). It is not possible to retrieve the actual raw datapoints.
See also:
Monitoring Amazon RDS metrics with Amazon CloudWatch - Amazon Relational Database Service
Amazon EKS and Kubernetes Container Insights metrics - Amazon CloudWatch
I want to stop my EC2 instances if the memory utilization is more than x% from my Lambda function(python) , is their any possibility to check the memory utilization of an EC2 instance
For EC2 by default it will only have the host level metrics be accessible, this includes CPU, Disk Performance and Network Performance but does not include other metrics such as Memory Utilization.
For this you will need to push a custom metric from the EC2 instance into AWS, this can be performed by installing the CloudWatch Agent.
Once you have the memory metric being pushed into CloudWatch you can create an alarm that will trigger on a specific threshold being exceeded, allowing you to trigger an SNS topic. This can have a Lambda subscribe to the topic to be triggered under the condition.
You need to install the CloudWatch agent on the EC2 instances, if it's not there already. Then the memory usage will be a metric in CloudWatch that your Lambda function can query.
There is an redis instance been created in ElasticCache and this will be used to store and retrieve data as usual.
Is there any max memory for this redis instance and how can that be checked?
All I need is say example if the data size in redis reaches above 100 mb then it should be auto scaled without me having to manually scale it or create a new instance and things like that.
And when the data size is reduced(example: From 300mb to 50 mb due to less traffic) then the instances should be reduced so that there is no extra cost incured.
How can this be configured in AWS ElastiCache?
unfortunately there is no auto-scaling policy attach with Elasticcache out of the box, amazon ElastiCache provides console, CLI, and API support for scaling your Redis (cluster mode disabled) replication group up.
One option that you can try is to set cloud watch alarm base on node memory and then trigger lambda function that will scale up and down base on metrics.
Create a CW alarm
Select Elastic cache metrics
Select Node level metrics
Select Free memory metrics
Trigger notification to SNS topic
Subscribe lambda function
scaleup/scaledown base on metrics
Now Elasticache supports autoscaling
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoScaling.html
I have an application running in EC2 that listen to many ports, some external devices connect to those ports to send data to my application. This is fine, but my client has a requirement that i must monitoring those ports and if one of them stop listening, the instance must be terminated and a new one started.
I was reading about couldwatch, but i didn't found an alarm that i can customize like this (doing requests to ports). Is it possible to do this using cloudwatch ? i'm looking for a direction to create this monitoring, using internal aws services or develop a new solution (maybe a sheel script).
thanks!
I'm not aware of any AWS provided EC2 healthcheck monitoring system for custom checks.
You could write an AWS lambda function which sends requests to the ports on the EC2 instance you require. You can then schedule that lambda to run periodically with whatever frequency you want with Cloudwatch Events. The lambda function could publish this as a metric to cloudwatch which would then make it possible for you to use it in an alarm and thus take action when whatever threshold you deem reasonable to spin up a new replacement instance.
One part of AWS that does have basically what you are looking for built-in though is ECS. Instead of an EC2 instance, you'd have a Docker instance (running on an EC2 instance or Fargate) which can have healthchecks defined.
There are many ways to do what you are asking for.
Simplest solution: I will write a boto3/shell script to monitor the port and call TerminateInstance API or use AWS CLI to terminate the current instance. Needless to say, you need to pass AWS credentials or attach instance profile with sufficient privileges to terminate the instance.
Using Cloudwatch: Have a script to check port status and send 1 or 0 (Dimension: Count) to Cloudwatch. Set a threshold in Cloudwatch if there is consecutive 0s or NoData, then terminate the instance. Or do not send any data to Cloudwatch if the port is not available and NoData in Cloudwatch can trigger TerminateInstance. See: Cloudwatch - AddingTerminateActions
Just looking the way to start/stop a AWS EC2 instance in case of CPU utilization increase or decrease on another EC2 instacne. I know there is service available Auto Scaling in AWS but I have a scenario where I can't take advantage of this service.
So just looking if it is possible or anyone can help me on this.
Just detailing the concern like suppose I have 2 EC2 instance on AWS account by name EC21 and EC22. By default, EC22 instance is stopped.
Now I need to setup CloudWatch or any other service to check if load/CPU utilization increase on EC21 instance by 70% then need to start EC22 server and similarly if load decrease on EC21 instance by 30% then stop EC22 server.
Please advice!
When your CloudWatch alarm is triggered, it will notify an SNS topic. You can have that SNS topic then invoke a Lambda function, which can then start your EC2 instance.
Create an AWS Lambda function that starts your EC2 instance.
Configure your SNS topic to invoke your Lambda function when it receives messages. You can read about that here: Invoking Lambda functions using Amazon SNS notifications
Finally, ensure your CloudWatch alert sends messages to the SNS topic.
Yes this is possible for certain types of EC2 instances. Check this detailed guide using which you can set up the triggers in your EC2 instances based on AWS Cloud Watch metrics.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/UsingAlarmActions.html
I think your problem might fit the scenario which I'm also trying to solve now - I have some functionality which cannot be solved with Lambdas because of their low lifetime, so I need a relatively short-lived EC2 instance to accomplish the task.
The solution is similar to the one described by Matt, but without SNS, using AWS triggers to launch a lambda function to start the instance. Added benefit is that the lambda function can itself verify whether the EC2 start is really needed.
How do I stop and start Amazon EC2 instances at regular intervals using AWS Lambda?
Issue
I want to reduce my Amazon Elastic Cloud Compute (Amazon EC2) usage by
stopping and starting instances at predefined times or utilization
thresholds. Can I configure AWS Lambda and Amazon CloudWatch to help
me do that automatically?
Short Description
You can use a CloudWatch Event to trigger a Lambda function to start
and stop your EC2 instances at scheduled intervals.
Source: AWS Knowledge Center