I want to stop my EC2 instances if the memory utilization is more than x% from my Lambda function(python) , is their any possibility to check the memory utilization of an EC2 instance
For EC2 by default it will only have the host level metrics be accessible, this includes CPU, Disk Performance and Network Performance but does not include other metrics such as Memory Utilization.
For this you will need to push a custom metric from the EC2 instance into AWS, this can be performed by installing the CloudWatch Agent.
Once you have the memory metric being pushed into CloudWatch you can create an alarm that will trigger on a specific threshold being exceeded, allowing you to trigger an SNS topic. This can have a Lambda subscribe to the topic to be triggered under the condition.
You need to install the CloudWatch agent on the EC2 instances, if it's not there already. Then the memory usage will be a metric in CloudWatch that your Lambda function can query.
Related
I want to monitor the average Disk Utilization across all the EC2 instances inside an auto scaling group.
I know I can monitor CPU Utilization across all instances inside an auto scaling group because there's a CloudWatch metrics for that. However, I need CloudWatch Agent to monitor the Disk Utilization. Is there a good way to monitor this? Or do I need to implement my own automation using Lambda?
CPU Utilization is managed by the Operating System, which runs 'inside' the EC2 instance. AWS has no access to your Operating System and, therefore, no knowledge of your disk utilization. For the same reason, an AWS Lambda function would not be able to access this information either.
Instead, Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent - Amazon CloudWatch. The CloudWatch is probably already installed on the instance and can be configured to collect metrics such as:
disk_total
disk_used
disk_used_percent
mem_available
mem_used
I need to create an Alarm in AWS which notifies me when my Storage used >= 80%
AWS has no visibility inside your Amazon EC2 instance. This is because the instance is run by the Operating System and AWS does not have a login to the instance.
However, you can Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent - Amazon CloudWatch, which is a piece of software you install on the instance. It then runs inside the instance and sends metrics (such as available disk space) to Amazon CloudWatch. You can then create an Alarm on that metric to receive notification when the disk spaces metric passes a threshold.
There is an redis instance been created in ElasticCache and this will be used to store and retrieve data as usual.
Is there any max memory for this redis instance and how can that be checked?
All I need is say example if the data size in redis reaches above 100 mb then it should be auto scaled without me having to manually scale it or create a new instance and things like that.
And when the data size is reduced(example: From 300mb to 50 mb due to less traffic) then the instances should be reduced so that there is no extra cost incured.
How can this be configured in AWS ElastiCache?
unfortunately there is no auto-scaling policy attach with Elasticcache out of the box, amazon ElastiCache provides console, CLI, and API support for scaling your Redis (cluster mode disabled) replication group up.
One option that you can try is to set cloud watch alarm base on node memory and then trigger lambda function that will scale up and down base on metrics.
Create a CW alarm
Select Elastic cache metrics
Select Node level metrics
Select Free memory metrics
Trigger notification to SNS topic
Subscribe lambda function
scaleup/scaledown base on metrics
Now Elasticache supports autoscaling
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoScaling.html
EC2 under Auto-Scaling won't scale maybe because due to a memory problem. My autoscaling setup checks the CPU since the memory by default can't be added to the cloudwatch metric. Any recommended ideas for me to have my servers auto scale with the above scenario?
syslogs
You can monitor the memory usage your EC2 instances by installing the CloudWatch Monitoring Scripts and setting them up as cron job to publish detailed metrics about CPU, memory/swap utilisation, and disk space utilisation. (Do it in CloudFormation so you have these scripts set up for every instance). Then you can create a normal CloudWatch alarm based on this data.
I am trying to create a graph for memory usage of an autoscaling group but I discovered that there is no such metric. Although there is Memory usage metric but it is for individual instances. It is useless since instances keep on changing in autoscaling group. I want to know the technical reason why AWS cloudwatch didn't provide it. Moreover I want to know the work around to achieve it.
The metrics that AWS provides can be collected at the hypervisor level. But memory metrics (like disk metrics) is from the OS level. So it is a custom metric that you have to periodically push to CloudWatch.
Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
shows how to push your metrics to CloudWatch. Install the scripts (along with credentials if you are not using IAM role) before creating your AMI and you are set. Each instance in AS will start pushing its memory metric to CloudWatch. Not sure how useful it will be for you.