EC2 under Auto-Scaling won't scale maybe because due to a memory problem. My autoscaling setup checks the CPU since the memory by default can't be added to the cloudwatch metric. Any recommended ideas for me to have my servers auto scale with the above scenario?
syslogs
You can monitor the memory usage your EC2 instances by installing the CloudWatch Monitoring Scripts and setting them up as cron job to publish detailed metrics about CPU, memory/swap utilisation, and disk space utilisation. (Do it in CloudFormation so you have these scripts set up for every instance). Then you can create a normal CloudWatch alarm based on this data.
Related
I want to monitor the average Disk Utilization across all the EC2 instances inside an auto scaling group.
I know I can monitor CPU Utilization across all instances inside an auto scaling group because there's a CloudWatch metrics for that. However, I need CloudWatch Agent to monitor the Disk Utilization. Is there a good way to monitor this? Or do I need to implement my own automation using Lambda?
CPU Utilization is managed by the Operating System, which runs 'inside' the EC2 instance. AWS has no access to your Operating System and, therefore, no knowledge of your disk utilization. For the same reason, an AWS Lambda function would not be able to access this information either.
Instead, Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent - Amazon CloudWatch. The CloudWatch is probably already installed on the instance and can be configured to collect metrics such as:
disk_total
disk_used
disk_used_percent
mem_available
mem_used
I want to stop my EC2 instances if the memory utilization is more than x% from my Lambda function(python) , is their any possibility to check the memory utilization of an EC2 instance
For EC2 by default it will only have the host level metrics be accessible, this includes CPU, Disk Performance and Network Performance but does not include other metrics such as Memory Utilization.
For this you will need to push a custom metric from the EC2 instance into AWS, this can be performed by installing the CloudWatch Agent.
Once you have the memory metric being pushed into CloudWatch you can create an alarm that will trigger on a specific threshold being exceeded, allowing you to trigger an SNS topic. This can have a Lambda subscribe to the topic to be triggered under the condition.
You need to install the CloudWatch agent on the EC2 instances, if it's not there already. Then the memory usage will be a metric in CloudWatch that your Lambda function can query.
I have multiple services running in my ECS cluster. Each service contains one or more tasks based on CPU utilization or a number of users.
I have deployed these containers with EC2 launch type.
Now, I want to increase/decrease the number of EC2 instances based on available resources in the cluster.
Let's say there are four ECS tasks running in two m5.large instances.
Now, if an ECS service increases the number of tasks and there aren't enough resources available in the cluster, how can I spin up an instance and add to the cluster?
And same goes for vice versa. If there is instance running with no ecs task in it, how can I destroy it automatically?
PS - I was using Fargate. Since it's cost is very high, I moved to EC2 instances.
you need to setup your ecs cluster instances in a ASG as #Nitesh says, second you need to set up a cloudwatch alert based in a key metric, with ecs is complex because you need to set up two autoscaling policies one by service another one to scale up your instances, for ec2 the metric that you could use is Cluster CPU reservation and /or Cluster memory reservation.
The scheme works like this your service increases the number of the desired container by an autoscaling rule using a key metric for your service as could be de CPU usage or the number the request in a load balancer and in consequence the Cluster CPU reservation increase this triggers the cloudwatch alert and your ASG increase the number of instaces.
Some tips scale up fast, and scale down slow this could by handle by setting up the time of the alerts
For the containers use Service Auto Scaling and Target tracking policies for more info see
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html#cluster_reservation
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html
https://aws.amazon.com/blogs/compute/automatic-scaling-with-amazon-ecs/
I hope this help
Regards
I am trying to create a graph for memory usage of an autoscaling group but I discovered that there is no such metric. Although there is Memory usage metric but it is for individual instances. It is useless since instances keep on changing in autoscaling group. I want to know the technical reason why AWS cloudwatch didn't provide it. Moreover I want to know the work around to achieve it.
The metrics that AWS provides can be collected at the hypervisor level. But memory metrics (like disk metrics) is from the OS level. So it is a custom metric that you have to periodically push to CloudWatch.
Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
shows how to push your metrics to CloudWatch. Install the scripts (along with credentials if you are not using IAM role) before creating your AMI and you are set. Each instance in AS will start pushing its memory metric to CloudWatch. Not sure how useful it will be for you.
Suppose I have an EC2 instance, which I understood is a VM instance. So if I enable CloudWatch for this EC2 instance, should this monitoring capabilitiy offered by CloudWatch added into my EC2 instance, or it is just running in the hypervisor like XEN?
Thanks.
CloudWatch monitoring is always enabled by default for every EC2 instance at 5-min granularity. What you can enable is detailed monitoring which means you get 1-min observation granularity and aggregate metrics. Default monitoring at 5-min level is free, but detailed monitoring costs money.
Out-of-the-box CloudWatch metrics are measured at hypervisor level and you do not need to do anything to turn them on. See more info on what metrics are available here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch.html
Things like memory utilization and disk space can't be measured at hypervisor level so CloudWatch distributes a simple package with scripts that can be installed on the instance (Linux or Windows.) Those scripts report the data as custom metrics which also costs money. See http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts.html
It is monitored at the hypervisor layer. Aamazon generally will not look into the instance at VM layer so they can't monitor some feature such as memroy usage at the VM.