I have setup different types of dynamic scaling policies within my autoscaling group that uses a launch template to try and see autoscaling launch new instances when these are triggered, but it doesn't do it. I just set up the following tracking policy, and here are the results:
The tracking policy
Alarm is in-alarm stated, and action was "Successfully executed"
Autoscaling activity is not reporting anything
The alarm always logs that an autoscaling action was triggered, however autoscaling does not log any activity.
This autoscaling group is set up with min: 1, max:6, currently there is only 1 instance running.
Where or how can I find the error that is causing this? Is it perhaps something related to permissions? All instances are healthy/in-service
Spent the past couple of days going through threads on stackoverflow and other forums and haven't found anything that can help me find where the issue is.
Also, the timestamps differ in the screenshots as I had just done a re-deployment, but it is the same behavior all the same before I had done that, there is never activity on the autoscaling instance from the action launched within the alarm...
I tried to replicate your scaling policy and environment:-
Created an ASG
assigned a tracking policy of 0.0001
My max capacity is 2, desired 1 and minimum 1.
The key point here is to wait for Cloudwatch ALarms to collect data from data points, for my scale-out activity it took around 5 minutes (the majority of the time is taken in warm up of instances too )
I just found the reason of this issue, under the autoscaling group, go to Details, then to Advanced configurations and then to Suspended processes, nothing should be selected in this field, in my case alarms was set there along a few other actions, that is the reason autoscaling wasn't running on cloudwatch alarms.
Autoscaling group advanced configurations section
Related
I am working with several np/dev environments. In each there is a minimum of 1 ECS cluster with an AutoScaling group. Each AutoScaling group runs between 2-5 EC2 instances. Currently there is no scheduling, everything up constantly. We want to start shutting down these EC2 instances over the weekend. Under AutoScaling Groups, I am unable to create a scheduled action that would adjust the desired capacity at specified intervals due to role limitations.
What are my options? Would it be possible to create a Lambda that looks for AutoScaling Group tags (ex. WeekendSchedule) and then sets the desired capacity for an interval of time (weekend)?
I have been trying to find a workaround for setting up Automatic Scaling scheduling since I do not have permissions to create a Scheduled Action in the console due to role restraints.
I have an AWS::AutoScaling::AutoScalingGroup configuration that runs two instances of EC2. My question is - is it possible to attach CloudWatch alarms for both instances? For example I want to observe StatusCheckFailed_Instance metric for each EC2 in a group?
Usually you can attach alarms through the EC2 Instance ID but how to know each EC2 Instance ID in AutoScalingGroup to attach alerts? or here should be another way to attach alerts? I really can't find something useful and workable over internet.
Option 1)
Create your own script that's triggered on launch/terminate events
the scripts will each be set to trigger a lambda that would read in the instance ID and create/delete an alarm
Option 2)
If you're not trying to use the auto-recover option (which you shouldn't need in an ASG, since the ASG will just replace the instances), then you can make 1 aggregate alarm for the ASG
Create the alarm based on the StatusCheckFailed_Instance metric with the ASGName=<> Dimension
Set it to trigger if the MAX statistic value is > 1 (since that means at least 1 instance is failing, each instance will push its own datapoints to ASG versions of EC2 metrics)
Since you only have 2 instances, you can just manually check both if it ever triggers. But for larger ASGs using the SEARCH() math expression on the CloudWatch metrics console (or a dashboard) would be a good way to look through all the ASG instances and view their metrics to see which one is failing
I am surprised to find little information regarding EC2 autoscaling with Phusion Passenger.
I actually discovered not so long ago a metric "Requests in queue" being exposed upon running passenger-status
I am wondering whether this stat would make a nice metric to help with autoscaling.
Right now most AWS EC2 Autoscaling guides mention using CPU and Memory to write autoscaling rules but I find this insufficient. When I think about the problem autoscaling should solve, that is being able to scale up to the demand, I'd rather base those rules on the number of pending/completed requests to report a node health or a cluster congestion, and Passenger "Requests in queue" (and also for each process, the "Last Used" and "Processed" count) seems to useful.
I am wondering it it would be possible to report this "Requests in queue" stat (and eventually others) periodically as an AWS metric. I was thinking the following rule would be ideal for autoscaling : If the average number of "requests in queue" on the autoscaled instances is to exceed a threshold value, this would trigger spawning a new machine from the autoscaling group.
Is this possible ?
Has anyone ever tried to implement autoscaling rules based on number of requests in queue this way ?
This is totally possible (and a good approach).
Step 1. Create custom CloudWatch metric for "Requests in queue".
You will have to write your own agent that runs passenger-status, extracts the value and sends it to CloudWatch. You can use any AWS SDK or just AWS CLI: http://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-data.html
Step 2. Create alarms for scale up and scale down based on your custom metric.
Step 3. Modify scaling policy for your Auto Scaling Group to use your custom alarms to scale up/down.
We have configured an Auto Scaling group in AWS. And it works fine. We have configured some alarms for the group, such as: send alarm if the average CPUUtilization > 60 for 2 minutes ... use AWS CLI.
The only problem is, if we want to monitoring each instance in the group. We have to configure them manually. Are they any way to do it automatically like config, template?
Amazon CloudWatch alarms can be created on the Auto Scaling group as a whole, such as Average CPUUtilization. This is because alarms are used to tell Auto Scaling when to add/remove instances and such decisions would be based upon the group as a whole. For example, if one machine is 100% busy but another is 0% busy, then on average the group is only 50% busy.
There should be no reason for placing an alarm on the individual instances in an auto-scaling group, at least as far as triggering a scaling action.
There is no in-built capability to specify an alarm that will be applied against each auto-scaled instance individually. You could do it programmatically by responding to an Amazon SNS notification whenever an instance is added/removed by Auto Scaling, but this would require your own code to be written.
You can accomplish this with lifecycle hooks and a little lambda glue. When you have lifecycle events for adding or terminating an instance, you can create an alarm on that individual instance or remove it (depending on the event) via a lambda function.
To John's point, this is a little bit of an anti-pattern with horizontal scaling and load balancing. However, theory and practice sometimes diverge.
I'm wanting to apply a CloudWatch alarm to stop instances which aren't being used in our pre-production environment. We often have instances being spun up, used and then left turned on which is really starting to cost us a fair amount of money.
CloudWatch alarms have a handy feature whereby we can stop based on some metrics - this is awesome and what I'd like to use to constantly keep an eye on the servers with but let it tidy up the instances for me.
The problem with this is that it appears that the CloudWatch alarms need to be created individually against each instance. Is there a way in which I can create one alarm which would share values across all current and future instances which will be started?
ETA - Alternatively, tell me that these options are better than CloudWatch and I'll be happy at that.
AWS EC2 stop all through PowerShell/CMD tools
Add a startup script that creates the CloudWatch alarm to the base image you use to generate your VMs.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CLIReference.html
I don't believe this is possible - CloudWatch seems designed to be 'very manual' or 'very automated'. i.e. You can't setup one alarm which would go off if any one instance is idle, you have to setup individual alarms for each instance.
A couple of possible solutions, which are probably not what you want to hear:
Script your instance creation, and add a call to cloudwatch to create an alarm for each instance.
Run a service continually, which looks for instances and checks to ensure that there is an alarm for the instance, create alarms for the new instances, and remove alarms for instances which have been terminated.
I think what you are actually looking for would be auto-scaling:
https://aws.amazon.com/documentation/autoscaling/