I configured for my AWS account the new AWS instance scheduler https://aws.amazon.com/answers/infrastructure-management/instance-scheduler/
The problem seems that, tagging ec2-instances through a scaling group the ec2-instances are correctly stopped, but since my scaling group has Min number set to 2 AWS scaling group restarts them anyway.
I would not like to set Min number to 0, just because is useful during application redeploy.
How to make the 2 services work fine?
When you stop your EC2 instances that are controlled by Auto Scaling, then Auto Scaling will see them as "unhealthy" and it will proceed to terminate and replace them.
You have 2 options.
Option 1: Pause Auto Scaling processing while your EC2 instances are stopped. By doing this, Auto Scaling won't care that your EC2 instances are stopped and won't terminate them. Just remember to resume processing after you restart your EC2 instances.
However, AWS Instance Scheduler will not manage this for you, so you'll need to find another way to schedule your EC2 instances to stop & restart.
Option 2: Scale your Auto Scaling group to 0 and back to 2. This will result in terminating your EC2 instances (when you don't need them) and re-creating them (when you want them). This will only work if your EC2 instances are ephemeral.
Again, AWS Instance Scheduler will not manage this for you. Auto Scaling scheduled actions may be able to help you with this.
Another option is to use asg standby feature before and after the aws instance scheduler. This will also let you work on the same Ami before the shutdown.
So high level solution is below:
Define ec2 instance schedule using aws instance scheduler
Define lambda that fetch the shutdown schedule and put the ec2 in standby mode before the planned shutdown.
Define lambda that fetch the startup schedule and put the ec2 instance out of standby after the ec2 planned restart.
Related
I am running a flask python program on EC2 which is under Load Balancer and autoscaling. In a scenario where is load increases on one Ec2 it creates another and if newly scaled Ec2 has been idle or not utilized it scales in or terminates it. The problem here is if a single user is accessing newly scaled instance which hardly takes any CPU utilization how autoscaling group will realize that it idle and if it doesn't it will terminate it leaving downtime for that user.
I have two scenarios in mind that it checks for a particular program for a amount of time in EC2 if it is running then don't, otherwise terminate it.
I see Step scaling policy but there option is only for CPU utilization that is hardly consumed if there is a single user, not even 0.1 %.
Can someone please tell me whats the best option for me and if these two options are possible then how to do it? I have been trying to ask developers since many days but could not get reliable answers in my case.
Amazon EC2 Auto-scaling does not know which of your instances are 'in use'.
Also, the decision to terminate an instance is typically made on a metric across all instances (eg CPU Utilization), rather than a metric on a specific instance.
When Auto Scaling decides to remove an instance from the Auto Scaling group, it picks an instance as follows:
It picks an Availability Zone with the most instances (to keep them balanced)
It then selects an instance based on the Termination Policy
See also: Control which Auto Scaling instances terminate during scale in - Amazon EC2 Auto Scaling
When using a Load Balancer with Auto Scaling, traffic going to the instance that will be terminated is 'drained', allowing a chance for the instance to complete existing requests.
You can further prevent an instance from terminating while it is still "in use"by implementing Amazon EC2 Auto Scaling lifecycle hooks that allow your own code to delay the Termination.
Or, if all of this is unsatisfactory, you can disable the automatic selection of an instance to terminate and instance have your own code call TerminateInstanceInAutoScalingGroup - Amazon EC2 Auto Scaling to terminate a specific instance of your choosing.
For an overview of Auto Scaling, I recommend this video from the AWS Reinvent conference: AWS re:Invent 2019: Capacity management made easy with Amazon EC2 Auto Scaling (CMP326-R1) - YouTube
I have created ec2 instance and it's part of the auto scaling group and i want to stop this instance so i can save money while this server being not used.Whenever i want to use this server that time i can start the instance.
AutoScaling Group is able to take care of this process for you.
You can change the size of the group manually, or you could schedule custom configurations to scale-up or scale-down as per your needs.
This is all supported through AWS APIs, so you can do this through AWS Console or automating the configuration.
Instances in ASG can't be stopped. If you want to stop one instance you have to detach it from ASG. When you are ready to start it, you can attach it again.
But really, you should reconsider how you use ASG. Detaching and re-attaching instances from ASG is meant for mostly troubleshooting purposes, not something you do regulatory just to save cost. Its better to set minimum and desired capacity of ASG to 0 if you don't want any instance in it.
I have an a scaling group of 2-5 instances to handle web traffic. I'm using a the rpush gem for push notifications, which requires a single daemon running to execute all the awaiting jobs. I'm already paying for the 2-5 instances, which have sufficient extra computing power to handle running the daemon, and I'd like to run the daemon on one of these instances.
The problem is, I can only use 1 API per auto-scaling group, so I'm having trouble finding a way to run the daemon on only one of the instances in the auto-scale group.
Is there a way to do this?
You could start your daemon manually on one of the instances and mark it as protected from termination. This way it won't be terminated during scaling in. While scaling out, the default new instances will be created without the deamon.
Keep in mind that while protected from termination in the auto-scaling
group, it may still be terminated by:
Manual termination through the Amazon EC2 console, the
terminate-instances command, or the TerminateInstances action. To
protect Auto Scaling instances from manual termination, enable
termination protection. For more information, see Enabling Termination
Protection in the Amazon EC2 User Guide for Linux Instances.
Health check replacement if the instance fails health checks.
Spot Instance interruption.
(source: AWS docs)
You can create new Launch Configuration (updating AMI or whatever) and attach this with an existing Autoscaling Group. Per AWS Docs: After you change the launch configuration for an Auto Scaling group, any new instances are launched using the new configuration options, but existing instances are not affected.
How do you force this? Meaning relaunch all new instances now (with the new AMI). Do I have to delete the existing Autoscaling Group and create a new Autoscaling Group (with new Config)? Or I simple delete existing instances (one by one manually) and then ASG relaunch with new AMI. Any best practices/gotchas?
CloudFormation has the RollingUpdate flag (not sure of this outside of CF)
Thanks
AWS has some OOTB solutions for this, CloudFormation (like you say), Elastic Beanstalk (built on top of CF), and CodeDeploy blue-green deployments (I've not tried this).
Personally for our SQS polling ASG, we do a "cold deploy" i.e. only "deploy" when there are no messages to process (and hence, due a scaling policy, no instances). It's been really effective.
A deploy can be done safely whilst there are messages, provided that you set scale-in-protection on the instance during message processing (and remove it and wait briefly before polling):
set desired-capacity to 0
wait a bit (for there to be no instances running)
set desired-capacity back to N.
Note: you can do this all in the console.
You can code a solution yourself that does this... but I probably wouldn't.
Be careful:
simple delete existing instances (one by one manually)
Whether you can do this, or depends on whether the instances are still handling requests/processing (usually you can't simply terminate an instance without dropping service).
I recommend Elastic Beanstalk which gives a rolling update feature for free and is very easy to get started. I've not tried the CodeDeploy blue-green but it looks interesting. If you want more advanced behavior (or are already using it) look into Cloud Formation... do not code your own solution for rolling deployments: just use CloudFormation.
if your issue is with "in flight" requests simply enable connection draining or increase de-registration delay of the ELB or "target groups" attached with the ASG. You can set a value up to one hour.
When you enable connection draining, you can specify a maximum time for the load balancer to keep connections alive before reporting the instance as de-registered. The maximum timeout value can be set between 1 and 3,600 seconds (the default is 300 seconds). When the maximum time limit is reached, the load balancer forcibly closes connections to the de-registering instance.
Then you can detached old instances.
If you detach an instance from an Auto Scaling group that has an attached load balancer, the instance is deregistered from the load balancer. If you detach an instance from an Auto Scaling group that has an attached target group, the instance is deregistered from the target group. If connection draining is enabled for your load balancer, Auto Scaling waits for in-flight requests to complete.
If you don't want to do any manual scaling I guess the best approach is to changing the termination policy to OldestInstance and leave the ASG as it is. When the scale-in activity happens ASG will automatically terminate the old instances.(in your case old launch config instances)
OldestInstance. Auto Scaling terminates the oldest instance in the group. This option is useful when you're upgrading the instances in the Auto Scaling group to a new EC2 instance type. You can gradually replace instances of the old type with instances of the new type.
I would like to prevent EC2 instance termination by Auto Scaling feature if that instance is in the middle of some sort of processing.
Background:
Suppose I have an Auto Scaling group that currently has 5 instances running.
I create an alarm on average CPU usage...
Suppose 4 of the instances are idle and one is doing some heavy processing...
The average CPU load will trigger the alarm and as a result the scale-down policy will execute.
How do I get Auto Scaling to terminate one of the idle instances and not the one that is in the middle of the processing?
Update
As noted by Ryan Walls (+1), AWS meanwhile provides Instance Protection to control whether Auto Scaling can terminate a particular instance when scaling in (see the introductory blog post Instance Protection for Auto Scaling for a walk through):
You can enable the instance protection setting on an Auto Scaling
group or an individual Auto Scaling instance. When Auto Scaling
launches an instance, the instance inherits the instance protection
setting of the Auto Scaling group. [...]
It's worth noting that this instance protection only applies to regular Auto Scaling scale in events:
Instance protection does not protect Auto Scaling instances from
manual termination through the Amazon EC2 console, the
terminate-instances command, or the TerminateInstances API. Instance
protection does not protect an Auto Scaling instance from termination
if it fails health checks and must be replaced. Also, instance
protection does not protect Spot instances in an Auto Scaling group
from interruption.
As usual, the feature is available via the AWS Management Console (menu Actions->Instance Protection->Set Scale In Protection)), the AWS CLI (set-instance-protection command), and the API (SetInstanceProtection API action).
The latter two options allow automation of the scenario at hand, i.e. one would need to enable instance protection before running 'heavy processing' jobs, and disable instance protection once they are finished so that the instance is eligible for termination again.
Initial Answer
This functionality is currently not available for Auto Scaling of Amazon EC2 instances - while you are indeed able to Configure [an] Instance Termination Policy for Your Auto Scaling Group, the available policies do not include such a (fairly advanced) concept:
Auto Scaling provides the following termination policy options for you
to choose from. You can specify one or more of these options in your
termination policy.
OldestInstance — Specify this if you want the oldest instance in your Auto Scaling group to be terminated. [...]
NewestInstance — Specify this if you want the last launched instance to be terminated. [...]
OldestLaunchConfiguration — Specify this if you want the instance launched using the oldest launch configuration to be
terminated. [...]
ClosestToNextInstanceHour — Specify this if you want the instance that is closest to completing the billing hour to be
terminated. [...]
Default — Specify this if you want Auto Scaling to use the default termination policy to select instances for termination.
I just successfully dealt with the problem of long-running jobs in an auto scaling group using the relatively recent lifecycle hook feature.
The problem with trying to choose an idle node to terminate, in my case, was that the process that chooses the idle node will race against processes that submit work to the nodes. In this case it's better to use a strategy where any node can be terminated, but termination happens gracefully so that no work is lost. You can then use all of the standard auto scaling policy stuff to manage scale-in and scale-out.
The termination lifecycle hook allows the user (or a process) to perform actions on the node after it has been placed into an intermediate state (labeled Terminating:Wait) by the auto scaling group. The user (or process) is then responsible for completing the lifecycle action via an AWS API call, resulting in the shutdown of the terminated EC2 instance.
The way I set this up, in short, is:
Create a role that allows auto scaling to post a message to an SQS queue.
Create an SQS queue for the termination messages.
Create a monitor script that runs as a service in each node. My script is a simple event-driven state machine that transitions in sequence from MONITORING (polling SQS for a termination message for the node) to DRAINING (polling a job queue until no work is being performed on the node) to TERMINATED (making the complete-lifecycle call).
Standard configuration for event-driven AWS auto-scaling; that is, creating CloudWatch alarms, and the auto-scaling policies for scale-in and scale-out.
One hinderance to this approach is that the lifecycle hook management isn't supported yet in the SDKs (boto, at least, doesn't support it AFAIK), nor are there Cloud Formation resources for the hooks.
The relevant AWS documentation is here:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AutoScalingGroupLifecycle.html
Amazon has finally addressed this issue in a simpler way. There is now "instance protection" where you can mark your instance as protected and it will not be terminated during a "scale in".
See https://aws.amazon.com/blogs/aws/new-instance-protection-for-auto-scaling
aws-cli is your best friend..
Disable your scale down policy on your autoscaling group.
Create a cron job or scheduled task using aws-cli to:
2a. Get the EC2 instances associated with the autoscaling group
http://docs.aws.amazon.com/cli/latest/reference/autoscaling/describe-auto-scaling-instances.html
2b. Next monitor the cloudwatch statistics on the EC2 instances
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/US_SingleMetricPerInstance.html
http://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html
2c. Terminate the idle EC2 instance(s) from your auto-scaling group
http://docs.aws.amazon.com/cli/latest/reference/autoscaling/terminate-instance-in-auto-scaling-group.html
You can use Amazon CloudWatch to achieve this:
http://aws.typepad.com/aws/2013/01/amazon-cloudwatch-alarm-actions.html. From the article:
You can use a similar strategy to get rid of instances that are tasked with handling compute-intensive batch processes. Once the CPU goes idle and the work is done, terminate the instance and save some money!
In this case, since you will be handling the termination, you will need to remove the scale-down policy. Also see another option: https://stackoverflow.com/a/19628453/432849.