In AWS EC2, both lifecycle hook and user data of launch configuration permits to execute some customised actions while launching instances.
Could you tell me is Lifecyle hook related actions are executed before or User data defined in launch configuration executed before ?
When do you chose which one ? What are their differences ?
User Data and Cloud-Init
When launching an Amazon EC2 instance, you can provide a User Data field. The information entered in this field is available to the instance via http://169.254.169.254/latest/user-data/.
This is an excellent way to pass information to the instance that is accessible to software running on the instance.
Then Canonical, maker of Ubuntu, conceived Cloud-Init as a way of running scripts during the startup of virtual machines. Cloud-Init takes a script passed via EC2 User Data and runs it as root during the first boot of an instance. It is a great way to install software and configure the machine when it is first used.
Amazon EC2 Auto-Scaling Lifecycle Hooks
Amazon EC2 Auto-Scaling is a method of automatically scaling a fleet of EC2 instances based on workload. Instances are launched or terminated based on a target capacity metric. Instances launched through Auto-Scaling are normal EC2 instances, so User Data can be used to configure these instances.
Sometimes, however, a more complex operation is required when launching/terminating instances. For example, when launching instances it might be necessary to contact an external configuration service, and when terminating instances it might be necessary to copy data off an instance. These tasks can be accomplished via Lifecycle Hooks.
From Amazon EC2 Auto Scaling Lifecycle Hooks - Amazon EC2 Auto Scaling:
Lifecycle hooks enable you to perform custom actions by pausing instances as an Auto Scaling group launches or terminates them. When an instance is paused, it remains in a wait state until either you complete the lifecycle action using the complete-lifecycle-action CLI command or CompleteLifecycleAction API action, or the timeout period ends (one hour by default).
Compared to User Data, Lifecycle Hooks are rarely used. They are typically required when a longer-running or external process is required before instances are ready to process requests. For example, there might be a long startup process required for new instances that exceeds the time normally allowed for health checks. Or, an external process (outside the instance) might need to be triggered before the instance can start processing traffic.
Lifecycle Hooks are more complicated because they involve a signalling mechanism. When an Auto Scaling instance is launched or terminated, Auto Scaling will send a message via Amazon SQS or Amazon SNS. You are then responsible for running a process that responds to this signal. When the process is complete, it must send a signal back to Auto Scaling so that the instance can be fully added to, or removed from, the Auto Scaling group. This will typically require something running external to the EC2 instance to process the Lifecycle Hook.
Bottom line: You want to use User Data. It is rare that you would use a Lifecycle Hook.
Thanks for your answers.
Actually, the LifeCycle hook is somehow useful. For example, in my situation, I have to deactivate Tableau Server licenses before the instance is terminated. This is a good situation to use it.
While, during starting up, user data is executed but the instance will be put into "In Service" states once the OS itself is up, even the user data is still not yet finished to be executed.
So to choose between the two approches, it all depends on what you want to do in the scripts.
Related
I am using AWS ECS in combination with EC2 instances.
Right now I am setting up Auto Scaling. How can I make sure that when, an EC2 instance gets terminated, all ECS tasks get migrated before the machine gets terminated?
Right now it is not automatically possible to achieve this. The best approach would be to have atleast 2 tasks running of each service, spread on different instances via a placement constraint.
Manually (or scripted) it is possible:
If you want to replace an instance attached to an ECS cluster, you can simply drain the instance. This will do the following
Start a new task of each running service on another instance in the cluster
Wait until the recently started task is 'steady'
shutdown the old task
To drain an instance using the AWS CLI, do the following:
Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.
In the navigation pane, choose Clusters and select the cluster.
Choose ECS Instances and select the check box for the container
instances.
Choose Actions, Drain instances.
After the instances are processed, choose Done.
This can also be done via the command line.
To do it automatically, you will need to add a lifecycle hook on termination.
Call the AWS CLI from the termination lifecycle hook to drain the instance, wait a fixed amount of time and then continue terminating the instance.
I'm looking to see if I can create an instance and deploy applications to athis instance dynamically via the API. I only want these instances to be created when my application needs them, or I request for them to be created.
I have two applications that I need to deploy to each created instance which require some set up and installation of dependencies prior to their launch. When I am finished with this application, I want to terminate the instance.
Am I able to do this? If so, could someone please point me to the right section of the documentation. I have searched on the documentation and found some information about creating images but I am unsure as to what exactly I will need to achieve this task.
Yes. Using an Autoscaling Group, you can create a launch configuration that will launch you instances. Using CodeDeploy, you would link your deployment group to the auto-scaling group.
See Integrating AWS CodeDeploy with Auto Scaling
AWS CodeDeploy supports Auto Scaling, an AWS service that can launch
Amazon EC2 instances automatically according to conditions you define.
These conditions can include limits exceeded in a specified time
interval for CPU utilization, disk reads or writes, or inbound or
outbound network traffic. Auto Scaling terminates the instances when
they are no longer needed. For more information, see What Is Auto
Scaling?.
Assuming you set your desired/minimum instances to 0, then the default state of the ASG will be to have no instances.
When you application needs an instance spun up, it would simply change the desired instance value to 1. When your application is completed with the instance, it would set your desired count to 0 again, thereby terminating that instance.
To develop this setup, start by running your instance normally (manually) and get the application deployment working. When that works, then create your auto scaling group. Finally, update your deployment group to refer to the ASG so that your code is deployed when you have scaling events.
How can I create a policy that will allow a single user in a group to create an ec2 instance in a particular VPC and AZ?
I need it to also be destroyable by the same user or destroyed when idle for more than 24 hours.
You can't. What you can do, however, is script the logic yourself to provide equivalent functionality. A good starting point would be to google AWS SDK.
This is not possible via IAM policies nor other configurations.
You would have to write your own application with this logic, which would launch the instance on the user's behalf and then terminate it later (although that depends on your definition of 'when idle'). The user would interact with this application when they want to launch the instance, rather than interacting directly with AWS.
To be more specific:
It is possible to limit the number of EC2 instances that an AWS Account can have running simultaneously (the default is 20)
It is not possible to limit the number of instances a single user may launch. Either they have permissions to launch an instance with certain attributes, or they don't, but this cannot be limited by whether/what instances are already running.
Some ways you could create a self-terminating instance:
A CloudWatch Alarm could terminate an instance when CPU or Network drops below a certain level for a period of time (eg when Network Out is below 10KB over a half-our period).
A script on the instance itself could decide when to Shutdown the instance. If the instance is launched with Shutdown Behaviour set to Terminate, then shutting down the instance from within the instance will cause it to terminate.
You could create an application that regularly looks at running instances and terminates them 24 hours after being launched. This could be triggered by a Scheduled Task (Windows) or cron job (Linux).
A variation on the previous option is to add a Tag to the instance to indicate when to terminate it. An application could regularly check the tag to determine which instances to terminate.
I am looking at using AWS auto-scaling to scale my infrastructure up and down based on various performance metrics (CPU, etc.). I understand how to set this up; however, I don't like that instances are terminated rather than stopped when it is scaled down. This means that when I scale back up, I have to start from scratch with a new instance and re-install my software, etc. I'd rather just start/stop my instances as needed rather than create/terminate. Is there a way to do this?
No, it is not possible to Stop an instance under Auto Scaling. When a Scaling Policy triggers the removal of an instance, Auto Scaling will always Terminate the instance.
However, here's some ideas to cope with the concept of Termination...
Option 1: Use pre-configured AMIs
You can configure an Amazon EC2 instance with your desired software, data and settings. Then, select the EC2 instance in the Management Console and choose the Create Image action. This will create a new Amazon Machine Image (AMI). You can then configure Auto Scaling to use this AMI when launching a new instance. Each new instance will contain exactly the same disk contents.
It's worth mentioning that EBS starts up very quickly from an AMI. Instead of copying the whole AMI to the boot disk, it copies it across on "first access". This means the new instance can start-up immediately rather than waiting for the whole disk to be copied.
Option 2: Use a startup (User Data) script
Each Amazon EC2 instance has a User Data field, which is accessible from the instance. A script can be passed through the User Data field, which is then executed when the instance starts. The script could be used to install software, download data and configure the instance.
The script could do something very simple, like download a configuration script from a source code repository, then execute the script. This means that machine configuration can be centrally managed and version-controlled. Want to update your app? Just launch a new instance with the updated script and throw away the old instance (which is much easier than "updating" an app).
Option 3: Add/Remove instances to an Auto Scaling group
Rather than using Scaling Policies to Launch/Terminate instances for an Auto Scaling group, it is possible to attach/detach specific instances. Thus, you could 'simulate' auto scaling:
When you want to scale-down, detach an instance from the Auto Scaling group, then stop it.
When you want to add an instance, start the instance then attach it to the Auto Scaling group.
This would require your own code, but it is very simple (basically two API calls). You would be responsible for keeping track of which instance to attach/detach.
You can suspend scaling processes, see documentation here:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html#as-suspend-resume
Add that instance to Scale in protection and then stop the instance then it will not delete your instance as it's having the scale in protection.
Actually you have three official AWS options to reboot or even stop an instance which belongs to an Auto Scaling Group:
Put the instance into the Standby state
Detach the instance from the group
Suspend the health check process
Ref.: https://aws.amazon.com/premiumsupport/knowledge-center/reboot-autoscaling-group-instance/
As of April 2021:
Option 4: Use Warm Pools and an Instance Reuse Policy
By default, Amazon EC2 Auto Scaling terminates your instances when your Auto Scaling group scales in. Then, it launches new instances into the warm pool to replace the instances that were terminated.
If you want to return instances to the warm pool instead, you can specify an instance reuse policy. This lets you reuse instances that are already configured to serve application traffic.
This mostly automates option 3 from John's answer.
Release announcement: https://aws.amazon.com/blogs/compute/scaling-your-applications-faster-with-ec2-auto-scaling-warm-pools/
Documentation: https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html
This is to expand a little on #mwalling's answer, because that is the right direction, but needs a little extra work to prevent instance termination.
There is now a way to stop or hibernate scaled in instances!
By default AWS Autoscaling scale in policy is to terminate an instance. Even if you have a warm pool configured. Autoscaling will create a fresh instance to put into the warm pool. Presumably to make sure you start with a fresh machine every time. However, with a instance reuse policy you can make AWS Autoscaling either stop or hibernate a running instance and store that instance in the warm pool.
Advantages include:
Local caches stay populated (use hibernate for in memory cache).
Burstable EC2 instances (those types with T*) keep built up burst credits instead of the newly created instance that have limited or no credits.
Practical example:
We use a burstable EC2 instance for CI/CD work that we scale to 0 instances outside working hours. With a reuse policy our local image repository stays populated with the most important Docker images. Also we keep the built up credit from the previous day and that speeds up automatic jobs we run first thing every morning.
How to implement:
There's currently no way of doing this completely via the management console. So you will need to use AWS CLI or SDK.
First create a warm pool as described in the AWS Documentation
Then execute this command to add a reuse policy:
aws autoscaling put-warm-pool --auto-scaling-group-name <Name-of-autoscaling-group> --instance-reuse-policy ReuseOnScaleIn=true
Reference docs for the command: AWS CLI Autoscaling put-warm-pool documentation
Flow diagram of possible life cycles of EC2 instances:
Image from AWS Documentation: Lifecycle state transitions for instances in a warm pool
I would like to prevent EC2 instance termination by Auto Scaling feature if that instance is in the middle of some sort of processing.
Background:
Suppose I have an Auto Scaling group that currently has 5 instances running.
I create an alarm on average CPU usage...
Suppose 4 of the instances are idle and one is doing some heavy processing...
The average CPU load will trigger the alarm and as a result the scale-down policy will execute.
How do I get Auto Scaling to terminate one of the idle instances and not the one that is in the middle of the processing?
Update
As noted by Ryan Walls (+1), AWS meanwhile provides Instance Protection to control whether Auto Scaling can terminate a particular instance when scaling in (see the introductory blog post Instance Protection for Auto Scaling for a walk through):
You can enable the instance protection setting on an Auto Scaling
group or an individual Auto Scaling instance. When Auto Scaling
launches an instance, the instance inherits the instance protection
setting of the Auto Scaling group. [...]
It's worth noting that this instance protection only applies to regular Auto Scaling scale in events:
Instance protection does not protect Auto Scaling instances from
manual termination through the Amazon EC2 console, the
terminate-instances command, or the TerminateInstances API. Instance
protection does not protect an Auto Scaling instance from termination
if it fails health checks and must be replaced. Also, instance
protection does not protect Spot instances in an Auto Scaling group
from interruption.
As usual, the feature is available via the AWS Management Console (menu Actions->Instance Protection->Set Scale In Protection)), the AWS CLI (set-instance-protection command), and the API (SetInstanceProtection API action).
The latter two options allow automation of the scenario at hand, i.e. one would need to enable instance protection before running 'heavy processing' jobs, and disable instance protection once they are finished so that the instance is eligible for termination again.
Initial Answer
This functionality is currently not available for Auto Scaling of Amazon EC2 instances - while you are indeed able to Configure [an] Instance Termination Policy for Your Auto Scaling Group, the available policies do not include such a (fairly advanced) concept:
Auto Scaling provides the following termination policy options for you
to choose from. You can specify one or more of these options in your
termination policy.
OldestInstance — Specify this if you want the oldest instance in your Auto Scaling group to be terminated. [...]
NewestInstance — Specify this if you want the last launched instance to be terminated. [...]
OldestLaunchConfiguration — Specify this if you want the instance launched using the oldest launch configuration to be
terminated. [...]
ClosestToNextInstanceHour — Specify this if you want the instance that is closest to completing the billing hour to be
terminated. [...]
Default — Specify this if you want Auto Scaling to use the default termination policy to select instances for termination.
I just successfully dealt with the problem of long-running jobs in an auto scaling group using the relatively recent lifecycle hook feature.
The problem with trying to choose an idle node to terminate, in my case, was that the process that chooses the idle node will race against processes that submit work to the nodes. In this case it's better to use a strategy where any node can be terminated, but termination happens gracefully so that no work is lost. You can then use all of the standard auto scaling policy stuff to manage scale-in and scale-out.
The termination lifecycle hook allows the user (or a process) to perform actions on the node after it has been placed into an intermediate state (labeled Terminating:Wait) by the auto scaling group. The user (or process) is then responsible for completing the lifecycle action via an AWS API call, resulting in the shutdown of the terminated EC2 instance.
The way I set this up, in short, is:
Create a role that allows auto scaling to post a message to an SQS queue.
Create an SQS queue for the termination messages.
Create a monitor script that runs as a service in each node. My script is a simple event-driven state machine that transitions in sequence from MONITORING (polling SQS for a termination message for the node) to DRAINING (polling a job queue until no work is being performed on the node) to TERMINATED (making the complete-lifecycle call).
Standard configuration for event-driven AWS auto-scaling; that is, creating CloudWatch alarms, and the auto-scaling policies for scale-in and scale-out.
One hinderance to this approach is that the lifecycle hook management isn't supported yet in the SDKs (boto, at least, doesn't support it AFAIK), nor are there Cloud Formation resources for the hooks.
The relevant AWS documentation is here:
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AutoScalingGroupLifecycle.html
Amazon has finally addressed this issue in a simpler way. There is now "instance protection" where you can mark your instance as protected and it will not be terminated during a "scale in".
See https://aws.amazon.com/blogs/aws/new-instance-protection-for-auto-scaling
aws-cli is your best friend..
Disable your scale down policy on your autoscaling group.
Create a cron job or scheduled task using aws-cli to:
2a. Get the EC2 instances associated with the autoscaling group
http://docs.aws.amazon.com/cli/latest/reference/autoscaling/describe-auto-scaling-instances.html
2b. Next monitor the cloudwatch statistics on the EC2 instances
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/US_SingleMetricPerInstance.html
http://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html
2c. Terminate the idle EC2 instance(s) from your auto-scaling group
http://docs.aws.amazon.com/cli/latest/reference/autoscaling/terminate-instance-in-auto-scaling-group.html
You can use Amazon CloudWatch to achieve this:
http://aws.typepad.com/aws/2013/01/amazon-cloudwatch-alarm-actions.html. From the article:
You can use a similar strategy to get rid of instances that are tasked with handling compute-intensive batch processes. Once the CPU goes idle and the work is done, terminate the instance and save some money!
In this case, since you will be handling the termination, you will need to remove the scale-down policy. Also see another option: https://stackoverflow.com/a/19628453/432849.