Before knowing exactly how AWS spot instances work I configured a spot request with the interruption behavior set to terminate. As I understand it my running instance state will be deleted on termination. So if I don't have an image backup I will not be able to start the server again in it's last state.
Since the spot request is fulfilled and the instance running, is it possible to change the interruption behavior to stop when I am outbid? I cant seem to find the option to change the interruption behavior.
For setting the interruption behavior to stop, we need to take care of some requirements:
For a Spot Instance request, the type must be persistent, not
one-time. You cannot specify a launch group in the Spot Instance
request.
For a Spot Fleet request, the type must be maintain, not request.
The root volume must be an EBS volume, not an instance store volume.
by following the above requirements, we can change the interruption behavior from terminate to stop.
Please refer the following url for reference:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
According to boto3 Spot instance creation documentation, you may also instruct the instance to stop or hibernate if you set the instance type to persistent. The default behaviour is terminate. This features is added in November 2017.
response = client.request_spot_fleet(
SpotFleetRequestConfig={
.....
Type='one-time'|'persistent',
'InstanceInterruptionBehavior': 'hibernate'|'stop'|'terminate'
}
Use them sparingly as each behaviour has some pros and cons, e.g. you must take care of process network connection interruption if yo use hibernate. For stop, you may want to store data into another mounted EBS.
Related
Imagine I have a persistent spot request of size 1. If I manually stop this instance, it will not restart until I start it again.
However, if the instance is interrupted by AWS because of a lack of spot capacity, it will restart automatically when capacity is available (as I understand it).
How can I put an instance in the stopped-because-interrupted state into a state where it will not restart automatically if capacity becomes available.
According to AWS documentation only EC2 can start interrupted spot instances so I think that the only option would be monitoring events/alerts which notify when instances restart and then stop them again with automation. Never tried this si this is only an educated guess/opinion.
I am using autoscale group for adding and removing additional instances for my application. I am using CPU Utilization as my scaling parameter and wondering what happens when an instance is running a program and the CPU Utilization comes below 65% (i.e threshold value).
Does it wait for the instance to finish the program or terminate the instance at that moment? If it terminates the instance at that moment then it might lead to data loss/data inconsistency.
Any help would be appreciated.
If you're looking to prevent or delay an instance during a scale in event you could take a look at lifecycle hooks.
By enabling this autoscaling can send a notification that a specific instance action is about to occur (scale out or scale in). Using a combination of services (such as SNS, Lambda, SSM etc) you would be able to programmatically notify the instance that is is about to be terminated which you can then take any necessary actions.
The instance termination will wait until there is a confirmation to the autoscaling group that it has been completed which will lead to it being terminated. Additionally a lifecycle hook will have a timeout, if no confirmation is received by the time the timeout has been exceeded then the termination will still occur.
I think that you are looking for termination policy
Look at this link:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-termination.html#default-termination-policy
And in my experience, the instance will be terminated no matter what it is running
Does it wait for the instance to finish the program or terminate the instance at that moment.
Sadly, I does not wait. ASG works outside of your instances and is not concerned with any programs running on your instance.
Having said that, there are few things you can do, some of which are described in:
Controlling which Auto Scaling instances terminate during scale in
Generally speaking you should develop your applications to be stateless. This means the applications should be "aware" that they can be terminated at any time. One way to achieve is by using external storage systems, such as S3 or EFS, which will persist data between terminations.
Other way, is to use termination protection. In this case, the application will put its instance in this state at the beginning of processing, and then whey the calcuation finishes, the termination protection will be removed.
Since today i can't access my instance, i tried stop and restart several times but the status is always : "1/2 checks passed"
I tried to create a snapshot, detach and reattach new volume but the result is the same.
I also tried to create another instance and attach the volume and it's not starting either.
Any help ?
The status checks automatically performed on Amazon EC2 instances are:
System Status Checks: These check the underlying systems used by the Amazon EC2 instance
Instance Status Checks: These check the configuration of the specific instance
See documentation: Status Checks for Your Instances
Often, an instance is available and ready to be used before these checks are complete -- this is especially the case for Linux instances because they boot very quickly.
If you receive a 1/2 checks passed message, either wait a little longer or Stop and Start the instance. Performing a Stop/Start will launch the instance on a different host, which will probably fix whatever problem was being experienced.
If the 1/2 checks passed message continues to appear after a Stop/Start, it is probably a misconfiguration of the AMI. I have seen this when the wrong virtualization type was selected for an AMI that was created from a Snapshot.
You might be able to get a hint about the problem by using the Get System Log command in the Actions menu, which shows the log while the instance is booting.
Worst case, launch a new instance from a known-good AMI, attach the non-booting volume as an additional disk and copy files to the new disk. You will still have access to your files even if it will not boot.
You can check the description of the checks here, and understand which one is not working...
While working with a g2.2xlarge spot instance, I have tried to set up an alarm that will notify me when the average CPU usage over a two hour period has dropped below 5% and will then automatically stop the instance. Here's a link to a nice article Amazon wrote up on how to use the stop/start instance feature. The AWS alarms seem to allow you to do this however after the trigger goes off I get this reply:
Dear AWS customer,
We are unable to execute the 'Stop' action on Amazon EC2 instance i-e60e21ec that you specified in the Amazon CloudWatch alarm awsec2-i-e60e21ec-Low-CPU-Utilization.
You may want to check the alarm configuration to ensure that it is compatible with your instance configuration. You can also attempt to execute the action manually.
These are some possible reasons for this failure and steps you can try to resolve it:
Incompatible action selected:
Your instance’s configuration may not be compatible with the selected action.
To execute the 'Terminate' action, your instance may have Termination Protection enabled. Disable this feature if you want to terminate your instance. Once you do that, the alarm will execute the action after the next applicable alarm state change.
To execute the 'Stop' action, your instance’s root device type must be an EBS volume. If the root device type is the instance store, select the 'Terminate' action instead. Once you do that, the alarm will execute the action after the next applicable alarm state change.
Temporary service interruption: There may have been an issue with Amazon CloudWatch or Amazon EC2. We have retried the action without success. You can try to execute the action manually, or wait for the next applicable alarm state change.
Sincerely, Amazon Web Services
Stop seems to be an option for the free micro instance but not for these other instances. When I try to change the shutdown behavior to stop in actions it says:
An error occurred while changing the shutdown behavior of this instance.
Modifying 'instanceInitiatedShutdownBehavior is not supported for spot instances.
Is there another way to get around this problem or will we have to wait until Amazon makes this feature available?
Use standard instances instead of spot instances. Spot instances allow you to bid on extra capacity within ec2. However, they may automatically shut down if the spot price exceeds your bid.
Its not really intended for an always on instance.
I have a long batch job that I'd like to run on AWS EC2 Spot Instances, to save money. However, I can't find the answer to two seemingly critical questions:
When a new instance is created, I need to upload the code onto it, configure it, and run the code. How does that get done for Spot Instances, which are created automatically and unattatendly?
When an instance is stopped, I would prefer having some type of notification, so that the state could be saved. (This is not critical, as the batch job will run fine if terminated suddenly - but a clean shutdown is preferred).
What is the standard way to deploy spot instances? Is there a way to do manual setup, turn it into a spot instance, and then let it hibernate until the spot price is available?
As to #1, if you create an AMI (amazon machine image), you can have everything you want pre-installed on a 'hibernating' image that you can use as a basis for the spot image you start:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances-getting-started.html
For #2, you can be notified when a spot instance terminates using SNS:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-autoscaling-notifications.html
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html
BTW: You can be notified the the instance was terminated, but only after it terminates. You can't get notified that an instance is about to be shutdown and gracefully save the state - you need to engineer your solution to be OK with unexpected shutdowns.
No matter how high you bid, there is always a risk that your Spot
Instance will be interrupted. We strongly recommend against bidding
above the On-Demand price or using Spot for applications that cannot
tolerate interruptions.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-protect-interruptions.html
You can use the user data settings to download from a specific repository a script and run it at the first instance startup.
As E.J. Brennan said: you can use SNS