On the personal health dashboard in the AWS Console, I've got this notification
EC2 persistent instance retirement scheduled
yesterday which says that one of my ec2 instances is scheduled to retire on 13th March 2019. The status was 'upcoming' while the start and end times both were set to 14-Mar-2019.
The content of the notification starts with:
Hello,
EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-xxxxxxxxxx) associated with your AWS account (AWS Account ID: xxxxxxxxxx) in the xxxx region. Due to this degradation your instance could already be unreachable. We will stop your instance after 2019-03-13 00:00 UTC.
....
I've got yet another notification today for the same instance and with the same subject line but the status has been changed to 'ongoing' and the start time is 27-Feb-2019 while the end time is 14-Mar-2019.
I was planning to do a start-stop of the instance next week but does the second notification tell me to do is ASAP?
Yes, it is better to do stop/start ASAP. Even in your message it says:
Due to this degradation your instance could already be unreachable
Related
I received an email from amazon aws
EC2 has detected degradation of the underlying hardware hosting your
Amazon EC2 instance (instance-ID: i-xxxxxxxx) associated with your AWS
account (AWS Account ID: xxxxxx) in the us-west-1 region. Due to this
degradation your instance could already be unreachable. We will stop
your instance after 2021-05-28 04:00:00 UTC. Please take appropriate
action before this time.
the instance itself is working fine at the moment but this email doesn't sound friendly.
what am i supposed to do this point. we are using docker and node on this instance.
Stopping the instance then starting the instance should migrate the VM off the degraded hardware. Note that you shouldn't "reboot" or "restart" it. You need to full stop, then start it.
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-degraded-hardware/
I got the mail from Amazon regarding my EC2 as follows:
EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: xxxxxxxx) associated with your AWS account (AWS Account ID: xxxxxx) in the ap-south-1 region. Due to this degradation your instance could already be unreachable. We will stop your instance after 2020-12-29 22:00:00 UTC. Please take appropriate action before this time.
The affected instances are listed below:
xxxxxxx
So what would I do now to keep my data safe ?
AWS provides guides on what to do when an instance is about to be retired:
What do I need to know when my Amazon EC2 instance is scheduled for retirement?
The easiest way would be to stop/start the instance:
You are required to stop and then start the instance at your preferred time before the instance retirement date. Stopping and starting the instance moves the instance to another healthy host.
However, exact details depend on your instance type (e.g. EBS based or instance store based), thus please read the guide in the link provided to understand your options which are mostly determined on your actual EC2 instance setup.
I'm using Ubuntu on Amazon EC2. If an EC2 instance stops running due to a monthly usage quota being exceeded (i.e., too many requests), does it start up automatically on the first of the next month or would one have to manually re-boot it?
I've created aws cloudwatch alarm based on ASG's group metrics cpuutilization. It sends an email alert email whenever cpuutilization exceeds more than 99% for more than an hour.
Is there a way to execute an event/action that will terminate specific ec2 instances that triggered the alarm? These instances hang and has to be terminated.
I would create an additional alarm that would terminate any instance that reaches 99% cpu for an hour. This is directly supported by CloudWatch.
From Create Alarms to Stop, Terminate, Reboot, or Recover an Instance:
Using Amazon CloudWatch alarm actions, you can create alarms that automatically stop, terminate, reboot, or recover your EC2 instances. You can use the reboot and recover actions to automatically reboot those instances or recover them onto new hardware if a system impairment occurs.
See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html
I feel possible solution for this requirement is to write AWS Cli script which would run probably every 15 mins and get list of all EC2 instances running and then terminate if needed. Also, need historical info for ec2's w/c cpu is at 100% for more than 45mins
While working with a g2.2xlarge spot instance, I have tried to set up an alarm that will notify me when the average CPU usage over a two hour period has dropped below 5% and will then automatically stop the instance. Here's a link to a nice article Amazon wrote up on how to use the stop/start instance feature. The AWS alarms seem to allow you to do this however after the trigger goes off I get this reply:
Dear AWS customer,
We are unable to execute the 'Stop' action on Amazon EC2 instance i-e60e21ec that you specified in the Amazon CloudWatch alarm awsec2-i-e60e21ec-Low-CPU-Utilization.
You may want to check the alarm configuration to ensure that it is compatible with your instance configuration. You can also attempt to execute the action manually.
These are some possible reasons for this failure and steps you can try to resolve it:
Incompatible action selected:
Your instance’s configuration may not be compatible with the selected action.
To execute the 'Terminate' action, your instance may have Termination Protection enabled. Disable this feature if you want to terminate your instance. Once you do that, the alarm will execute the action after the next applicable alarm state change.
To execute the 'Stop' action, your instance’s root device type must be an EBS volume. If the root device type is the instance store, select the 'Terminate' action instead. Once you do that, the alarm will execute the action after the next applicable alarm state change.
Temporary service interruption: There may have been an issue with Amazon CloudWatch or Amazon EC2. We have retried the action without success. You can try to execute the action manually, or wait for the next applicable alarm state change.
Sincerely, Amazon Web Services
Stop seems to be an option for the free micro instance but not for these other instances. When I try to change the shutdown behavior to stop in actions it says:
An error occurred while changing the shutdown behavior of this instance.
Modifying 'instanceInitiatedShutdownBehavior is not supported for spot instances.
Is there another way to get around this problem or will we have to wait until Amazon makes this feature available?
Use standard instances instead of spot instances. Spot instances allow you to bid on extra capacity within ec2. However, they may automatically shut down if the spot price exceeds your bid.
Its not really intended for an always on instance.