I'm having an issue with AWS boxes (EC2) where I terminate the box and it re-spawns. To give context, there is no autoscaling group. Anywhere I can search for some config that might be triggering the launch?
I would make sure you don't have a persistent spot request active in your account, and also check to see if you perhaps installed the AWS Instance scheduler - either or both of those could be starting instances on your behalf - (double check the autoscaling group, that is the most obvious reason though)
If you terminate a running Spot Instance that was launched by a
persistent Spot request, the Spot request returns to the open state so
that a new Spot Instance can be launched. To cancel a persistent Spot
request and terminate its Spot Instances, you must cancel the Spot
request first and then terminate the Spot Instances. Otherwise, the
persistent Spot request can launch a new instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
https://aws.amazon.com/solutions/instance-scheduler/
So I found out the culprit, maybe it can help more people. Apparently, there is a service from AWS called OpsWorks that automates things like Cheff of Puppet, which my company had configured some time ago. That would be checking for instances running and triggering re-provisioning when it didn't see the instance running. OpsWorks is here
Related
I would like to use AWS EC2 spot requests for machine learning that I do in spots.
If you don't configure any settings for spot requests, they will be set to "fleet" and an EC2 instance will be created that cannot be paused (it can only be terminated).
If you know how to create an instance with a spot request that can be paused, please let me know.
Spot instances launched in a fleet can’t be paused or stopped. To stop a Spot instance, it must be launched by a Spot instance request for a single instance, you can create such requests using Launch Instance wizard, you may follow steps at this link. Make sure to select "Persistent request" in the wizard.
This request will launch a Spot instance that can be Stopped but still can’t be Paused. Hibernate/Pause option is only available as a behavior at the event of interruption and can't be requested by the user.
I just figured out about AWS CloudWatch that would let you terminate your EC2 instance after some time in inactivity. So I created an alarm that would terminate the instance when the CPU usage is less the 1% for 2 hours. Which ultimately ended up putting my instance into an alarm state right away and it prevented me from starting it up to test the feature out.
I then deleted the CloudWatch alarm again in order to be able to launch the EC2 instance gain but even after I deleted the CloudWatch, the state is set to Terminated but the but the Start option from the Actions drop down button is still disabled.
How do I get the instance to start again?
You can't restart a terminated instance. The instance no longer exists. It is just listed as "terminated" in your web console for a little bit so you can see that it was deleted. You have to create a new instance now.
I have foreseen a problem that could happen with my application but I am unsure if it is possible to solve, and perhaps the architecture needs to be redesigned.
I am using an AutoScalingGroup (ASG) on AWS to create EC2 instances that host game servers that players can join. At the moment, the ASG is scaled manually via a matchmaking API which changes the desired capacity based on its needs. The problem occurs when a game server is finished.
When a game finishes, it signals to the matchmaker that it is finished and needs terminating, and the matchmaker will then scale down the ASG accordingly, however, it doesn't seem to know exactly which instance to remove, and it won't necessarily be the one that needs terminating.
I can terminate the instance, but then as the ASG desired capacity is never changed when the instance is terminated, another server is created.
Is there a way I can scale down the ASG, as well as specifying which servers to remove from the group?
In a nutshell, the default termination policy during scale in is designed to remove instances that use the oldest launch configuration.
Currently, Amazon EC2 Auto Scaling supports the following termination policie:
OldestInstance Terminate the oldest instance in the group. This option is useful when you're upgrading the instances in the Auto Scaling group to a new EC2 instance type. You can gradually replace instances of the old type with instances of the new type.
NewestInstance Terminate the newest instance in the group. This policy is useful when you're testing a new launch configuration but don't want to keep it in production.
OldestLaunchConfiguration Terminate instances that have the oldest launch configuration. This policy is useful when you're updating a group and phasing out the instances from a previous configuration.
ClosestToNextInstanceHour Terminate instances that are closest to the next billing hour. This policy helps you maximize the use of your instances and manage your Amazon EC2 usage costs.
Default Terminate instances according to the default termination policy. This policy is useful when you have more than one scaling policy for the group.
Instance protection
One of the possible solutions could be to use Instance protection. The auto-scaling provides an instance protection to control whether instance can be terminated when scaling-in.
Therefore, enable the instance protection for ASG to protect instances from scaling-in by default. Once you are done with you server, decrease a value of desired number of instances, remove instance protection from particular instance (either using CLI or SDK; note that this protection remains enabled for the rest of instances) and auto-scaling will terminate that exact instance.
For more information about instance protection, see Instance Protection
The oldest server is removed. If you want to scale down a specific server, you will have to kill that server before changing desired capacity.
I am creating AWS spot instances on demand and wanted to keep interruption behavior as STOP, so that AWS can restart my VM as soon as price/demand meets.
But the problem is AWS GUI doesn't have any option to STOP running spot instances.
It just have option to terminate instances or cancel spot requests.
I am developing some application which will resume itself on reboot.
I wanted to test the application by simulating AWS stop and restart behaviour.
Thx in advance
~Ashish
As of now you cannot Stop a Spot Instance.
To simulate your application behavior, bid for minimum and wait for AWS to interrupt the instance.
Make sure that your spot instance is Stop Interruption ready.
Reference:
Spot Instance Interruptions
You do not have an option to restart a SPOT instance. SPOT instance is not the right candidate for this kind of application.
I wrote a terraform script to spin up an EC2 spot instance, and every time I terminate it, it automatically spins up a new instance. I don't understand this behavior. Why is this happening and how do I stop it?
When you requested your spot instance, you probably didn't specify the spot-type as 'one-time' so the request persists beyond instance termination. Or in other words, when your instance is terminated the request begins anew and launches a new instance based on the same bid price, launch specification, etc.
From the terraform documentation:
spot_type - (Optional; Default: "persistent") If set to "one-time",
after the instance is terminated, the spot request will be closed.
Also, Terraform can't manage one-time spot requests, just launch them.
From the AWS documentation:
This will continue until you cancel the spot instance request. You can do this via the AWS Console, CLI (cancel-spot-instance-requests), or via terraform by destroying the resource.
Further Reading
AWS Documentation - Cancelling a Spot Instance Request