We are having multiple test servers which are running idle and needs to terminate/stop to reduce cost
Scenario is we are using Jenkins to run a project where developers will build the project and everytime new server will spin up, if there is any stopped server due to the previous build it will first terminate the stopped one and then new instance will spin up.
I tried using
Cloudwatch CPU utilization function to stop/terminate idle instances but as every time a new instance ID will generate this option is out.
We are using ASG, but when I tried using simple step scaling to reduce it to zero it didn't work
I am looking to create lambda function so idle instance will stop/terminate without considering instance ID
Related
I have an ECS cluster running service and task that sits on an EC2 machine. The task is big and takes up the whole machine its running on. Is there any way to set up ECS & EC2 to scale temporally on deployments to create a new instance, run the new task then stop the old pre-deployment task?
So far tried to play with the auto-scaling on both EC2 & ECS separately but it seems to me that the conditions for scaling are down to CPU or memory utilization, however as my task takes up the whole instance there are no alarms that could be triggered on deployment as there's just no suitable instance to start the task on.
Right now I have the services running as DAEMON, so it's one per instance, and autoscaling seems to be disabled for it. During deployments, it drains the old task before starting the new one creating downtime. To set the service type to REPLICA, scaling can be enabled, however, it's based on instance resource utilization and I can't seem to figure out how I would create a new instance on deployments.
I am missing something or interpreting how these auto-scalers work wrong, as I can't seem to see a nice way to deploy without affecting the current task sort of having just an unused instance sitting there running in the background all the time(wasting money).
Background
I got the following setup with AWS code deploy:
Currently we have our EC2 application servers connected to an auto-scaling group, but there is a missing step: once a new server is fired up, we don't automatically deploy the latest code on it from our git repo
Question
I was going over this tutorial:
Basically i want to run a bunch of commands as soon as an instance is launched but before it's hooked up to the load balancer.
The above tutorial describes things in general, but I couldn't answer the following questions:
Where do I save the script on the ec2 instance?
How is that script executed once the instance is scaled in but before its connected to the load balancer?
I think you do not need to life cycle hook, the life cycle is useful when you want to perform an action in different stats like stop, start and terminate but you just to pull the latest code and some other commands.
To answer your Question I will suggest below approach, as there are many many more approaches for the same task.
You do not need to save the script or command, place them on s3 or you can run commands just put them in the user data in your launch configuration. You can run them as bash script or you can pull your scripts from aws s3.
This can be the simplest example to handle pull code case. So this will run whenever a new instance launch in this auto-scaling group.
Another example can be to run a complex script, place them on s3 and pull them during scaling up.
I assume you already set permission for s3 and bitbucket. You can run any complex during this time.
The second steps are a bit tricky, you can use a different approach, the instance will never receive traffic until its healthy so start your application once your code updated and all the required scripts done execution than at the end you can run your application.
Another approach can be
a):Health Check Grace Period
Frequently, an Auto Scaling instance that has just come into service
needs to warm up before it can pass the health check. Amazon EC2
Auto Scaling waits until the health check grace period ends before
checking the health status of the instance.
b)Custom Health Checks
If you have your own health check system, you can send the instance's
health information directly from your system to Amazon EC2 Auto
Scaling.
Use the following set-instance-health command to set the health state
of the specified instance to Unhealthy.
aws autoscaling set-instance-health --instance-id i-123abc45d --health-status healthy
You can get instance-id using curl call, the script that we place in the userdata.
If you have custom health checks, you can send the information from your health checks to Amazon EC2 Auto Scaling so that Amazon EC2 Auto Scaling can use this information. For example, if you determine that an instance is not functioning as expected, you can set the health status of the instance to Unhealthy. The next time that Amazon EC2 Auto Scaling performs a health check on the instance, it will determine that the instance is unhealthy and then launch a replacement instance.
c)Instance Warmup
With step scaling policies, you can specify the number of seconds that
it takes for a newly launched instance to warm up. Until its specified
warm-up time has expired, an instance is not counted toward the
aggregated metrics of the Auto Scaling group. While scaling out, AWS
also does not consider instances that are warming up as part of the
current capacity of the group. Therefore, multiple alarm breaches that
fall in the range of the same step adjustment result in a single
scaling activity. This ensures that we don't add more instances than
you need.
Again, the second step is not that big deal, you can control the flow using your script and start the application at the end so then it will mark healthy,
You can also try as-enter-exit-standby but I think custom health checks for warm up can do this job.
I am unable to figure out how to protect my ECS task instances when using Auto Scaling in Amazon AWS. I have a long running task that can scale out as required but I want to mark task instances that are running as "not destroyable". I have found several resources that talk about instance protection such as:
https://aws.amazon.com/blogs/aws/new-instance-protection-for-auto-scaling/
and (because I am using python) the API documentation is here:
http://boto3.readthedocs.io/en/latest/reference/services/autoscaling.html#AutoScaling.Client.set_instance_protection
This method requires an InstanceId and so I attempt to get the instance id of the current container by using a command like:
curl http://169.254.169.254/latest/meta-data/instance-id
However, this method just returns the instance id of the EC2 machine the task is running on. So my question is: Is there a way to get the instance id of a docker task instance (if that even exists)? If not is there another way I can prevent Auto Scaling from terminating a task that is still running? Do I have to write my own task manager that manages Scaling In?
To solve the same issue, we developed a simple application which is started when a job is done on one of the Auto Scaling Group (ASG) instances. This application checks the queue and if there is no job in the queue (for let's say 10 minutes or 10 times) it terminates its instance and decrements the Desired value of the ASG. This provides us with a reliable mechanism for scaling in. Scaling out on the other hand, is done by the ASG itself based on the number of jobs in the queue.
I want to start and stop an ec2 instance daily at a given time. I am using Auto scaling module for doing the same. But it is terminating the instance instead stopping (shutting down) the instance and while starting the instance each time launching a new instance. Auto scaling is taking inputs as image ID of the instance, AWS access key ID and AWS secret key. I want to start and stop same instance everyday. How can it be accomplished?
There are 2 ways in which you can achieve this. Yes auto scaling terminates the instances and doesn't stops the instances.
With Auto-Scaling :
You need to modify your code / app logic to handle the difference between stop & terminate instance. You need to make the application deployed in your EC2 instance to be stateless.
Without Auto-Scaling :
You can run a separate process / scheduled script which can run either on-premises or inside EC2 which fires the script. The Script should have the instance ID and start and stop.
PS : Looking at your scenario, I suggest to pick the "With-Auto Scaling" approach; I am not sure how it would differ or where does it affect because of the instance behavior or STOP vs. TERMINATE.
We have set up our Auto Scaling Group to scale up the number of instances when a certain load threshold is met.
The problem is that when the new instance has finished booting we have a bootstrap script that runs to configure the machine (the boot strap script launches puppet, which configures the machine accordingly).
During the run of this script (which can take a few minutes), the load on this machine is high, which causes the ASG to launch yet another machine, which in turn causes yet another instance to get created, etc. etc.
Is there a way to tell the ASG not to start collecting metrics from this machine until x amount of time has gone by (or better yet, when the boot strap script is done)?
You probably need to set the healthcheckgraceperiod higher:
Length of time in seconds after a new Amazon EC2 instance comes into service that Auto Scaling starts checking its health. During this time any health check failure for the that instance is ignored.
This is required if you are adding ELB health check. Frequently, new instances need to warm up, briefly, before they can pass a health check. To provide ample warm-up time, set the health check grace period of the group to match the expected startup period of your application.
http://docs.aws.amazon.com/AutoScaling/latest/APIReference/API_CreateAutoScalingGroup.html
In the meantime, there is such a setting for scaling policies: Instance warm-up
It tells the autoscaling system not to take newly launched instances' metrics into consideration for a specified amount of time (the warm up time).
Since it defaults to the ASG's "cooldown" time, there is correlation to that. You might need to adjust that as well to not trigger too many scaling activites just because your instances aren't ready yet. The documentation says it's not respected by target tracking or step scaling policies but I cannot confirm that in my tests and had to adjust it as well.