In Jenkins there are two similar plugins available:
Both are linked to the same Jenkins wiki page
I haven't found any documentation for the scalable version of the plugin and I have the following question:
Is it possible to add ECS instances in cluster from 0(none) to 1 using this plugin?
I want to have active ECS instances only when there are jobs to be done.
I will appreciate any help.
Try to uninstall these plugins and compile ecs-slave plugin manually from the branch autoscaling https://github.com/cbamelis/amazon-ecs-plugin
I found a workaround to scale out the number of ECS instances from zero.
I created a new job with the following shell code:
result=$(aws ecs list-container-instances --cluster ${cluster-name} | grep -c arn:aws:ecs:${aws-region}) || true
if [ "$result" = '0' ]
then aws autoscaling set-desired-capacity --auto-scaling-group-name ${asg-name} --desired-capacity 1
else
echo "Container already exists"
fi
replace variables ${cluster-name}, ${aws-region}, ${asg-name} with actual values
This job increases the number of ECS VMs to 1 if it was 0.
Scaling in can be done using Cloudwatch alarm and Autoscaling policy.
Related
I have this bash script:
#!/bin/bash
myClusterId="myCluster"
for service in $(aws ecs list-services --cluster $myClusterId --query "serviceArns[*]" | jq -r 'to_entries[] | .value | sub(".*/";"")'); do
for task in $( aws ecs list-tasks --cluster $myClusterId --service-name $service --desired-status 'RUNNING' --no-paginate --output text --query 'taskArns[*]' ) ; do
aws ecs stop-task --cluster $myClusterId --task $task --reason "Restarted using bash script" > /dev/null 2>&1
done
done
In short, it will restart all my ECS Fargate tasks under myCluster (excluding scheduled tasks triggered by CloudWatch Rules). It's working fine so far.
All my services have minHealthyPercent set to 100 and maxHealthyPercent set to 200. But, I noticed that it didn't keep any healthy tasks during the restart process. All tasks get killed immedietly and my load balancer throws 503 Service Temporarily Unavailable error when new tasks are in pending/provisioning state.
Am I missing something in my script? How do I correctly perform no-downtime services restart process using AWS CLI?
The parameters maximumPercent and minimumHealthyPercent are only used during rolling updates of your ECS service:
The number of tasks that Amazon ECS adds or removes from the service during a rolling update is controlled by the deployment configuration. A deployment configuration consists of the minimum and maximum number of tasks allowed during a service deployment.
Restarting a task is not considered as a new deployment.
To rectify the issue, there are few choices:
include a sleep in your for loop. Its the crudest way, but fastest to implement for testing.
use describe-tasks in the for loop to pull the state of the task just terminated. Proceed with restarting next task only when the state of the most recently restarted one will be RUNNING.
I think your best option would be to do a blue/green deployment through CodeDeploy, assuming you use an Elastic Load Balancer. The blue/green deployment will automatically detect any error and stop the deployment if required.
https://aws.amazon.com/blogs/devops/use-aws-codedeploy-to-implement-blue-green-deployments-for-aws-fargate-and-amazon-ecs/
I have simple query : what is the best way to simply recycle/reboot a service having 2 tasks using AWS ECS console without any actual change being deployed ?
Currently I need to update service and set tasks count from 2 to 0 and wait for tasks to drain out. Then I will set tasks count from 0 to 2 to bring it up. This is how recycle/reboot 2 tasks of a service.
I need to do this sometimes due to internal app error and just want to reboot them without any actual change which resolves my problem.
AWS provides one option (Force new deployment checkbox) which is not helping and it works if there is a change in image ? Wish AWS could provide one option as "Recycle a service(tasks)" which will start 2 new tasks and drain out 2 existing tasks.
What could be the best and easiest way do it using AWS Console or even AWS API/CLI ?
If you stop the tasks, ECS will launch new ones to satisfy the desired count. That's fairly easy in the ECS console, just select the tasks in the list of tasks and choose Stop in the Action dropdown.
Using the aws CLI you can get a list of the tasks to kill using:
aws ecs list-tasks --service-name my-service
to delete each task use:
aws ecs stop-task --task %1
where %1 is the ARN of the task as provided by the first command.
Here is a command that combines both commands above. It will kill all the tasks of a given service:
SVC=your-service-name-here
aws ecs list-tasks --service-name $SVC --output text | cut -f2 | perl -ne 'system("aws ecs stop-task --task $_")'
I wonder if that would be possible to temporary stop the worker VM instances so they are not running at night time when I am not working on a cluster development. So far the only way I am aware of to "stop" the instances from running is to delete the cluster itself which I don't want to do. Any suggestions are highly appreciated.
P.S. Edited later
The cluster was created following steps outlined in this guide.
I'm just learning myself but this might help. If you have eksctl installed, you can use it from the command line to scale your cluster. I scale mine down to the min size when I'm not using it:
eksctl get cluster
eksctl get nodegroup --cluster CLUSTERNAME
eksctl scale nodegroup --cluster CLUSTERNAME --name NODEGROUPNAME --nodes NEWSIZE
To completely scale down the nodes to zero use this (max=0 threw errors):
eksctl scale nodegroup --cluster CLUSTERNAME --name NODEGROUPNAME --nodes 0 --nodes-max 1 --nodes-min 0
Go to EC2 instances dashboard of your Node Group and from right panel in bottom click on Auto Scaling Groups then select your group by click on checkbox and click edit button and change Desired, Min & Max capacities to 0
Edit the autoscaling group and set the instances to 0.
This will shut down all worker nodes.
Now you can use AWS Automation to schedule a repetitive action through automation documents that will be stopping/starting at given periods of time.
You can't stop the master nodes as they are managed by AWS.
Take a look at the kube-downscaler which can be deployed to cluster to scale in and out the deployments based on time of day.
More cost reduction techniques in this blog.
Haven't been able to find this in docs. Can I just pause an ECS service so it stopped creating new tasks? Or do I have to delete it to stop that behavior?
I just want to temporarily suspend it from creating new tasks on the cluster
It is enough to set the desired number of tasks for a service to 0.
ECS will automatically remove all running tasks.
aws ecs update-service --desired-count 0 --cluster "ecs-my-ClusterName" --service "service-my-ServiceName-117U7OHVC5NJP"
You can accomplish a "pause" by adjusting your service configuration to match your current number of running tasks. For example, if you currently have 3 running tasks in your service, you'd configure the service as below:
This tells the service:
The number of tasks I want is [current-count]
I want you to maintain at least [current-count]
I don't want more than [current-count
These combined effectively halt your service from making any changes.
The accepted answer is incorrect.
If you set both "Minimum healthy percent" and "Maximum healthy percent" to 100, AWS will give you an error similar to following:
To stop service from creating new tasks, you have to update service by updating task definition and setting desired number of tasks to 0. After that you can use AWS CLI (fastest option) to stop existing running tasks , for example:
aws ecs list-services --cluster "ecs-my-ClusterName"
aws ecs list-tasks --cluster "ecs-my-ClusterName" --service "service-my-ServiceName-117U7OHVC5NJP"
After that you will get the list of the running tasks for the service, such as:
{
"taskArns": [
"arn:aws:ecs:us-east-1:XXXXXXXXXXX:task/12e13d93-1e75-4088-a7ab-08546d69dc2c",
"arn:aws:ecs:us-east-1:XXXXXXXXXXX:task/35ed484a-cc8f-4b5f-8400-71e40a185806"
]
}
Finally use below to stop each task:
aws ecs stop-task --cluster "ecs-my-ClusterName" --task 12e13d93-1e75-4088-a7ab-08546d69dc2c
aws ecs stop-task --cluster "ecs-my-ClusterName" --task 35ed484a-cc8f-4b5f-8400-71e40a185806
UPDATE: By setting the desired number of running tasks to 0, ECS will stop and drain all running tasks in that service. There is no need to stop them individually afterwards using CLI commands originally posted above.
I am trying to restart an AWS service (basically stop and start all tasks within the service) without making any changes to the task definition.
The reason for this is because the image has the latest tag attached with every build.
I have tried stopping all tasks and having the services recreate them but this means that there is some temporarily unavailable error when the services are restarting in my instances (2).
What is the best way to handle this? Say, A blue-green deployment strategy so that there is no downtime?
This is what I have currently. It'shortcomings is that my app will be down for a couple of seconds as the service's tasks are being rebuilt after deleting them.
configure_aws_cli(){
aws --version
aws configure set default.region us-east-1
aws configure set default.output json
}
start_tasks() {
start_task=$(aws ecs start-task --cluster $CLUSTER --task-definition $DEFINITION --container-instances $EC2_INSTANCE --group $SERVICE_GROUP --started-by $SERVICE_ID)
echo "$start_task"
}
stop_running_tasks() {
tasks=$(aws ecs list-tasks --cluster $CLUSTER --service $SERVICE | $JQ ".taskArns | . []");
tasks=( $tasks )
for task in "${tasks[#]}"
do
[[ ! -z "$task" ]] && stop_task=$(aws ecs stop-task --cluster $CLUSTER --task "$task")
done
}
push_ecr_image(){
echo "Push built image to ECR"
eval $(aws ecr get-login --region us-east-1)
docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/repository:$TAG
}
configure_aws_cli
push_ecr_image
stop_running_tasks
start_tasks
Use update-service and the --force-new-deployment flag:
aws ecs update-service --force-new-deployment --service my-service --cluster cluster-name
Hold on a sec.
If I understood you usecase correctly, this is addressed in the official docs:
If your updated Docker image uses the same tag as what is in the existing task definition for your service (for example, my_image:latest), you do not need to create a new revision of your task definition. You can update the service using the procedure below, keep the current settings for your service, and select Force new deployment....
To avoid downtime, you should manipulate 2 parameters: minimum healthy percent and maximum percent:
For example, if your service has a desired number of four tasks and a maximum percent value of 200%, the scheduler may start four new tasks before stopping the four older tasks (provided that the cluster resources required to do this are available). The default value for maximum percent is 200%.
This basically mean, that regardless of whether your task definition changed and to what extent, there can be an "overlap" between the old and the new ones, and this is the way to achieve resilience and reliability.
UPDATE:
Amazon has just introduced External Deployment Controllers for ECS(both EC2 and Fargate). It includes a new level of abstraction called TaskSet. I haven't tried it myself yet, but such fine grain control over service and task management(both APIs are supported) can potentially solve the problem akin this one.
After you push your new image to your Docker repository, you can create a new revision of your task definition (it can be identical to the existing task definition) and update your service to use the new task definition revision. This will trigger a service deployment, and your service will pull the new image from your repository.
This way your task definition stays the same (although updating the service to a new task definition revision is required to trigger the image pull), and still uses the "latest" tag of your image, but you can take advantage of the ECS service deployment functionality to avoid downtime.
The fact that I have to create a new revision of my task definition every time even when there is no change in the task definition itself is not right.
There are a bunch of crude bash implementations on this which means that AWS should have the ECS service scheduler listen for changes/updates in the image, especially for an automated build process.
My crude work-around to this was have two identical task definitions and switch between them for every build. That way I don't have redundant revisions.
Here is the specific script snippet that does that.
update_service() {
echo "change task definition and update service"
taskDefinition=$(aws ecs describe-services --cluster $CLUSTER --services $SERVICE | $JQ ".services | . [].taskDefinition")
if [ "$taskDefinition" = "$TASK_DEF_1" ]; then
newDefinition="$TASK_DEF_2"
else
newDefinition="$TASK_DEF_1"
fi
rollUpdate=$(aws ecs update-service --cluster $CLUSTER --service $SERVICE --task-definition $newDefinition)
}
Did you have this question solved? Perhaps this will work for you.
With a new release image pushed to ECR with a version tag, i.e. v1.05, and the latest tag, the image locator in my task definition needed to be explicitly updated to have this version tag postfixed like :v1.05.
With :latest, this new image did not get pulled by the new container after aws ecs update-service --force-new-deployment --service my-service.
I was doing tagging and pushing like this:
docker tag ${imageId} ${ecrRepoUri}:v1.05
docker tag ${imageId} ${ecrRepoUri}:latest
docker push ${ecrRepoUri}
...where as this is the proper way of pushing multiple tags:
docker tag ${imageId} ${ecrRepoUri}
docker push ${ecrRepoUri}:v1.05
docker push ${ecrRepoUri}:latest
This was briefly mentioned in the official docs without a proper example.
Works great https://github.com/fdfk/ecsServiceRestart
python ecsServiceRestart.py restart --services="app app2" --cluster=test
The quick and dirty way:
login to EC2 instance running the task
find your container with docker container list
use docker restart [container]