I"m running an EC2 cluster on AWS ECS.
I launch my service like so:
ecs-cli compose -f docker-compose-base.yml -f docker-compose-prod.yml --ecs-profile root service up --create-log-groups
In my ecs-params.yml file I specified desiredCount: 2:
version: 1
task_definition:
services:
api:
desiredCount: 2
However, it always get a default desired count of 1:
INFO[0000] Using ECS task definition TaskDefinition="api:5"
WARN[0000] No log groups to create; no containers use 'awslogs'
INFO[0016] (service api) has started 1 tasks: (task decf9405-63b1-4ddf-ba12-69018299e157). timestamp="2020-05-16 12:03:46 +0000 UTC"
INFO[0077] Service status desiredCount=1 runningCount=1 serviceName=api
How do I change the default desired count without having to run service scale N command?
That's right, the default desired count is 1. You'll need to use ecs-cli compose service scale This command sets the desired count of the service to the specified count. Here's the full syntax, where n is the desired count:
ecs-cli compose service scale [--deployment-max-percent n] [--deployment-min-healthy-percent n] [--timeout value] n [--help]
And here's an example that sets desired count to 2:
ecs-cli compose --project-name hello-world --file hello-world.yml service scale 2
Related
I'm using ECS through ecs-cli to deploy my API.
I start by launching a cluster of spot instances using this command :
sudo ecs-cli up --region MY REGION --keypair MY KEY PAIR --instance-type t2.micro --capability-iam --size 1 --cluster MY CLUSTER NAME --spot-price 0.01
Then, using the following docker-compose.yml and ecs-params.yml files :
version: '3'
services:
selenium:
image: selenium/standalone-chrome
...etc
api:
image: myapithatusesselenium/myapithatusesselenium
ports:
- 3000:3000
links:
- selenium
...etc
version: 1
task_definition:
task_execution_role: ROLE ID
services:
selenium:
cpu_shares: 600
mem_limit: 700000000
api:
repository_credentials:
credentials_parameter: REPO CREDENTIALS
cpu_shares: 400
mem_limit: 300000000
I'm deploying a service with a load balancer using this command :
sudo ecs-cli compose --file docker-compose.yml --ecs-params ecs-params.yml --project-name MY PROJECT NAME service up --cluster MY CLUSTER NAME --target-group-arn LOAD BALANCER RESSOURCE ID --container-name api --container-port 3000
So, When my API is under a lot of load (When it starts notifying me that the API is going down) I add additional instances by scaling using these commands:
# 1 - scale the number of ec2 instances in the cluster
sudo ecs-cli scale --size 3 --capability-iam
# 2 - scale the number of tasks
sudo ecs-cli compose --file docker-compose.yml --project-name MY PROJECT NAME service scale 3
As you can see the number of tasks and ec2 instances is the same because each container can handle a single task.
When there isn't a lot of load I reduce the size again.
What I need right now is a way to make this automatic (Auto scaling in and out). I can't figure out how to do that.
Thank you !
ECS doesn't nativly autoscaling. You have to use the application autoscaling service for that. You'll need to use the regular aws CLI and call register-scalable-target, and then create a scaling policy
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html
https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/register-scalable-target.html
https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scaling-policy.html
I have simple query : what is the best way to simply recycle/reboot a service having 2 tasks using AWS ECS console without any actual change being deployed ?
Currently I need to update service and set tasks count from 2 to 0 and wait for tasks to drain out. Then I will set tasks count from 0 to 2 to bring it up. This is how recycle/reboot 2 tasks of a service.
I need to do this sometimes due to internal app error and just want to reboot them without any actual change which resolves my problem.
AWS provides one option (Force new deployment checkbox) which is not helping and it works if there is a change in image ? Wish AWS could provide one option as "Recycle a service(tasks)" which will start 2 new tasks and drain out 2 existing tasks.
What could be the best and easiest way do it using AWS Console or even AWS API/CLI ?
If you stop the tasks, ECS will launch new ones to satisfy the desired count. That's fairly easy in the ECS console, just select the tasks in the list of tasks and choose Stop in the Action dropdown.
Using the aws CLI you can get a list of the tasks to kill using:
aws ecs list-tasks --service-name my-service
to delete each task use:
aws ecs stop-task --task %1
where %1 is the ARN of the task as provided by the first command.
Here is a command that combines both commands above. It will kill all the tasks of a given service:
SVC=your-service-name-here
aws ecs list-tasks --service-name $SVC --output text | cut -f2 | perl -ne 'system("aws ecs stop-task --task $_")'
Haven't been able to find this in docs. Can I just pause an ECS service so it stopped creating new tasks? Or do I have to delete it to stop that behavior?
I just want to temporarily suspend it from creating new tasks on the cluster
It is enough to set the desired number of tasks for a service to 0.
ECS will automatically remove all running tasks.
aws ecs update-service --desired-count 0 --cluster "ecs-my-ClusterName" --service "service-my-ServiceName-117U7OHVC5NJP"
You can accomplish a "pause" by adjusting your service configuration to match your current number of running tasks. For example, if you currently have 3 running tasks in your service, you'd configure the service as below:
This tells the service:
The number of tasks I want is [current-count]
I want you to maintain at least [current-count]
I don't want more than [current-count
These combined effectively halt your service from making any changes.
The accepted answer is incorrect.
If you set both "Minimum healthy percent" and "Maximum healthy percent" to 100, AWS will give you an error similar to following:
To stop service from creating new tasks, you have to update service by updating task definition and setting desired number of tasks to 0. After that you can use AWS CLI (fastest option) to stop existing running tasks , for example:
aws ecs list-services --cluster "ecs-my-ClusterName"
aws ecs list-tasks --cluster "ecs-my-ClusterName" --service "service-my-ServiceName-117U7OHVC5NJP"
After that you will get the list of the running tasks for the service, such as:
{
"taskArns": [
"arn:aws:ecs:us-east-1:XXXXXXXXXXX:task/12e13d93-1e75-4088-a7ab-08546d69dc2c",
"arn:aws:ecs:us-east-1:XXXXXXXXXXX:task/35ed484a-cc8f-4b5f-8400-71e40a185806"
]
}
Finally use below to stop each task:
aws ecs stop-task --cluster "ecs-my-ClusterName" --task 12e13d93-1e75-4088-a7ab-08546d69dc2c
aws ecs stop-task --cluster "ecs-my-ClusterName" --task 35ed484a-cc8f-4b5f-8400-71e40a185806
UPDATE: By setting the desired number of running tasks to 0, ECS will stop and drain all running tasks in that service. There is no need to stop them individually afterwards using CLI commands originally posted above.
I have pushed the initial docker image into repository and created AWS Fargate using the image, is there any way to update the image as there are certain changes in my docker image.
As simple as:
aws ecs update-service --cluster <cluster> --service <service> --force-new-deployment
See AWS documentation:
If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service.
Note
If your updated Docker image uses the same tag as what is in the existing task definition for your service (for example, my_image:latest), you do not need to create a new revision of your task definition. You can update the service using the procedure below, keep the current settings for your service, and select Force new deployment. The new tasks launched by the deployment pull the current image/tag combination from your repository when they start. The Force new deployment option is also used when updating a Fargate task to use a more current platform version when you specify LATEST. For example, if you specified LATEST and your running tasks are using the 1.0.0 platform version and you want them to relaunch using a newer platform version.
https://docs.aws.amazon.com/AmazonECS/latest/userguide/update-service.html#update-service
Create a new version of the task definition and update the container with latest labels and update the service.
You can write the configuration file once you created a cluster using the default-launch-type as FARGATE for your application and define the respective parameters in your task definition i.e. ecs-params.yaml
Here is one file for the nginx:latest image which is stored in Amazon ECR.
version: '2'
services:
web:
image: account-id.dkr.ecr.ap-southeast-1.amazonaws.com/nginx:latest
ports:
- "80:80"
logging:
driver: awslogs
options:
awslogs-group: awslogs-web
awslogs-region: ap-southeast-1
awslogs-stream-prefix: web-nginx
You simply change the image and you could get the updated image into your deployment as you update the service inside your cluster.
If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service. The service scheduler uses the minimum healthy percent and maximum percent parameters (in the service's deployment configuration) to determine the deployment strategy.
Note: The Execution Role in task definition gives permissions to pull the images from container registry.
You could find the doc guide here, AWS ECS Update Service
I want to build a script to deploy a docker container to ecs.
This is the command I am using.
ecs-cli compose --file src/main/docker/docker-compose-export.yml -p export service up
It works about 60% of the time. The other 40% of the time the command stalls.
This is the compose file
version: '2'
services:
export:
image: 1234567890lalala.dkr.ecr.eu-central-1.amazonaws.com/export:${VERSION}
cpu_shares: 200
mem_limit: 100000000
I have uploaded the image to the ecs registry before.
This is the log I am getting:
WARN[0000] Skipping unsupported YAML option... option name=networks
WARN[0000] Skipping unsupported YAML option for service... option name=networks service name=export
INFO[0000] Using ECS task definition TaskDefinition="ecscompose-export:3"
INFO[0000] Updated the ECS service with a new task definition. Old containers will be stopped automatically, and replaced with new ones desiredCount=1 serviceName=ecscompose-service-export taskDefinition="ecscompose-export:3"
INFO[0000] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
INFO[0030] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
INFO[0061] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
INFO[0091] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
The running count goes to 2 and then back to 1 (which is expected). But then it does not stop like it does when everything works but it keeps checking the status a while and finally just stalls.
The service on the cluster is in a good state. The new docker image is running and everything is find. Its just that the command doesn't stop.
Has anyone an idea how to fix this? Are there maybe other commands that I could use to achieve the same in a more reliable fashion?