I want to build a script to deploy a docker container to ecs.
This is the command I am using.
ecs-cli compose --file src/main/docker/docker-compose-export.yml -p export service up
It works about 60% of the time. The other 40% of the time the command stalls.
This is the compose file
version: '2'
services:
export:
image: 1234567890lalala.dkr.ecr.eu-central-1.amazonaws.com/export:${VERSION}
cpu_shares: 200
mem_limit: 100000000
I have uploaded the image to the ecs registry before.
This is the log I am getting:
WARN[0000] Skipping unsupported YAML option... option name=networks
WARN[0000] Skipping unsupported YAML option for service... option name=networks service name=export
INFO[0000] Using ECS task definition TaskDefinition="ecscompose-export:3"
INFO[0000] Updated the ECS service with a new task definition. Old containers will be stopped automatically, and replaced with new ones desiredCount=1 serviceName=ecscompose-service-export taskDefinition="ecscompose-export:3"
INFO[0000] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
INFO[0030] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
INFO[0061] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
INFO[0091] Describe ECS Service status desiredCount=1 runningCount=1 serviceName=ecscompose-service-export
The running count goes to 2 and then back to 1 (which is expected). But then it does not stop like it does when everything works but it keeps checking the status a while and finally just stalls.
The service on the cluster is in a good state. The new docker image is running and everything is find. Its just that the command doesn't stop.
Has anyone an idea how to fix this? Are there maybe other commands that I could use to achieve the same in a more reliable fashion?
Related
I would like to launch an interactive shell into a public Docker image on my AWS ECS/Fargate cluster to run network/connectivity tests from inside the cluster.
It seems the official way to do this is with aws ecs run-task followed by aws ecs execute-command [1][2]
I'd like to use existing, public Docker Hub images rather than build custom images if possible.
If I run do run-task with no command or the default command, the task exits and execute-command won't work on an exited task.
"Essential container in task exited"
If I set a Docker command of sleep 10000, I get:
"CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \"sleep 10000\": executable file not found in $PATH: unknown",
Ideally, run-task and execute-command would be combined in one step. I don't want a background task running indefinitely, I want a shell to run a few commands interactively, that is cleaned up when I'm finished. How would I achieve this?
[1] https://aws.amazon.com/blogs/containers/new-using-amazon-ecs-exec-access-your-containers-fargate-ec2/
[2] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html
I had the same issue. I was finally able to get a container to sit "idle" with the following command inside the Task Definition:
"tail", "-F", "/dev/null"
Then I could connect in with an interactive execute-command.
Is there easy way to run an ECS Task attached or to follow the logs only while the container is Running (ie. Detach after displaying all of the logs associated)?
Using the AWS CLI (1.17.0) and ecs-cli (1.21.0), I have gotten decently close with the following two commands:
aws ecs run-task --cluster "mycluster" --task-definition testhelloworldjob --launch-type FARGATE --network-configuration etc.etc.etc.
ecs-cli logs --task-id {TASK_ID_HERE_FROM_OUTPUT_OF_PREVIOUS_COMMAND} --follow
I am currently have two issues with the above approach:
There is a race condition being that the logs are not available when the task is in a pre "running" state. Instead of ecs-cli logs waiting for the logs to exist, there is an error immediately thrown.
Even after waiting for the task to be in a running state, and issuing the ecs-cli logs the command refuses to detach even AFTER the task is finished and in a Post Running status.
For the first issue I could poll until there is a post activating/pending status, prior to calling logs. For the second issue I could draft some type of threaded call that would poll to stop the following of a log after the container in question is no longer running.... But there has to be an easier way?
To clarify I am coming from numerous other container orchestration tools/technologies that seemingly supported this very seamlessly. Here are some examples of tools and their associated commands that would yield me my intended results:
Docker CLI:
docker run hello-world
Docker-Compose Yaml:
docker-compose up
K8 Kubectl Yaml:
kubectl apply -f ./hello-k8.yaml && kubectl logs --follow hello-world
I think ecs-cli is the best option available at the moment.
Apart from that, you can change the logs driver of the AWS ECS task to syslog and then watch the logs file from the terminal after doing SSH into the EC2 container instance in which it is running.
Another thing you can do is SSH into the EC2 container instance in which it was running before and then run the container of that AWS ECS task by yourself in it using docker run and once the testing is done, you can stop and remove that container and then get that task started via AWS ECS.
Note: You can use AWS SSM Session Manager in order to avoid using EC2 key pair and adding an inbound rule for SSH.
I"m running an EC2 cluster on AWS ECS.
I launch my service like so:
ecs-cli compose -f docker-compose-base.yml -f docker-compose-prod.yml --ecs-profile root service up --create-log-groups
In my ecs-params.yml file I specified desiredCount: 2:
version: 1
task_definition:
services:
api:
desiredCount: 2
However, it always get a default desired count of 1:
INFO[0000] Using ECS task definition TaskDefinition="api:5"
WARN[0000] No log groups to create; no containers use 'awslogs'
INFO[0016] (service api) has started 1 tasks: (task decf9405-63b1-4ddf-ba12-69018299e157). timestamp="2020-05-16 12:03:46 +0000 UTC"
INFO[0077] Service status desiredCount=1 runningCount=1 serviceName=api
How do I change the default desired count without having to run service scale N command?
That's right, the default desired count is 1. You'll need to use ecs-cli compose service scale This command sets the desired count of the service to the specified count. Here's the full syntax, where n is the desired count:
ecs-cli compose service scale [--deployment-max-percent n] [--deployment-min-healthy-percent n] [--timeout value] n [--help]
And here's an example that sets desired count to 2:
ecs-cli compose --project-name hello-world --file hello-world.yml service scale 2
I have recently changed AMI on which my ECS EC2 instances are running from Amazon Linux to Amazon Linux 2 (in both cases I am using ECS optimized images). I am deploying my instances using cloudformation and having a real headache as those new instances sometimes are being run successfully and sometimes not (same stack, no updates, same code).
On the failed instances I see that there is an issue with ECS service itself after executing ecs-logs-collector.sh I see in ecs file log "warning: The Amazon ECS Container Agent is not running". Also directory "/var/log/ecs" doesn't even exist!.
I have correct IAM role attached to an instance.
Also as mentioned, it is the same code being run, and on 75% of attempts it fails with ECS service, I have no more ideas, where else to look for some issues/logs/errors.
AMI: ami-0650e7d86452db33b (eu-central-1)
Solved. If someone will fall into this issue adding this to my userdata helped:
cp /usr/lib/systemd/system/ecs.service /etc/systemd/system/ecs.service
sed -i '/After=cloud-final.service/d' /etc/systemd/system/ecs.service
systemctl daemon-reload
I have pushed the initial docker image into repository and created AWS Fargate using the image, is there any way to update the image as there are certain changes in my docker image.
As simple as:
aws ecs update-service --cluster <cluster> --service <service> --force-new-deployment
See AWS documentation:
If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service.
Note
If your updated Docker image uses the same tag as what is in the existing task definition for your service (for example, my_image:latest), you do not need to create a new revision of your task definition. You can update the service using the procedure below, keep the current settings for your service, and select Force new deployment. The new tasks launched by the deployment pull the current image/tag combination from your repository when they start. The Force new deployment option is also used when updating a Fargate task to use a more current platform version when you specify LATEST. For example, if you specified LATEST and your running tasks are using the 1.0.0 platform version and you want them to relaunch using a newer platform version.
https://docs.aws.amazon.com/AmazonECS/latest/userguide/update-service.html#update-service
Create a new version of the task definition and update the container with latest labels and update the service.
You can write the configuration file once you created a cluster using the default-launch-type as FARGATE for your application and define the respective parameters in your task definition i.e. ecs-params.yaml
Here is one file for the nginx:latest image which is stored in Amazon ECR.
version: '2'
services:
web:
image: account-id.dkr.ecr.ap-southeast-1.amazonaws.com/nginx:latest
ports:
- "80:80"
logging:
driver: awslogs
options:
awslogs-group: awslogs-web
awslogs-region: ap-southeast-1
awslogs-stream-prefix: web-nginx
You simply change the image and you could get the updated image into your deployment as you update the service inside your cluster.
If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service. The service scheduler uses the minimum healthy percent and maximum percent parameters (in the service's deployment configuration) to determine the deployment strategy.
Note: The Execution Role in task definition gives permissions to pull the images from container registry.
You could find the doc guide here, AWS ECS Update Service