AWS Fargate can be used either with AWS Batch or with ECS. In both cases, you provide Fargate with a Docker Image, it executes it, and then stops. Hence, could someone help me understand the advantages/disadvantages of using either of these services?
Related
I have a docker image. I would like to create a container periodically and execute as a job, say every 1 hour, by creating CloudWatch Rule.
As we are using AWS cloud, I am looking at the AWS Batch service. Interestingly there is also a ECS Scheduled task.
What is the difference between these 2?
Note: I have an init container - that is I have 2 docker containers to run one after another. It seems to be possible with ECS Scheduled Task. But not with Batch.
AWS Batch is for batch jobs, such as processing numerous images or videos in parallel (one container per image/video). This is mostly useful in batch-type workloads for research purposes.
AWS Batch is based on ECS (also supports EC2), and it allows you to simply to run your containers. It does not have specific use-case, it is more generic. If you don't have batch-type projects, then ECS would be probably better choice for you.
The other answers are spot on. I just wanted to add that we (AWS container team) ran a session at re:Invent last year that covered these options and provided hints about when using one over the other. The session covers the relationship between ECS, EC2 and Fargate (something that is often missed) as well as when to use "raw" ECS, Vs Step Functions Vs Batch as an entry point for running your batch jobs. This is the link to the session.
If you want to run two containers in sequence, using AWS Fargate, then you probably want to orchestrate it with AWS Step Functions. Step Functions will allow you to call arbitrary tasks in serial, and it has direct integration with AWS Fargate.
Amazon EventBridge Rule (hourly) ----- uses AWS IAM role to gain permission to trigger Step Functions
|
| triggers
|
AWS Step Functions ----- Uses AWS IAM role to gain permission to trigger Fargate
|
| triggers
|
AWS Fargate (Amazon ECS) Task Definition
AWS Batch is designed for data processing tasks that need to scale out across many nodes. If your use case is simply to spin up a couple of containers in sequence, then AWS Batch will be overkill.
CloudWatch Event Rules
FYI CloudWatch Event Rules still work, but the service has been rebranded as Amazon EventBridge. I'd recommend using the Amazon EventBridge console and APIs instead of Amazon CloudWatch Events APIs going forward.
I want to know what is the best way to run a scheduled task on AWS. Specifically I would like to only pay for the computation that is needed to run the task. So if the tasks runs once per day for 2h, then I only will pay for the 2h computation. So I don't want to have an EC2 instance running all the time, when the task is not running.
Could an AWS expert please explain how to realize this on AWS?
AWS ECS Scheduled Tasks is perfect solution for you. You can run ECS Tasks on top on AWS Fargate, so, you don't need to provision any EC2 instances. It's complete Serverless solution with simple type of configuration.
Related information:
AWS ECS
AWS Fargate
Scheduling Amazon ECS Tasks
Also you can just run and stop EC2 instances by AWS CloudWatch Event (cron scheduler)
I used ECS Fargate and it provides containerization, auto-scaling based on request count, CPU and Memory.
It is working as expected.
I start to explore the AWS EKS feature and I didn't see any advantage in using this as all are provided by ECS Fargate.
Could someone help me understand where to use ECS Fargate and Where to use AWS EKS?
Anyhelp is appreciated.
Thanks,
Harry
You would use AWS EKS if you want to use Kubernetes.
Since Kubernetes is a standard, you could in theory move your application from AWS EKS to other cloud providers like Azure, Google Cloud, or DigitalOcean easily since they all support Kubernetes.
If you don't care about Kubernetes then I find that AWS ECS with the AWS Fargate [Serverless compute for containers] deployment type is currently the easiest method of running Docker containers on AWS.
Note that Amazon is actively working on adding the Fargate deployment type to the EKS service.
I would check back after the AWS re:invent conference next month to see how things have changed in this area.
We hear these questions often and I tried to capture some of the core principles of these comparisons/positioning in this blog post.
Has anyone been able to configure selenoid on aws ecs ? I am able to run the selenoid-ui container but the selenoid hub image keeps throwing an error regarding the browsers.json however I have not been able to find a way to add the browsers.json file because it stops before it executes the CMD command
There is no point to run selenoid on AWS ECS, as your setup won't scale (your browser containers will be launched on the same EC2 instance where your selenoid container is running). With ECS, you run your service on a cluster, so either your cluster contains on 1 EC2 instance, or you waste your compute resources.
If you don't need scaling, I'd suggest you run selenoid on simple EC2 instance with docker installed. If you do want to have scaling, then I suggest you to take a look at a commercial version of selenoid (called Moon), which you can run on AWS EKS.
I am trying to migrate a set of microservices from Docker Swarm, to AWS ECS using Fargate.
I have created an ECS cluster. Moreover, I have initialized repositories using the ECR, each of which contains an image of a microservice.
I have successfully came up with a way to create new images, and push them into the ECR. In fact, with each change in the code, a new docker image is built, tagged, and pushed.
Moreover, I have created a task definition that is linked to a service. This task definition contains one container, and all the necessary information. Moreover, its service defines that the task will run in a VPC, and is linked to a load balancer, and has a target group. I am assuming that every new deployment uses the image with the "latest" tag.
So far with what I have explained, everything is clear and is working well.
Below is the part that is confusing me. After every new build, I would like to update the service in order for new tasks with the update image get deployed. I am using the cli to do so with the following command:
aws ecs update-service --cluster <cluster-name> --service <service-name>
Typically, after performing the command, I am monitoring the deployment logs, under the event tab, and checking the state of the service using the following command:
aws ecs describe-services --cluster <cluster-name> --service <service-name>
Finally, I tried to simulate a case where the newly created image contains a bad code. Thus, the new tasks will not be able to get deployed. What I have witnessed is that Fargate will keep trying (without stopping) to deploy the new tasks. Moreover, aside the event logs, the describe-services command does not contain relevant information, other than what Fargate is doing (e.g., registering/deregistering tasks). I am surprised that I could not find any mechanism that instructs Fargate, or the service to stop the deployment and rollback to the already existing one.
I found this article (https://aws.amazon.com/blogs/compute/automating-rollback-of-failed-amazon-ecs-deployments/ ), which provides a solution. However, it is a fairly complicated one, and assumes that each new deployment is triggered by a new task definition, which is not what I want.
Therefore, considering what I have described above, I hope you can answer the following questions:
1) Using CLI commands (For automation purposes) Is there a way to instruct Fargate to automatically stop the current deployment, after failing to deploy new tasks after a few tries?
2) Using the CLI commands, is there a way to monitor the current status of the deployment? For instance, when performing a service update on a service on Docker swarm, the terminal generates live logs on the update process
3) After a failed deployment, is there a way for Fargate to signal an error code, or flag, or message?
At the moment, ECS does not offer deployment status directly. Once you issue a deployment, there is no way to determine its status other than to continually poll for updates until you have enough information to infer from them. Plus unexpected container exits are not logged anywhere. You have to search through failed tasks. The way I get them is by cloudwatch rule that triggers a lambda upon task state change.
I recommend you read: https://medium.com/#aaron.kaz.music/monitoring-the-health-of-ecs-service-deployments-baeea41ae737
As of now, you have a way to do this:
aws ecs wait services-stable --cluster MyCluster --services MyService
The previous example pauses and continues only after it can confirm that the service running on the cluster is stable. Will return 255 exit code after 40 failed checks.
To cancel a deployment, enable ECS Circuit Breaker when creating your service:
aws ecs create-service \
--service-name MyService \
--deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}" \
{...}
References:
Service deployment check.
Circuit Breaker