I have an application running using docker-compose.
Now I'm migrating the application to be hosted on ECS.
I'm translating the docker-compose settings to the boto3 ECS equivalents.
Unfortunately I don't find an equivalent of docker-compose's command in the AWS CLI.
You can use container transform with boto3, that will convert docker-compose to equivalent ECS task definition. this is also base on python.
container-transform is a small utility to transform various docker container formats to one another.
Currently, container-transform can parse and convert:
Kubernetes Pod specs
ECS task definitions
Docker-compose configuration files
Marathon Application Definitions or Groups of Applications
Chronos Task Definitions
container-transform
cat docker-compose.yml | container-transform -v
compose-to-ecs
Also suggested tool by AWS ECS road map.
we're unlikely to support the docker-compose format directly in our APIs. But, would a tool like container-transform to transform a docker-compose file into an ECS task definition work for you? Then you can use the resulting ECS task definition file in boto.
ECS does not contain a docker-compose command. Instead you will specify a task definition file that contains all the definitions of a service and the containers that reside within it.
The ECS service will then deploy this based on the task definition, you simply define parameters such as how many of these tasks are operating at once.
You can however use the ecs-cli tool to perform this migration for you, using the ecs-cli compose command it can take the docker-compose file and perform those translations.
Take a look at the Using Docker Compose File Syntax page to see which parameters are supported from a docker-compose file.
You can also use ECS ComposeX which will allow you to keep using your docker-compose definitions as they exist for local purposes as it does not introduce any unsupported extensions for docker-compose, but will also allow you to defined RDS/DocDB/DynamoDB/Kinesis and plenty other options that you can automatically link to your services.
When ready, ComposeX will transform all of that in CFN templates, containing AWS ECS definitions and all necessary resources, that are logically linked to work together but equally self-sufficient (so you can deploy things separately, like DBs for example).
All templates are automatically parsed and validated through cloudformation API (to the best of its abilities).
It is purposely aimed at working with AWS services, and follow all best practices, including allowing you to define least-privileged access from/to services and AWS resources.
It supports autoscaling, creation/use of existing ECS clusters and is aimed to make workloads primarly on Fargate but also on EC2 instances.
Related
I need to deploy an application in AWS using ECS Fargate. This application has multiple services and a docker-compose file. I see there are two main ways to do this:
Using Docker's Context ECS cli, the official docs I found: Docker doc and AWS doc.
Using Amazon's ECS cli as described here.
I am trying to understand the following but didn't find any comparison on the web:
Which are the advantages/disadvantages of each way?
Can the same result be achieved with both options, or is there something one can do that the other can't?
What should I take in consideration when I choose one?
Thanks,
So I've been trying this this past week, both approaches, and here's what I have found.
ecs-cli and docker support different sets of tags for nontrivial features, even things such as how much CPU and memory your container needs.
For example docker wants the deploy config in a deploy tag under the service. See https://docs.docker.com/compose/compose-file/deploy/#memory
However ECS-CLI wants things in a acs-params.yml. There are almost no examples out there other than trivial ones though, and the ones published don't actually work with the current tooling. What they publish doesn't work.
Docker ECS integration works and handles tons of details for you, including VPC, subnet creation, LBs, security groups, everything. This is an amazing part.
ecs-cli offers tons more options than Docker CLI but you need to do a lot more work yourself, manual security group config, etc.
I was never able to get ecs-cli to really work. It kept choking on cpu config, and what was written in AWS docs did not actually work\
docker compose logs doesn't work
Overall neither CLI seems to be in production shape, but docker one seems to be far ahead of where the ecs-cli is IMO.
I have multi container application and using docker-compose.yml file to start the application. I'm planning to use AWS Elastic Container Service for deployment. From the tutorials, I came across to create a container definitions in task definitions instead of using user defined compose file in AWS ECS console.
How to use existing docker-compose.yml file in AWS Elastic Container Service?
It does seem that there is support to use the ECS CLI to use a docker-compose file using the ecs-cli-compose command.
This will translate your docker-compose file and create a task definition that is able to be used with ECS
The task definition in ECS is a mapping file for all container based configuration. Within this file you can define the container definitions for each container within the task, define port mappings, any custom commands, volume mounting etc.
I'm fairly new to AWS ECS and am unsure if you are supposed to commit into version control some things that AWS already hosts for you. Specifically, should task-definitions created on AWS be committed to GitHub? Or do we just use AWS ECS/ECR as the version control FOR ECS Task definitions?
First ECS/ECR not used for version control like GithHub.
ECS is container management service where ECR is nothing but docker registry on AWS.
Task definition used to run docker container in ECS.You can create task definition on or after creating ECS cluster.You can create/modify task definition in many ways like console,aws cli, aws cloud formation,terraform etc. It depends on how you want to do this and how frequently you change task definition. yes, you can keep your task definition GitHub and create automated job to execute every time from there but there is no need to store task definition anywhere once your task running.
ECR is Elastic container registry which is used to store the container images.
ECS Task definition A task definition is required to run Docker containers in Amazon ECS. Some of the parameters you can specify in a task definition & include: The Docker images to use with the containers in your & task.
You have to provide your ECR URI while creating task definition or it will look for the default docker hub registry for the container image.
Also, You can keep your task definition json on any version control system if you want to use them lateron.
I have a basic django/postgres app running locally, based on the Docker Django docs. It uses docker compose to run the containers locally.
I'd like to run this app on Amazon Web Services (AWS), and to deploy it using the command line, not the AWS console.
My Attempt
When I tried this, I ended up with:
this yml config for ecs-cli
these notes on how I deployed from the command line.
Note: I was trying to fire up the Python dev server in a janky way, hoping that would work before I added nginx. The cluster (RDS+server) would come up, but then the instances would die right away.
Issues I Then Failed to Solve
I realized over the course of this:
the setup needs another container for a web server (nginx) to run on AWS (like this blog post, but the tutorial uses the AWS Console, which I wanted to avoid)
ecs-cli uses a different syntax for yml/json config than docker-compose, so you need to have some separate/similar code from your local docker.yml (and I'm not sure if my file above was correct)
Question
So, what ecs-cli commands and config do I use to deploy the app, or am I going about things all wrong?
Feel free to say I'm doing it all wrong. I could also use Elastic Beanstalk - the tutorials on this don't seem to use docker/docker-compose, but seem easier overall (at least well documented).
I'd like to understand why any given approach is a good way to do this.
One alternative you may wish to consider in lieu of ECS, if you just want to get it up in the amazon cloud, is to make use of docker-machine using the amazonec2 driver.
When executing the docker-compose, just ensure the remote Amazon host machine is ACTIVE which can be viewed with a docker-machine ls
One item you will have to revisit with the Amazon Mmgt Console is to open the applicable PORTS such as Port 80 and any other ports exposed in the compose file. Once the security group is in place for the VPC, you should be able to simply refer to the VPC ID on subsequent executions bypassing any need to use the Mgmt console to add the ports. You may wish to bump up the instance size from the default t2.micro to match the t2.medium specified in your NOTES.
If ECS orchestration is needed, then a task definition will need to be created containing the container definitions you require as defined in your docker compose file. My recommendation would be to take advantage of the Mgmt console to construct the definition and then grab the accompanying JSON defintion which is made available and store in your source code repository for future executions on the command line where they can be referenced in registering task definitions, executing tasks and services within a given cluster.
Currently I'm using task-definitions that refer to custom images in dockerhub to deploy my webapp on ECS (Amazon EC2 Container Service). Is there a way to do this without going through dockerhub i.e. build/deploy the dockerfile locally across cluster nodes?
At the moment, I can only think of sending shell commands over ssh or using a tool like ansible.
Perhaps I'm missing something totally obvious here...
This is a little late for an answered question, but I just figured this out myself. The EC2 Container Registry (ECR, Amazon's repository equivalent) is working well for me, maybe didn't exist at the time?
I build the containers locally. Tag them and push them to Amazon's ECR using the AWS CLI (later versions of which include support for ECR), and then refer to them at that location in the task definitions in ECS. Works like a charm.
http://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html
ECS is a service to run containers, not to build them. It has no native support for it, so you're not missing something obvious.
As you suggest, you could distribute a Dockerfile to the container instances and build locally, but that will actually be more difficult since the container instances must have everything needed to build the image, plus you'd have to distribute the image to the other container instances.
You could run a repository yourself and specify a different repository-url for the image parameter in your ECS task definition. You'd still be responsible for building the images and now the added burden of running a repository as well.
Sorry to be the bearer of bad news but there's not a simpler workflow for this at the moment.