There are so many options:
Docker-compose with ECS cli looks the easiest solution
Terraform
CloudFormation (looks complex!)
Ansible
I am only interested in setting up a basic ECS docker set-up with ELB and easily updating the Docker image version.
We all love technology here, but we're not all super geniuses when it comes to tech. So I'm looking to keep my set-up as simple as possible. We run Jenkins, 2 NodeJS applications, 2 Java applications in ECS and I know it involves IAM, Security Groups, EBS, ELB, ECS Service/Task, ECS Task Definition, but that already gets complex quickly in CloudFormation.
What are good technologies that will allow us to use Docker, keep things simple and don't require us to be very intelligent to understand our own programming code?
I would suggest you start by trying to setup your pipeline using Terraform. Learning it will give you experience in a non-vendor specific infrastructure as code.
Another possibility is to avoid using CloudFormation directly and prefer using the AWS CDK (https://docs.aws.amazon.com/cdk/latest/guide/home.html) as IaC.
Best regards
Related
I am trying to automate the deployment of the AWS ECS and couldn't find much information I could do that and will like to see if there is any advice on what I can explore. Currently, we have an Azure DevOps pipeline that will push the containerized image to the ECR and we will manually create the task definition at ecs and update the service afterwards. Is there anyway that I can automate this with azure devops release?
A bit open ended for a Stackoverflow style question but the short answer is that there are a lot of AWS native alternatives to this. This is an example that implements the blue-green pattern (it can be simplified with a more generic rolling update deployment). If you are new to ECS you probably want to consider using Copilot. This is a entry level blog that hints about how to deploy an application and build a pipeline for it.
I am at a bit of a crossroads here. My goal is to automate creating my ECS architecture and deploying my docker-compose services to ECS Fargate, but there are so many ways to do it!
Hoping to get some insight from the community on picking the right tool for the job. What are the use cases for each of these? When should I pick one over the other?
Docker ECS integration
https://docs.docker.com/cloud/ecs-integration/
ecs-cli
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html
AWS Copilot
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Copilot.html
AWS CLI ecs
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecs/index.html
This is how I am breaking it down:
Docker Compose / ECS integration: use this if you are deep into Docker already are in love with the compose syntax. If you are not already in love with it you should because it's a very simple syntax
ECS CLI: this is the previous version of the Copilot CLI. ECS CLI had native ECS components/workflows to it + some level of docker compose compatibility. We have since improved the workflows in Copilot and forked compose support into the Docker Compose / ECS integration above. TL/DR: don't use ECS CLI
Copilot CLI: this is a very easy way to start with containers on AWS ECS. If you don't know much about Docker and you don't need to become an expert in containers Copilot is the right approach. Only drawback (IMO) is that it doesn't really support a proper IaC pattern yet (example).
AWS CLI (ecs namespace): this is just the CLI manifestation of the 1:1 mapping of the core ECS APIs. This is most likely way too low level for you to extract value (unless you are doing very deep things). For example one Copilot command or a few Docker Compose lines could easily be exploded in dozens of AWS CLI commands.
I'd add CDK to your list: CDK is a representation of AWS constructs (including ECS) that could be represented with standard programming languages. The good thing about CDK is that it could map 1:1 raw API/Cloudformation constructs but it also ships with higher level and more abstracted libraries that allows you to express in a few lines of code content that would require hundreds of lines of CFN.
If you are deep into compose (as it seems from your original message) you may want to have a look at this approach.
Is there any known downside to creating resources on aws through the CLI? Is it more reliable/easier/error prone/largely accepted/recommended to use one method over the other? While setting up recurring scripts, is there a reason why i would want to use CloudFormation or the AWS Console over the AWS CLI to run commands directly?
For example, if I were to create an ECS Fargate Task Definition, is there any reason why I might want to use AWS CloudFormation or the Console over AWS CLI? Cli syntax is straightforward and easy to use, and there are a few things (like setting up event rules/targets for a fargate task specifically) that are not supported via cloudformation yet.
The AWS CLI and AWS CloudFormation are two different tools that can be used to create infrastructure on AWS. The CLI is more powerful and has finer grained control than CloudFormation. CloudFormation makes it very easy to use yaml or json text files that can describe an entire enterprise in the cloud.
One of the strong benefits of CloudFormation is the automatic support for rolling back changes if anything fails while deploying a stack. The CLI in comparison would require you to figure out the details of what went wrong and how to get back to where your state was. Updating infrastructure using CloudFormation is another benefit. Make the change in the template and update the stack.
For small setups, using the CLI is fine. However, once you get past launching an EC2 instance and start building VPCs, Instances, KeyPairs, Security Groups, RDS, etc. etc. you will find that the CLI has some real limitations: mostly being too manual of a process, not easily repeatable, difficult to put the process into version control, ....
If you are constantly building, testing and deleting complex setups, CloudFormation is absolutely one of the best tools from AWS. Note that there are a number of third party solutions that have a huge number of followers such as Bamboo, Octopus, Jenkins, Chef, etc.
If your job is SysOps or DevOps then you absolutely want to master the CLI and CloudFormation. These are amazing tools for working with AWS. Also master Beanstalk, maybe OpsWorks and one of the third party tools like Jenkins.
I want to confirm my approach to setting up a VPC using cloudformation/scepter and seeding instances with docker container is correct.
Create an aws ec2 instance.
Create a docker image on that instance
Create a cloudformation VPC template (.yaml )
-reference docker image in template?
Create a sceptre project using the template above and run script from ec2 instance
So as I understand if the majority of the work will be in the cloudformation template. Currently I'm stuck on sceptre errors, but I wanted to make sure I was approaching the problem correctly. Does this look like the right approach?
There are a lot of ways of doing what you want:
Run sceptre locally on your development machine
This is easier, but not best practice for important environments as
having a build server, gives a better trail of what was done when (especially in shared environments)
Use CodeBuild to save you having to do steps 1 & 2 yourself (AWS maintain a docker image with python installed)
It also avoids the chicken and egg problem of how you deploy the EC2 instance in the first place.
Configure Jobs on a build server such as Jenkins
CodeDeploy is good for simple setups, but a well configured build server, can have dashboards to track what is deployed where
as sceptre is just a way of generating/managing deploying templates across environments, there are lots of other ways of doing this including what you outlined.
p.s Apologies that the getting started documentation isn't great at the moment, it is something we are focusing on for release 2.0.
The purpose is production-level deployment of a 8-container application, using swarm.
It seems (ECS aside) we are faced with 2 options:
Use the so called docker-for-aws that does (swarm) provisioning via a cloudformation template.
Set up our VPC as usual, install docker engines, bootstrap the swarm (via init/join etc) and deploy our application in normal EC2 instances.
Is the only difference between these two approaches the swarm bootstrap performed by docker-for-aws?
Any other benefits of docker-for-aws compared to a normal AWS VPC provisioning?
Thx
If you need to provide a portability across different cloud providers - go with AWS CloudFormation template provided by Docker team. If you only need to run on AWS - ECS should be fine. But you will need to spend a bit of time on figuring out how service discovery works there. Benefit of Swarm is that they made it fairly simple, just access your services via their service name like they were DNS names with built-in load-balancing.
It's fairly easy to automate new environment creation with it and if you need to go let's say Azure or Google Cloud later - you simply use template for them to get your docker cluster ready.
Docker team has put quite a few things into that template and you really don't want to re-create them yourself unless you really have to. For instance if you don't use static IPs for your infra (fairly typical scenario) and one of the managers dies - you can't just restart it. You will need to manually re-join it to the cluster. Docker for AWS handles that through IPs sync via DynamoDB and uses other provider specific techniques to make failover / recovery work smoothly. Another example is logging - they push your logs automatically into CloudWatch, which is very handy.
A few tips on automating your environment provisioning if you go with Swarm template:
Use some infra automation tool to create VPC per environment. Use some template provided by that tool so you don't write too much yourself. Using a separate VPC makes all environment very isolated and easier to work with, less chance to screw something up. Also, you're likely to add more elements into those environments later, such as RDS. If you control your VPC creation it's easier to do that and keep all related resources under the same one. Let's say DEV1 environment's DB is in DEV1 VPC
Hook up running AWS Cloud Formation template provided by docker to provision a Swarm cluster within this VPC (they have a separate template for that)
My preference for automation is Terraform. It lets me to describe a desired state of infrastructure rather than on how to achieve it.
I would say no, there are basically no other benefits.
However, if you want to achieve all/several of the things that the docker-for-aws template provides I believe your second bullet point should contain a bit more.
E.g.
Logging to CloudWatch
Setting up EFS for persistence/sharing
Creating subnets and route tables
Creating and configuring elastic load balancers
Basic auto scaling for your nodes
and probably more that I do not recall right now.
The template also ingests a bunch of information about related resources to your EC2 instances to make it readily available for all Docker services.
I have been using the docker-for-aws template at work and have grown to appreciate a lot of what it automates. And what I do not appreciate I change, with the official template as a base.
I would go with ECS over a roll your own solution. Unless your organization has the effort available to re-engineer the services and integrations AWS offers as part of the offerings; you would be artificially painting yourself into a corner for future changes. Do not re-invent the wheel comes to mind here.
Basically what #Jonatan states. Building the solutions to integrate what is already available is...a trial of pain when you could be working on other parts of your business / application.