I could not find a definite 'yes' or 'no' anywhere, so I thought maybe I ask here. Is it possible to run a custom script which would automatically create a new environment on AWS with all the settings like (Network, Capacity, Security etc.)? I need to create a lot of new environments as I am switching from individual load balancers to shared ones, and all the settings are the same (apart from the environment and application name), so it involves a lot of manual work.
From What is AWS CloudFormation? - AWS CloudFormation:
AWS CloudFormation is a service that helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; CloudFormation handles that.
If you want to create the CloudFormation template programmatically, you can use AWS CDK - AWS Cloud Development Kit (CDK):
The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.
The AWS CDK supports TypeScript, JavaScript, Python, Java, C#/.Net, and Go. Developers can use one of these supported programming languages to define reusable cloud components known as Constructs. You compose these together into Stacks and Apps.
Or, you can simply write your own script in a programming language that calls an AWS SDK to individually create resources in AWS. Everything in AWS can be done via API calls.
I would like to understand the need of tools like Terraform. When we do have Cloudformation template available and one can create/update all AWS services with that , What is the point in using a service like Terraform.
Please Suggest.
CloudFormation (CFN) and Terraform (CF) are both Infrastructure as Code (IaC) development tools.
However, CFN is only for AWS. You can't use it with Azure, GCP or anything else outside of AWS ecosystem. In contrast, TF is cloud agnostic. You can use it across not only multiple cloud providers, but also to work with non-cloud products, such as docker, various databases and even domino pizza if you want.
So the main advantage of TF is that once you learn it only once, you can apply it to a number of cloud providers. CFN is only useful in AWS, and once you stop using CFN, you have to learn something new to work with other cloud.
There are also difference in how TF and CFN work. Both have their strengths and weekends. For example:
when you deploy using CFN all resources are available to view in one central location in AWS along with template's source code. Whereas with TF there is no such place. If you login to the AWS console, you have no idea what was created by TF, what was the source code used, etc.
TF has loops and complex data structures and condtions, while CFN does not.
CFN has creation policies and update policies, TF has not.
You can control access to CFN using CFN policies and IAM policies. You can't do same with TF as it "lives" outside of AWS.
There are a couple of reasons why you might choose Terraform over CloudFormation:
Vendor Agnostic: There might be a point in the future where you need to migrate your cloud infrastructure. This could be due to several reasons (e.g. costs, regulatory compliance, etc.). With Terraform you are still able to use the same tool to deploy the new infrastructure. With smart use of Terraform modules you can even leave large parts of your infrastucture as code repository in tact.
Support for other tools: This also builds a bit on the previous point, but Terraform can deploy a lot more then just AWS resources. For example, you can use Terraform to orchestrate the deployment of an EC2 machine that is then configured with Ansible. Or you could use Terraform to deploy applications on top of your Kubernetes cluster. While CloudFormation supports custom resources via the creation of custom Lambdas, it is quite a lot of work to maintain.
Wider ecosystem: Due to the Open Source nature of Terraform, there is a huge ecosystem of tools that help you solve all kinds of issues, such as testing the infrastructure as code or building in compliance in a continuous fashion.
Arguably a better language: Personally I think Terraform is a way more suited for Infrastructure as Code then CloudFormation. Terraform has a lot more flexibility build in to the language (HCL) and their module system allows for a lot more composability then what can be achieved in CloudFormation.
Is there any known downside to creating resources on aws through the CLI? Is it more reliable/easier/error prone/largely accepted/recommended to use one method over the other? While setting up recurring scripts, is there a reason why i would want to use CloudFormation or the AWS Console over the AWS CLI to run commands directly?
For example, if I were to create an ECS Fargate Task Definition, is there any reason why I might want to use AWS CloudFormation or the Console over AWS CLI? Cli syntax is straightforward and easy to use, and there are a few things (like setting up event rules/targets for a fargate task specifically) that are not supported via cloudformation yet.
The AWS CLI and AWS CloudFormation are two different tools that can be used to create infrastructure on AWS. The CLI is more powerful and has finer grained control than CloudFormation. CloudFormation makes it very easy to use yaml or json text files that can describe an entire enterprise in the cloud.
One of the strong benefits of CloudFormation is the automatic support for rolling back changes if anything fails while deploying a stack. The CLI in comparison would require you to figure out the details of what went wrong and how to get back to where your state was. Updating infrastructure using CloudFormation is another benefit. Make the change in the template and update the stack.
For small setups, using the CLI is fine. However, once you get past launching an EC2 instance and start building VPCs, Instances, KeyPairs, Security Groups, RDS, etc. etc. you will find that the CLI has some real limitations: mostly being too manual of a process, not easily repeatable, difficult to put the process into version control, ....
If you are constantly building, testing and deleting complex setups, CloudFormation is absolutely one of the best tools from AWS. Note that there are a number of third party solutions that have a huge number of followers such as Bamboo, Octopus, Jenkins, Chef, etc.
If your job is SysOps or DevOps then you absolutely want to master the CLI and CloudFormation. These are amazing tools for working with AWS. Also master Beanstalk, maybe OpsWorks and one of the third party tools like Jenkins.
The purpose is production-level deployment of a 8-container application, using swarm.
It seems (ECS aside) we are faced with 2 options:
Use the so called docker-for-aws that does (swarm) provisioning via a cloudformation template.
Set up our VPC as usual, install docker engines, bootstrap the swarm (via init/join etc) and deploy our application in normal EC2 instances.
Is the only difference between these two approaches the swarm bootstrap performed by docker-for-aws?
Any other benefits of docker-for-aws compared to a normal AWS VPC provisioning?
Thx
If you need to provide a portability across different cloud providers - go with AWS CloudFormation template provided by Docker team. If you only need to run on AWS - ECS should be fine. But you will need to spend a bit of time on figuring out how service discovery works there. Benefit of Swarm is that they made it fairly simple, just access your services via their service name like they were DNS names with built-in load-balancing.
It's fairly easy to automate new environment creation with it and if you need to go let's say Azure or Google Cloud later - you simply use template for them to get your docker cluster ready.
Docker team has put quite a few things into that template and you really don't want to re-create them yourself unless you really have to. For instance if you don't use static IPs for your infra (fairly typical scenario) and one of the managers dies - you can't just restart it. You will need to manually re-join it to the cluster. Docker for AWS handles that through IPs sync via DynamoDB and uses other provider specific techniques to make failover / recovery work smoothly. Another example is logging - they push your logs automatically into CloudWatch, which is very handy.
A few tips on automating your environment provisioning if you go with Swarm template:
Use some infra automation tool to create VPC per environment. Use some template provided by that tool so you don't write too much yourself. Using a separate VPC makes all environment very isolated and easier to work with, less chance to screw something up. Also, you're likely to add more elements into those environments later, such as RDS. If you control your VPC creation it's easier to do that and keep all related resources under the same one. Let's say DEV1 environment's DB is in DEV1 VPC
Hook up running AWS Cloud Formation template provided by docker to provision a Swarm cluster within this VPC (they have a separate template for that)
My preference for automation is Terraform. It lets me to describe a desired state of infrastructure rather than on how to achieve it.
I would say no, there are basically no other benefits.
However, if you want to achieve all/several of the things that the docker-for-aws template provides I believe your second bullet point should contain a bit more.
E.g.
Logging to CloudWatch
Setting up EFS for persistence/sharing
Creating subnets and route tables
Creating and configuring elastic load balancers
Basic auto scaling for your nodes
and probably more that I do not recall right now.
The template also ingests a bunch of information about related resources to your EC2 instances to make it readily available for all Docker services.
I have been using the docker-for-aws template at work and have grown to appreciate a lot of what it automates. And what I do not appreciate I change, with the official template as a base.
I would go with ECS over a roll your own solution. Unless your organization has the effort available to re-engineer the services and integrations AWS offers as part of the offerings; you would be artificially painting yourself into a corner for future changes. Do not re-invent the wheel comes to mind here.
Basically what #Jonatan states. Building the solutions to integrate what is already available is...a trial of pain when you could be working on other parts of your business / application.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 months ago.
The community reviewed whether to reopen this question 3 months ago and left it closed:
Not suitable for this site
Improve this question
I have developed a .NET MVC application and have started playing around with AWS and deploying it via the Visual Studio Toolkit. I have successfully deployed the application using the Elastic Beanstalk option in the toolkit.
As I was going over the tutorials for deploying .NET apps to AWS with the toolkit, I noticed there are tutorials for deploying with both Elastic Beanstalk and CloudFormation. What is the difference between these two?
From what I can tell, it seems like they both essentially are doing the same thing - making it easier to deploy your application to the AWS cloud (setting up EC2 instances, load balancer, auto-scaling, etc). I have tried reading up on them both, but I can't seem to get anything other than a bunch of buzz-words that sound like the same thing to me. I even found an FAQ on the AWS website that is supposed to answer this exact question, yet I don't really understand.
Should I be using one or the other? Both?
They're actually pretty different. Elastic Beanstalk is intended to make developers' lives easier. CloudFormation is intended to make systems engineers' lives easier.
Elastic Beanstalk is a PaaS-like layer on top of AWS's IaaS services which abstracts away the underlying EC2 instances, Elastic Load Balancers, auto-scaling groups, etc. This makes it a lot easier for developers, who don't want to be dealing with all the systems stuff, to get their application quickly deployed on AWS. It's very similar to other PaaS products such as Heroku, EngineYard, Google App Engine, etc. With Elastic Beanstalk, you don't need to understand how any of the underlying magic works.
CloudFormation, on the other hand, doesn't automatically do anything. It's simply a way to define all the resources needed for deployment in a huge JSON/YAML file. So a CloudFormation template might actually create two Elastic Beanstalk environments (production and staging), a couple of ElasticCache clusters, a DynamoDB table, and then the proper DNS in Route53. I then upload this template to AWS, walk away, and 45 minutes later everything is ready and waiting. Since it's just a plain-text JSON/YAML file, I can stick it in my source control which provides a great way to version my application deployments. It also ensures that I have a repeatable, "known good" configuration that I can quickly deploy in a different region.
For getting started quickly deploying a standard .NET web-application, Elastic Beanstalk is the right service for you.
AWS CloudFormation: "Template-Driven Provisioning"
AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.
CloudFormation (CFn) is a lightweight, low-level abstraction over existing AWS APIs. Using a static JSON/YAML template document, you declare a set of Resources (such as an EC2 instance or an S3 bucket) that correspond to CRUD operations on the AWS APIs.
When you create a CloudFormation stack, CloudFormation calls the corresponding APIs to create the associated Resources, and when you delete a stack, CloudFormation calls the corresponding APIs to delete them. Most (but not all) AWS APIs are supported.
AWS Elastic Beanstalk: "Web Apps Made Easy"
AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS.
You can simply upload your code and Elastic Beanstalk automatically handles the deployment, from capacity provisioning, load balancing, auto-scaling to application health monitoring.
Elastic Beanstalk (EB) is a higher-level, managed 'platform as a service' (PaaS) for hosting web applications, similar in scope to Heroku. Rather than deal with low-level AWS resources directly, EB provides a fully-managed platform where you create an application environment using a web interface, select which platform your application uses, create and upload a source bundle, and EB handles the rest.
Using EB, you get all sorts of built-in features for monitoring your application environment and deploying new versions of your application.
Under the hood, EB uses CloudFormation to create and manage the application's various AWS resources. You can customize and extend the default EB environment by adding CloudFormation Resources to an EB configuration file deployed with your application.
Conclusion
If your application is a standard web-tier application using one of Elastic Beanstalk's supported platforms, and you want easy-to-manage, highly-scalable hosting for your application, use Elastic Beanstalk.
If you:
Want to manage all of your application's AWS resources directly;
Want to manage or heavily customize your instance-provisioning or deployment process;
Need to use an application platform not supported by Elastic Beanstalk; or
Just don't want/need any of the higher-level Elastic Beanstalk features
then use CloudFormation directly and avoid the added configuration layer of Elastic Beanstalk.
Cloud Formation is a service that lets you deploy AWS services. You create a template file that describes which services you want. When you deploy that template, Cloud Formation creates the resources for you as a "package". All the resources you defined in your template are started and terminated together. Examples of types of resources that can be created with Cloud Formation are: S3, EC2 instances, AutoScaling, DynamoDb, etc. For EC2, Cloud Formation also gives you the ability to make use of "cfn-init" scripts; which can be used in conjunction with the template to boot strap your instances.
Elastic Beanstalk uses Cloud Formation templates and scipts to: 1. Create a Load Balancer and Auto Scaling Group, 2. Copy your code to S3, 3. Bootstrap an Ec2 instance to Download the code from S3 and deploy it.
Cloud Formation is not as easy to use as EB, but it is much more powerful, because you can create resources other than EC2 instances, control how the cfn-init script, and etc.
There are other differences worth noting. Elastic beanstalk is designed as a container for a single app. I've a set of several websites and services but found it very difficult to deploy multiple websites with beanstalk and was advised, after several attempts, by AWS help to use cloud formation in this situation as it has the extra flexibility.
Theres a really helpful article on bootstrapping AWS cloud formation and updating a running site here thats much clearer than the AWS pages. Still trying to work out if we can deploy from VS straight to the cloud formation template stored on S3 and get it to auto update like beanstalk...
These services are designed to complement each other. AWS Elastic Beanstalk provides an environment to easily deploy and run applications in the cloud. It is integrated with developer tools and provides a one-stop experience for you to manage the lifecycle of your applications. AWS CloudFormation is a convenient provisioning mechanism for a broad range of AWS and third party resources. It supports the infrastructure needs of many different types of applications such as existing enterprise applications, legacy applications, applications built using a variety of AWS resources and container-based solutions (including those built using AWS Elastic Beanstalk).
AWS CloudFormation supports Elastic Beanstalk application environments as one of the AWS resource types. This allows you, for example, to create and manage an AWS Elastic Beanstalk–hosted application along with an RDS database to store the application data. In addition to RDS instances, any other supported AWS resource can be added to the group as well.
Both are for provisioning infrastructure; but they differ in their approach.
Beanstalk: The starting point is the code. I have a NodeJs code I want to upload & run it; please provision the infrastructure for me. (PaaS) Platform as a Service
CloudFormation: The starting point is the infrastructure. Please create an EC2 instance, with one LoadBalancer, Security Group etc so that I can uploaded my NodeJs code to it. Infrastructure as Code (IaC).
Elastic Beanstalk automatically handles the deployment, from capacity provisioning, load balancing, auto-scaling to application health monitoring based on the code you upload to it, where as CloudFormation is an automated provisioning engine designed to deploy entire cloud environments via a JSON script.
Beanstalk: Gives the developer the ability to manage only code and not systems
Cloud Formation: Simplifies and makes everything easier for a Systems Engineer
If a developer or the dev team is looking for a quick MVP testing, the best option is to quickly get deployed with Beanstalk and check.
When a AWS migration happens, systems engineer will get involved in provisioning and Cloud Formation will help a lot and give much more granular control.
Beanstack internally uses cloudformation.
Beanstalk - Basically helpful for software developers.
Example : You want to start the PC quickly and run an application. You don't buy the PC items (harddisk, ram, Processor) separately. You buy a whole CPU or a laptop of a required config. You dont care how its running inside as you want your application to run for you. Beanstalk gives you this feature of everything ready made with no worries.
Cloudformation - Basically helpful for system engineer/ Hardware.
Example : You want to assemble 100's of PC's and give it to the developers then instead of assembling so many PC's you can just give a list of items and the PC is assembled for you by the retailer.
Similarly create a template and send it to cloudformation it will finish your work with no effort.