I am trying to setup DEIS in AWS. So I am in the process of learning and it is expected that I will have to provision and setup many times before I master things.
Let's say I run provision-ec2-cluster and for some reason, I want to revert everything that was done in AWS (delete VPC, dele instances, scaling rules, security groups, etc.).
What is the easiest way to do that? Does Deis come with an script that can help me in this respect?
This is an ideal use case for AWS CloudFormation. Either use the official Deis published info on CloudFormation at http://docs.deis.io/en/latest/installing_deis/aws/. Note that with CloudFormation, you can with a single command launch all resources, or destroy all resources.
Or, alternatively, for a more sophisticated example with Terraform, a third-party alternative to CloudFormation, you can use something like https://github.com/brandfolder/infrastructure.
Related
I have created an ec2 instance and configured it as a target behind my load balancer. I want to convert this entire environment as something that can be deployed repeatedly in automation. I have looked at cloudformation but don't know if it can help me with converting this environment to a cloudformation template. Is there any other way to achieve this. If there is an approach outside of Cloudformation, that's fine too.
PS: I am new to AWS and it's capabilities
Thanks
Have a look at Former2.
Former2 allows you to generate Infrastructure-as-Code outputs from your existing resources within your AWS account.
You need an IaC (Infrastructure as Code) tool. Cloudformation is one of them, but there are plenty others. Terraform, Pulumi or even the AWS CDK.
Look at Infrastructure as Code try AWS CDK, Terraform, you should also look at methods for replacing existing infrastructure. Spinning a fresh set of infrastructure along side the existing one and swapping out in DNS is the most common of approach.
Is there any known downside to creating resources on aws through the CLI? Is it more reliable/easier/error prone/largely accepted/recommended to use one method over the other? While setting up recurring scripts, is there a reason why i would want to use CloudFormation or the AWS Console over the AWS CLI to run commands directly?
For example, if I were to create an ECS Fargate Task Definition, is there any reason why I might want to use AWS CloudFormation or the Console over AWS CLI? Cli syntax is straightforward and easy to use, and there are a few things (like setting up event rules/targets for a fargate task specifically) that are not supported via cloudformation yet.
The AWS CLI and AWS CloudFormation are two different tools that can be used to create infrastructure on AWS. The CLI is more powerful and has finer grained control than CloudFormation. CloudFormation makes it very easy to use yaml or json text files that can describe an entire enterprise in the cloud.
One of the strong benefits of CloudFormation is the automatic support for rolling back changes if anything fails while deploying a stack. The CLI in comparison would require you to figure out the details of what went wrong and how to get back to where your state was. Updating infrastructure using CloudFormation is another benefit. Make the change in the template and update the stack.
For small setups, using the CLI is fine. However, once you get past launching an EC2 instance and start building VPCs, Instances, KeyPairs, Security Groups, RDS, etc. etc. you will find that the CLI has some real limitations: mostly being too manual of a process, not easily repeatable, difficult to put the process into version control, ....
If you are constantly building, testing and deleting complex setups, CloudFormation is absolutely one of the best tools from AWS. Note that there are a number of third party solutions that have a huge number of followers such as Bamboo, Octopus, Jenkins, Chef, etc.
If your job is SysOps or DevOps then you absolutely want to master the CLI and CloudFormation. These are amazing tools for working with AWS. Also master Beanstalk, maybe OpsWorks and one of the third party tools like Jenkins.
I have an AWS website that is running inside a private subnet and I am not sure what the best way is to update it.
I would like something that is non-burdensome, ideally it would be nice to have some EC2 Box (with security groups only allowing select IP's to connect too) running the development page, and then I could simply copy it over to the private EC2 Box with a click of a button.
I am not too familiar with best practices, but the idea of connecting through several EC2 boxes seems burdensome.
Thank You!
Sounds like you might want to make use of AWS CodeDeploy. There are other tools as well, but since you are already on/using AWS this might be a good one to start with:
AWS CodeDeploy is a service that automates code deployments to any
instance, including Amazon EC2 instances and instances running
on-premises. AWS CodeDeploy makes it easier for you to rapidly release
new features, helps you avoid downtime during application deployment,
and handles the complexity of updating your applications. You can use
AWS CodeDeploy to automate software deployments, eliminating the need
for error-prone manual operations, and the service scales with your
infrastructure so you can easily deploy to one instance or thousands.
https://aws.amazon.com/codedeploy/
I want to understand the exact limitations of OpsWorks- things that we simply may not be able to do or not optimally. That would require to go to the next level - CloudFormation. Of course we can use OpsWorks + CloudFormation - get best of both worlds - a best practice.
Some limitations of OpsWorks that I am aware of are - it cant provision everything - like EMR, S3 etc - but have never found on exhaustive list. Also OpsWorks the AutoScaling configuration has limitations. And CloudFormation lets us version control the environment unlike OpsWorks. Do understand that there is good amount of overlap and CloudFormation does add to complexity.
There was a previous discussion but the demarcations were not made clear.
OpsWorks is a totally different service from CloudFormation.
OpsWorks is focused at managing applications layered as stacks, and taking advantadge of chef recipes for setting up and deploying applications.
CloudFormation is a descriptive language to create sets of AWS infrastructure.
It may look obvious, but the thing is that whenever you prefer to manage an application and its deployment cycle, the OpsWorks service is better suited. You can of course use cloud formation to define entire applications and layers in OpsWorks and that will allow you to replicate entire applications layer sets (for test environments and so)
The only good way to learn the boundaries of every service is using them for your needs, then you will find where Opsworks is strong and where Cloudformation complements or allows you to automate your Opsworks setups.
About versioning, CF lets you version the infrastructure stack, which may have nothing to do with the version of code that you are managing through opsworks.
regards
In many cases CloudFormation is used with OpsWorks; where CloudFormation is used to provision the infrastructure and OpsWorks to configure the resources created. OpsWorks (via Chef or Puppet) provides a richer set of abilities when it comes to configuring the application stack then the simple shell scripting offered by CloudFormation. So, generally CloudFormation is used to deploy AWS resources, and OpsWorks is used to do the detailed configuration of the application(s)/operating system(s).
In some ways one can say that CloudFormation is focused more with a collection of AWS infrastructure resources rather than the application itself; but as you have alluded to some things can be done by both services interchangeably.
This AWS link may help where it states:
“Compared to AWS CloudFormation, AWS OpsWorks Stacks supports a narrower range of application-oriented AWS resource types including Amazon EC2 instances, Amazon EBS volumes, Elastic IPs, and Amazon CloudWatch metrics.”
and this is certainly not a limitation for CloudFormation.
The purpose is production-level deployment of a 8-container application, using swarm.
It seems (ECS aside) we are faced with 2 options:
Use the so called docker-for-aws that does (swarm) provisioning via a cloudformation template.
Set up our VPC as usual, install docker engines, bootstrap the swarm (via init/join etc) and deploy our application in normal EC2 instances.
Is the only difference between these two approaches the swarm bootstrap performed by docker-for-aws?
Any other benefits of docker-for-aws compared to a normal AWS VPC provisioning?
Thx
If you need to provide a portability across different cloud providers - go with AWS CloudFormation template provided by Docker team. If you only need to run on AWS - ECS should be fine. But you will need to spend a bit of time on figuring out how service discovery works there. Benefit of Swarm is that they made it fairly simple, just access your services via their service name like they were DNS names with built-in load-balancing.
It's fairly easy to automate new environment creation with it and if you need to go let's say Azure or Google Cloud later - you simply use template for them to get your docker cluster ready.
Docker team has put quite a few things into that template and you really don't want to re-create them yourself unless you really have to. For instance if you don't use static IPs for your infra (fairly typical scenario) and one of the managers dies - you can't just restart it. You will need to manually re-join it to the cluster. Docker for AWS handles that through IPs sync via DynamoDB and uses other provider specific techniques to make failover / recovery work smoothly. Another example is logging - they push your logs automatically into CloudWatch, which is very handy.
A few tips on automating your environment provisioning if you go with Swarm template:
Use some infra automation tool to create VPC per environment. Use some template provided by that tool so you don't write too much yourself. Using a separate VPC makes all environment very isolated and easier to work with, less chance to screw something up. Also, you're likely to add more elements into those environments later, such as RDS. If you control your VPC creation it's easier to do that and keep all related resources under the same one. Let's say DEV1 environment's DB is in DEV1 VPC
Hook up running AWS Cloud Formation template provided by docker to provision a Swarm cluster within this VPC (they have a separate template for that)
My preference for automation is Terraform. It lets me to describe a desired state of infrastructure rather than on how to achieve it.
I would say no, there are basically no other benefits.
However, if you want to achieve all/several of the things that the docker-for-aws template provides I believe your second bullet point should contain a bit more.
E.g.
Logging to CloudWatch
Setting up EFS for persistence/sharing
Creating subnets and route tables
Creating and configuring elastic load balancers
Basic auto scaling for your nodes
and probably more that I do not recall right now.
The template also ingests a bunch of information about related resources to your EC2 instances to make it readily available for all Docker services.
I have been using the docker-for-aws template at work and have grown to appreciate a lot of what it automates. And what I do not appreciate I change, with the official template as a base.
I would go with ECS over a roll your own solution. Unless your organization has the effort available to re-engineer the services and integrations AWS offers as part of the offerings; you would be artificially painting yourself into a corner for future changes. Do not re-invent the wheel comes to mind here.
Basically what #Jonatan states. Building the solutions to integrate what is already available is...a trial of pain when you could be working on other parts of your business / application.