I manage an established AWS ECS application with terraform. The terraform also manages all other aspects of each of 4 AWS environments, including VPCs, subnets, bastion hosts, RDS databases, security groups and so on.
We manage our 4 environments by putting all the common configuration in modules which are parameterised with variables derived from the environment specific terraform files.
Now, we are trying to migrate to using Kubernetes instead of Amazon ECS for container orchestration and I am trying to do this incrementally rather than with a big bang approach. In particular, I'd like to use terraform to provision the Kubernetes cluster and link it to the other AWS resources.
What I'd initially hoped to do was capture the terraform output from kops create cluster, generalise it by parameterising it with environment specific variables and then use this one kubernetes module across all 4 environments.
However, I now realise this isn't going to work because the k8s nodes and masters all reference the kops state bucket (in s3) and it seems like I am going to have clone that bucket and rewrite the files contained therein. This seems like a rather fragile way to manage the kubernetes environment - if I recreate the terraform environment, the related state kops state bucket is going to be inconsistent with the AWS environment.
It seems to me that kops generated terraform may be useful for managing a single instance of an environment, but it isn't easily applied to multiple environments - you effectively need one kops generated terraform per environment and there is noway to reuse the terraform to establish a new environment - for this you must fall back from a declarative approach and resort to an imperative kops create cluster command.
Am I missing a good way to manage the definition of multiple similar kubernetes environments with a single terraform module?
I'm not sure how you reached either conclusions, Terraform (which will cause more trouble than it will ever solve, that's definitely a tool to get rid of asap), or having to duplicate S3 buckets.
I'm pretty sure you'd be interested in kops's cluster template feature.
You won't need to generate, hack, and launch (and debug...) Terraform, and kops templates are just as easy if not significantly easier (and more specific...) to maintain than Terraform.
When kops releases new versions, you won't have to re-generate and re-hack your Terraform scripts either!
Hope this helps!
Related
I'm trying to create an AWS EKS cluster with Pulumi and it seems two components exists:
#pulumi/eks providing a Cluster component
#pulumi/aws providing an eks/Cluster component
#pulumi/eks seems to be higher level but I cannot find a documentation specifying the concrete difference between those, and if one is preferred depending on use cases.
What's the difference between those two components?
#pulumi/eks/Cluster is a component resource that is built on top of #pulumi/aws/eks/Cluster and other resources to simplify provisioning of EKS clusters. Its goal is to make common scenarios achievable with a handful of lines of code, as opposed to the involved model of raw AWS resources.
You can find some usage examples in
AWS Crosswalk: AWS Elastic Kubernetes Service
Easily Create and Manage AWS EKS Kubernetes Clusters with Pulumi.
I suggest you start with #pulumi/eks and see if it works well for you.
What does AWS' Elastic Kubernetes Service (EKS) do exactly if so much configuration is needed in CloudFormation which is (yet) another AWS service?
I followed the AWS EKS Getting Started in the docs at (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf) where it seems CloudFormation knowledge is heavily required to run EKS.
Am I mistaken or something?
So in addition to learning the Kubernetes .yaml manifest definitions, to run k8s on EKS, AWS expects you to learn their CloudFormation .yaml configuration manifests as well (which are all PascalCase as opposed to k8s' camelCase i might add)?
I understand that EKS does some management of latest version of k8s and control plane, and is "secure by default" but other than that?
Why wouldn't I just run k8s on AWS using kops then, and deal with the slightly outdated k8s versions?
Or am I supposed to do EKS + CloudFormation + kops at which point GKE looks like a really tempting alternative?
Update:
At this point I'm really thinking EKS is just a thin wrapper over CloudFormation after searching on EKS in detail and how it is so reliant on CloudFormation manifests.
Likely a business response to the alarming popularity of k8s, GKE in general with no substance to back the service.
Hopefully this helps save the time of anyone evaluating the half-baked service that is EKS.
To run Kubernetes on AWS you have basically 2 options:
using kops, it will create Master nodes + workers node under the hood, in plain EC2 machines
EKS + Cloudformation workers stack (you can use also Terraform as an alternative to deploy the workers, or eksctl, that will create both the EKS cluster and the workers. I recommend you to follow this workshop)
EKS alone provides only the master nodes of a kubernetes cluster, in a highly available setup. You still need to add the worker nodes, where your containers will be created.
I tried both kops and EKS + Workers, and I ended up using EKS, because I found it easier to setup and maintain and more fault-tolerant.
I feel the same difficulties earlier, and none of article could give me requirement in a glance for things that need to be done. Lot of people just recommend using eksctl which in my opinion will create a bloated and hard to manage kind of CloudFormation.
Basically both EKS is just a wrapper of Kubernetes, there's some points of integration between Kubernetes and AWS that still need to be done manually.
I've wrote an article that hope could help you understand all the process that need to be inplaces
EKS is the managed control plane for kubernetes , while Cloud-formation is a infrastructure templating service .
Instead of EKS you can run and manage the control plane(master nodes) on top of EC2 machines if you want to optimize for costs.For using EKS you have to pay for the underlying infra(EC2+networking..) and managed service fee(EKS price) .
Cloud-formation provides a nice interface to template and automate your infrastructure.You may use terraform in place of CF
Terraform allows provisionning aws infrastructures with custom ansible scripts.
Since the function ami_from_instance from terraform,
allow convert an Instance into an AMI, and aws_instance the opposit.
I am quite new to that tools and I might not understand their subtilities but why should the common pattern of using Packer to generate the ami instanciated by Terraform be used ?
I am not going to repeat what "ydaetskcoR" had already mentioned. Grt points. Another usecase that Packer really does is sharing the AMI With multiple accounts. In our setup, we create AMI in one account and shared it other accounts to be used. Packer is specifically build to create AMI's and so has many features than the simple Terraform's ami_from_instance. My 2 cents
Because Packer creates an AMI with Configuration as Code, you will have a reproducible recipe for how you AMI's are created.
If you would use Terraforms ami_from_instance you instead creates clones of an non-reproducible source, thus creating snowflake servers (all are slightly different).
Also on important feature of a public cloud is autoscaling and for that you want to start off with AMI's that includes as much as possible so the startup time is small. This makes a pre-baked AMI better than a generic with a initialisation script that installs and configure all adaptations to you production environment.
I am sitting with a situation where I need to provision EC2 instances with some packages on startup. There are a couple of (enterprise/corporate) constraints that exist:
I need to provision on top of a specific AMI, which adds enterprisey stuff such as LDAP/AD access and so on
These changes are intended to be used for all internal development machines
Because of mainly the second constraint, I was wondering where is the best place to place the provisioning. This is what I've come up with
Provision in Terraform
As it states, I simply provision in terraform for the necessary instances. If I package these resources into modules, then provisioning won't "leak out". The disadvantages
I won't be able to add a different set of provisioning steps on top of the module?
A change in the provisioning will probably result in instances being destroyed on apply?
Provisioning takes a long time because of the packages it tries to install
Provisioning in Packer
This is based on the assumption that Packer allows you to provision on top of AMIs so that AMIs can be "extended". Also, this will only be used in AWS so it won't use other builders necessarily. Provisioning in Packer makes the Terraform Code much simpler and terraform applies will become faster because it's just an AMI that you fire up.
For me both of these methods have their place. But what I really want to know is when do you choose Packer Provisioning over Terraform Provisioning?
Using Packer to create finished (or very nearly finished) images drastically shortens the time it takes to deploy new instances and also allows you to use autoscaling groups.
If you have Terraform run a provisioner such as Chef or Ansible on every EC2 instance creation you add a chunk of time for the provisioner to run at the time you need to deploy new instances. In my opinion it's much better to do the configuration up front and ahead of time using Packer to bake as much as possible into the AMI and then use user data scripts/tools like Consul-Template to provide environment specific differences.
Packer certainly can build on top of images and in fact requires a source_ami to be specified. I'd strongly recommend tagging your AMIs in a way that allows you to use source_ami_filter in Packer and Terraform's aws_ami data source so when you make changes to your AMIs Packer and Terraform will automatically pull those in to be built on top of or deployed at the next opportunity.
I personally bake a reasonably lightweight "Base" AMI that does some basic hardening and sets up monitoring and logging that I want for all instances that are deployed and also makes sure that Packer encrypts the root volume of the AMI. All other images are then built off the latest "Base" AMI and don't have to worry about making sure those things are installed/configured or worry about encrypting the root volume.
By baking your configuration into the AMI you are also able to move towards the immutable infrastructure model which has some major benefits in that you know that you can always throw away an instance that is having issues and very quickly replace it with a new one. Depending on your maturity level you could even remove access to the instances so that it's no longer possible to change anything on the instance once it has been deployed which, in my experience, is a major factor in operational issues.
Very occasionally you might come across something that makes it very difficult to bake an AMI for and in those cases you might choose to run your provisioning scripts in a Terraform provisioner when it is being created. Sometimes it's simply easier to move an existing process over to using provisioners with Terraform than baking the AMIs but I would push to move things over to Packer where possible.
I have come across the same situation. As per my understanding
If you bring up your EC2 instances very frequently say 2 to 3 times a
day then go with creating an customized AMI with packer and then call
the ami through terraform.
If your base image ( AMI created by packer) change frequently based
on your requirements then its good to go with packer. But for me
running a packer scripts are very time consuming.
You can do the same with packer as well. You can write your
requirements in a script and call it in terraform. By having everything
incorporated in a terraform script would reduce some time
Finally its your decision and your frequency of bringing up EC2 instances.
I am trying to setup DEIS in AWS. So I am in the process of learning and it is expected that I will have to provision and setup many times before I master things.
Let's say I run provision-ec2-cluster and for some reason, I want to revert everything that was done in AWS (delete VPC, dele instances, scaling rules, security groups, etc.).
What is the easiest way to do that? Does Deis come with an script that can help me in this respect?
This is an ideal use case for AWS CloudFormation. Either use the official Deis published info on CloudFormation at http://docs.deis.io/en/latest/installing_deis/aws/. Note that with CloudFormation, you can with a single command launch all resources, or destroy all resources.
Or, alternatively, for a more sophisticated example with Terraform, a third-party alternative to CloudFormation, you can use something like https://github.com/brandfolder/infrastructure.