When using infrastructure as code tools like Terraform to create resources, what are the best practices for creating of one-off creation resources like VPCs and databases via an automated pipeline?
For example, when creating an Lambda function, we need to provide a VPC ID. This VPC ID can only be supplied if the VPC is already created via the Terraform. So should there be 2 different pipelines based on the resource type created? For one-off resources a separate pipeline which will be ideally only run once and another pipeline to create disposable resources?
If you're talking about general best practices for IaC (Infrastructure as Code), check out the following:
Codify everything.
Document as little as possible.
Maintain version control.
Continuously test, integrate, and deploy.
Make your infrastructure code modular.
Make your infrastructure immutable whenever possible.
I would recommend using AWS CloudFormation as it is free and provides you an interface to manage stacks and get visibility of the resources created. Also, it has the auto-rollback capability on the stack create or update failure.
This VPC ID can only be supplied if the VPC is already created via the Terraform. So should there be 2 different pipelines based on the resource type created?
Not really as you can use the VPC ID as a parameter to your AWS CloudFormation template or Terraform template.
For one-off resources a separate pipeline which will be ideally only run once and another pipeline to create disposable resources?
It is always better to have a separate stack for continuous deployment or disposable resources and a separate stack for one-off resources defining your base infrastructure.
Related
I have created a terraform module that create different aws ressources when applied.
The resources are :
An ec2
A security group
An Elastic IP
I would love to automate resource deployment via Gitlab Pipeline as it support Terraform.
However I have a question concerning my case.
This module has been created in the goal of creating an instance on demand for a specific person. So I would like to be able to have individual control over each resources for each person as opposed to a global "inventory" handled by Terraform state.
I was thinking about creating a branch for each person so I can destroy this person's resources if they are not needed anymore without doing it by hand on AWS console.
Let's say I run this module for personA and personB. If personA doesn't need these resources anymore it would be nice to be able to destroy it with terraform without impacting personB's resources.
Is my way to do it correct or is there a better one ?
If it's a general purpose module, then I would create a TF stack for each person. This way each instance would have independent state, and each user could manage their own instance.
Had a question regarding infrastructure as code on AWS.
Wondering how to do this (the process of deploying) and also why is this an efficient method for architecture? Also, are there other methods that should be looked at over this?
I am looking to deploy this for a startup I am working for and need assistance in getting this going. Any help is appreciated.
Thank you.
From What is AWS CloudFormation?:
AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; AWS CloudFormation handles all of that.
So, instead of manually creating each bit of architecture (network, instances, queues, storage, etc), you can define them in a template and CloudFormation will deploy them. It is smart enough to mostly know the correct order of creation (eg creating the network before creating an Amazon EC2 instance within the network) and it can also remove resources when the 'stack' is no longer required.
Other benefits:
The template effectively documents the infrastructure
Infrastructure can be checked into a source code repository, and versioned
Infrastructure can be repeatedly and consistently deployed (eg Test environment matches Production environment)
Changes can be made to the template and CloudFormation can update the 'stack' by just deploying the changes
There are tools (eg https://former2.com/) that can generate the template from existing infrastructure, or just create it from code snippets taken from the documentation.
Here's an overview: Simplify Your Infrastructure Management Using AWS CloudFormation - YouTube
There are also tools like Terraform that can deploy across multiple cloud services.
I would like to understand the need of tools like Terraform. When we do have Cloudformation template available and one can create/update all AWS services with that , What is the point in using a service like Terraform.
Please Suggest.
CloudFormation (CFN) and Terraform (CF) are both Infrastructure as Code (IaC) development tools.
However, CFN is only for AWS. You can't use it with Azure, GCP or anything else outside of AWS ecosystem. In contrast, TF is cloud agnostic. You can use it across not only multiple cloud providers, but also to work with non-cloud products, such as docker, various databases and even domino pizza if you want.
So the main advantage of TF is that once you learn it only once, you can apply it to a number of cloud providers. CFN is only useful in AWS, and once you stop using CFN, you have to learn something new to work with other cloud.
There are also difference in how TF and CFN work. Both have their strengths and weekends. For example:
when you deploy using CFN all resources are available to view in one central location in AWS along with template's source code. Whereas with TF there is no such place. If you login to the AWS console, you have no idea what was created by TF, what was the source code used, etc.
TF has loops and complex data structures and condtions, while CFN does not.
CFN has creation policies and update policies, TF has not.
You can control access to CFN using CFN policies and IAM policies. You can't do same with TF as it "lives" outside of AWS.
There are a couple of reasons why you might choose Terraform over CloudFormation:
Vendor Agnostic: There might be a point in the future where you need to migrate your cloud infrastructure. This could be due to several reasons (e.g. costs, regulatory compliance, etc.). With Terraform you are still able to use the same tool to deploy the new infrastructure. With smart use of Terraform modules you can even leave large parts of your infrastucture as code repository in tact.
Support for other tools: This also builds a bit on the previous point, but Terraform can deploy a lot more then just AWS resources. For example, you can use Terraform to orchestrate the deployment of an EC2 machine that is then configured with Ansible. Or you could use Terraform to deploy applications on top of your Kubernetes cluster. While CloudFormation supports custom resources via the creation of custom Lambdas, it is quite a lot of work to maintain.
Wider ecosystem: Due to the Open Source nature of Terraform, there is a huge ecosystem of tools that help you solve all kinds of issues, such as testing the infrastructure as code or building in compliance in a continuous fashion.
Arguably a better language: Personally I think Terraform is a way more suited for Infrastructure as Code then CloudFormation. Terraform has a lot more flexibility build in to the language (HCL) and their module system allows for a lot more composability then what can be achieved in CloudFormation.
I have created an ec2 instance and configured it as a target behind my load balancer. I want to convert this entire environment as something that can be deployed repeatedly in automation. I have looked at cloudformation but don't know if it can help me with converting this environment to a cloudformation template. Is there any other way to achieve this. If there is an approach outside of Cloudformation, that's fine too.
PS: I am new to AWS and it's capabilities
Thanks
Have a look at Former2.
Former2 allows you to generate Infrastructure-as-Code outputs from your existing resources within your AWS account.
You need an IaC (Infrastructure as Code) tool. Cloudformation is one of them, but there are plenty others. Terraform, Pulumi or even the AWS CDK.
Look at Infrastructure as Code try AWS CDK, Terraform, you should also look at methods for replacing existing infrastructure. Spinning a fresh set of infrastructure along side the existing one and swapping out in DNS is the most common of approach.
I am trying to programmatically recreate a bunch of AWS resources that were created/configured manually via AWS consoles.
The AWS consoles do a lot for you.
For example, you can create a Lambda function with an Api-Gateway trigger in about 10 seconds using the AWS console.
The console is doing a lot of magic under the covers, defining and configuring resources such as policies, stages, permissions, models, etc.
In theory, CloudTrail is supposed to allow me to see what exactly is happening under the covers, but it seems to be silent in this case (i.e. Lambda function with Api-Gateway trigger).
I can play hide and seek and do extensive dumps using the CLI to list stages, policies, export api definitions, etc. etc. and look for the differences but is there an easier way? - like some way to trace the REST calls that the console is creating when it does all its magic?
Note: CloudFormer could have helped but it is only half-written software (Hey Amazon!) and only covers about a third of the resources I have defined. Does embracing Cloudformation imply not using these great time-saving consoles?
CloudFormation and other Infrastructure as code services are there to lessen the clicks you make while using AWS console or any other cloud console to manage your resources.
But these come in handy when you have to spin up resources which will be having almost the same configurations and software stack.
If you use CloudFormation you will be able to define the policies according to your need, which OS image to use, which stack to install etc. etc. it provides you minute control over your resources.
I suggest if you have to deploy these resources multiple times then create a CloudFormation template and use it.
So, I would suggest that rather than finding a way to recreate the code from your current infrastructure, create a CloudFormation template and use it for future needs.
If you need something easier than your current flow, this is it, as you just have to write your required configuration once.
Hashicorp Terraform is also a good alternative to AWS CloudFormation. You can use Terraforming to export the current infrastructure into Terraform readable files.