I have created an infrastructure with terraform and now I need to setup a CI... I'm thinking of using terraform also. Is it possible to extract certain part of tf code to place to Pipeline in order of updating ECS tasks ignoring the rest of infrastructure?
As ydaetskcoR suggested in a comment, if you really want to run parts of your Terraform configuration independently of the rest, you're better off splitting it up.
What I'd suggest is several terraform projects grouped in the way you'd organize responsibility and releases, (e.g. projects might have their own, VPC might be on its own, some shared infrastructure in its own), and use Terraform remote state to connect them all.
Related
I am not quite clear on the best practices related to using CDK to deploy private Github repos to AWS. I understand that a pipeline should be created by CDK and the pipeline should invoke CodeDeploy to deploy the assets, but beyond that the details are murky.
I also want to understand if for this use case it would make sense to have a separate CDK repo which is responsible for the infrastructure for the entire backend of my project, or if it would make more sense to have CDK code included in each individual component repo. As I will be utilizing a microservice/cell based approach for building out components, the overhead required in adding CDK configuration for each component might be substantial.
You can think of CDK as a compiler that takes a given language and 'compiles' it down to CloudFormation templates. Those templates are then uploaded to Cloudformation by the CDK framework, and run for you when you execute the deploy command.
So. To answer your question more directly - if you can figure out how to do it in cloudformation, you can do it. That may involve spinning up a code build and running a script that executes some api calls or prepares a package for an ec2 server that then uses that package in the next step or any number of things.
But remember that CDK synths its cloudformation template all at once, and is only creating the template. It does not run any scripts that may be part of your code builds, and it does not 'wait' for certain things to be complete - because it isn't doing anything like that. If you have a sequence of events that need to occur, you want to use CodePipeline to orchestrate those events for you - but you can set up your CodePipeline with CDK for certain!
As for Overhead, maybe at first. But trust me when I say it becomes very quick and easy to generate a CDK stack for a given microservice with experience, and its super handy. Being able to spin up an ad-hoc on demand testing environment is super useful. Being able to deploy individual stacks on demand and make quick changes with just a line of code is handy as all get out. Having a single source of code for both your prod and development environments that you make a change in one and on next deployment in each is automatically reflected is super handy.
CDK is a very powerful tool - but it is a very low level one. It creates the template that will create your resources. Thats it. If your resources need to do something after being created for something else to happen you have to make use of other services to orchestrate that (CodePipeline, StepFunctions, Cloudwatch Events, ect)
I am new to Terraform and building a CI setup. When I want to create a CodePipeline that is going to be connected to a GitHub repo, do I run specific commands inside my Terraform codebase that will reach out to AWS and create the CodePipeline config/instance for me? Or would I set this CodePipeline up manually inside AWS console and hook it up to Terraform after the fact?
do I run specific commands inside my Terraform codebase that will reach out to AWS and create the CodePipeline config/instance for me?
Yes, you use aws_codepipeline which will create new pipeline in AWS.
Or would I set this CodePipeline up manually inside AWS console and hook it up to Terraform after the fact?
You can also import existing resources to terraform.
I see you submitted this eight months ago, so I am pretty sure you have your answer, but for those searching that comes across this question, here are my thoughts on it.
As most of you have researched, terraform is infrastructure as code (IaC). As IaC it needs to be executed somewhere. This means that you either execute locally or inside a pipeline. A pipeline consists of docker containers that emulate a local environment and run commands for you to deploy your code. There is more to that, but the premise of understanding how terraform runs remains the same.
So to the magic question, Terraform is Code, and if you intend to use a pipeline, Jenkins, AWS, GitLab, and more, then you need a code repository to put all your code into. In this case, a code repository where you can store your terraform code so a pipeline can consume it when deploying your code. There are other reasons why you should use a code repository, but your question is directed to terraform and its usage with the pipeline.
Now the magnificent argument, the chicken or the egg, when to create your pipeline and how to do it. To your original question, you could do both. You could store all your terraform code in a repository (i recommend), clone it down, and locally run terraform to create your pipeline. This would be ideal for you to save time and leverage automation. Newbies, you will have to research terraform state files which is an element you need to backup in some form or shape once the pipeline is deployed for you.
If you are not so comfortable with Terraform, the GUI in AWS is also fine, and you can configure it easily to hook your pipeline into Github to run jobs.
You must set up Terraform and AWS locally on your machine or within the pipeline to deploy your code in both scenarios. This article is pretty good and will give you the basic understanding of setting up terraform
Don't forget to configure AWS on your local machine. For you Newbies using pipeline, you can leverage some of the pipeline links to get you started. Remember one thing, within AWS Codepipeine; you have to use IAM roles and not access keys. That will make more sense once you have gone through the first link. Please also go to youtube and search Terraform for beginners in AWS. Various videos can provide a lot more substance to help you get started.
Recently, while trying to build a terraform IaC, I found that I couldn’t get the API Gateway to route to the Lambda properly. It turned out that when using the console AWS automatically assigns the permissions the gateway needs for the Lambda, but with IaC in terraform this must be assigned explicitly.
The above is understandable but for a newbie, to both AWS and terraform, confusing.
Is there documentation which explains the required components within an infrastructure connection, such as that above?
I know of the AWS docs and the terraform docs are particularly well thought out but none of it actually explains (as far as I’ve seen) that a certain resource is required in any particular (however common or obscure) setup. Inferring these connections from general searching is not a great replacement.
I don't think that there is a documentation that lists "all of the required components" in one single page/area. But you can get different pieces of information from different docs, and as you mentioned AWS and Terraform do both a great job at this.
Talking about AWS, in the case of permissions in API gateway, I can think of two useful links (the 1st one is referenced from the 2nd one though):
How API Gateway resource policies affect authorization workflow
Control access for invoking an API
I agree in the fact that sometimes it's a lot of guesses to translate AWS into terraform if you don't really know what you are trying to achieve. Usually when I am blocked on something that "should theoritically work" in IaC vs AWS console, I step back from the problem and try to figure out what kind of components am I really trying to glue together in AWS world. Then usually things become more obvious.
Because in terraform it's really creating small independant pieces of infrastructure and make them work together. Comparing with other IaC, in my experience it's a lot more granular than CloudFormation for instance.
A personal practice that helps me figure out things faster is to read every single intro doc of the components I am working on in Terraform. For instance, if I am writing lambda in terraform IaC, I would quickly read all the lambda_xxxx_yyyy intro parts to get less stuck and react faster when something fails. It usually works for me.
I haven't see such a documentation, but I can share my work-around for similar cases.
You can make changes you need using AWS console - manually, using UI. Then you can define resources you just created in your TF files, defining only/required required set of properties, even random values will work. Then you import what you created manually into resources you defined.
By running terraform plan you will see the differences, that will allow you to adjust your TF files accordingly.
After few iterations you will replicate what you have just done in the UI using TF. As a final test you can manually revert your changes, run terraform apply and ensure that everything works as expected.
When deciding between Terraform and AWS CloudFormation, one advantage generally associated with Terraform is the separation of planning (terraform plan) and execution (terraform apply).
I recently heard about CloudFormation's create-change-set feature, which is described as:
Creates a list of changes that will be applied to a stack so that you can review the changes before executing them.
In Updating Stacks Using Change Sets from the AWS documentation, they describe the recommended update process.
Now I wonder:
Doesn't create-change-set also allow to separate planning from execution?
If not, what are the shortcomings in comparison with Terraform's plan & apply?
Background: Examples where the separation is emphasized as a significant advantage:
Official comparison on terraform.io:
Terraform also separates the planning phase from the execution phase, by using the concept of an execution plan. By running terraform plan, the current state is refreshed and the configuration is consulted to generate an action plan. The plan includes all actions to be taken: which resources will be created, destroyed or modified. It can be inspected by operators to ensure it is exactly what is expected.
[...]
Terraform lets operators apply changes with confidence, as they know exactly what will happen beforehand.
(Possibly outdated) Blog post from 2014 comparing Terraform and CloudFormation:
Infrastructure Updates: This is the absolute killer feature of Terraform. Terraform has a separate planning and execution phase. The planning phase shows which resources will be created, modified and destroyed. It gives you complete control of how your changes will affect the existing environment, which is quite crucial. This was one of the main reasons why we went ahead with Terraform. CloudFormation does not show you what changes it is going to make to the environment.
A CloudFormation Change Set is an analogous object to a Terraform plan saved in a plan file. Its core functionality is the same: inspect a report of the changes required to reach a new desired state, and then apply those changes.
When Terraform was originally created, CloudFormation did not have this feature. The documentation you have found was accurate at the time it was written, but with the release of Change Sets there is not a significant difference in this regard.
The other difference covered in the official comparison still applies, however:
Terraform similarly uses configuration files to detail the
infrastructure setup, but it goes further by being both cloud-agnostic
and enabling multiple providers and services to be combined and
composed. For example, Terraform can be used to orchestrate an AWS and
OpenStack cluster simultaneously, while enabling 3rd-party providers
like Cloudflare and DNSimple to be integrated to provide CDN and DNS
services. This enables Terraform to represent and manage the entire
infrastructure with its supporting services, instead of only the
subset that exists within a single provider. It provides a single
unified syntax, instead of requiring operators to use independent and
non-interoperable tools for each platform and service.
Is there a way to create the terraform state file, from the existing infrastructure. For example, an AWS account comes with a some services already in place ( for ex: default VPC).
But terraform, seems to know only the resources, it creates. So,
What is the best way to migrate an existing AWS Infrastructure to Terraform code
Is it possible to add a resource manually and modify the state file manually (bad effects ?)
Update
Terraform 0.7.0 supports importing single resource.
For relatively small things I've had some success in manually mangling a state file to add stubbed resources that I then proceeded to Terraform over the top (particularly with pre-existing VPCs and subnets and then using Terraform to apply security groups etc).
For anything more complicated there is an unofficial tool called terraforming which I've heard is pretty good at generating Terraform state files but also merging with pre-existing state files. I've not given it a go but it might be worth looking into.
Update
Since Terraform 0.7, Terraform now has first class support for importing pre-existing resources using the import command line tool.
As of 0.7.4 this will import the pre-existing resource into the state file but not generate any configuration for the resource. Of course if then attempt a plan (or an apply) Terraform will show that it wants to destroy this orphaned resource. Before running the apply you would then need to create the configuration to match the resource and then any future plans (and applys) should show no changes to the resource and happily keep the imported resource.
Use Terraforming https://github.com/dtan4/terraforming , To date it can generate most of the *.tfstate and *.tf file except for vpc peering.