I have a Terraform script that sets up 30 resources in all, including roles, policies and ECS infrastructure.
The state file was destroyed when switching git branches - I should've backed it up, but now I do not have a state file referencing the already existing resources.
I can't use terraform apply because the names already exist, and I can't clear the existing resources with terraform destroy because the state file now thinks the resources don't exist.
Is there a way I could sync with existing resources? I am aware of terraform import but to my knowledge that only works with one resource at a time.
One of the functions of Terraform "state" is to allow Terraform to recognize which subset of objects in your account it's responsible for managing, and so unfortunately there is no built-in mechanism in Terraform to automatically recover that information: terraform import is the way to explicitly tell Terraform that it should begin managing an existing object.
However, there are some external tools that make use of additional knowledge about particular cloud platforms in order to provide a higher-level interface to importing. One example is terraformer, which allows you to import a set of objects matching a query, as long as the objects you want to import are in the subset of types it supports.
Related
I am required to modify pre-existing resources using cdk. I understand that there are methods to call upon that allows you to import a given resource, but how can I go about modifying that resource? For example, I have an RDS that was manually created and I want to change the instance type after creation. How do I go about doing that using cdk/cloue formation?
If you'd like to 'take ownership' of a resource in AWS with CloudFormation you can follow the steps outline here. In short:
Create a CloudFormation template that only has the one RDS resource you'd like to take ownership of. Have the template match the resource as much as possible. You should use the CDK to do this, just make sure your CDK code ONLY includes those resources you want to import. That can sometimes be tricky since L2 constructs often create more than just one CloudFormation resource. Trim back the synthesized CloudFormation template as much as needed to get just the one RDS resource you want to import.
Create a new Stack using the 'import' option. It's important that the only resources in the template are resources you are trying to import and 'take ownership of'.
Run Drift Detection and correct anything that is out of sync by updating your template and then running additional Update Stack steps.
You can, of course, have the CDK generate this template. Same rules apply, though. You need to make sure you have only the RDS instance.
Please refer to this post as I go into more detail there.
Additionally, there is a command link option, cdk import which can help do this (not detailed here or in the blog, though).
Once you have the resource imported into a stack you can continue making future changes using the CDK.
There is a terraform code to configure an MWAA Environment in AWS. When it runs second time, no need to create IAM role or policy again. So it gives an error.
How to ignore the creation of existing resources in TF?
I assume that you applied a Terraform plan which created resource "MWAA", then you somehow lost the state (locally stored and lost?, or the state wasn't shared with a different client?), then you re-apply the plan again, and Terraform informs you that it created "MWAA", again.
In that case, your main problem is that you lost the state, and you need to make sure that you do persist it, e.g., by storing it in a bucket.
However, if you really need to make Terraform aware about an already created resource, you need to put it in Terraform's state. One tool to do that is "terraform import", about which you can read more here: https://www.terraform.io/cli/import
If you already have the statefile and if terraform is trying to re-install it again, then may be some tag change or modified timestamp value change...
In order to avoid it, you can specify the resource you want to apply using a terraform apply command..
terraform apply --target=resource
Could you please suggest easy way to copy/move already created AWS stuffs from current VPC to another ?
There is no "easy" way but you can use different tools to import existing resources and deploy them to different VPCs.
There is a free tool called Former2 (https://github.com/iann0036/former2) which can be used to scan existing resources and produce outputs. These outputs can be used to deploy new resources. I tested this tool and it seems to be quite intuitive to use to gather information about existing resources and produce outputs in different template languages (Cloudformation, Terraform, CDK, Stroposphere, Pulumi).
Terraform can be used to import existing resources to the current state and after that copy resources to configuration. Future versions of Terraform will be able to also update the configuration. To use terraform you must know every resource you want to import and use import command with their ids. Terraform does not support nested imports so importing vpc does not add subnets or other resources to the state but they have to be imported separately.
I think CloudFormation has also import feature but specific resource template must be written beforehand and submitted during the import. This is not an easy or fast way to copy resources but should work and as an end product there should a template which can be used to deploy resources to another VPCs.
When deciding between Terraform and AWS CloudFormation, one advantage generally associated with Terraform is the separation of planning (terraform plan) and execution (terraform apply).
I recently heard about CloudFormation's create-change-set feature, which is described as:
Creates a list of changes that will be applied to a stack so that you can review the changes before executing them.
In Updating Stacks Using Change Sets from the AWS documentation, they describe the recommended update process.
Now I wonder:
Doesn't create-change-set also allow to separate planning from execution?
If not, what are the shortcomings in comparison with Terraform's plan & apply?
Background: Examples where the separation is emphasized as a significant advantage:
Official comparison on terraform.io:
Terraform also separates the planning phase from the execution phase, by using the concept of an execution plan. By running terraform plan, the current state is refreshed and the configuration is consulted to generate an action plan. The plan includes all actions to be taken: which resources will be created, destroyed or modified. It can be inspected by operators to ensure it is exactly what is expected.
[...]
Terraform lets operators apply changes with confidence, as they know exactly what will happen beforehand.
(Possibly outdated) Blog post from 2014 comparing Terraform and CloudFormation:
Infrastructure Updates: This is the absolute killer feature of Terraform. Terraform has a separate planning and execution phase. The planning phase shows which resources will be created, modified and destroyed. It gives you complete control of how your changes will affect the existing environment, which is quite crucial. This was one of the main reasons why we went ahead with Terraform. CloudFormation does not show you what changes it is going to make to the environment.
A CloudFormation Change Set is an analogous object to a Terraform plan saved in a plan file. Its core functionality is the same: inspect a report of the changes required to reach a new desired state, and then apply those changes.
When Terraform was originally created, CloudFormation did not have this feature. The documentation you have found was accurate at the time it was written, but with the release of Change Sets there is not a significant difference in this regard.
The other difference covered in the official comparison still applies, however:
Terraform similarly uses configuration files to detail the
infrastructure setup, but it goes further by being both cloud-agnostic
and enabling multiple providers and services to be combined and
composed. For example, Terraform can be used to orchestrate an AWS and
OpenStack cluster simultaneously, while enabling 3rd-party providers
like Cloudflare and DNSimple to be integrated to provide CDN and DNS
services. This enables Terraform to represent and manage the entire
infrastructure with its supporting services, instead of only the
subset that exists within a single provider. It provides a single
unified syntax, instead of requiring operators to use independent and
non-interoperable tools for each platform and service.
Is there a way to create the terraform state file, from the existing infrastructure. For example, an AWS account comes with a some services already in place ( for ex: default VPC).
But terraform, seems to know only the resources, it creates. So,
What is the best way to migrate an existing AWS Infrastructure to Terraform code
Is it possible to add a resource manually and modify the state file manually (bad effects ?)
Update
Terraform 0.7.0 supports importing single resource.
For relatively small things I've had some success in manually mangling a state file to add stubbed resources that I then proceeded to Terraform over the top (particularly with pre-existing VPCs and subnets and then using Terraform to apply security groups etc).
For anything more complicated there is an unofficial tool called terraforming which I've heard is pretty good at generating Terraform state files but also merging with pre-existing state files. I've not given it a go but it might be worth looking into.
Update
Since Terraform 0.7, Terraform now has first class support for importing pre-existing resources using the import command line tool.
As of 0.7.4 this will import the pre-existing resource into the state file but not generate any configuration for the resource. Of course if then attempt a plan (or an apply) Terraform will show that it wants to destroy this orphaned resource. Before running the apply you would then need to create the configuration to match the resource and then any future plans (and applys) should show no changes to the resource and happily keep the imported resource.
Use Terraforming https://github.com/dtan4/terraforming , To date it can generate most of the *.tfstate and *.tf file except for vpc peering.