Recently, we had issues with tfstate being deleted on S3.
As a result, there are a number of EC2 instances still running (duplicates if you will)
Is there a way to query Terraform and list which EC2 instances (and other resources) Terraform has under its control? I want to delete the duplicate AWS resources without messing up Terraform state.
Depending on whether you care about availability you could just delete everything and let Terraform recreate it all.
Or you could use terraform state list and then iterate through that with terraform state show (eg. terraform state list | xargs terraform state show) to show everything.
terraform import is for importing stuff that exists back in to your state which doesn't sound like what you want because it sounds like you've already recreated some things so have duplicates. If you had caught the loss of the resources from your state file before Terraform recreated it (for example by seeing an unexpected creation in the plan and seeing that the resource already existed in the AWS console) then you could have used that to import the resources back into the state file so that Terraform would then show an empty plan for these resources.
Iin the future make sure you use state locking to prevent this from happening again!
Related
There is a terraform code to configure an MWAA Environment in AWS. When it runs second time, no need to create IAM role or policy again. So it gives an error.
How to ignore the creation of existing resources in TF?
I assume that you applied a Terraform plan which created resource "MWAA", then you somehow lost the state (locally stored and lost?, or the state wasn't shared with a different client?), then you re-apply the plan again, and Terraform informs you that it created "MWAA", again.
In that case, your main problem is that you lost the state, and you need to make sure that you do persist it, e.g., by storing it in a bucket.
However, if you really need to make Terraform aware about an already created resource, you need to put it in Terraform's state. One tool to do that is "terraform import", about which you can read more here: https://www.terraform.io/cli/import
If you already have the statefile and if terraform is trying to re-install it again, then may be some tag change or modified timestamp value change...
In order to avoid it, you can specify the resource you want to apply using a terraform apply command..
terraform apply --target=resource
I have a module with various batch service resources such as job definition, compute environment, and job queue.
Today I was unable to perform terraform apply due to an error with updating one of these Batch related resources (unfortunately I have lost those initial error messages).
I have run into something like this before and cleared the hurdle by manually disabling/deleting the job queue. So I did that again via the AWS Console and ran terraform apply, however this time the plan/apply still didn't work, perhaps because the queue had a single outstanding job in runnable but unstarted state (the previous times I've done this successfully the job queue was empty). So I did the same thing with the compute environment (disabled/deleted via AWS Console) and now I get messages such as this with terraform plan:
2020/04/30 15:15:43 [ERROR] module.batch_data_load: eval: *terraform.EvalRefresh, err: One compute environment is expected, but AWS return no compute environment
2020/04/30 15:15:43 [ERROR] module.batch_data_load: eval: *terraform.EvalSequence, err: One compute environment is expected, but AWS return no compute environment
So it seems that I've bungled my state file by monkeying with the Terraform managed resources via AWS Console.
How can I apply or destroy after removing resources via AWS Console?
Terraform does try to resynchronize its own idea of the state with the remote objects as part of creating a plan, but it doesn't always work because sometimes remote objects interact in ways Terraform can't account for, or other reasons where Terraform doesn't have enough information to fully resynchronize.
If you know that the remote object associated with a particular resource instance in your configuration is deleted, you can tell Terraform to "forget" that object and remove it from the state altogether, using the terraform state rm command:
terraform state rm aws_batch_job_queue.example
You should do this only if you know for certain that the object no longer exists, or if you intend to continue managing the existing object using some other system, because once Terraform has forgotten about the object it will never plan to destroy it and so the object can be left active forever.
A contractor built an application's AWS infrastructure on his local laptop, never commited his code then left (wiping the notebook's HD). But he did create the infrastructure with Terraform and stored the remote state in an s3 bucket, s3://analytics-nonprod/analytics-dev.tfstate.
This state file includes all of the VPC, subnets, igw, nacl, ec2, ecs, sqs, sns, lambda, firehose, kinesis, redshift, neptune, glue connections, glue jobs, alb, route53, s3, etc. for the application.
I am able to run Cloudformer to generate cloudformation for the entire infrastructure, and also tried to import the infrastructure using terraformer but terraformer does not include neptune and lambda components.
What is the best way/process to recreate a somewhat usable terraform just from the remote state?
Should I generate some generic :
resource "aws_glue_connection" "dev" { }
and run "terraform import aws_glue_connection.dev"
then run "terraform show"
for each resource?
Terraform doesn't have a mechanism specifically for turning existing state into configuration, and indeed doing so would be lossy in the general case because the Terraform configuration likely contained expressions connecting resources to one another that are not captured in the state snapshots.
However, you might be able to get a starting point -- possibly not 100% valid but hopefully a better starting point than nothing at all -- by configuring Terraform just enough to find the remote state you have access to, running terraform init to make Terraform read it, and then run terraform show to see the information from the state in a human-oriented way that is designed to resemble (but not necessarily exactly match) the configuration language.
For example, you could write a backend configuration like this:
terraform {
backend "s3" {
bucket = "analytics-nonprod"
key = "analytics-dev.tfstate"
}
}
If you run terraform init with appropriate AWS credentials available then Terraform should read that state snapshot, install the providers that the resource instances within it belong to, and then leave you in a situation where you can run Terraform commands against that existing state. As long as you don't take any actions that modify the state, you should be able to inspect it with commands like terraform show.
You could then copy the terraform show output into another file in your new Terraform codebase as a starting point. The output is aimed at human consumption and is not necessarily all parsable by Terraform itself, but the output style is similar enough to the configuration language that hopefully it won't take too much effort to massage it into a usable shape.
One important detail to watch out for is the handling of Terraform modules. If the configuration that produced this state contained any module "foo" blocks then in your terraform show output you will see some things like this:
# module.foo.aws_instance.bar
resource "aws_instance" "bar" {
# ...
}
In order to replicate the configuration for that, it is not sufficient to paste the entire output into one file. Instead, any resource block that has a comment above it indicating that it belongs to a module will need to be placed in a configuration file belonging to that module, or else Terraform will not understand that block as relating to the object it can see in the state.
I'd strongly suggest taking a backup copy of the state object you have before you begin, and you should be very careful not to apply any plans while you're in this odd state of having only a backend configuration, because Terraform might (if it's able to pick up enough provider configuration from the execution environment) plan to destroy all of the objects in the state in order to match the configuration.
I am creating AWS infrastructure using Terraform, and using S3 backend configuration. Now the issue is, someone deleted the S3 bucket storing the state, and now every time I run terraform it fails saying the resources already exist. The old tfstate is lost, and new has no information about existing resources.
Note: I do not have write access to the AWS environment. I trigger terraform via Jenkins CD pipeline, so I cannot manually modify the infrastructure or run any terraform command.
Is there a way to cleanup existing resources or force recreating resources(if they already exist) with tf file? This is the only place I can make changes.
You really are in a mess. You need to restore the S3 bucket or make a new one and point your code at that.
You then need to recreate the state you lost, that or delete every object you created via Terraform and start again. Most objects have the ability to import existing objects via the Terraform import command.
This could be a significantly large task.
And you'd be needing write access to the bucket? Terraform refresh is only going to help if you still had the state file. You don't.
If you haven't got permission to do that, then maybe give up that or persist in getting sufficient privilege.
If you can't run Terraform locally then you are also wasting your time.
Good luck.
However....
You don't want to be here again. How did you delete/lose the bucket?
You really need that never to happen again as #ydaetskcoR said some MFA protection on the bucket - definitely do that and adding versioning to it is a REALLY good idea.
Also if you haven't added DynamoDB locking to the bucket do so, its really worth it.
This may also be a good time to think/dwell about tagging your infrastructure, then you might be able to identify what infra belonged to the code. That would also help - well next time.
If you're working in a production account, follow the advice of others and do not mess with anything in the account manually!
If you are just starting out with terraform or terragrunt and you're trying to reset the terraform state:
Ensure that you are logged into the correct AWS account, it is not a production account, and it is not in use by anyone else
Terraform state is saved in two places under the same name: S3 and DynamoDB. In order to reset the state, you will need to delete both.
If there is anything else that was previously created in the account, you will need to delete those manually. Depending on how much you created, this could take a very long time.
Once you have deleted the S3 and DynamoDB named after your terraform state bucket and deleted the infrastructure created by terraform, you should be able to terraform init and terraform plan without further issue.
Is there a way to create the terraform state file, from the existing infrastructure. For example, an AWS account comes with a some services already in place ( for ex: default VPC).
But terraform, seems to know only the resources, it creates. So,
What is the best way to migrate an existing AWS Infrastructure to Terraform code
Is it possible to add a resource manually and modify the state file manually (bad effects ?)
Update
Terraform 0.7.0 supports importing single resource.
For relatively small things I've had some success in manually mangling a state file to add stubbed resources that I then proceeded to Terraform over the top (particularly with pre-existing VPCs and subnets and then using Terraform to apply security groups etc).
For anything more complicated there is an unofficial tool called terraforming which I've heard is pretty good at generating Terraform state files but also merging with pre-existing state files. I've not given it a go but it might be worth looking into.
Update
Since Terraform 0.7, Terraform now has first class support for importing pre-existing resources using the import command line tool.
As of 0.7.4 this will import the pre-existing resource into the state file but not generate any configuration for the resource. Of course if then attempt a plan (or an apply) Terraform will show that it wants to destroy this orphaned resource. Before running the apply you would then need to create the configuration to match the resource and then any future plans (and applys) should show no changes to the resource and happily keep the imported resource.
Use Terraforming https://github.com/dtan4/terraforming , To date it can generate most of the *.tfstate and *.tf file except for vpc peering.