I'm attempting to figure out if there's a way to check if a resources is already created or enabled vi terraform, and if so, not to have terraform preform the resource creation. My issue is that there was no constancy when the AWS resources were built as it was done all manually, and I don't want to recreate resources or attempt to enable resource/services if they have been preformed/created. So I'm wondering if there is a way, I can find if a resource has been created or enabled, and skip that. For example, is there a way to check if CloudTrail already present or if Guardduty enabled or if Config enabled. I have multiple accounts, and can't go through them manually, and trying to find a programmatic way of doing this. I know I can use "count" but I need to explicitly identify what I'm looking to validate and was hoping there's a simpler way for terraform to pull information and in which case I can validate and use count.
Related
There is a terraform code to configure an MWAA Environment in AWS. When it runs second time, no need to create IAM role or policy again. So it gives an error.
How to ignore the creation of existing resources in TF?
I assume that you applied a Terraform plan which created resource "MWAA", then you somehow lost the state (locally stored and lost?, or the state wasn't shared with a different client?), then you re-apply the plan again, and Terraform informs you that it created "MWAA", again.
In that case, your main problem is that you lost the state, and you need to make sure that you do persist it, e.g., by storing it in a bucket.
However, if you really need to make Terraform aware about an already created resource, you need to put it in Terraform's state. One tool to do that is "terraform import", about which you can read more here: https://www.terraform.io/cli/import
If you already have the statefile and if terraform is trying to re-install it again, then may be some tag change or modified timestamp value change...
In order to avoid it, you can specify the resource you want to apply using a terraform apply command..
terraform apply --target=resource
As described in the documentation, I think CloudFormation will update with no interruption just by changing to add a tag.
In my case, another team uses terraform to add tag resources with generic tags, and my team uses CloudFormation to update the application-specific tags. Does this cause replacement...?
Also, this only occurred for limited resources such as security groups.
Does anyone know anything about this issue?
Cloudformation is a Jealous Tool - in otherwords, if it doesn't control the Resource in entirety it will overwrite changes, assuming that the changes are in a location it thinks it has to update.
In order for CloudFormation stacks not to do work that is not needed they generate ChangeSets - indicating what resources need updating/rebuilding/ect. If you don't change anything in the template (or your cdk stack) then the ChangeSet won't update that resource cause nothing needs to change. Depending on the resource (and it varies from service to service) sometimes changes made outside of CloudFormation control (by either other services like Terraform or by manual changes in the console - both of which are referred to as 'Drift') dont get overwritten by the stack updating (most common I can think of is API gateway - adding / deleting resources or methods doesn't always get re done by a redeploy of the stack.
However, for things like tags, yes - it will overwrite any changes made in there at any time and reapply the tags as of the moment it deploys - (again, depending on the Service) often even if there are no other changes to deploy for that given resource. I suspect the reason for this is because of the way AWS uses tags on the backend to sort and search resources.
Your best bet is to pick one service and do everything with it.
Each of these two tools will overwrite the tags the other created. You can configure Terraform to ignore certain tags However I'm not sure you can configure CloudFormation to ignore any tags. I think CloudFormation will always delete tags that it doesn't manage. In general it's not going to work well using both CloudFormation and Terraform to manage the same resources.
For a couple of days I am working on a way to generate a list of non-terraform created resources. Currently I am using the tagging method but this means that AWS resources that can't be tagged should also be excluded from the AWS Config query.
The tagging part is also quite cumbersome as this must be done manually.
Is there a way to tell Terraform (or Terragrunt) to automatically use general tags on resources?
Is there a way to ask Terraform or any other tool to create a list of resources that are not available in the state files?
Is there a way to tell Terraform (or Terragrunt) to automatically use general tags on resources?
No, this is not possible. Arguably, it would go against the declarative of Terraform's conceptual model, so this would not likely be made a feature of Terraform.
Terraform Enterprise has policy enforcement which could be used to compel users to provide specific tags, but I don't think it could automatically inject tags.
Is there a way to ask Terraform or any other tool to create a list of resources that are not available in the state files?
No. Terraform cares about the resources in its state file. It does not care about anything else.
Personally I think you are approaching this problem from the wrong angle. Asking Terraform "what do you not know about" would be convenient if it were possible (which it is not) but you would get a better answer if you consult a list of what it *does know about and infer from that whether a resource is or is not part of Terraform.
I don't know the tools you are using or the workflow you are hoping for, but you could do something like this:
Use the terraform show -json command to generate a JSON document representing all of your known infrastructure in the Terraform state file.
Use the tool jq to parse the JSON for a list of resource IDs
Use the aws resourcegroupstaggingapi get-resources command to generate a list of all known resources, or perhaps even a tool like aws-list-all
Loop through each entry in the 'all resources' list; for each one, loop through each entry in the 'Terraform-managed' list; if there's no match, add the resource ID to a list of 'not-managed-by-Terraform' resources
for people still stumbling upon this. It seems that a community has created a tool named Driftctl that would check environments against your Terraform state.
Since there is no good way to know if a resource is managed by terraform or not, I would try the following approach:
Use terraformer to import all your resources (not all resources are supported by terraformer, so this is not a complete solution), and after you have this list just substract all the resources that are managed by your terraform states from it. When running terraform plan you can see your resources ids.
If you have tags for all your terraform managed resources you can filter them with terraformer
I agree with Chuppa Chump's answer & will add if it's not already obvious, to ensure all Terraform created resources have a tag indicating it was created by Terraform. For example, CreatedBy = "Terraform" OR Automation = "Terraform". And then any resource that doesn't have the Automation or CreatedBy Tag, was not created by terraform.
Would anybody have any clues as to how AWS CloudFormation works under the hood?
Also, would anybody know an open-source equivalent to AWS CF (and I don't mean tooling that may be using CloudFormation)?
It's clearly a powerful orchestrator, but I'd be keen to explore the inner workings of such tools.
AWS Cloudformation has multiple pre-defined set of schemas for each of the components that are supported. When you upload a Cloudformation template for creating resources, it performs the below steps:
It validates the templates against the schema
It generates dynamic form for gathering parameters
It validates the values of parameters
Once it has all it needs, you can click Create to begin with the resource creation
Under the hood, it starts creation of resources using the internal coding for which is keeps echoing the status and progress continuously on the console.
We need to understand here that internally Cloudformation in itself is a product that does use AWS SDK/CLI as needed. However, under the hood, it maintains its own data to compare the attributes and resources when you run an update.
An open source alternative to this is Terraform. Terraform is the most widely used open source replacement of Cloudformation. Terraform is known for its Cloud independent architecture. Terraform works with multiple clouds with minimal changes in the templates.
The under-the-hood working of terraform involves creation of a State file/directory where it stores the current state of any stack identified uniquely by the name provided by the user. Terraform creates resources majorly using Python SDK (boto3) and some other APIs as needed. We need to pass the access key and secrets to the Terraform configuration in order to enable it to access the AWS Cloud environment.
If you are looking to build a smart new alternative, it should be fairly simple considering that AWS strictly follows standard design patterns in its SDK and CLI interface design. This makes it easier to convert template into executable code.
More information about working of Cloudformation can be found here
Is it possible to use AWS CloudFormation Templates to validate that already existing AWS resources are configured properly? What I would like to do is create a JSON file according to the AWS EC2 Instance resource specification and validate that the instance is configured properly using BOTO3. I haven't been able to find this type of functionality in the AWS API, and was wondering if there was something that I missed, or there is a 3rd party tool that will do this for me.
Thank you for your time.
I don't believe this is possible - you'd run into some interesting issues in a lot of cases, as CloudFormation doesn't always name resources exactly the way you'd expect. Cloudformation is meant more for building resources and maintaining them, not verifying existing outside-of-cloudformation resources.
Cool idea for a tool though, not sure if anything like that already exists. It should be pretty easy to write something that snapshots various resources, then compares them over time using boto.
Good luck! If you write a tool (or find one) let me know - though almost all of our resources are managed in CloudFormation already (via stacker).