How to instantiate contents of rendered template_file? - amazon-web-services

I have a template file that I use to on a list:
variable "users" {
type = "list"
default = [
"blackwidow",
"hulk",
"marvel",
]
}
// This will loop through the users list above and render out code for
// each item in the list.
data "template_file" "init" {
template = file("user_template.tpl")
count = length(var.users)
vars = {
username = var.users[count.index]
bucketid = aws_s3_bucket.myFTP_Bucket.id
}
}
The template file has multiple aws resources like
- "aws_transfer_user"
- "aws_s3_bucket_object"
- "aws_transfer_ssh_key"
etc... In fact it can have more stuff than just that. It also has some terraform variables in there too.
This data template works great in rendering out the contents of the template file, substituting in the names of my users.
But that's all terraform does.
Terraform doesn't instantiate the rendered content of the template file. It just merely keeps it as a string and keeps it in memory. Kind of like the C preprocessor doing substitution, but not 'including' the file.
Kind of frustrating. I'd like Terraform to instantiate the contents of my rendered template file. How do I do this?

The template_file data source (along with the templatefile function that has replaced it for Terraform 0.12) are for string templating, not for modular Terraform configuration.
To produce a set of different resource instances per item in a collection, we use resource for_each:
variable "users" {
type = set(string)
default = [
"blackwidow",
"hulk",
"marvel",
]
}
resource "aws_transfer_user" "example" {
for_each = var.users
# ...
}
resource "aws_transfer_user" "example" {
for_each = var.users
# ...
}
resource "aws_s3_bucket_object" "example" {
for_each = var.users
# ...
}
resource "aws_transfer_ssh_key" "example" {
for_each = aws_transfer_user.example
# ...
}
Inside each of those resource blocks you can use each.key to refer to each one of the usernames. Inside the resource "aws_transfer_ssh_key" "example" block, because I used aws_transfer_user.example as the repetition expression, you can also use each.value to access the attributes of the corresponding aws_transfer_user object. That for_each expression also serves to tell Terraform that aws_transfer_ssh_key.example depends on aws_transfer_user.example.

Related

How to use 'depends_on' with 'for_each' in terraform?

I've got a terraform plan that creates a number of resources in a for_each loop, and I need another resource to depend_on those first ones. How can I do it without having to explicitly list them?
Here's the first resource (AWS API Gateway resource):
locals {
apps = toset(["app1", "app2", "app3"])
}
resource "aws_api_gateway_integration" "lambda" {
for_each = local.apps
rest_api_id = aws_api_gateway_rest_api.primary.id
resource_id = aws_api_gateway_resource.lambda[each.key].id
http_method = aws_api_gateway_method.lambda_post[each.key].http_method
integration_http_method = "POST"
type = "AWS_PROXY"
uri = aws_lambda_function.lambda[each.key].invoke_arn
}
Now I need to wait for all the 3 apps integrations before creating an API Gateway deployment:
resource "aws_api_gateway_deployment" "primary" {
rest_api_id = aws_api_gateway_rest_api.primary.id
depends_on = [
aws_api_gateway_integration.lambda["app1"],
aws_api_gateway_integration.lambda["app2"],
aws_api_gateway_integration.lambda["app3"],
]
However the list of apps keeps growing and I don't want to maintain it manually over here. Everywhere else I can simply use for or for_each together with local.apps but I can't figure out how to dynamically build the list for depends_on? Any ideas?
You have to do it manually. TF docs clearly explain that depends_on must be explicitly defined:
You only need to explicitly specify a dependency
Dependencies in Terraform are always between static blocks (resource, data, and module blocks mainly) and not between individual instances of those objects.
Therefore listing individual instances like you did in your example is redundant:
depends_on = [
aws_api_gateway_integration.lambda["app1"],
aws_api_gateway_integration.lambda["app2"],
aws_api_gateway_integration.lambda["app3"],
]
The above is exactly equivalent to declaring a dependency on the resource as a whole:
depends_on = [
aws_api_gateway_integration.lambda,
]
The for_each argument itself typically has its own dependencies (e.g. local.apps in your example) and so Terraform needs to construct the dependency graph before evaluating for_each. This means that there is only one node in the dependency graph representing the entire resource (including its for_each expression), and the individual instances are not represented in the plan-time dependency graph at all.
There is another way:
Create a module with for_each approach called aws_api_gateway_integration
Modify module to support all parameters needed
Then you can just call new objects to create - example
At the end you can put depends_on on a module - depends_on = [
module.logic_app
]

How can I use var in resource calling

I'm importing roles which already have been created in AWS console and unfortunately the names are strange. So in order to use those roles I am trying like this
I've two IAM roles as follows
data "aws_iam_role" "reithera-rtcov201" {
name = "exomcloudrosareitherartcov-YRX1M2GJKD6H"
}
data "aws_iam_role" "dompe-rlx0120" {
name = "exomcloudrosadomperlx0120p-1SCGY0RG5JXFF"
}
In this file I have 2 variables as follows:
sponsor = ["reithera", "dompe"]
study = ["rtcov201", "rlx0120"]
I'm trying in the following way, but terraform doesn't allow to use $.
data.aws_iam_role.${var.sponsor}-${var.study}.arn
Do you know any solution for this.
Its not possible. You can dynamically create references to resources.
Instead of two separate data sources you should create one:
variable "iam_roles"
default = ["exomcloudrosareitherartcov-YRX1M2GJKD6H", "exomcloudrosadomperlx0120p-1SCGY0RG5JXFF"]
}
and then
data "aws_iam_role" "role" {
for_each = toset(var.iam_roles)
name = each.key
}
and you can refer to them using role name:
data.aws_iam_role["exomcloudrosareitherartcov-YRX1M2GJKD6H"].arn

iterate over data source for a list of secrets arn

I have a few secerts in aws that were created manually. Is there a way to list them with data "aws_secretsmanager_secret"?
My goal is to get an list/index of the ARNs and then use it in a daymnic block. I want to try and avoid writing multiple data source blocks.
All the sercerts have a similar naming prefix:
db-credentials/${var.env-name}/<db-user>
The <db-user> changes of course from user to user.
So I guess I'm looking to iterate with data source over all secrets which falls into this naming pattern and get a list of their ARN. After that use each ARN indie a daymnic block
The daynic block will be used inside resource "aws_db_proxy" in the auth block
if anyone will find this useful I manged to do it like this:
locals {
secrets_list = [
"db-credentials/${var.env-name}/user1",
"db-credentials/${var.env-name}/user2",
"db-credentials/${var.env-name}/user3"
]
}
data "aws_secretsmanager_secret" "rds_secrets" {
for_each = toset(local.secrets_list)
name = each.key
}
resource "aws_db_proxy" "rds_db_proxy" {
name = "${var.env-name}-rds-proxy"
engine_family = "MYSQL"
idle_client_timeout = 900
require_tls = true
.
.
.
.
dynamic "auth" {
for_each = local.secrets_list
content {
secret_arn = data.aws_secretsmanager_secret.rds_secrets[auth.value].arn
auth_scheme = "SECRETS"
iam_auth = "REQUIRED"
}
}
}

JSON Syntax error in Terraform aws_iam_role_policy

So with Terraform, I'm creating an IAM policy and attaching it to a role. I am currently running:
Terraform v0.12.16
provider.aws v2.40.0
provider.template v2.1.2
When executing the code, I am able to initialize terraform with no issue. When running terraform plan, I am being presented with the following error:
Error: "policy" contains an invalid JSON: invalid character '}' looking for beginning of value
on ec2-iam.tf line 8, in resource "aws_iam_role_policy" "s3_ec2_policy":
8: resource "aws_iam_role_policy" "s3_ec2_policy" {
I am getting stuck with this error. Any advice would be helpful. Below is my code:
data "template_file" "s3_web_policy" {
template = file("scripts/iam/web-ec2-policy.json")
vars = {
s3_bucket_arn = "arn:aws:s3:::${var.my_app_s3_bucket}/*"
}
}
resource "aws_iam_role_policy" "s3_ec2_policy" {
name = "s3_ec2_policy"
role = aws_iam_role.s3_ec2_role.id
policy = data.template_file.s3_web_policy.rendered
}
resource "aws_iam_role" "s3_ec2_role" {
name = "s3_ec2_role"
assume_role_policy = file("scripts/iam/web-ec2-assume-role.json")
}
It's common to encounter syntax errors when generating JSON from string templates, because the template language isn't aware of JSON syntax and so you as the tempate author must take care to ensure that brackets are all nested correctly, there are no missing or extra commas, etc.
You can usually avoid problems of this type by generating JSON with Terraform's jsonencode function instead:
resource "aws_iam_role_policy" "s3_ec2_policy" {
name = "s3_ec2_policy"
role = aws_iam_role.s3_ec2_role.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
# etc, etc
]
})
}
If the policy definition seems too large to include inline in your resource block, you can still factor it out into a separate template file if desired:
resource "aws_iam_role_policy" "s3_ec2_policy" {
name = "s3_ec2_policy"
role = aws_iam_role.s3_ec2_role.id
policy = templatefile("${path.module}/scripts/iam/web-ec2-policy.json.tmpl", {
s3_bucket_arn = "arn:aws:s3:::${var.my_app_s3_bucket}/*"
})
}
...but inside the template, rather than using individual template interpolations, just write the entire template as a single call to jsonencode, like this:
${jsonencode({
Version = "2012-10-17"
Statement = [
{
# ...
Resource = s3_bucket_arn
# ...
},
# etc, etc
]
})}
Note that the template_file data source is for Terraform 0.11 and earlier, and is in Terraform 0.12 only for backward compatibility. You should use the templatefile function instead, which serves the same purpose but is integrated directly into the Terraform language.

Preventing destroy of resources when refactoring Terraform to use indices

When I was just starting to use Terraform, I more or less naively declared resources individually, like this:
resource "aws_cloudwatch_log_group" "image1_log" {
name = "${var.image1}-log-group"
tags = module.tagging.tags
}
resource "aws_cloudwatch_log_group" "image2_log" {
name = "${var.image2}-log-group"
tags = module.tagging.tags
}
resource "aws_cloudwatch_log_stream" "image1_stream" {
name = "${var.image1}-log-stream"
log_group_name = aws_cloudwatch_log_group.image1_log.name
}
resource "aws_cloudwatch_log_stream" "image2_stream" {
name = "${var.image2}-log-stream"
log_group_name = aws_cloudwatch_log_group.image2_log.name
}
Then, 10-20 different log groups later, I realized this wasn't going to work well as infrastructure grew. I decided to define a variable list:
variable "image_names" {
type = list(string)
default = [
"image1",
"image2"
]
}
Then I replaced the resources using indices:
resource "aws_cloudwatch_log_group" "service-log-groups" {
name = "${element(var.image_names, count.index)}-log-group"
count = length(var.image_names)
tags = module.tagging.tags
}
resource "aws_cloudwatch_log_stream" "service-log-streams" {
name = "${element(var.image_names, count.index)}-log-stream"
log_group_name = aws_cloudwatch_log_group.service-log-groups[count.index].name
count = length(var.image_names)
}
The problem here is that when I run terraform apply, I get 4 resources to add, 4 resources to destroy. I tested this with an old log group, and saw that all my logs were wiped (obviously, since the log was destroyed).
The names and other attributes of the log groups/streams are identical- I'm simply refactoring the infrastructure code to be more maintainable. How can I maintain my existing log groups without deleting them yet still refactor my code to use lists?
You'll need to move the existing resources within the Terraform state.
Try running terraform show to get the strings under which the resources are stored, this will be something like [module.xyz.]aws_cloudwatch_log_group.image1_log ...
You can move it with terraform state mv [module.xyz.]aws_cloudwatch_log_group.image1_log '[module.xyz.]aws_cloudwatch_log_group.service-log-groups[0]'.
You can choose which index to assign to each resource by changing [0] accordingly.
Delete the old resource definition for each moved resource, as Terraform would otherwise try to create a new group/stream.
Try it with the first import and check with terraform plan if the resource was moved correctly...
Also check if you need to choose some index for the image_names list jsut to be sure, but I think that won't be necessary.