I'd like to iterate over a map variable and concatenate individual strings to a single string for an S3 subfolder.
My expected output is:
s3://my-bucket/name1/k1/abc/
s3://my-bucket/name1/k2/def/
s3://my-bucket/name2/k3/xyz/
But with the current setup I get a Cannot include the given value in a string template: string required. Is there another way to loop over my source_names variable or should I set it up differently?
main.tf s3_buckets modules
resource "aws_s3_bucket" "bucket" {
bucket = var.bucket_name
acl = "private"
tags = {
Name = var.tag
}
}
resource "aws_s3_object" "new_folders" {
for_each = var.source_names
bucket = aws_s3_bucket.bucket.id
acl = "private"
key = "${each.key}/${each.value.key_name}/${each.value.key_value}"
}
resource "aws_s3_bucket_public_access_block" "example" {
bucket = aws_s3_bucket.bucket.id
block_public_acls = true
block_public_policy = true
}
variables.tf s3_buckets_module
variable "bucket_name" {
description = "Name of the bucket"
type = string
}
variable "tag" {
description = "Resource tag"
type = string
}
variable "source_names" {
description = "key"
type = map(object({
key_name = list(string)
key_value = list(string)
}))
}
main.tf pipeline module
module "s3_bucket" {
source = "../s3_buckets"
bucket_name = "tf-list-keys-bucket"
tag = "tf"
source_names = {"name1" = {"key_name" = ["k1", "k2"], "key_value" = ["abc/", "def/"]},
"name2" = {"key_name" = ["k3"], "key_value" = ["xyz/"]}
}
}
main.tf
module "pipeline" {
source = "../modules/pipeline"
}
Iterating over your source_names requires you to have 2 embedder for loops:
resource "aws_s3_object" "new_folders" {
for_each = toset(flatten([
for key, value in var.source_names : [
for key_name, key_values in value : [
for key_value in key_values :
"${key}/${key_name}/${key_value}"
]]
]))
bucket = aws_s3_bucket.bucket.id
acl = "private"
key = each.value
}
You have to flatten out your data structure, otherwise Terraform wont know how many resources to provision. You cannot just put an array in a string and expecting Terraform to iterate over it (example: key = "${each.key}/${each.value.key_name}/${each.value.key_value}" - each.value.key_value is an array).
Related
I am deploying an array of crawlers from terraform, I want to have a list of 2 or more crawlers to deploy at the same time. I am using a foreach but for the dynamodb_target I can't find how to associate it to a specific crawler, that is, crawler 1 has table_name 1 as dynamodb_target an so on.
main.tf
resource "aws_glue_crawler" "example" {
for_each = var.crawlerList
database_name = each.value.database_name
name = each.value.crawler_name
role = each.value.role
dynamodb_target {
path = var.table_name
}
}
variables.tf
variable "table_name" {
type = string
description = "The name of the DynamoDB table to crawler."
}
variable "crawlerList" {
type = map(object({
database_name = string
crawler_name = string
role = string
}))
}
var.tfvars
table_name = "ref_master_files"
crawlerList = {
fisrt = ({
database_name = "dm_web"
crawler_name = "example_one"
role = "xxxx"
}),
second = ({
database_name = "dm_web"
crawler_name = "example_two"
role = "xxxx"
}),
third = ({
database_name = "dm_web"
crawler_name = "example_third"
role = "xxxx"
}),
}
Result
It's little difficult to get what you want to achieve so I have two options.
You want to assign every crawler different table_name. Then - expand your variable for another value:
variable "crawlerList" {
type = map(object({
database_name = string
crawler_name = string
role = string
table_name = string
}))
}
You have some table_names variable of multiple variables which you didn't attach. Then what you need to is somehow match them.
It could be something like this:
tables = ["table1", "table2"]
matches = {"table1": "first", "table2":"second"}
Then you would iterate like this:
resource "aws_glue_crawler" "example" {
for_each = var.matches
database_name = var.crawler_list[each.value].database_name
name = var.crawler_list[each.value].crawler_name
role = var.crawler_list[each.value].role
dynamodb_target {
path = var.tables[each.key]
}
}
We are currently in the process of editing our core s3 module to adapt to the new V4.0 changes that terraform released -
Existing main.tf
bucket = local.bucket_name
tags = local.tags
force_destroy = var.force_destroy
dynamic "logging" {
for_each = local.logging
content {
target_bucket = logging.value["target_bucket"]
target_prefix = logging.value["target_prefix"]
}
}
....
}
I am trying to convert this to use the resource aws_s3_bucket_logging as below
resource "aws_s3_bucket" "bucket" {
bucket = local.bucket_name
tags = local.tags
force_destroy = var.force_destroy
hosted_zone_id = var.hosted_zone_id
}
resource "aws_s3_bucket_logging" "logging" {
bucket = aws_s3_bucket.bucket.id
dynamic "logging" {
for_each = local.logging
content {
target_bucket = logging.value["target_bucket"]
target_prefix = logging.value["target_prefix"]
}
}
locals.tf
locals {
logging = var.log_bucket == null ? [] : [
{
target_bucket = var.log_bucket
target_prefix = var.log_prefix
}
]
....
variables.tf
type = string
default = null
description = "The name of the bucket that will receive the log objects."
}
variable "log_prefix" {
type = string
default = null
description = "To specify a key prefix for log objects."
}
And I receive the error
Error: Unsupported block type
Blocks of type "logging" are not expected here.
Any help is greatly appreciated. TA
If you check TF docs aws_s3_bucket_logging, you will find that aws_s3_bucket_logging does not have any block nor attribute called logging. Please have a look at the docs linked, and follow the examples and the documentation.
Although, Is it the right usage of the resource if i remove the locals and simplify the resource to be as below? All am i trying to is pass null as default value
resource "aws_s3_bucket_logging" "logging" {
bucket = aws_s3_bucket.bucket.id
target_bucket = var.log_bucket
target_prefix = var.log_prefix
}
main.tf
module "iam_assumable_role" {
for_each = var.service_accounts
source = "../../../../../../modules/iam-assumable-role-with-oidc/"
create_role = true
role_name = each.value.name
provider_url = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
// role_policy_arns = [for i in each.value.policies : "aws_iam_policy.${i}.arn"]
oidc_fully_qualified_subjects = each.value.wildcard == "" ? ["system:serviceaccount:${each.value.namespace}:${each.value.name}"] : []
oidc_subjects_with_wildcards = each.value.wildcard != "" ? ["system:serviceaccount:${each.value.namespace}:${each.value.wildcard}"] : []
tags = var.tags
}
resource "aws_iam_policy" "dev-policy1" {
name_prefix = "dev-policy"
description = "some description"
policy = data.aws_iam_policy_document.dev-policy1.json
}
variable "service_accounts" {
type = map(object({
name = string
namespace = string
wildcard = string
policies = list(any)
}))
}
tfvars
service_accounts = {
"dev-sa" = {
"name" = "dev-sa",
"namespace" = "dev",
"wildcard" = "*",
"policies" = ["dev-policy1", "dev-policy2"]
},
"qa-sa" = {
"name" = "qa-sa",
"namespace" = "qa",
"wildcard" = "*",
"policies" = ["qa-policy1", "qa-policy2"]
}
}
My code is iterating over service_accounts variable and creates appropriate resources. The problem is that in the commented line I cannot get the list of aws_iam_policy.arn s for the provided policy names (policy names are provided through service_account variable). My current code returns the aws_iam_policy.PolicyName.arn as string and not the actual value. Note that dev-policy1 resource s just one of the all policy resources. All policy documents exist as well. module itself is working correctly when I provide policy list directly and not through variable.
Is it possible to achieve the desired in terraform at all?
You have to use for_each, to create your policies, as you can't dynamically references individual resources the way you are trying to do:
# get all policy names. Your names are unique, so its fine to use list
locals {
policy_names = flatten(values(var.service_accounts)[*]["policies"])
}
# create policy for each name in `policy_names`
resource "aws_iam_policy" "policy" {
for_each = local.policy_names
name_prefix = "dev-policy"
description = "some description"
# similar must be done below
# policy = data.aws_iam_policy_document.dev-policy1.json
}
Then you refer to them as:
role_policy_arns = [for i in each.value.policies: aws_iam_policy[${i}].arn]
I am trying to pass the values s3 name and create_user into local block in main.tf so that both of them have the value in list and then I am passing list_of_bucket in local block in module s3 to create the buckets and looping of user_to_create in module s3_user to create the user if the boolean is set to true. All of these values are passed to variable.tf and then to main.tf
dev.tfvars
wea-nonprod = {
services = {
s3 = [
sthree = {
create_user = true,
}
sfour = {
create_user = true,
}
sfive = {
create_user = true,
}
]
}
}
variable.tf
variable "s3_buckets" {
type = list(map)
}
main.tf
locals {
users_to_create = ""
list_of_buckets = ""
}
module "s3" {
source = "../../s3"
name = join("-", [var.name_prefix, "s3"])
tags = merge(var.tags, {Name = join("-", [var.name_prefix, "s3"])})
buckets = list_of_buckets
sse_algorithm = "AES256"
access_log_bucket_name = var.access_log_bucket_name
}
module "s3_user" {
for_each = local.users_to_create
source = "./service-s3-bucket-user"
name = join("-", [var.name_prefix, each.key])
tags = var.tags
bucket_arn = module.s3.bucket_arns[each.key]
depends_on = [module.s3]
}
Just iterate over your wea-nonprod map:
locals {
users_to_create = [ for name in var.wea-nonprod.services.s3 if name.create_user == true ]
list_of_buckets = [ for bucket in var.wea-nonprod.services.s3 ]
}
And a few changes to your module blocks:
module "s3" {
source = "../../s3"
name = "${var.name_prefix}-s3"
tags = merge(var.tags, { Name = "${var.name_prefix}-s3" })
buckets = local.list_of_buckets
sse_algorithm = "AES256"
access_log_bucket_name = var.access_log_bucket_name
}
module "s3_user" {
count = length(local.users_to_create)
source = "./service-s3-bucket-user"
name = "${var.name_prefix}${local.users_to_create[count.index]}"
tags = var.tags
bucket_arn = module.s3.bucket_arns[local.users_to_create[count.index]]
depends_on = [module.s3]
}
Creating a bucket is pretty simple.
resource "aws_s3_bucket" "henrys_bucket" {
bucket = "${var.s3_bucket_name}"
acl = "private"
force_destroy = "true"
}
Initially I thought I could create a list for the s3_bucket_name variable but I get an error:
Error: bucket must be a single value, not a list
-
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
How can I create multiple buckets without duplicating code?
You can use a combination of count & element like so:
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
resource "aws_s3_bucket" "henrys_bucket" {
count = "${length(var.s3_bucket_name)}"
bucket = "${element(var.s3_bucket_name, count.index)}"
acl = "private"
force_destroy = "true"
}
Edit: as suggested by #ydaetskcoR you can use the list[index] pattern rather than element.
variable "s3_bucket_name" {
type = "list"
default = ["prod_bucket", "stage-bucket", "qa_bucket"]
}
resource "aws_s3_bucket" "henrys_bucket" {
count = "${length(var.s3_bucket_name)}"
bucket = "${var.s3_bucket_name[count.index]}"
acl = "private"
force_destroy = "true"
}