i'm trying to account for an annoying case when creating my own private hosted zones where the kinesis-streams endpoint has several non-standard DNS records. I am aiming to create a custom map of objects for each endpoint with information that I need (below):
locals {
endpoint_service_names = {
ec2 = {
name = "com.amazonaws.${data.aws_region.current.name}.ec2"
type = "Interface"
private_dns = false
phz_names = ["ec2.${data.aws_region.current.name}.amazonaws.com"]
phz_wildcard = false
}
"ecr.dkr" = {
name = "com.amazonaws.${data.aws_region.current.name}.ecr.dkr"
type = "Interface"
private_dns = false
phz_names = ["dkr.ecr.${data.aws_region.current.name}.amazonaws.com"]
phz_wildcard = true
}
"ecr.api" = {
name = "com.amazonaws.${data.aws_region.current.name}.ecr.api"
type = "Interface"
private_dns = false
phz_names = ["api.ecr.${data.aws_region.current.name}.amazonaws.com"]
phz_wildcard = false
}
kinesis-streams = {
name = "com.amazonaws.${data.aws_region.current.name}.kinesis-streams"
type = "Interface"
private_dns = false
phz_names = [
"kinesis.${data.aws_region.current.name}.amazonaws.com",
"data-kinesis.${data.aws_region.current.name}.amazonaws.com",
"control-kinesis.${data.aws_region.current.name}.amazonaws.com"
]
phz_wildcard = true
}
}
In order to use this in my impl, however, I would need to perform a nested for_each which Terraform does not directly allow, so I need to merge my map info into a new map with the service and DNS names in the one object. My problem is similar to: Looping in for_each nested resources with terraform
except I do not have a list in each map element, only on the phz_names. I can't figure out how to get the syntax right to produce something like this:
endpoint_service_dns_list = {
kinesis-streams = [
"kinesis.${data.aws_region.current.name}.amazonaws.com",
"data-kinesis.${data.aws_region.current.name}.amazonaws.com",
"control-kinesis.${data.aws_region.current.name}.amazonaws.com"
]
[...]
}
}
My attempt:
endpoint_service_dns_list = merge([
for servicename, service in local.endpoint_service_names : {
for phz_name in service[0].phz_names :
"${servicename}-${phz_name}" => {
service_name = servicename
phz_name = phz.phz_names
}
}
])
but the syntax highlighting/attempt to indexing obviously fails because I do not have a list for each service, but I am not sure what to replace it with.
Maybe I have missed the point but based on your comments and your kind of expected result in the question you could have something like this. I have added a format that maps then service name to all its urls as a list. Or incase you wanted a mapping of each url to its service name I added an additional local var.
locals {
endpoint_service_names = {
ec2 = {
name = "com.amazonaws.${data.aws_region.current.name}.ec2"
type = "Interface"
private_dns = false
phz_names = ["ec2.${data.aws_region.current.name}.amazonaws.com"]
phz_wildcard = false
}
"ecr.dkr" = {
name = "com.amazonaws.${data.aws_region.current.name}.ecr.dkr"
type = "Interface"
private_dns = false
phz_names = ["dkr.ecr.${data.aws_region.current.name}.amazonaws.com"]
phz_wildcard = true
}
"ecr.api" = {
name = "com.amazonaws.${data.aws_region.current.name}.ecr.api"
type = "Interface"
private_dns = false
phz_names = ["api.ecr.${data.aws_region.current.name}.amazonaws.com"]
phz_wildcard = false
}
kinesis-streams = {
name = "com.amazonaws.${data.aws_region.current.name}.kinesis-streams"
type = "Interface"
private_dns = false
phz_names = [
"kinesis.${data.aws_region.current.name}.amazonaws.com",
"data-kinesis.${data.aws_region.current.name}.amazonaws.com",
"control-kinesis.${data.aws_region.current.name}.amazonaws.com"
]
phz_wildcard = true
}
}
endpoint_service_dns_list = {for service, attrs in local.endpoint_service_names : service => attrs.phz_names}
endpoint_dns_service_list = merge([for service, attrs in local.endpoint_service_names : {for url in attrs["phz_names"] : url => service}]...)
}
data "aws_region" "current" {}
output "endpoint_service_dns_list" {
value = local.endpoint_service_dns_list
}
output "endpoint_dns_service_list" {
value = local.endpoint_dns_service_list
}
This would map the phz_names with the service as seen in the output, or map each url to its service
endpoint_dns_service_list = {
"api.ecr.eu-west-2.amazonaws.com" = "ecr.api"
"control-kinesis.eu-west-2.amazonaws.com" = "kinesis-streams"
"data-kinesis.eu-west-2.amazonaws.com" = "kinesis-streams"
"dkr.ecr.eu-west-2.amazonaws.com" = "ecr.dkr"
"ec2.eu-west-2.amazonaws.com" = "ec2"
"kinesis.eu-west-2.amazonaws.com" = "kinesis-streams"
}
endpoint_service_dns_list = {
"ec2" = [
"ec2.eu-west-2.amazonaws.com",
]
"ecr.api" = [
"api.ecr.eu-west-2.amazonaws.com",
]
"ecr.dkr" = [
"dkr.ecr.eu-west-2.amazonaws.com",
]
"kinesis-streams" = [
"kinesis.eu-west-2.amazonaws.com",
"data-kinesis.eu-west-2.amazonaws.com",
"control-kinesis.eu-west-2.amazonaws.com",
]
}
Hopefully one of these is the kind of thing you were looking to do
Related
Im trying to add retention policy but I want to enable it conditionally, as you can see from the code
buckets.tf
locals {
team_buckets = {
arc = { app_id = "20390", num_buckets = 2, retention_period = null }
ana = { app_id = "25402", num_buckets = 2, retention_period = 631139040 }
cha = { app_id = "20391", num_buckets = 2, retention_period = 631139040 } #20 year
}
}
module "team_bucket" {
source = "../../../../modules/gcs_bucket"
for_each = {
for bucket in flatten([
for product_name, bucket_info in local.team_buckets : [
for i in range(bucket_info.num_buckets) : {
name = format("%s-%02d", product_name, i + 1)
team = "ei_${product_name}"
app_id = bucket_info.app_id
retention_period = bucket_info.retention_period
}
]
]) : bucket.name => bucket
}
project_id = var.project
name = "teambucket-${each.value.name}"
app_id = each.value.app_id
team = each.value.team
retention_period = each.value.retention_period
}
root module is defined as follows
main.tf
resource "google_storage_bucket" "bucket" {
project = var.project_id
name = "${var.project_id}-${var.name}"
location = var.location
labels = {
app_id = var.app_id
ei_team = var.team
cost_center = var.cost_center
}
uniform_bucket_level_access = var.uniform_bucket_level_access
dynamic "retention_policy" {
for_each = var.retention_policy == null ? [] : [var.retention_period]
content {
retention_period = var.retention_period
}
}
}
but I can't seem to make the code pick up the value,
for example as you see below the value doesn't get implemented
~ resource "google_storage_bucket" "bucket" {
id = "teambucket-cha-02"
name = "teambucket-cha-02"
# (11 unchanged attributes hidden)
- retention_policy {
- is_locked = false -> null
- retention_period = 3155760000 -> null
}
}
variables.tf for retention policy is as follows
variable "retention_policy" {
description = "Configuation of the bucket's data retention policy for how long objects in the bucket should be retained"
type = any
default = null
}
variable "retention_period" {
default = null
}
Your var.retention_policy is always null, as its default value. You are not changing the default value at all. Probably you wanted the following:
for_each = var.retention_period == null ? [] : [var.retention_period]
instead of
for_each = var.retention_policy == null ? [] : [var.retention_period]
I'm trying to create certain BigQuery tables with time_partitioning with the dynamic block and I want to use the values from tfvars in runtime as follows:
./tables/tables.tf:
resource "google_bigquery_table" "tables" {
for_each = var.tables == [] ? [] : toset(var.tables)
dataset_id = var.db_id
deletion_protection = false
table_id = each.key
dynamic "time_partitioning" {
for_each = var.partitioned_tables
content {
type = "DAY"
field = time_partitioning.value.field
}
}
labels = {
environment = var.environment
application = var.application
}
schema = fileexists("${path.module}/${var.db_id}/${each.key}.json") ? file("${path.module}/${var.db_id}/${each.key}.json") : null
}
main.tf:
resource "google_bigquery_dataset" "database" {
count = length(var.dbs)
dataset_id = var.dbs[count.index].db_id
friendly_name = var.dbs[count.index].db_name
description = "TF"
location = "US"
delete_contents_on_destroy = var.delete_contents_on_destroy
labels = {
environment = var.environment
application = var.dbs[count.index].app_name
}
}
module "tables" {
source = "./tables"
count = length(var.dbs)
db_id = google_bigquery_dataset.database[count.index].dataset_id
environment = var.environment
application = var.dbs[count.index].app_name
tables = var.dbs[count.index].tables
partitioned_tables = var.dbs[count.index].partitioned_tables
}
module "iam" {
source = "./iam"
count = length(var.dbs)
db_id = google_bigquery_dataset.database[count.index].dataset_id
iam_members = var.dbs[count.index].iam_members
}
dev.tfvars:
region = "us-central1"
project_id = "some-project"
dbs = [
{
db_id = "dataset1"
db_name = "dataset1"
app_name = "hello"
iam_members = [
{
role = "roles/bigquery.dataEditor"
member = "serviceAccount:ser-sa#some-project.iam.gserviceaccount.com",
}
]
tables = ["daily_inventory", "dc_inventory", "products", "daily_sales", "planned_inventory", "stores", "stores_in_program"]
partitioned_tables = [
{
table = "daily_sales"
field = "sales_timestamp"
},
{
table = "daily_inventory"
field = "inventory_timestamp"
}
]
},
{
db_id = "dataset2"
db_name = "dataset2"
app_name = "hello"
iam_members = [
{
role = "roles/bigquery.dataEditor"
member = "serviceAccount:ser-sa#some-project.iam.gserviceaccount.com"
}
]
tables = []
}
]
environment = "development"
delete_contents_on_destroy = true
var.dbs is type = list(any)
Getting:
The given value is not suitable for var.dbs declared at
variables.tf:9,1-15: all list elements must have the same type.
Thanks in advance!
list(any) does not mean that you can have elements of "any" type in your list. All elements must have same type, and you can't mix types, as you do now (i.e. second element is missing partitioned_tables). any only means that TF will infer the single type for the elements, but all elements must be of that single type. So you have three choices:
remove type = list(any)
Fully define your type with optional arguments, instead of using any
Add partitioned_tables to the second element:
[
{
db_id = "dataset1"
db_name = "dataset1"
app_name = "hello"
iam_members = [
{
role = "roles/bigquery.dataEditor"
member = "serviceAccount:ser-sa#some-project.iam.gserviceaccount.com",
}
]
tables = ["daily_inventory", "dc_inventory", "products", "daily_sales", "planned_inventory", "stores", "stores_in_program"]
partitioned_tables = [
{
table = "daily_sales"
field = "sales_timestamp"
},
{
table = "daily_inventory"
field = "inventory_timestamp"
}
]
},
{
db_id = "dataset2"
db_name = "dataset2"
app_name = "hello"
iam_members = [
{
role = "roles/bigquery.dataEditor"
member = "serviceAccount:ser-sa#some-project.iam.gserviceaccount.com"
}
]
partitioned_tables = []
tables = []
}
]
I working on a module, provided below, to manage AWS KMS keys via Terraform and I'm using the flatten function but the output I'm getting is empty when I call this module.
Any thought why I'm getting empty output?
module
main.tf
locals {
kms_keys = flatten([
for key, kms_key in var.kms_key_list : [
for index in range(kms_key.key_id) : {
key_id = index
aws_kms_alias = kms_key.alias
is_rotating = kms_key.enable_key_rotation
deletion_window_in_days = kms_key.deletion_window_in_days
is_enabled = kms_key.is_enabled
description = kms_key.description
policy = kms_key.policy
}
]
])
}
resource "aws_kms_key" "main" {
for_each = {
for k, v in local.kms_keys: k => v if v.key_id > 0
}
deletion_window_in_days = each.value.deletion_window_in_days
is_enabled = each.value.is_enabled
enable_key_rotation = each.value.enable_key_rotation
description = each.value.description
policy = each.value.policy
tags = merge({
Name = each.value.aws_kms_alias
}, var.common_tags)
}
resource "aws_kms_alias" "alias" {
for_each = aws_kms_key.main
name = "alias/${each.value.tags.Name}"
target_key_id = each.value.key_id
}
variables.tf
variable "kms_key_list" {
type = map(object({
key_id = number
deletion_window_in_days = number
is_enabled = bool
enable_key_rotation = bool
description = string
policy = string
key_usage = string
customer_master_key_spec = string
alias = string
}))
}
calling the module in main.tf
module "kms_keys" {
source = "../module/kms"
kms_key_list = local.kms_keys
}
kms_keys.tf
locals {
kms_keys = {
name_1 = {
key_id = 1
deletion_window_in_days = 7
is_enabled = true
enable_key_rotation = true
description = "description_1"
policy = ""
key_usage = "ENCRYPT_DECRYPT"
customer_master_key_spec = "SYMMETRIC_DEFAULT"
alias = "alias_1"
}
}
}
TF Plan Output looks like this:
Changes to Outputs:
+ kms_info = {
+ kms_key = {}
}
This seems odd:
for index in range(kms_key.key_id)
This is going to loop through all values from 0 to the key_id value; is that really what you want? To add an entry into kms_keys for each value from 0 to key_id?
I doubt it, because the way you have this coded, if your var.kms_key_list contains a key config with key_id = 10, it's going to create 10 different KMS keys, all with the same configuration values.
Essentially, I'm not understanding the purpose of the nested for loop.
If you can provide samples of:
The input variable, but with a key_id > 1
The output that you expect to see
Then we might be able to help. Also, I don't see any output declared either in the module or in the parent file, so those must be missing; please include them.
I am trying to pass the values s3 name and create_user into local block in main.tf so that both of them have the value in list and then I am passing list_of_bucket in local block in module s3 to create the buckets and looping of user_to_create in module s3_user to create the user if the boolean is set to true. All of these values are passed to variable.tf and then to main.tf
dev.tfvars
wea-nonprod = {
services = {
s3 = [
sthree = {
create_user = true,
}
sfour = {
create_user = true,
}
sfive = {
create_user = true,
}
]
}
}
variable.tf
variable "s3_buckets" {
type = list(map)
}
main.tf
locals {
users_to_create = ""
list_of_buckets = ""
}
module "s3" {
source = "../../s3"
name = join("-", [var.name_prefix, "s3"])
tags = merge(var.tags, {Name = join("-", [var.name_prefix, "s3"])})
buckets = list_of_buckets
sse_algorithm = "AES256"
access_log_bucket_name = var.access_log_bucket_name
}
module "s3_user" {
for_each = local.users_to_create
source = "./service-s3-bucket-user"
name = join("-", [var.name_prefix, each.key])
tags = var.tags
bucket_arn = module.s3.bucket_arns[each.key]
depends_on = [module.s3]
}
Just iterate over your wea-nonprod map:
locals {
users_to_create = [ for name in var.wea-nonprod.services.s3 if name.create_user == true ]
list_of_buckets = [ for bucket in var.wea-nonprod.services.s3 ]
}
And a few changes to your module blocks:
module "s3" {
source = "../../s3"
name = "${var.name_prefix}-s3"
tags = merge(var.tags, { Name = "${var.name_prefix}-s3" })
buckets = local.list_of_buckets
sse_algorithm = "AES256"
access_log_bucket_name = var.access_log_bucket_name
}
module "s3_user" {
count = length(local.users_to_create)
source = "./service-s3-bucket-user"
name = "${var.name_prefix}${local.users_to_create[count.index]}"
tags = var.tags
bucket_arn = module.s3.bucket_arns[local.users_to_create[count.index]]
depends_on = [module.s3]
}
Im trying to iterate through a variable type map and i'm not sure how to
This is what i have so far
In my main.tf:
resource "aws_route_53_record" "proxy_dns" {
count = "${length(var.account_name)}"
zone_id = "${infrastructure.zone_id}"
name = "proxy-${element(split(",", var.account_name), count.index)}-dns
type = CNAME
ttl = 60
records = ["{records.dns_name}"]
}
And in my variables.tf
variable "account_name" {
type = "map"
default = {
"account1" = "accountA"
"account2" = "accountB"
}
}
I want to be able to create multiple resources with the different account names
If you are using Terraform 0.12.6 or later then you can use for_each instead of count to produce one instance for each element in your map:
resource "aws_route53_record" "proxy_dns" {
for_each = var.account_name
zone_id = infrastructure.zone_id
name = "proxy-${each.value}-dns"
# ... etc ...
}
The primary advantage of for_each over count is that Terraform will identify the instances by the key in the map, so you'll get instances like aws_route53_record.proxy_dns["account1"] instead of aws_route53_record.proxy_dns[0], and so you can add and remove elements from your map in future with Terraform knowing which specific instance belongs to each element.
each.key and each.value in the resource type arguments replace count.index when for_each is used. They evaluate to the key and value of the current map element, respectively.
You can use a combination of map, keys function,index function, and count. This terraform creates 3 acls with various rules.
The names of the acl's are determined by the keys.
The number of acl's is determined by the count of the keys.
The index of each rule (priority) is determined by the index function
The name of each rule is from the CONTAINS_WORD or CONTAINS property in the map
=>
variable "acls" {
type = map(any)
default = {
"acl1" = {
"CONTAINS_WORD" = ["api","aaa", "bbb", "ccc"]
"CONTAINS" = ["xxx","yyy"]
}
"acl2" = {
"CONTAINS_WORD" = [ "url1,"url2","url3"]
"CONTAINS" = ["url4"]
}
"acl3" = {
"CONTAINS_WORD" = ["xxx"]
"CONTAINS" = []
}
}
}
resource "aws_wafv2_web_acl" "acl" {
name = keys(var.acls)[count.index]
scope = "REGIONAL"
count = length(keys(var.acls))
default_action {
block {}
}
dynamic "rule" {
for_each = toset(var.acls[keys(var.acls)[count.index]].CONTAINS_WORD)
content {
name = rule.key
priority = index(var.acls[keys(var.acls)[count.index]].CONTAINS_WORD, rule.key)
action {
allow {}
}
statement {
#https://docs.aws.amazon.com/waf/latest/APIReference/API_ByteMatchStatement.html
byte_match_statement {
positional_constraint = "CONTAINS_WORD"
search_string = lower(rule.key)
field_to_match {
uri_path {}
}
text_transformation {
priority = 0
type = "LOWERCASE"
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "waf-${keys(var.acls)[count.index]}-${rule.key}"
sampled_requests_enabled = true
}
}
}
dynamic "rule" {
for_each = toset(var.acls[keys(var.acls)[count.index]].CONTAINS)
content {
name = replace(rule.key, ".", "_")
priority = index(var.acls[keys(var.acls)[count.index]].CONTAINS, rule.key) + length(var.acls[keys(var.acls)[count.index]].CONTAINS_WORD)
action {
allow {}
}
statement {
#https://docs.aws.amazon.com/waf/latest/APIReference/API_ByteMatchStatement.html
byte_match_statement {
positional_constraint = "CONTAINS"
search_string = lower(rule.key)
field_to_match {
uri_path {}
}
text_transformation {
priority = 0
type = "LOWERCASE"
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "waf-${keys(var.acls)[count.index]}-${replace(rule.key, ".", "_")}"
sampled_requests_enabled = true
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "waf-${keys(var.acls)[count.index]}"
sampled_requests_enabled = true
}
}
Make the variable a list instead of a map. Maps are used to reference a name to a value. Lists are better for iterating over via a count method.
variable "account_name" {
type = "list"
default = {"accountA","accountB"}
}
resource "aws_route_53_record" "proxy_dns" {
count = "${length(var.account_name)}"
zone_id = "${infrastructure.zone_id}"
name = "proxy-${element(var.account_name, count.index)}-dns
type = CNAME
ttl = 60
records = ["{records.dns_name}"]
}