Terraform missing when value should be passed in from tfvar file - amazon-web-services

I have a terraform project which contains a main module and a submodule. Something like this:
\modules
dev.tfvars
\app
\main
main.tf, output.tf, providers.tf, variables.tf
\sub-modules
\eventbridge
main.tf, output.tf, variables.tf
Both variables.tf files have this variable defined:
variable "secrets" {
description = "map for secret manager"
type = map(string)
}
The top level main.tf has this defined:
module "a-eventbridge-trigger" {
source = "../sub-modules/eventbridge"
secrets = var.secrets
}
The submodule main.tf has this:
resource "aws_cloudwatch_event_connection" "auth" {
name = "request-token"
description = "Gets token"
authorization_type = "OAUTH_CLIENT_CREDENTIALS"
auth_parameters {
oauth {
authorization_endpoint = "${var.apiurl}"
http_method = "POST"
oauth_http_parameters {
body {
key = "grant_type"
value = "client_credentials"
is_value_secret = true
}
body {
key = "client_id"
value = var.secrets.Client_Id
is_value_secret = true
}
body {
key = "client_secret"
value = var.secrets.Client_Secret
is_value_secret = true
}
}
}
}
}
However, when run it throws this error:
Error: error creating EventBridge connection (request-token): InvalidParameter: 2 validation error(s) found.
- missing required field, CreateConnectionInput.AuthParameters.OAuthParameters.ClientParameters.ClientID.
- missing required field, CreateConnectionInput.AuthParameters.OAuthParameters.ClientParameters.ClientSecret.
A file dump ahead of the terrform apply command successfully dumps out the contents of the tfvars file, so I know it exists at time of execution.
The top level output.tf successfully writes out the complete values of the secrets variable after execution, so I know the top level module receives the variables.
In the submodule, the resources defined after the aws_cloudwatch_event_connection block do get created and they also use variables received from the same tfvars file.
Is this a problem with how I am providing the variables or with my definition of the resources itself? (Or something else?)

client_parameters is missing on your configuration, you need to set it in auth_parameters.oauth
resource "aws_cloudwatch_event_connection" "auth" {
name = "request-token"
description = "Gets token"
authorization_type = "OAUTH_CLIENT_CREDENTIALS"
auth_parameters {
oauth {
authorization_endpoint = "${var.apiurl}"
http_method = "POST"
client_parameters {
client_id = var.secrets.Client_Id
client_secret = var.secrets.Client_Secret
}
oauth_http_parameters {
body {
key = "grant_type"
value = "client_credentials"
is_value_secret = true
}
}
}
}
}

Related

Terraform build APIgateway2 authoriser jwt configuration using AWS SSM parameters

I am creating an AWS http APIgateway with JWT authorisation from cognito using Terraform.
I have built this out, using one app client and it works.
But i would like to add another app client. i call the clientid from AWS SSM.
below is the jwt_configuration extract from the module we have built.
extprovider is passed from the main resource tf as a variable.
eg extprovider=production
jwt_configuration {
audience = [ data.aws_ssm_parameter.clientid.value ]
issuer = data.aws_ssm_parameter.user_pool_endpoint.value
}
my data.tf
data aws_ssm_parameter clientid {
name = "/${var.workflow}/cognito/${var.extprovider}/clientid"
}
The new setup will have the extprovider name of production_odd and production_even, this part must be done within the module, as the name of production is used elsewhere.
i had thought my approach would be;
locals {
extprovider_new = flatten([ for e in var.extprovider : tolist(["${e}odd","${e}even"])])
}
data aws_ssm_parameter clientid1 {
name = "/${var.workflow}/cognito/${var.extprovider_new[0]}/clientid"
}
data aws_ssm_parameter clientid2 {
name = "/${var.workflow}/cognito/${var.extprovider_new[1]}/clientid"
}
jwt_configuration {
audience = [ data.aws_ssm_parameter.clientid1.value,data.aws_ssm_parameter.clientid2.value ]
issuer = data.aws_ssm_parameter.user_pool_endpoint.value
}
However, i am getting the error;
Error: Iteration over non-iterable value
on ../modules/apiresource/locals.tf line 3, in locals:
3: extprovider_new = flatten([ for e in var.extprovider : tolist(["${e}odd","${e}even"])])
├────────────────
│ var.extprovider is "foo"
A value of type string cannot be used as the collection in a 'for' expression.
Error: Iteration over non-iterable value
on ../modules/apiresource/locals.tf line 3, in locals:
3: extprovider_new = flatten([ for e in var.extprovider : tolist(["${e}odd","${e}even"])])
├────────────────
│ var.extprovider is "bar"
A value of type string cannot be used as the collection in a 'for' expression.
Variable is a string, it needs to be a list;
extprovider_new = ["${var.extprovider}odd", "${var.extprovider}even"]
}```

Using Terraform, how can I pass in a list of users from module to module

I am trying to use Terraform's public modules and I created a lot of IAM users like so:
module "iam_finance_users" {
source = "terraform-aws-modules/iam/aws//modules/iam-user"
version = "5.9.1"
for_each = var.finance_users
name = each.value
force_destroy = true
create_iam_access_key = false
password_reset_required = true
}
My variables.tf file has the following:
variable "finance_users" {
type = set(string)
default = ["mary.johnson#domain.com","mike.smith#domain.com"....]
}
Now I am trying to add these users to a group like so
module "iam_group_finance" {
source = "terraform-aws-modules/iam/aws//modules/iam-group-with-policies"
version = "5.9.1"
name = "finance"
group_users = ["${module.iam_finance_users.iam_user_name}"]
attach_iam_self_management_policy = true
custom_group_policy_arns = [
"${module.iam_read_write_policy.arn}",
]
}
But no matter what I try I still keep getting errors. I have a list of users in a variable and I want to add all those users created in that module to the group module. I know I'm close but I can't quite seem to close this.
Since you used for_each in the module, you have to use values to access elements of all instances of the module created:
group_users = values(module.iam_finance_users)[*].iam_user_name
It's often helpful to know what the error is. In versions of Terraform v1.0 or newer, the errors become fairly explanatory. The error you were likely getting was something similar to:
│ Error: Unsupported attribute
│
│ on main.tf line 14, in output "iam_finance_users":
│ 14: value = module.iam_finance_users.iam_user_name
│ ├────────────────
│ │ module.iam_finance_users is object with 2 attributes
│
│ This object does not have an attribute named "iam_user_name".
The solutions, which both work, can be tested using the following example code:
Module 1: modules/iam-user
variable "name" {
type = string
}
output "iam_user_name" {
value = "IAM-${var.name}"
}
Module 2: modules/iam-group-with-policies
variable "group_users" {
type = set(string)
}
output "iam-users" {
value = var.group_users
}
Root module (your code):
variable "finance_users" {
type = set(string)
default = ["mary.johnson#domain.com", "mike.smith#domain.com"]
}
module "iam_finance_users" {
source = "../modules/iam-user"
for_each = var.finance_users
name = each.value
}
module "iam_group_finance" {
source = "../modules/iam-group-with-policies"
# This works...
group_users = values(module.iam_finance_users)[*].iam_user_name
# This also works...
group_users = [for o in module.iam_finance_users : o.iam_user_name]
}
output "group_users" {
value = module.iam_group_finance.iam-users
}
You can easily test each module along the way using terraform console.
E.g.:
terraform console
> module.iam_finance_users
{
"mary.johnson#domain.com" = {
"iam_user_name" = "IAM-mary.johnson#domain.com"
}
"mike.smith#domain.com" = {
"iam_user_name" = "IAM-mike.smith#domain.com"
}
}
Here you can see why it didn't work. The module doesn't spit out a list in that way, so you have to iterate through the module to get your variable. This is why both the for method as well as the values method works.
Look at how differently a single module is handled:
module "iam_finance_users" {
source = "../modules/iam-user"
name = tolist(var.finance_users)[1]
}
terraform console
> module.iam_finance_users
{
"iam_user_name" = "IAM-mike.smith#domain.com"
}
Notice that when not inside a loop (foreach), that the single iam_user_name is accessible.
You could try something like below:
This is untested, so you might need to tweak a bit. Reference thread: https://discuss.hashicorp.com/t/for-each-objects-to-list/36609/2
module "iam_group_finance" {
source = "terraform-aws-modules/iam/aws//modules/iam-group-with-policies"
version = "5.9.1"
name = "finance"
group_users = [ for o in module.iam_finance_users : o.iam_user_name ]
attach_iam_self_management_policy = true
custom_group_policy_arns = [
module.iam_read_write_policy.arn
]
}

How to skip declaring values in root module (for_each loop)

I am trying to build a reusable module that creates multiple S3 buckets. Based on a condition, some buckets may have lifecycle rules, others do not. I am using a for loop in the lifecycle rule resource and managed to do it but not on 100%.
My var:
variable "bucket_details" {
type = map(object({
bucket_name = string
enable_lifecycle = bool
glacier_ir_days = number
glacier_days = number
}))
}
How I go through the map on the lifecycle resource:
resource "aws_s3_bucket_lifecycle_configuration" "compliant_s3_bucket_lifecycle_rule" {
for_each = { for bucket, values in var.bucket_details : bucket => values if values.enable_lifecycle }
depends_on = [aws_s3_bucket_versioning.compliant_s3_bucket_versioning]
bucket = aws_s3_bucket.compliant_s3_bucket[each.key].bucket
rule {
id = "basic_config"
status = "Enabled"
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
transition {
days = each.value["glacier_ir_days"]
storage_class = "GLACIER_IR"
}
transition {
days = each.value["glacier_days"]
storage_class = "GLACIER"
}
expiration {
days = 2555
}
noncurrent_version_transition {
noncurrent_days = each.value["glacier_ir_days"]
storage_class = "GLACIER_IR"
}
noncurrent_version_transition {
noncurrent_days = each.value["glacier_days"]
storage_class = "GLACIER"
}
noncurrent_version_expiration {
noncurrent_days = 2555
}
}
}
How I WOULD love to reference it in the root module:
module "s3_buckets" {
source = "./modules/aws-s3-compliance"
#
bucket_details = {
"fisrtbucketname" = {
bucket_name = "onlythefisrtbuckettesting"
enable_lifecycle = true
glacier_ir_days = 555
glacier_days = 888
}
"secondbuckdetname" = {
bucket_name = "onlythesecondbuckettesting"
enable_lifecycle = false
}
}
}
So when I reference it like that, it cannot validate, because I am not setting values for both glacier_ir_days & glacier_days - understandable.
My question is - is there a way to check if the enable_lifecycle is set to false, to not expect values for these?
Currently, as a workaround, I am just setting zeroes for those and since the resource is not created if enable_lifecycle is false, it does not matter, but I would love it to be cleaner.
Thank you in advance.
The forthcoming Terraform v1.3 release will include a new feature for declaring optional attributes in an object type constraint, with the option of declaring a default value to use when the attribute isn't set.
At the time I'm writing this the v1.3 release is still under development and so not available for general use, but I'm going to answer this with an example that should work with Terraform v1.3 once it's released. If you wish to try it in the meantime you can experiment with the most recent v1.3 alpha release which includes this feature, though of course I would not recommend using it in production until it's in a final release.
It seems that your glacier_ir_days and glacier_days attributes are, from a modeling perspective, attribtues that are required when the lifecycle is enabled and not required when lifecycle is disabled.
I would suggest modelling that by placing these attributes in a nested object called lifecycle and implementing it such that the lifecycle resource is enabled when that attribute is set, and disabled when it is left unset.
The declaration would therefore look like this:
variable "s3_buckets" {
type = map(object({
bucket_name = string
lifecycle = optional(object({
glacier_ir_days = number
glacier_days = number
}))
}))
}
When an attribute is marked as optional(...) like this, Terraform will allow omitting it in the calling module block and then will quietly set the attribute to null when it performs the type conversion to make the given value match the type constraint. This particular declaration doesn't have a default value, but it's also possible to pass a second argument in the optional(...) syntax which Terraform will then use instead of null as the placeholder value when the attribute isn't specified.
The calling module block would therefore look like this:
module "s3_buckets" {
source = "./modules/aws-s3-compliance"
#
bucket_details = {
"fisrtbucketname" = {
bucket_name = "onlythefisrtbuckettesting"
lifecycle = {
glacier_ir_days = 555
glacier_days = 888
}
}
"secondbuckdetname" = {
bucket_name = "onlythesecondbuckettesting"
}
}
}
Your resource block inside the module will remain similar to what you showed, but the if clause of the for expression will test if the lifecycle object is non-null instead:
resource "aws_s3_bucket_lifecycle_configuration" "compliant_s3_bucket_lifecycle_rule" {
for_each = {
for bucket, values in var.bucket_details : bucket => values
if values.lifecycle != null
}
# ...
}
Finally, the references to the attributes would be slightly different to traverse through the lifecycle object:
transition {
days = each.value.lifecycle.glacier_days
storage_class = "GLACIER"
}

Terraform count calling module output(return) error

I need to return output of name and id only
root main.tf
# Azure Provider configuration
provider "azurerm" {
features {}
}
module "rg" {
source = "../../modules/rg"
count = length(var.resource_groups)
resource_group_name = var.resource_groups[count.index].name
tags = {}
location = westus2
lookup = false
}
module/rg/main.tf
resource "azurerm_resource_group" "rg" {
count = var.lookup == true ? 0 : 1
name = var.resource_group_name
location = var.location
tags = var.tags
}
output "rg_id" {
description = "The id of the newly created rg"
value = azurerm_resource_group.rg[count_index].id # azurerm_resource_group.rg, this statement will work
}
output "rg_name" {
description = "The name of the newly created rg"
value = azurerm_resource_group.rg[count_index].name # azurerm_resource_group.rg, this statement will work
}
I am getting below error:
A reference to a resource type must be followed by at least one attribute access, specifying the resource name.
I think the following should address your issue:
output "rg_id" {
description = "The id of the newly created rg"
value = var.lookup == true ? azurerm_resource_group.rg[0].id : null
}
output "rg_name" {
description = "The name of the newly created rg"
value = var.lookup == true ? azurerm_resource_group.rg[0].name : null
}

Value for Terraform Composer airflow_config_override secrets-backend_kwargs

I need to change, using Terraform, the default project_id in my Composer environment so that I can access secrets from another project. To do so, according to Terraform, I need the variable airflow_config_overrides. I guess I should have something like this:
resource "google_composer_environment" "test" {
# ...
config {
software_config {
airflow_config_overrides = {
secrets-backend = "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend",
secrets-backend_kwargs = {"project_id":"9999999999999"}
}
}
}
}
The secrets-backend section-key seems to be working. On the other hand, secrets-backend_kwargs is returning the following error:
Inappropriate value for attribute "airflow_config_overrides": element "secrets-backend_kwargs": string required
It seems that the problem is that GCP expects a JSON format and Terraform requires a string. How can I get Terraform to provide it in the format needed?
You can convert a map such as {"project_id":"9999999999999"} into a JSON encoded string by using the jsonencode function.
So merging the example given in the google_composer_environment resource documentation with your config in the question you can do something like this:
resource "google_composer_environment" "test" {
name = "mycomposer"
region = "us-central1"
config {
software_config {
airflow_config_overrides = {
secrets-backend = "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend",
secrets-backend_kwargs = jsonencode({"project_id":"9999999999999"})
}
pypi_packages = {
numpy = ""
scipy = "==1.1.0"
}
env_variables = {
FOO = "bar"
}
}
}
}