I am trying to build a reusable module that creates multiple S3 buckets. Based on a condition, some buckets may have lifecycle rules, others do not. I am using a for loop in the lifecycle rule resource and managed to do it but not on 100%.
My var:
variable "bucket_details" {
type = map(object({
bucket_name = string
enable_lifecycle = bool
glacier_ir_days = number
glacier_days = number
}))
}
How I go through the map on the lifecycle resource:
resource "aws_s3_bucket_lifecycle_configuration" "compliant_s3_bucket_lifecycle_rule" {
for_each = { for bucket, values in var.bucket_details : bucket => values if values.enable_lifecycle }
depends_on = [aws_s3_bucket_versioning.compliant_s3_bucket_versioning]
bucket = aws_s3_bucket.compliant_s3_bucket[each.key].bucket
rule {
id = "basic_config"
status = "Enabled"
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
transition {
days = each.value["glacier_ir_days"]
storage_class = "GLACIER_IR"
}
transition {
days = each.value["glacier_days"]
storage_class = "GLACIER"
}
expiration {
days = 2555
}
noncurrent_version_transition {
noncurrent_days = each.value["glacier_ir_days"]
storage_class = "GLACIER_IR"
}
noncurrent_version_transition {
noncurrent_days = each.value["glacier_days"]
storage_class = "GLACIER"
}
noncurrent_version_expiration {
noncurrent_days = 2555
}
}
}
How I WOULD love to reference it in the root module:
module "s3_buckets" {
source = "./modules/aws-s3-compliance"
#
bucket_details = {
"fisrtbucketname" = {
bucket_name = "onlythefisrtbuckettesting"
enable_lifecycle = true
glacier_ir_days = 555
glacier_days = 888
}
"secondbuckdetname" = {
bucket_name = "onlythesecondbuckettesting"
enable_lifecycle = false
}
}
}
So when I reference it like that, it cannot validate, because I am not setting values for both glacier_ir_days & glacier_days - understandable.
My question is - is there a way to check if the enable_lifecycle is set to false, to not expect values for these?
Currently, as a workaround, I am just setting zeroes for those and since the resource is not created if enable_lifecycle is false, it does not matter, but I would love it to be cleaner.
Thank you in advance.
The forthcoming Terraform v1.3 release will include a new feature for declaring optional attributes in an object type constraint, with the option of declaring a default value to use when the attribute isn't set.
At the time I'm writing this the v1.3 release is still under development and so not available for general use, but I'm going to answer this with an example that should work with Terraform v1.3 once it's released. If you wish to try it in the meantime you can experiment with the most recent v1.3 alpha release which includes this feature, though of course I would not recommend using it in production until it's in a final release.
It seems that your glacier_ir_days and glacier_days attributes are, from a modeling perspective, attribtues that are required when the lifecycle is enabled and not required when lifecycle is disabled.
I would suggest modelling that by placing these attributes in a nested object called lifecycle and implementing it such that the lifecycle resource is enabled when that attribute is set, and disabled when it is left unset.
The declaration would therefore look like this:
variable "s3_buckets" {
type = map(object({
bucket_name = string
lifecycle = optional(object({
glacier_ir_days = number
glacier_days = number
}))
}))
}
When an attribute is marked as optional(...) like this, Terraform will allow omitting it in the calling module block and then will quietly set the attribute to null when it performs the type conversion to make the given value match the type constraint. This particular declaration doesn't have a default value, but it's also possible to pass a second argument in the optional(...) syntax which Terraform will then use instead of null as the placeholder value when the attribute isn't specified.
The calling module block would therefore look like this:
module "s3_buckets" {
source = "./modules/aws-s3-compliance"
#
bucket_details = {
"fisrtbucketname" = {
bucket_name = "onlythefisrtbuckettesting"
lifecycle = {
glacier_ir_days = 555
glacier_days = 888
}
}
"secondbuckdetname" = {
bucket_name = "onlythesecondbuckettesting"
}
}
}
Your resource block inside the module will remain similar to what you showed, but the if clause of the for expression will test if the lifecycle object is non-null instead:
resource "aws_s3_bucket_lifecycle_configuration" "compliant_s3_bucket_lifecycle_rule" {
for_each = {
for bucket, values in var.bucket_details : bucket => values
if values.lifecycle != null
}
# ...
}
Finally, the references to the attributes would be slightly different to traverse through the lifecycle object:
transition {
days = each.value.lifecycle.glacier_days
storage_class = "GLACIER"
}
Related
I have a terraform project which contains a main module and a submodule. Something like this:
\modules
dev.tfvars
\app
\main
main.tf, output.tf, providers.tf, variables.tf
\sub-modules
\eventbridge
main.tf, output.tf, variables.tf
Both variables.tf files have this variable defined:
variable "secrets" {
description = "map for secret manager"
type = map(string)
}
The top level main.tf has this defined:
module "a-eventbridge-trigger" {
source = "../sub-modules/eventbridge"
secrets = var.secrets
}
The submodule main.tf has this:
resource "aws_cloudwatch_event_connection" "auth" {
name = "request-token"
description = "Gets token"
authorization_type = "OAUTH_CLIENT_CREDENTIALS"
auth_parameters {
oauth {
authorization_endpoint = "${var.apiurl}"
http_method = "POST"
oauth_http_parameters {
body {
key = "grant_type"
value = "client_credentials"
is_value_secret = true
}
body {
key = "client_id"
value = var.secrets.Client_Id
is_value_secret = true
}
body {
key = "client_secret"
value = var.secrets.Client_Secret
is_value_secret = true
}
}
}
}
}
However, when run it throws this error:
Error: error creating EventBridge connection (request-token): InvalidParameter: 2 validation error(s) found.
- missing required field, CreateConnectionInput.AuthParameters.OAuthParameters.ClientParameters.ClientID.
- missing required field, CreateConnectionInput.AuthParameters.OAuthParameters.ClientParameters.ClientSecret.
A file dump ahead of the terrform apply command successfully dumps out the contents of the tfvars file, so I know it exists at time of execution.
The top level output.tf successfully writes out the complete values of the secrets variable after execution, so I know the top level module receives the variables.
In the submodule, the resources defined after the aws_cloudwatch_event_connection block do get created and they also use variables received from the same tfvars file.
Is this a problem with how I am providing the variables or with my definition of the resources itself? (Or something else?)
client_parameters is missing on your configuration, you need to set it in auth_parameters.oauth
resource "aws_cloudwatch_event_connection" "auth" {
name = "request-token"
description = "Gets token"
authorization_type = "OAUTH_CLIENT_CREDENTIALS"
auth_parameters {
oauth {
authorization_endpoint = "${var.apiurl}"
http_method = "POST"
client_parameters {
client_id = var.secrets.Client_Id
client_secret = var.secrets.Client_Secret
}
oauth_http_parameters {
body {
key = "grant_type"
value = "client_credentials"
is_value_secret = true
}
}
}
}
}
I really stuck today on the following error:
MalformedXML: The XML you provided was not well-formed
when applying aws_s3_bucket_lifecycle_configuration via Terraform using hashicorp/aws v4.38.0.
I wanted to set a rule that would expire files after 365 days with file size greater than 0 bytes for a my_prefix prefix so the definition of the resource looks like that:
resource "aws_s3_bucket_lifecycle_configuration" "my-bucket-lifecycle-configuration" {
depends_on = [aws_s3_bucket_versioning.my-bucket-versioning]
bucket = aws_s3_bucket.my_bucket.id
rule {
id = "my_prefix_current_version_config"
filter {
and {
prefix = "my_prefix/"
object_size_greater_than = 0
}
}
expiration {
days = 365
}
status = "Enabled"
}
}
Anyone has idea what's wrong with the above definition? :nerd_face:
Documentation: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_lifecycle_configuration
Remark: the following definition can be applied without problem (no and block):
resource "aws_s3_bucket_lifecycle_configuration" "my-bucket-lifecycle-configuration" {
depends_on = [aws_s3_bucket_versioning.my-bucket-versioning]
bucket = aws_s3_bucket.my_bucket.id
rule {
id = "my_prefix_current_version_config"
filter {
prefix = "my_prefix/"
}
expiration {
days = 365
}
status = "Enabled"
}
}
From the documentation, you have to specify both the object size range (which I guess mean, you have to specify both object_size_greater_than and object_size_less_than) and prefix, for example:
filter {
and {
prefix = "my_prefix/"
object_size_greater_than = 0
object_size_less_than = 500
}
}
I have two conditions need to be fulfilled:
Grant users permission to specific project-id based on env. For example: my-project-{env} (env: stg/prd)
I want to loop over the variables, instead of writing down repetitive resource for each user.
Example:
variable some_ext_users {
type = map(any)
default = {
user_1 = { email_id = "user_1#gmail.com" }
user_2 = { email_id = "user_2#gmail.com" }
}
}
To avoid repetitive resource made on each user (imagine 100++ users), I decided to list them in variable as written above.
Then, I'd like to assign these user GCS permission, e.g:
resource "google_storage_bucket_iam_member" "user_email_access" {
for_each = var.some_ext_users
count = var.env == "stg" ? 1 : 0
provider = google-beta
bucket = "my-bucketttt"
role = "roles/storage.objectViewer"
member = "user:${each.value.email_id}"
}
The error I'm getting is clear :
Error: Invalid combination of "count" and "for_each" on
../../../modules/my-tf.tf line 54, in resource
"google_storage_bucket_iam_member" "user_email_access": 54:
for_each = var.some_ext_users The "count" and "for_each"
meta-arguments are mutually-exclusive, only one should be used to be
explicit about the number of resources to be created.
My question is, what is the workaround in order to satisfy the requirements above if count and for_each can't be used together?
You could control the user list according to the environment, rather than trying to control the resource. So, something like this:
resource "google_storage_bucket_iam_member" "user_email_access" {
for_each = var.env == "stg" ? var.some_ext_users : {}
provider = google-beta
bucket = "my-bucketttt"
role = "roles/storage.objectViewer"
member = "user:${each.value.email_id}"
}
The rule for for_each is to assign it a map that has one element per instance you want to declare, so the best way to think about your requirement here is that you need to write an expression that produces a map with zero elements when your condition doesn't hold.
The usual way to project and filter collections in Terraform is for expressions, and indeed we can use a for expression with an if clause to conditionally filter out unwanted elements, which in this particular case will be all of the elements:
resource "google_storage_bucket_iam_member" "user_email_access" {
for_each = {
for name, user in var.some_ext_users : name => user
if var.env == "stg"
}
# ...
}
Another possible way to structure this would be to include the environment keywords as part of the data structure, which would keep all of the information in one spot and potentially allow you to have entries that apply to more than one environment at once:
variable "some_ext_users" {
type = map(object({
email_id = string
environments = set(string)
}))
default = {
user_1 = {
email_id = "user_1#gmail.com"
environments = ["stg"]
}
user_2 = {
email_id = "user_2#gmail.com"
environments = ["stg", "prd"]
}
}
}
resource "google_storage_bucket_iam_member" "user_email_access" {
for_each = {
for name, user in var.some_ext_users : name => user
if contains(user.environments, var.env)
}
# ...
}
This is a variation of the example in the "Filtering Elements" documentation I linked above, which uses an is_admin flag in order to declare different resources for admin users vs. non-admin users. In this case, notice that the if clause refers to the symbols declared in the for expression, which means we can now get a different result for each element of the map, whereas the first example either kept all elements or no elements.
I need to change, using Terraform, the default project_id in my Composer environment so that I can access secrets from another project. To do so, according to Terraform, I need the variable airflow_config_overrides. I guess I should have something like this:
resource "google_composer_environment" "test" {
# ...
config {
software_config {
airflow_config_overrides = {
secrets-backend = "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend",
secrets-backend_kwargs = {"project_id":"9999999999999"}
}
}
}
}
The secrets-backend section-key seems to be working. On the other hand, secrets-backend_kwargs is returning the following error:
Inappropriate value for attribute "airflow_config_overrides": element "secrets-backend_kwargs": string required
It seems that the problem is that GCP expects a JSON format and Terraform requires a string. How can I get Terraform to provide it in the format needed?
You can convert a map such as {"project_id":"9999999999999"} into a JSON encoded string by using the jsonencode function.
So merging the example given in the google_composer_environment resource documentation with your config in the question you can do something like this:
resource "google_composer_environment" "test" {
name = "mycomposer"
region = "us-central1"
config {
software_config {
airflow_config_overrides = {
secrets-backend = "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend",
secrets-backend_kwargs = jsonencode({"project_id":"9999999999999"})
}
pypi_packages = {
numpy = ""
scipy = "==1.1.0"
}
env_variables = {
FOO = "bar"
}
}
}
}
I want to set multiple paths with aws_alb_listener_rule resource
but aws_alb_listener_rule resource should not be able to accept multiple values in the condition object?
Below is my resource. however, the error written in the title is out,
how can i fix that
resource "aws_alb_listener_rule" "admin_static" {
listener_arn = "${aws_alb_listener.web_http.arn}"
priority = 99
action {
type = "forward"
target_group_arn = "${aws_alb_target_group.ec2_web.arn}"
}
condition {
field = "host-header"
values = ["example.com"]
}
condition {
field = "path-pattern"
values = ["/admin/*"]
}
condition {
field = "path-pattern"
values = ["/static/*"]
}
}
I added the new source code and solved that Error modifying LB Listener Rule: ValidationError