I have a global Aurora RDS cluster that takes automated snapshot everyday. My DB instance looks like this :-
new-test ( Global Database )
new-test-db ( Primary Cluster )
new-test-db-0 ( Writer Instance )
I have enabled automated snapshots for the db. What i am trying to achieve is to get the ARN for my snapshot using data resource. My ARN is something like this :-
arn:aws:rds:us-west-2:123456789101:cluster-snapshot:rds:new-test-db-2022-08-23-08-06
This is what my data resource looks like :-
data "aws_db_cluster_snapshot" "db" {
for_each = toset(var.rds_sources)
db_cluster_identifier = each.key
most_recent = true
}
where var.rds_sources is a list of strings. But when i try to access the arn using :-
data.aws_db_cluster_snapshot.db[*].db_cluster_snapshot_arn
I keep running into
Error: Unsupported attribute
│
│ on ../main.tf line 73, in resource "aws_iam_policy" "source_application":
│ 73: cluster_data_sources = jsonencode(data.aws_db_cluster_snapshot.db[*].db_cluster_snapshot_arn)
│
│ This object does not have an attribute named "db_cluster_snapshot_arn".
Which is weird since the attribute is laid out in the official docs. Thank you for the help.
This is my provider file :-
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.75"
}
archive = "~> 2.2.0"
}
required_version = "~> 1.2.6"
}
Since the data source is using for_each, the result will be a map of key value pairs. In terraform, there is a built-in function values [1] which can be used to fetch the values of a map. The return value is a list, so in order to get all the values for all the keys the splat operator is used [2]. Then, since the data source is returning multiple attributes and only one is required (namely db_cluster_snapshot_arn), the final expression needed is as follows:
jsonencode(values(data.aws_db_cluster_snapshot.db)[*].db_cluster_snapshot_arn)
[1] https://www.terraform.io/language/functions/values
[2] https://www.terraform.io/language/expressions/splat
Related
I current have code that I have been using for quiet sometime that calls a custom S3 module. Today I tried to run the same code and I started getting an error regarding the provider.
╷ │ Error: Failed to query available provider packages │ │ Could not
retrieve the list of available versions for provider hashicorp/s3:
provider registry registry.terraform.io does not have a provider named
│ registry.terraform.io/hashicorp/s3 │ │ All modules should specify
their required_providers so that external consumers will get the
correct providers when using a module. To see which modules │ are
currently depending on hashicorp/s3, run the following command: │
terraform providers
Doing some digging seems that terraform is looking for a module registry.terraform.io/hashicorp/s3, which doesn't exist.
So far, I have tried the following things:
Validated that the S3 Resource code meets the standards of the upgrade Hashicorp did to 4.x this year. Plus I have been using it for a couple of months with no issues.
Delete .terraform directory and rerun terraform init (No success same error)
Delete .terraform directory and .terraform.hcl lock and run terraform init -upgrade (No Success)
I have tried to update my provider's file to try to force an upgrade (no Success)
I tried to change the provider to >= current version to pull the latest version with no success
Reading further, it refers to a caching problem of the terraform modules. I tried to run terraform providers lock and received this error.
Error: Could not retrieve providers for locking │ │ Terraform failed
to fetch the requested providers for darwin_amd64 in order to
calculate their checksums: some providers could not be installed: │ -
registry.terraform.io/hashicorp/s3: provider registry
registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/s3.
Kind of at my wits with what could be wrong. below is a copy of my version.tf which I changed from providers.tf based on another post I was following:
version.tf
# Configure the AWS Provider
provider "aws" {
region = "us-east-1"
use_fips_endpoint = true
}
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.9.0"
}
local = {
source = "hashicorp/local"
version = "~> 2.2.1"
}
}
required_version = ">= 1.2.0" #required terraform version
}
S3 Module
I did not include locals, outputs, or variables unless someone thinks we need to see them. As I said before, the module was running correctly until today. Hopefully, this is all you need for the provider's issue. Let me know if other files are needed.
resource "aws_s3_bucket" "buckets" {
count = length(var.bucket_names)
bucket = lower(replace(replace("${var.bucket_names[count.index]}-s3", " ", "-"), "_", "-"))
force_destroy = var.bucket_destroy
tags = local.all_tags
}
# Set Public Access Block for each bucket
resource "aws_s3_bucket_public_access_block" "bucket_public_access_block" {
count = length(var.bucket_names)
bucket = aws_s3_bucket.buckets[count.index].id
block_public_acls = var.bucket_block_public_acls
ignore_public_acls = var.bucket_ignore_public_acls
block_public_policy = var.bucket_block_public_policy
restrict_public_buckets = var.bucket_restrict_public_buckets
}
resource "aws_s3_bucket_acl" "bucket_acl" {
count = length(var.bucket_names)
bucket = aws_s3_bucket.buckets[count.index].id
acl = var.bucket_acl
}
resource "aws_s3_bucket_versioning" "bucket_versioning" {
count = length(var.bucket_names)
bucket = aws_s3_bucket.buckets[count.index].id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_lifecycle_configuration" "bucket_lifecycle_rule" {
count = length(var.bucket_names)
bucket = aws_s3_bucket.buckets[count.index].id
rule {
id = "${var.bucket_names[count.index]}-lifecycle-${count.index}"
status = "Enabled"
expiration {
days = var.bucket_backup_expiration_days
}
transition {
days = var.bucket_backup_days
storage_class = "GLACIER"
}
}
}
# AWS KMS Key Server Encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "bucket_encryption" {
count = length(var.bucket_names)
bucket = aws_s3_bucket.buckets[count.index].id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.bucket_key[count.index].arn
sse_algorithm = var.bucket_sse
}
}
}
Looking for any other ideas I can use to fix this issue. thank you!!
Although you haven't included it in your question, I'm guessing that somewhere else in this Terraform module you have a block like this:
resource "s3_bucket" "example" {
}
For backward compatibility with modules written for older versions of Terraform, terraform init has some heuristics to guess what provider was intended whenever it encounters a resource that doesn't belong to one of the providers in the module's required_providers block. By default, a resource "belongs to" a provider by matching the prefix of its resource type name -- s3 in this case -- to the local names chosen in the required_providers block.
Given a resource block like the above, terraform init would notice that required_providers doesn't have an entry s3 = { ... } and so will guess that this is an older module trying to use a hypothetical legacy official provider called "s3" (which would now be called hashicorp/s3, because official providers always belong to the hashicorp/ namespace).
The correct name for this resource type is aws_s3_bucket, and so it's important to include the aws_ prefix when you declare a resource of this type:
resource "aws_s3_bucket" "example" {
}
This resource is now by default associated with the provider local name "aws", which does match one of the entries in your required_providers block and so terraform init will see that you intend to use hashicorp/aws to handle this resource.
My colleague and I finally found the problem. Turns out that we had a data call to the S3 bucket. Nothing was wrong with the module but the place I was calling the module had a local.tf action where I was calling s3 in a legacy format see the change below:
WAS
data "s3_bucket" "MyResource" {}
TO
data "aws_s3_bucket" "MyResource" {}
Appreciate the responses from everyone. Resource was the root of the problem but forgot that data is also a resource to check.
I have a Terraform module that I would like to modify. Currently my module creates a service account. I would like to modify it so that someone could pass in an existing service account OR if one is not passed in, then the module creates a service account is it would have originally.
Originally my service account looked like this:
resource "google_service_account" "scheduler" {
account_id = "${var.prefix}-scheduler"
project = var.project
}
I've added the following variable to my variables.tf file:
variable "service_account_email" {
default = null
description = "Existing service account for running ... jobs. If null a new service account will be created."
}
What I originally thought to do was to add some locals
locals {
service_account_count = var.service_account_email == null ? 1 : 0
service_account_email = var.service_account_email == null ? google_service_account.scheduler.email : var.service_account_email
}
Then I could change my service account to look like
resource "google_service_account" "scheduler" {
count = local.service_account_count
account_id = "${var.prefix}-scheduler"
project = var.project
}
And then wherever I would have referenced google_service_account.scheduler.email I can instead reference local.service_account_email .. It doesn't look like I'm able to do this, however, for a few reasons.
I get the following error if I try to use the locals block that mentioned above:
│ Because google_service_account.scheduler has "count" set, its attributes must be accessed on specific instances.
│
│ F`or example, to correlate with indices of a referring resource, use:
│ google_service_account.scheduler[count.index]
╵
If I change it so that I'm using google_service_account.scheduler[count.index].email instead, I get the following error:
│ Because google_service_account.scheduler has "count" set, its attributes must be accessed on specific instances.
│
│ For example, to correlate with indices of a referring resource, use:
│ google_service_account.scheduler[count.index]
╵
Now I'm sort of stuck, because I can't force any resources that would originally have referenced google_service_account.scheduler.email to instead reference the var.service_account_email variable that is being passed in for cases where we would prefer to use an existing service account.
Since you are using count, you have to use [0] to access your resource:
service_account_email = var.service_account_email == null ? google_service_account.scheduler[0].email : var.service_account_email
I am working quite a bit with terraform in hopes of building an in-house solution to standing up infra. So far, I have written most of the terraform code and am building out the Ansible code for post-processing of the instances that are stood up. I am shuttling over the dynamic inventory from terraform to Ansible using this little Go app that can be found here, https://github.com/adammck/terraform-inventory. All of that works well.
As I get more into the terraform code, I am trying to use a ternary conditional operator on the ssh key for Linux instances. The goal is to "reuse" this resource on multiple instances.
My resource looks like this ..
resource "aws_key_pair" "key" {
key_name = var.ssh_key
count = var.create_ssh_key ? 1 : 0
public_key = file("~/.ssh/${var.ssh_key}")
}
I've included the [count.index] within the key argument here ...
resource "aws_instance" "linux" {
ami = var.linux_ami
instance_type = var.linux_instance_type
count = var.linux_instance_number
subnet_id = data.aws_subnet.itops_subnet.id
key_name = aws_key_pair.key[count.index].key_name
...
$ terraform validate comes back clean.
$ terraform plan -var-file response-file.tfvars -var "create_ssh_key=false" does not.
The std error is as follows ...
$ terraform plan -var-file response-file.tfvars -var "create_ssh_key=false"
╷
│ Error: Invalid index
│
│ on instances.tf line 16, in resource "aws_instance" "linux":
│ 16: key_name = aws_key_pair.key[count.index].key_name
│ ├────────────────
│ │ aws_key_pair.key is empty tuple
│ │ count.index is 0
│
│ The given key does not identify an element in this collection value.
What am I missing?
Thanks for the feedback!
if count in aws_key_pair is 0, there is no key to reference later on at all.
So you have to check for that and use null to eliminate key_name in such a case:
resource "aws_instance" "linux" {
ami = var.linux_ami
instance_type = var.linux_instance_type
count = var.linux_instance_number
subnet_id = data.aws_subnet.itops_subnet.id
key_name = var.create_ssh_key ? aws_key_pair.key[0].key_name : null
I think the root problem in your example here is that your resource "aws_key_pair" "key" block and your resource "aws_instance" "linux" block both have different values for count, and so therefore it isn't valid to use the count.index of the second to access an instance of the first.
In your case, you seem to have zero key pairs (it says aws_key_pair.key is empty tuple) but you have at least one EC2 instance, and so your expression is trying to access the zeroth instance of the key pair resource, which then fails because it doesn't have a zeroth instance.
If you are using Terraform v0.15 or later then you can use the one function to concisely handle both the zero- and one-instance cases of the key pair resource, like this:
resource "aws_instance" "linux" {
# ...
key_name = one(aws_key_pair.key[*].key_name)
# ...
}
Breaking this down into smaller parts:
aws_key_pair.key[*].key_name is a splat expression which projects from the list of objects aws_key_pair.key into a list of just strings containing key names, by accessing .key_name on each of the objects. In your case, because your resource can only have count 0 or 1, this'll be either a zero-or-one-element list of key names.
one then accepts both of those possible results as follows:
If it's a one-element list, it'll return just that single element no longer wrapped in a list.
If it's a zero-element list, it'll return null which, in a resource argument, means the same thing as not specifying that argument at all.
The effect, then, will be that if you have one key pair then it'll associate that key pair, but if you have no key pairs then it'll leave that argument unset and thus create an instance that doesn't have a key pair at all.
I have a terraform module that does domain delegation. For several variables there is some validation against a hard-coded value to check that a user is using valid inputs, for example:
resource "null_resource" "validate_region" {
count = contains(local.regions, var.region) == true ? 0 : "Please provide a valid AWS region. E.g. (us-west-2)"
}
with local.regions being hard-coded and var.region being a user-set variable. The above code works in that when a user sets the variable wrong, it throws an error like this:
Error: Incorrect value type
on .terraform/foo/main.tf line 46, in resource "null_resource" "validate_region":
46: count = contains(local.regions, var.region) == true ? 0 : "Please provide a valid AWS region. E.g. (us-west-2)"
Invalid expression value: a number is required.
I now need to validate that the AWS account the user is currently using is the correct one. In this case it's up to the user to set the account id of the correct account in their variables, and my code needs to pull the account id of the account that's running the module and compare it against the user's variable. I've tried something like this:
data "aws_caller_identity" "account" {}
resource "null_resource" "validate_account" {
count = data.aws_caller_identity.account.account_id == var.primary_account_id ? 0 : "Please check that you are using the AWS creds for the primary account for this domain."
}
data "aws_route53_zone" "primary" {
name = local.primary_name
}
with various syntax changes on the "{data.aws_caller_identity.account.account_id == var.primary_account_id}" ? 0 part in an effort to get the logic to work, but no luck. I would like it to throw an error like the region validation does, where it will show the error message I wrote. Instead(depending on syntax), it will work as expected for the correct account and throw a Error: no matching Route53Zone found error for the incorrect account, OR it will throw a completely different error presumably because the syntax is screwing things up.
How do I get this to work? is it possible?
What I do is create an if statement in the locals block and source a file with the error message I want to display.
variable "stage" {
type = string
desciption = "The stage to run the deployment in"
}
locals {
stage_validation = var.stage == "prod" || var.stage == "dev"
? var.stage
: file("[Error] this module should only be ran for stages ['prod' or 'dev' ]")
}
The output of setting the stage variable to anything other than 'dev' or 'prod' is as bellow
╷
│ Error: Invalid function argument
│
│ on main.tf line 10, in locals:
│ 10: stage_validation = var.stage == "prod" || var.stage == "dev"
│ ? var.stage
│ : file("[Error] this module should only be ran for stages ['prod' or 'dev' ]")
│
│ Invalid value for "path" parameter: no file exists at This module should only be run for stages ['prod' or 'dev']; this function works only
│ with files that are distributed as part of the configuration source code, so if this file will be created by a resource in this
│ configuration you must instead obtain this result from an attribute of that resource.
╵
This is helpful because it allows you to write an error message that will be shown to the person trying to run the code.
An alternative option here that should simplify what you are doing is to set the region and account constraints up so that Terraform will automatically use the correct region and fail if the credentials are not for the correct account.
You can define this in the aws provider block. An example might look like this:
provider "aws" {
region = "eu-west-1"
allowed_account_ids = ["123456789012"]
}
Now if you attempt to use credentials for a different AWS account then Terraform will fail during the plan stage:
Error: AWS Account ID not allowed: 234567890123
I figured out that this block:
data "aws_route53_zone" "primary" {
name = local.primary_name
}
was running before the account validation resource block. Add in a depends_on like so:
data "aws_route53_zone" "primary" {
name = local.primary_name
depends_on = [null_resource.validate_account,
]
}
And it's all good.
I'm getting the following error when trying to initially plan or apply a resource that is using the data values from the AWS environment to a count.
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
Error: Invalid count argument
on main.tf line 24, in resource "aws_efs_mount_target" "target":
24: count = length(data.aws_subnet_ids.subnets.ids)
The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.
$ terraform --version
Terraform v0.12.9
+ provider.aws v2.30.0
I tried using the target option but doesn't seem to work on data type.
$ terraform apply -target aws_subnet_ids.subnets
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
The only solution I found that works is:
remove the resource
apply the project
add the resource back
apply again
Here is a terraform config I created for testing.
provider "aws" {
version = "~> 2.0"
}
locals {
project_id = "it_broke_like_3_collar_watch"
}
terraform {
required_version = ">= 0.12"
}
resource aws_default_vpc default {
}
data aws_subnet_ids subnets {
vpc_id = aws_default_vpc.default.id
}
resource aws_efs_file_system efs {
creation_token = local.project_id
encrypted = true
}
resource aws_efs_mount_target target {
depends_on = [ aws_efs_file_system.efs ]
count = length(data.aws_subnet_ids.subnets.ids)
file_system_id = aws_efs_file_system.efs.id
subnet_id = tolist(data.aws_subnet_ids.subnets.ids)[count.index]
}
Finally figured out the answer after researching the answer by Dude0001.
Short Answer. Use the aws_vpc data source with the default argument instead of the aws_default_vpc resource. Here is the working sample with comments on the changes.
locals {
project_id = "it_broke_like_3_collar_watch"
}
terraform {
required_version = ">= 0.12"
}
// Delete this --> resource aws_default_vpc default {}
// Add this
data aws_vpc default {
default = true
}
data "aws_subnet_ids" "subnets" {
// Update this from aws_default_vpc.default.id
vpc_id = "${data.aws_vpc.default.id}"
}
resource aws_efs_file_system efs {
creation_token = local.project_id
encrypted = true
}
resource aws_efs_mount_target target {
depends_on = [ aws_efs_file_system.efs ]
count = length(data.aws_subnet_ids.subnets.ids)
file_system_id = aws_efs_file_system.efs.id
subnet_id = tolist(data.aws_subnet_ids.subnets.ids)[count.index]
}
What I couldn't figure out was why my work around of removing aws_efs_mount_target on the first apply worked. It's because after the first apply the aws_default_vpc was loaded into the state file.
So an alternate solution without making change to the original tf file would be to use the target option on the first apply:
$ terraform apply --target aws_default_vpc.default
However, I don't like this as it requires a special case on first deployment which is pretty unique for the terraform deployments I've worked with.
The aws_default_vpc isn't a resource TF can create or destroy. It is the default VPC for your account in each region that AWS creates automatically for you that is protected from being destroyed. You can only (and need to) adopt it in to management and your TF state. This will allow you to begin managing and to inspect when you run plan or apply. Otherwise, TF doesn't know what the resource is or what state it is in, and it cannot create a new one for you as it s a special type of protected resource as described above.
With that said, go get the default VPC id from the correct region you are deploying in your account. Then import it into your TF state. It should then be able to inspect and count the number of subnets.
For example
terraform import aws_default_vpc.default vpc-xxxxxx
https://www.terraform.io/docs/providers/aws/r/default_vpc.html
Using the data element for this looks a little odd to me as well. Can you change your TF script to get the count directly through the aws_default_vpc resource?