Terraform - Conditional Data Source - amazon-web-services

In terraform, is there any way to conditionally use a data source? For example:
data "aws_ami" "application" {
most_recent = true
filter {
name = "tag:environment"
values = ["${var.environment}"]
}
owners = ["self"]
}
I'm hoping to be able to pass in an environment variable via the command line, and based on that, determine whether or not to fetch this data source.
I know with resources you can use the count property, but it doesn't seem you can use that with data sources.
I would consider tucking this code away in a module, but modules also can't use the count parameter.
Lastly, another option would be to provide a "Default" value for the data source, if it returned null, but I don't think that's doable either.
Are there any other potential solutions for this?

You can use a conditional on data sources the same as you can with resources and also from Terraform 0.13+ on modules as well:
variable "lookup_ami" {
default = true
}
data "aws_ami" "application" {
count = var.lookup_ami ? 1 : 0
most_recent = true
filter {
name = "tag:environment"
values = [var.environment]
}
owners = ["self"]
}
One use case for this in Terraform 0.12+ is to utilise the lazy evaluation of ternary statements like with the following:
variable "internal" {
default = true
}
data "aws_route53_zone" "private_zone" {
count = var.internal ? 1 : 0
name = var.domain
vpc_id = var.vpc_id
private_zone = var.internal
}
data "aws_route53_zone" "public_zone" {
count = var.internal ? 0 : 1
name = var.domain
private_zone = var.internal
}
resource "aws_route53_record" "www" {
zone_id = var.internal ? data.aws_route53_zone.private_zone.zone_id : data.aws_route53_zone.public_zone.zone_id
name = "www.${var.domain}"
type = "A"
alias {
name = aws_elb.lb.dns_name
zone_id = aws_elb.lb.zone_id
evaluate_target_health = false
}
}
This would create a record in the private zone when var.internal is true and instead create a record in the public zone when var.internal is false.
For this specific use case you could also use Terraform 0.12+'s null to rewrite this more simply:
variable "internal" {
default = true
}
data "aws_route53_zone" "zone" {
name = var.domain
vpc_id = var.internal ? var.vpc_id : null
private_zone = var.internal
}
resource "aws_route53_record" "www" {
zone_id = data.aws_route53_zone.zone.zone_id
name = "www.${data.aws_route53_zone.zone.name}"
type = "A"
alias {
name = aws_elb.lb.dns_name
zone_id = aws_elb.lb.zone_id
evaluate_target_health = false
}
}
This would only pass the vpc_id parameter to the aws_route53_zone data source if var.internal is set to true as you can't set vpc_id when private_zone is false.
Old Terraform 0.11 and earlier answer:
You can in fact use a conditional on the count of data sources but I've yet to manage to work out a good use case for it when I've tried.
As an example I successfully had this working:
data "aws_route53_zone" "private_zone" {
count = "${var.internal == "true" ? 1 : 0}"
name = "${var.domain}"
vpc_id = "${var.vpc_id}"
private_zone = "true"
}
data "aws_route53_zone" "public_zone" {
count = "${var.internal == "true" ? 0 : 1}"
name = "${var.domain}"
private_zone = "false"
}
But then had issues in how to then select the output of it because Terraform will evaluate any variables in the ternary conditional before deciding which side of the ternary to use (instead of lazy evaluation). So something like this doesn't work:
resource "aws_route53_record" "www" {
zone_id = "${var.internal ? data.aws_route53_zone.private_zone.zone_id : data.aws_route53_zone.public_zone.zone_id}"
name = "www.example.com"
type = "A"
alias {
name = "${aws_elb.lb.dns_name}"
zone_id = "${aws_elb.lb.zone_id }"
evaluate_target_health = "false"
}
}
Because if internal is true then you get the private_zone data source but not the public_zone data source and so the second half of the ternary fails to evaluate because data.aws_route53_zone.public_zone.zone_id isn't defined and equally with the other way around too.
In your case you probably just want to conditionally use the data source so might be able to do something like this:
variable "dynamic_ami" { default = "true" }
variable "default_ami" { default = "ami-123456" }
data "aws_ami" "application" {
most_recent = true
filter {
name = "tag:environment"
values = ["${var.environment}"]
}
owners = ["self"]
}
resource "aws_instance" "app" {
ami = "${var.dynamic_ami == "true" ? data.aws_ami.application.id : var.default_ami}"
instance_type = "t2.micro"
}

Related

Terraform: Iterating list for AWS Certificate validation with Cloudflare DNS

I have a map in a tfvars file that contains, Cloudflare zone id, site address, and zone (domain), I am wanting to iterate through that map, generating an ACM certificate, with a certificate validation DNS record being created in Cloudflare.
My map looks like this;
my_domains = {
example1 = {
cloudflare_zone_id = "00000000000000000000000000001"
address = "dev.example1.com"
domain = "example1.com"
}
example2 = {
cloudflare_zone_id = "0000000000000000000000000000002"
address = "dev.example2.com"
domain = "example2.com"
}
example3 = {
cloudflare_zone_id = "0000000000000000000000000000003"
address = "dev.library.example3.com"
domain = "example3.com"
}
}
I then have the following code for the certificate creation and validation:
resource "aws_acm_certificate" "my_certs" {
for_each = var.my_domains
domain_name = each.value.address
validation_method = "DNS"
subject_alternative_names = [
"*.${each.value.address}"
]
lifecycle {
create_before_destroy = true
}
}
resource "cloudflare_zone" "my_zone" {
for_each = var.my_domains
zone = each.value.domain
type = "full"
}
resource "cloudflare_record" "my_certificate_validation" {
for_each = {
for dvo in aws_acm_certificate.my_certs.domain_validation_options : dvo.domain_name => {
name = dvo.resource_record_name
record = dvo.resource_record_value
type = dvo.resource_record_type
}
}
zone_id = cloudflare_zone.my_zone.id
name = each.value.name
value = trimsuffix(each.value.record, ".")
type = each.value.type
ttl = 1
proxied = false
}
When I run a plan, I get the following errors:
Error: Missing resource instance key
on cfcertvalidation.tf line 23, in resource "cloudflare_record" "my_certificate_validation":
23: for dvo in aws_acm_certificate.my_certs.domain_validation_options : dvo.domain_name => {
Because aws_acm_certificate.my_certs has "for_each" set, its attributes must be
accessed on specific instances.
For example, to correlate with indices of a referring resource, use:
aws_acm_certificate.my_certs[each.key]
Error: Missing resource instance key
on cfcertvalidation.tf line 30, in resource "cloudflare_record" "my_certificate_validation":
30: zone_id = cloudflare_zone.my_zone.id
Because cloudflare_zone.cdt has "for_each" set, its attributes must be
accessed on specific instances.
For example, to correlate with indices of a referring resource, use:
cloudflare_zone.my_zone[each.key]
Note: I added the cloudflare_zone resource rther than using the zone id already in the map as a way to simplify things in troubleshooting.
I am sure the answer is in the suggestion for using a [each.key], but I'm not sure how to implement it.
Any assistance would be greatly appreciated.
I have changed the map somewhat for my solution, so for completeness I have included the changed map here:
variable "my_domains" {
type = map(any)
default = {
example1 = {
cf_zone_id = "0000000000000000000000000000"
address = "example1.com"
zone = "example1.com"
}
example2 = {
cf_zone_id = "0000000000000000000000000000"
address = "example2.com"
zone = "example2.com"
}
example3 = {
cf_zone_id = "0000000000000000000000000000"
address = "library.example3.com"
zone = "example3.com"
}
}
}
What follows is the working solution, we start out by creating a local variable of type list, looping through the my_domains map to get the cert validation records we need. That then gets converted into a map, which is then used by the cloudflare_record resource to create the relevant DNS entries.
resource "aws_acm_certificate" "my_certs" {
for_each = var.my_domains
domain_name = "${var.env_url_prefix}${var.my_domains[each.key] ["address"]}"
validation_method = "DNS"
subject_alternative_names = ["*.${var.env_url_prefix}${var.my_domains[each.key]["address"]}"]
lifecycle {
create_before_destroy = true
}
}
locals {
validation = [
for certs in keys(var.my_domains) : {
for dvo in aws_acm_certificate.my_certs[certs].domain_validation_options : dvo.domain_name => {
name = dvo.resource_record_name
value = trimsuffix(dvo.resource_record_value, ".")
type = dvo.resource_record_type
zone_id = var.my_domains[certs]["cf_zone_id"] # Get the zone id
}
}
]
# transform the list into a map
validation_map = { for item in local.validation : keys(item)[0] => values(item)[0]
}
}
resource "cloudflare_record" "my_cert_validations" {
for_each = local.validation_map
zone_id = local.validation_map[each.key]["zone_id"]
name = local.validation_map[each.key]["name"]
value = local.validation_map[each.key]["value"]
type = local.validation_map[each.key]["type"]
ttl = 1
proxied = false #important otherwise validation will fail
}

for_each value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created

i am trying to create route 53 using module concept. but getting below error.
"The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created."
for_each = var.create-route53 ? local.recordsets : tomap({})
local.recordsets will be known only after apply
var.create-route53 is true
can someone guide me on this ?
actual code:
module "route53" {
...
...
records = [
{
name = "test-name"
type = "A"
ttl = 300
records = [for instance in module.ec2: instance.ec2-IP
]
},
]
vpc_id = "${module.vpc.vpc_id}"
}
Inside modules folder below code will be there in route53 folder:
locals {
records = try(jsondecode(var.records), var.records)
recordsets = {
for rs in local.records :
join(" ", compact(["${rs.name} ${rs.type}", lookup(rs, "set_identifier", "")])) => merge(rs, {
records = jsonencode(try(rs.records, null))
})
}
}
resource "aws_route53_record" "this" {
for_each = var.create-route53 ? local.recordsets : tomap({})
zone_id = aws_route53_zone.private.zone_id
name = each.value.name != "" ? "${each.value.name}" : "test"
type = each.value.type
ttl = lookup(each.value, "ttl", null)
records = jsondecode(each.value.records)
set_identifier = lookup(each.value, "set_identifier", null)
health_check_id = lookup(each.value, "health_check_id", null)
multivalue_answer_routing_policy = lookup(each.value, "multivalue_answer_routing_policy", null)
allow_overwrite = lookup(each.value, "allow_overwrite", false)
}

Terraform fail due output for_each values

When I trying to dynamically allocate subnet ids by running this tf file:
data "aws_availability_zones" "available" {}
resource "aws_vpc" "wpl_vpc" {
cidr_block = var.vpc_cidr
enable_dns_hostnames = true
tags = {
Name = "WPL-VPC"
CreatedBy = var.created_by
}
}
resource "aws_subnet" "wpl_public_subnet" {
for_each = { for index, az_name in data.aws_availability_zones.available.names : index => az_name }
vpc_id = aws_vpc.wpl_vpc.id
cidr_block = cidrsubnet(var.vpc_cidr, 8, each.key + 10)
availability_zone = data.aws_availability_zones.available.names[each.key]
map_public_ip_on_launch = true
tags = {
Name = "WPL-PublicSubnet"
CreatedBy = var.created_by
}
}
output "wpl_public_subnet_ids" {
value = [aws_subnet.wpl_public_subnet.*.id]
}
I get an error:
Error: Unsupported attribute
on ../modules/vpc/outputs.tf line 5, in output "wpl_public_subnet_ids":
5: value = [aws_subnet.wpl_public_subnet.*.id]
This object does not have an attribute named "id".
However, everything works like charm, if I replace the output with this one:
aws_subnet.wpl_public_subnet.0.id
How could I output all subnet ids values?
Since you are using for_each, you will have a map, not a list. Thus, you have to get values first, before getting the ids. Also you don't need extra square brackets:
output "wpl_public_subnet_ids" {
value = values(aws_subnet.wpl_public_subnet)[*].id
}

aws_acm_certificate seems to have changed its state output possibly due to a provider update- am I doing this wrong?

Terraform v0.12.12
+ provider.aws v3.0.0
+ provider.template v2.1.2
Before I was doing this:
resource "aws_route53_record" "derps" {
name = aws_acm_certificate.mycert[0].resource_record_name
type = aws_acm_certificate.mycert[0].resource_record_type
zone_id = var.my_zone_id
records = aws_acm_certificate.mycert[0].resource_record_value
ttl = 60
}
And that worked fine for me about a week ago.
I just did a plan and got an error:
records = [aws_acm_certificate.mycert.domain_validation_options[0].resource_record_value]
This value does not have any indices.
Now I don't pin provider versions, so I'm assuming I pulled a newer version and the resource changed.
After fighting with this and realizing it's not a list (even though when doing show state it sure looked like one) I am now doing this to make it a list:
resource "aws_route53_record" "derps" {
name = sort(aws_acm_certificate.mycert.domain_validation_options[*].resource_record_name)[0]
type = sort(aws_acm_certificate.mycert.domain_validation_options[*].resource_record_type)[0]
zone_id = var.my_zone_id
records = [sort(aws_acm_certificate.mycert.domain_validation_options[*].resource_record_value)[0]]
ttl = 60
}
This resulted in no changes which is good. But if I use the example for doing this from the docs they now use for_each: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acm_certificate_validation
resource "aws_route53_record" "example" {
for_each = {
for dvo in aws_acm_certificate.example.domain_validation_options : dvo.domain_name => {
name = dvo.resource_record_name
record = dvo.resource_record_value
type = dvo.resource_record_type
zone_id = dvo.domain_name == "example.org" ? data.aws_route53_zone.example_org.zone_id : data.aws_route53_zone.example_com.zone_id
}
}
allow_overwrite = true
name = each.value.name
records = [each.value.record]
ttl = 60
type = each.value.type
zone_id = each.value.zone_id
}
resource "aws_acm_certificate_validation" "example" {
certificate_arn = aws_acm_certificate.example.arn
validation_record_fqdns = [for record in aws_route53_record.example : record.fqdn]
}
Is the above the correct way to do this now? Am I going to run into issues doing it the way I currently am? Doing it like above would result in a destroy/recreate (i guess I could import it myself but that's painful).
Is doing it my way not going to result in unexpected diffs?
Edit
So, more specific for my issue. This is what I see when I look at the state:
terraform state show aws_acm_certificate.mycert
...
domain_name = "*.mydom.com"
domain_validation_options = [
{
domain_name = "*.mydom.com"
resource_record_name = "_11111111111.mydom.com."
resource_record_type = "CNAME"
resource_record_value = "_1111111111.11111111.acm-validations.aws."
},
{
domain_name = "mydom.com"
resource_record_name = "_11111111111.mydom.com."
resource_record_type = "CNAME"
resource_record_value = "_1111111111.111111111.acm-validations.aws."
},
]
...
By using sort I'm effectively using count which of course results in a destroy/recreate if the order changes. But in my case I think that's unlikely?? I also don't fully understand the difference between just using the values from the wildcard validation config and using both of them.
The AWS Terraform provider was recently upgraded to version 3.0. This version comes with a list of breaking changes. I recommend consulting the AWS provider 3.0 upgrade guide.
The issue you are encountering is because the domain_validation_options attribute is now a set instead of a list. From that guide:
Since the domain_validation_options attribute changed from a list to a set and sets cannot be indexed in Terraform, the recommendation is to update the configuration to use the more stable resource for_each support instead of count
I recommend using the new foreach syntax, as the upgrade guide recommends, in order to avoid unexpected diffs. The guide states that you will need to use terraform state mv to move the old configuration state to the new configuration, in order to prevent the resources from being recreated.
This is the same problem we were facing just now.. we use for_each to define hosting for multiple sites according to provided local variables, now, since we already use for_each we can't use it for workaround of their changes.. unfortunate.
I didn't want to go with sort so I checked what Jimmy wrote but it didn't work for this case due to output being index, I fixed it by using [0] instead of [*]:
resource "aws_route53_record" "cert_validation" {
for_each = local.web_pages
allow_overwrite = true
name = tolist(aws_acm_certificate.cert[each.key].domain_validation_options)[0].resource_record_name
type = tolist(aws_acm_certificate.cert[each.key].domain_validation_options)[0].resource_record_type
records = [tolist(aws_acm_certificate.cert[each.key].domain_validation_options)[0].resource_record_value]
zone_id = var.aws_hosted_zone
ttl = 60
}
works for us now ;)
These 3 sets of code below work. (I used Terraform v0.15.0)
*The difference between 1st and 2nd is [0] and .0
1st:
resource "aws_route53_record" "myRecord" {
zone_id = aws_route53_zone.myZone.zone_id
name = tolist(aws_acm_certificate.myCert.domain_validation_options)[0].resource_record_name
type = tolist(aws_acm_certificate.myCert.domain_validation_options)[0].resource_record_type
records = [tolist(aws_acm_certificate.myCert.domain_validation_options)[0].resource_record_value]
ttl = "60"
allow_overwrite = true
}
2nd:
resource "aws_route53_record" "myRecord" {
zone_id = aws_route53_zone.myZone.zone_id
name = tolist(aws_acm_certificate.myCert.domain_validation_options).0.resource_record_name
type = tolist(aws_acm_certificate.myCert.domain_validation_options).0.resource_record_type
records = [tolist(aws_acm_certificate.myCert.domain_validation_options).0.resource_record_value]
ttl = "60"
allow_overwrite = true
}
3rd:
resource "aws_route53_record" "myRecord" {
for_each = {
for dvo in aws_acm_certificate.myCert.domain_validation_options : dvo.domain_name => {
name = dvo.resource_record_name
record = dvo.resource_record_value
type = dvo.resource_record_type
}
}
name = each.value.name
records = [each.value.record]
ttl = 60
type = each.value.type
zone_id = aws_route53_zone.myZone.zone_id
allow_overwrite = true
}
You can also cheat with tolist.
e.g. rewrite aws_acm_certificate.mycert.domain_validation_options[*].resource_record_name as tolist(aws_acm_certificate.mycert.domain_validation_options)[*].resource_record_name
I had to do this in a module, as this particular module resource already had a count in it, and I knew I'd only ever have one entry.

Dynamic data source for fetching AMI using Terraform

So the following data source will fetch the latest AMI with component:web tagged. Let's say I have a handful of components with their own AMI. Instead of creating this same block 5 times, is there a way to make this more dynamic where I can pass in the value of component? I can't seem to think of a unique value I can pass in. Do I need to refactor my code a bit?
data "aws_ami" "web" {
filter {
name = "state"
values = ["available"]
}
filter {
name = "tag:component"
values = ["web"]
}
most_recent = true
}
I have a defaults module that acts as a metadata lookup where it fetches and outputs basic things like AMI ID and VPC IDs.
Default Module
# defaults/main.tf
data "aws_ami" "web" {
filter {
name = "state"
values = ["available"]
}
filter {
name = "tag:component"
values = ["web"]
}
most_recent = true
}
output "web_ami" {
value = "${data.aws_ami.web.id}"
}
Main code
# service_name/main.tf
module "defaults" {
source = "../defaults"
region = "${var.region}"
environment = "${var.environment}"
}
module "ftpserver" {
source . = "../ec2_instance"
ami_id = "${module.defaults.web_ami}"
...
}
I'd move the aws_ami data source into the module and have it look up the AMI directly rather than have it passed in from outside.
So I would change the ec2_instance module to look like:
variable "ami_component" {}
data "aws_ami" "selected" {
filter {
name = "state"
values = ["available"]
}
filter {
name = "tag:component"
values = ["${var.ami_component"]
}
most_recent = true
}
resource "aws_instance" "instance" {
ami = "${data.aws_ami.selected.id}"
instance_type = "t2.micro"
tags {
Name = "HelloWorld"
}
}
If you then felt like you needed to be able to override the AMI in the ec2_instance module you could change that to instead be:
variable "ami_component" {}
variable "override_ami" {
default = ""
}
data "aws_ami" "selected" {
filter {
name = "state"
values = ["available"]
}
filter {
name = "tag:component"
values = ["${var.ami_component"]
}
most_recent = true
}
resource "aws_instance" "instance" {
ami = "${var.override_ami != "" ? var.override_ami : data.aws_ami.selected.id}"
instance_type = "t2.micro"
tags {
Name = "HelloWorld"
}
}
This uses a conditional to check if the override_ami variable has been set to something else in which case it will use that, otherwise it will use the ami_component variable to look up the appropriate AMI and use that instead.
This has the benefit of moving the AMI selection logic into the Terraform module making the interface to that module much simpler.