AWS ACM certificate not validating - amazon-web-services

Question

Before I begin let me say that I read thoroughly all the stack overflow posts and resources in the appendix, and could not find a solution to my problem.
I am trying to create, validate and connect a subdomain through Route53 and AWS Certificate Manager. The subdomain is challenge.sre.mycompany.com.
The terraform plan looks something like this:
# module.project_challenge.module.challenge-certificate.aws_acm_certificate.cert will be created
+ resource "aws_acm_certificate" "cert" {
+ arn = (known after apply)
+ domain_name = "challenge.sre.mycompany.com"
+ domain_validation_options = [
+ {
+ domain_name = "challenge.sre.mycompany.com"
+ resource_record_name = (known after apply)
+ resource_record_type = (known after apply)
+ resource_record_value = (known after apply)
},
]
+ id = (known after apply)
+ status = (known after apply)
+ subject_alternative_names = (known after apply)
+ tags_all = (known after apply)
+ validation_emails = (known after apply)
+ validation_method = "DNS"
}
# module.project_challenge.module.challenge-certificate.aws_acm_certificate_validation.cert will be created
+ resource "aws_acm_certificate_validation" "cert" {
+ certificate_arn = (known after apply)
+ id = (known after apply)
+ validation_record_fqdns = (known after apply)
}
# module.project_challenge.module.challenge-certificate.aws_route53_record.cert["challenge.sre.mycompany.com"] will be created
+ resource "aws_route53_record" "cert" {
+ allow_overwrite = true
+ fqdn = (known after apply)
+ id = (known after apply)
+ name = (known after apply)
+ records = (known after apply)
+ ttl = 60
+ type = (known after apply)
+ zone_id = (known after apply)
}
# module.project_challenge.module.vpc.aws_route53_zone.public will be created
+ resource "aws_route53_zone" "public" {
+ arn = (known after apply)
+ comment = "Managed by Terraform"
+ force_destroy = false
+ id = (known after apply)
+ name = "sre.mycompany.com"
+ name_servers = (known after apply)
+ tags_all = (known after apply)
+ zone_id = (known after apply)
}
As you can see, it create a public hosted zone, an acm certificate and even the validation record. The problem here is that the certificate is stuck on 'Pending Validation` for about 48 hours.
Some details:
The domain is registered through our production account, where I am working on our development account for this.
Both accounts are in the same AWS organisation (if this matters)
Terraform created a public hosted zone sre.mycompany.com with the following attributes:
sre.mycompany.com NS Records:
ns-001.awsdns-01.com.
ns-002.awsdns-02.net.
ns-003.awsdns-03.co.uk.
ns-004.awsdns-04.org.
sre.mycompany.com SOA Simple Record:
ns-001.awsdns-01.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
CNAME Simple Record
_g938534f3gfe03832h34.challenge.sre.mycompany.com _89432htieh4934hw043f.tkfpekghn.acm-validations.aws.
Obviously the real values are obfuscated*
When I dig sre.mycompany.com or dig challenge.sre.mycompany.com I get:
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16577
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
When I dig just mycompany.com I get:
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61857
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 5
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mycompany.com. IN A
;; ANSWER SECTION:
mycompany.com. 300 IN A <some-ip-hidden>
;; AUTHORITY SECTION:
mycompany.com. 169554 IN NS ns-555.awsdns-55.com.
mycompany.com. 169554 IN NS ns-666.awsdns-66.net.
mycompany.com. 169554 IN NS ns-777.awsdns-77.org.
mycompany.com. 169554 IN NS ns-888.awsdns-88.co.uk.
Notice that the nameservers here are different from the ones I see in the console of my terraform created hosted zone (scroll above ns-001.awsdns-01.com. etc)
I cannot seem to fetch the CNAME record from my terminal.
In AWS everything seems to work fine on the other hand. When I go to:
Route 53> Hosted zones > Test Record I do get the value of the CNAME record:
Response returned by Route 53
Response from Route 53 based on the following options.
Hosted zone: sre.mycompany.com
Record name: _g938534f3gfe03832h34.challenge.
Record type: CNAME
DNS response code: No Error
Protocol: UDP
Response returned by Route 53: _89432htieh4934hw043f.tkfpekghn.acm-validations.aws.
At last if I , the response is:
;; Received 888 bytes from <some-ip-hidden>#53(ns-666.awsdns-66.net) in 3 ms
mycompany.com. 169201 IN NS ns-666.awsdns-66.net.
mycompany.com. 169201 IN NS ns-777.awsdns-77.org.
mycompany.com. 169201 IN NS ns-888.awsdns-88.co.uk.
mycompany.com. 169201 IN NS ns-555.awsdns-55.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-888.awsdns-88.co.uk) in 4 ms
mycompany.com. 169201 IN NS ns-777.awsdns-77.org.
mycompany.com. 169201 IN NS ns-666.awsdns-66.net.
mycompany.com. 169201 IN NS ns-888.awsdns-88.co.uk.
mycompany.com. 169201 IN NS ns-555.awsdns-55.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-555.awsdns-55.com) in 4 ms
mycompany.com. 169201 IN NS ns-666.awsdns-66.net.
mycompany.com. 169201 IN NS ns-777.awsdns-77.org.
mycompany.com. 169201 IN NS ns-555.awsdns-55.com.
mycompany.com. 169201 IN NS ns-888.awsdns-88.co.uk.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-888.awsdns-88.co.uk) in 4 ms
mycompany.com. 169201 IN NS ns-777.awsdns-77.org.
mycompany.com. 169201 IN NS ns-666.awsdns-66.net.
mycompany.com. 169201 IN NS ns-888.awsdns-88.co.uk.
mycompany.com. 169201 IN NS ns-555.awsdns-55.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-777.awsdns-77.org) in 5 ms
mycompany.com. 169201 IN NS ns-777.awsdns-77.org.
mycompany.com. 169201 IN NS ns-888.awsdns-88.co.uk.
mycompany.com. 169201 IN NS ns-555.awsdns-55.com.
mycompany.com. 169201 IN NS ns-666.awsdns-66.net.
;; BAD (HORIZONTAL) REFERRAL
Key takeaways:
I cannot get the CNAME with any command from my terminal
The certificate is not validating
Appendix
Certificate in Pending state in AWS Certificate Manager
Certificate with DNS Validation is stuck in Pending Validation
AWS ACM certificate state is pending validation and not changing to issues
My domain is pending validation in AWS Certificate Manager
AWS ACM Stuck in Pending Validation Unless NS Changed in Domain
Resolve ACM certificate still pending

Answer 1

Everything your terraform is creating is fine, however when you create a new zone in AWS you need to add the nameservers on the ROOT DNS Panel (most likely where you bought the domain mycompany.com).
You need to add the NS entry for the subdomain you want to use (the new zone you're creating)
You can reference this article https://webmasters.stackexchange.com/questions/93897/can-i-use-different-nameservers-for-different-subdomains

Answer 2

When you have multiple route53 hosted zones for domain and subdomains you need to link them together.
This can be done by adding the subdomain Nameservers in the domain hosted zone.
You cannot break the domain hosted zone by adding records, you will only break the link with the subdomain hosted zone.
So to clarify with an example, let's say you have a domain route53 hosted zone for
mycomany.gr
Record Name
Type
Value
mycomany.gr
NS
ns-xxxx.org ns-xxx.uk
sre.mycompany.com
NS
ns-yyy.org ns-yyy.uk
The first row is created when you create the route53 hosted zone. After you need to take the nameservers and add them to your domain provider. With this way you link the domain with AWS and it knows its valid.
The second row you will need to add it manually after you have create your subdomain route53 hosted zone (sre.mycompany.com). The nameservers now are the ones that route53 subdomain created for you. With this way you say to route53 this domain (mycompany.com) owns this subdomain (sre.mycompany.com).
All those stuff need to be done before and ACM certificates are created and the reason is that the ACM has a domain validation which tries to create a record to your valid domain or subdomain. If your domain or subdomain isnt linked to a valid domain the acm will throw an error.

Answer 3

Your CNAME in your zone file has a mycompany.com on the end. That's not the normal way to do a CNAME. Should probably be:
CNAME Simple Record
_g938534f3gfe03832h34.challenge.sre _89432htieh4934hw043f.tkfpekghn.acm-validations.aws.
If you add the mycompany.com into the CNAME, then the actual resolve address is _g938534f3gfe03832h34.challenge.sre.mycompany.com.mycompany.com
The only way I have found to get correct validation records into route53 through terraform looks like this:
resource "aws_route53_record" "cert-verify" {
for_each = {
for dvo in aws_acm_certificate.cert_name.domain_validation_options : dvo.domain_name => {
name = dvo.resource_record_name
record = dvo.resource_record_value
type = dvo.resource_record_type
}
name = each.value.name
records = [each.value.record]
ttl = 60
type = each.value.type
zone_id = aws_route53_zone.zone.zone_id
}
makes for a messy state file, but it works