AWS Client VPN DNS Resolution for RDS - amazon-web-services

Question

I've set up Client-VPN and can't seem to reach my RDS instance in a private subnet. I can reach EC2 instances using IP but not by DNS. My setup looks a little like this:
VPC:
CIDR: 10.0.0.0/16
DNS Resolution: Enabled
DNS Hostnames: Enabled
Client-Vpn:
DNS Servers: 10.0.0.2 (have also tried empty)
Security Group: vpn-sg (ingress all from my IP, egress all)
Client CIDR: 10.1.0.0/16
Transport: UDP 443
Associations: 3x private subnets (all have access to RDS instance)
Split-tunnel: Enabled
RDS Instance:
Security Group: rds-sg
Security Group Ingress: All traffic from vpn-sg
I believe that there is a problem with DNS resolution and that for some reason, DNS for the RDS instance is not being resolved. From my EC2 instance I can connect to RDS which suggests DNS resolution is working within the VPC.
I'm running Ubunutu 20.04 and I'm using the AWS VPN client (which I believe uses openvpn underneath). I'm using the openvpn configuration downloaded from the VPN settings in the AWS control panel.
Can someone help explain why the DNS isn't being resolved? Debugging informaiton is below.
Debugging when connected to the VPN
$ ping ip-10-0-0-177.eu-west-1.compute.internal
ping: ip-10-0-0-177.eu-west-1.compute.internal: Name or service not known
$ ping 10.0.0.177
PING 10.0.0.177 (10.0.0.177) 56(84) bytes of data.
64 bytes from 10.0.0.177: icmp_seq=1 ttl=254 time=22.8 ms
64 bytes from 10.0.0.177: icmp_seq=2 ttl=254 time=22.5 ms
64 bytes from 10.0.0.177: icmp_seq=3 ttl=254 time=24.1 ms
--- 10.0.0.177 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 22.472/23.841/25.161/1.046 ms
$ systemd-resolve --status
Global
LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 22 (tun0)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 10.0.0.2
DNS Servers: 10.0.0.2
Link 3 (wlp0s20f3)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 192.168.1.254
DNS Servers: 192.168.1.254
DNS Domain: ~.
home
$ traceroute google.com
traceroute to google.com (216.58.212.238), 30 hops max, 60 byte packets
1 eehub.home (192.168.1.254) 2.327 ms 2.225 ms 3.201 ms
2 * * *
3 * * *
4 213.121.98.128 (213.121.98.128) 14.432 ms 14.407 ms 14.380 ms
5 87.237.20.130 (87.237.20.130) 20.563 ms 20.538 ms 20.992 ms
6 74.125.52.216 (74.125.52.216) 16.718 ms 12.813 ms 12.728 ms
7 * * *
8 142.251.52.148 (142.251.52.148) 13.044 ms 209.85.248.240 (209.85.248.240) 11.870 ms 142.251.54.26 (142.251.54.26) 13.344 ms
9 ams16s22-in-f14.1e100.net (216.58.212.238) 13.257 ms 216.239.63.219 (216.239.63.219) 14.388 ms 14.360 ms
$ traceroute ip-10-0-0-177.eu-west-1.compute.internal
ip-10-0-0-177.eu-west-1.compute.internal: Name or service not known
Cannot handle "host" cmdline arg `ip-10-0-0-177.eu-west-1.compute.internal' on position 1 (argc 1)
Edit 1: I just learned how to run a dig command with a specific nameserver and have confirmed that the DNS resolution does work when the system uses the right server:
$ dig #10.0.0.2 ip-10-0-0-177.eu-west-1.compute.internal
; <<>> DiG 9.16.1-Ubuntu <<>> #10.0.0.2 ip-10-0-0-177.eu-west-1.compute.internal
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2950
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ip-10-0-0-177.eu-west-1.compute.internal. IN A
;; ANSWER SECTION:
ip-10-0-0-177.eu-west-1.compute.internal. 60 IN A 10.0.0.177
;; Query time: 24 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Sat Mar 05 22:38:15 GMT 2022
;; MSG SIZE rcvd: 85
Edit 2: After reading some troubleshooting tips I have managed to get EC2 DNS resolution but not RDS. Still hoping someone can help decipher this :)
$ dig ip-10-0-0-177.eu-west-1.compute.internal
; <<>> DiG 9.16.1-Ubuntu <<>> ip-10-0-0-177.eu-west-1.compute.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3681
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;ip-10-0-0-177.eu-west-1.compute.internal. IN A
;; ANSWER SECTION:
ip-10-0-0-177.eu-west-1.compute.internal. 54 IN A 10.0.0.177
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sat Mar 05 22:46:10 GMT 2022
;; MSG SIZE rcvd: 85
dig ***.***.eu-west-1.rds.amazonaws.com
; <<>> DiG 9.16.1-Ubuntu <<>> ***.***.eu-west-1.rds.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44468
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;***.***.eu-west-1.rds.amazonaws.com. IN A
;; Query time: 20 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sat Mar 05 22:48:26 GMT 2022
;; MSG SIZE rcvd: 82
Again when I perform this directly against the correct nameserver, it resolves.
dig #10.0.0.2 ***.***.eu-west-1.rds.amazonaws.com
; <<>> DiG 9.16.1-Ubuntu <<>> #10.0.0.2 ***.***.eu-west-1.rds.amazonaws.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5532
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;***.***.eu-west-1.rds.amazonaws.com. IN A
;; ANSWER SECTION:
***.***.eu-west-1.rds.amazonaws.com. 5 IN A 10.0.1.233
;; Query time: 24 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Sat Mar 05 22:49:23 GMT 2022
;; MSG SIZE rcvd: 98

Answer 1

The poster gave the answer in his reddit post, which also worked for me, so I am leaving it here in case someone else stumble on this
I had to set my DNS to something useful like Google's (8.8.8.8, 4.4.4.4) on my studio card and reboot. That was it. I don't know why but my other DNS wouldn't resolve.
I am on Comcast internet and I had the exact same problem as the OP. As soon as I switched to Google DNS, per his suggestion, everything worked!
Thanks OP.