AWS RDS Proxy unavailalbe - how to debug? - amazon-web-services

Question

I have created an RDS Proxy using Terraform. However, it does not seem to be working.
My application code cannot connect to the proxy (timeout) and aws rds describe-db-proxy-targets gives the following:
{
"Targets": [
{
"Endpoint": "mydb.aaaaaaaaaaaa.eu-west-2.rds.amazonaws.com",
"RdsResourceId": "mydb",
"Port": 5432,
"Type": "RDS_INSTANCE",
"TargetHealth": {
"State": "UNAVAILABLE",
"Description": "DBProxy Target unavailable due to an internal error"
}
}
]
}
How can I go about debugging this?
Here is the Terraform script for the proxy. The RDS instance is described elsewhere, but is working.
data "aws_subnet" "mydb_rds" {
filter {
name = "availability-zone"
values = [ aws_db_instance.mydb.availability_zone ]
}
}
resource "aws_secretsmanager_secret" "mydb_rds_proxy" {
name = "mydb-rds-proxy"
}
resource "aws_secretsmanager_secret_version" "mydb_rds_proxy" {
secret_id = aws_secretsmanager_secret.mydb_rds_proxy.id
secret_string = var.db_password
}
resource "aws_iam_role" "mydb_rds_proxy" {
name = "mydb-rds-proxy"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "rds.amazonaws.com"
}
}
]
}
EOF
}
resource "aws_iam_policy" "mydb_rds_proxy_policy" {
name = "mydb-rds-proxy"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GetSecretValue",
"Action": [
"secretsmanager:GetSecretValue"
],
"Effect": "Allow",
"Resource": [
"${aws_secretsmanager_secret.mydb_rds_proxy.arn}"
]
},
{
"Sid": "DecryptSecretValue",
"Action": [
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": [
"${aws_secretsmanager_secret.mydb_rds_proxy.arn}"
]
}
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "mydb_rds_proxy_policy_attachment" {
role = aws_iam_role.mydb_rds_proxy.name
policy_arn = aws_iam_policy.mydb_rds_proxy_policy.arn
}
resource "aws_db_proxy" "mydb" {
name = "mydb-rds-proxy"
debug_logging = false
engine_family = "POSTGRESQL"
idle_client_timeout = 1800
require_tls = true
role_arn = aws_iam_role.mydb_rds_proxy.arn
vpc_security_group_ids = [ aws_security_group.mydb_rds.id ]
vpc_subnet_ids = [
data.aws_subnet.mydb_rds.id,
aws_default_subnet.subnet_a.id,
aws_default_subnet.subnet_b.id
]
auth {
auth_scheme = "SECRETS"
iam_auth = "DISABLED"
secret_arn = aws_secretsmanager_secret.mydb_rds_proxy.arn
}
}
resource "aws_db_proxy_default_target_group" "mydb" {
db_proxy_name = aws_db_proxy.mydb.name
connection_pool_config {
connection_borrow_timeout = 120
max_connections_percent = 100
max_idle_connections_percent = 50
}
}
resource "aws_db_proxy_target" "mydb" {
db_instance_identifier = aws_db_instance.mydb.id
db_proxy_name = aws_db_proxy.mydb.name
target_group_name = aws_db_proxy_default_target_group.mydb.name
}
locals {
proxied_pg_connection_string = "postgres://${aws_db_instance.mydb.username}:${var.db_password}#${aws_db_proxy.mydb.endpoint}:5432/postgres?client_encoding=UTF8"
}

Answer 1

There are several things you need to get right for this to work:
Username / password stored in the secret
Security group rules from Lambda -> RDS Proxy
Security group rules from RDS Proxy -> RDS
RDS Proxy, Lambda and RDS in the same VPC
RDS Proxy role can access the secret
A useful debugging query is:
aws rds describe-db-proxy-targets --db-proxy-name <proxy-name>
To understand the error message it gives back, see this page.
The username / password is the hardest thing to discover, since Terraform does not support it yet. What you need to do is construct a JSON string in Terraform that matches what RDS Proxy can understand:
resource "aws_secretsmanager_secret_version" "my_db_proxy" {
secret_id = aws_secretsmanager_secret.my_db_proxy.id
secret_string = jsonencode({
"username" = aws_db_instance.my_db.username
"password" = var.db_password
"engine" = "postgres"
"host" = aws_db_instance.my_db.address
"port" = 5432
"dbInstanceIdentifier" = aws_db_instance.my_db.id
})
}
You then need ensure these security group rules allowing TCP traffic on port 5432 (for Postgres) exist:
ingress Lambda to RDS Proxy
ingress RDS Proxy to RDS
egress RDS Proxy to "0.0.0.0/0"
The RDS Proxy role should have a policy like this:
resource "aws_iam_policy" "my_rds_proxy_policy" {
name = "my-rds-proxy"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Action": [
"rds:*"
],
"Effect": "Allow",
"Resource": [
"${aws_db_instance.my_db.arn}"
]
},
{
"Sid": "GetSecretValue",
"Action": [
"secretsmanager:GetSecretValue"
],
"Effect": "Allow",
"Resource": [
"${aws_secretsmanager_secret.my_rds_proxy.arn}"
]
},
{
"Sid": "DecryptSecretValue",
"Action": [
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Sid": "DecryptKms",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:ViaService": "secretsmanager.${var.aws_region}.amazonaws.com"
}
}
}
]
}
EOF
}
Good luck!