Secrets Manager rotation timeout - amazon-web-services

Question

Alright, so after 3 days of trying to get this working I finally give up.
I have:
a private VPC with a subnet that contains an RDS MySQL instance
a Lambda rotator function based on the AWS Python template for single user MySQL rotation
an Endpoint for Secrets Manager, Private DNS enabled.
I have security groups, but for debugging, I've allowed all traffic for all security groups and Network ACLs.
The Lambda rotator function has permissions for secrets manager for all resources, logging to Cloudwatch and the relevant VPC permissions to execute in a VPC.
What does work:
logging to CloudWatch works. I have turned on DEBUG mode
Secrets Manager is invoking the lambda function successfully
a few requests to Secrets Manager appear to work
What doesn't work:
after a few requests, any subsequent requests start timing out
After some time, it manages to send a few more requests. Could this have something to do with Python, networking timeouts and lambda connections being held or dropped due to timeouts?
I can see it does a DescribeSecret request.
Then I can see a GetSecretValue request for an AWSCURRENT stage.
Then I can see a GetSecretValue request for an AWSPENDING stage.
This one returns:
Secrets Manager can't find the specified secret value for VersionId: xxxxxxxx"
Then I can see a GetRandomPassword request.
After that, I see the following in the logs:
Resetting dropped connection: secretsmanager.ap-southeast-2.amazonaws.com
The lambda function now times out.
From this point on, it can't even successfully do a DescribeSecret without the lambda timing out. After maybe 10-15 minutes it starts working again up to the GetRandomPassword part and then drops the connection again.
I don't think it's a security group, ACL or endpoint config issue, because it would either work or not work, not sometimes work.
I also don't think I'm stressing out the API that much - a few requests in a period of a few seconds and then nothing for many minutes should be fine for AWS.
I found a little clue here maybe after GetSecretValue is called?
[DEBUG] 2022-04-09T10:49:20.073Z 34585068-3f21-4471-9035-f9368a3094dd Response headers: {'x-amzn-RequestId': 'f443766f-921c-4772-997f-b150643c4909', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '156', 'Date': 'Sat, 09 Apr 2022 10:49:19 GMT', 'Connection': 'close'}
Looks like the response header contains Connection: close, but that's coming back FROM Secrets Manager.
When I look at other people's logs I can see the headers that the boto3 client sends usually contains Connection: keep-alive, yet looking at my logs none of them contain that header.
I did a bit of an experiment by injecting that header.
session = boto3.session.Session()
session.events.register('before-call.secrets-manager.*', inject_header)
...
def inject_header(params, **kwargs):
params['headers']['Connection'] = 'keep-alive'
However, even if I send that header to the Secrets Manager API it makes no difference.
There's got to be something else going on, I just don't understand the intermittent nature of it!
For reference, the lambda role policy. As you can see for debugging and troubleshooting I've left the secrets manager policy wide open.
{
"Statement": [
{
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"secretsmanager:GetRandomPassword"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface",
"ec2:AssignPrivateIpAddresses",
"ec2:UnassignPrivateIpAddresses"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": "logs:CreateLogGroup",
"Effect": "Allow",
"Resource": "arn:aws:logs:ap-southeast-2:xxxxxxxxxxxx:*"
},
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": [
"arn:aws:logs:ap-southeast-2:xxxxxxxxxxxx:log-group:/aws/lambda/rcf-apse2-dev-onsite-rds-secret-rotator-function:*"
]
}
],
"Version": "2012-10-17"
}
The Python code is as per the following template:
https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/master/SecretsManagerRDSMySQLRotationSingleUser/lambda_function.py
The secret value is being stored in this format as required:
{
"dbClusterIdentifier": "rcf-apse2-dev-onsite",
"engine": "mysql",
"host": "rcf-apse2-dev-onsite.cluster-xxxxxxxxx.ap-southeast-2.rds.amazonaws.com",
"password": "xxxxxxx",
"username": "xxxxx"
}

Answer 1

I think I finally understand what's going on!
The default timeout of boto3 is 60 seconds, but the Lambda execution timeout was only set to 30 seconds, which is why the boto3 retry logic never had a chance to kick in and the Lambda function would keep timing out.
Obviously a crude way to fix this is to increase the Lambda timeout, but a better solution in my opinion is to add the following to the Python code and adjust the timeouts as you see fit.
from botocore.config import Config
...
config = Config(
connect_timeout=2,
read_timeout=2,
retries = {
'max_attempts': 10,
'mode': 'standard'
}
)
service_client = boto3.client('secretsmanager', config=config, endpoint_url=os.environ['SECRETS_MANAGER_ENDPOINT'])
I'm still not sure why the connection resets happen in the first place, but I suspect it's probably because AWS doesn't want hold on to open connections for too long as they cost memory and resources.
Oh the joys of AWS!