How to check AWS secretsmanager rotation is completed successfully - amazon-web-services

Question

Created a secret in AWS secretsmanager, enabled automatic rotation with lambda function.
when I trigger rotation for the first time from cli, It's not completed. This is the initial state of secret when updated secret in aws console manually.
# aws secretsmanager list-secret-version-ids --secret-id ******
{
"Versions": [
{
"VersionId": "9e82b9e2-d074-478e-83a5-baf4e578cb49",
"VersionStages": [
"AWSCURRENT"
],
"LastAccessedDate": 1592870400.0,
"CreatedDate": 1592889913.431
},
{
"VersionId": "e32ddaf8-7f21-40e2-adf8-f976b8f3f104",
"VersionStages": [
"AWSPREVIOUS"
],
"LastAccessedDate": 1592870400.0,
"CreatedDate": 1592887518.46
}
],
"ARN": "arn:aws:secretsmanager:us-east-1:***********:secret:***********",
"Name": "*******"
}
Now I triggered rotation from aws cli
aws secretsmanager rotate-secret --secret-id ******
# aws secretsmanager list-secret-version-ids --secret-id ********
{
"Versions": [
{
"VersionId": "704102f3-b36d-4529-b257-0457354d3c93",
"VersionStages": [
"AWSPENDING"
],
"CreatedDate": 1592890351.334
},
{
"VersionId": "e32ddaf8-7f21-40e2-adf8-f976b8f3f104",
"VersionStages": [
"AWSPREVIOUS"
],
"LastAccessedDate": 1592870400.0,
"CreatedDate": 1592887518.46
},
{
"VersionId": "9e82b9e2-d074-478e-83a5-baf4e578cb49",
"VersionStages": [
"AWSCURRENT"
],
"LastAccessedDate": 1592870400.0,
"CreatedDate": 1592889913.431
}
],
"ARN": "arn:aws:secretsmanager:us-east-1:**********:secret:********",
"Name": "********"
}
Cloudwatch log stopped at this createSecret: Successfully put secret for ARN arn:aws:secretsmanager:xxxxxxx.. looks like only createsecret function is called.
When I rotate the secret again, Gets this output in cli
An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.
Unable to understand what's happening. Can someone help?

Answer 1

Unfortunately there is no out-of-the-box way for that, as Secrets Manger does not have build in SNS notification nor CloudWatch Events for when rotation completes.
Thus, you have to construct a solution yourself, which can be done using SDK or CLI.
For CLI you can use describe-secret and pull secret details in a loop. In the loop, you have to look into AWSPENDING and AWSCURRENT labels for the versions.
From the docs:
If instead the AWSPENDING staging label is present but is not attached to the same version as AWSCURRENT then any later invocation of RotateSecret assumes that a previous rotation request is still in progress and returns an error.
So basically, looking at your output:
{
"VersionId": "704102f3-b36d-4529-b257-0457354d3c93",
"VersionStages": [
"AWSPENDING"
],
"CreatedDate": 1592890351.334
}
you have a version with AWSPENDING label, which is not attached to the same version as AWSCURRENT. This indicates that the rotation is in progress.
The rotation completes, when a version is in one of the two states:
The AWSPENDING and AWSCURRENT staging labels are attached to the same version of the secret, or The AWSPENDING staging label is not attached to any version of the secret.

Answer 2

Secrets Manager will publish an event via CloudTrail - 'RotationSucceeded' when there is a successful rotation.
See this for more information on how to setup a Cloudwatch alarm off that CloudTrail event - https://docs.aws.amazon.com/secretsmanager/latest/userguide/monitoring.html