Cannot create PostgreSQL with CloudFormation but works with web interface - amazon-web-services

I'm trying to create a Serverless V2 Aurora PostgreSQL cluster and an instance with CloudFormation.
It works fine when using the AWS web interface but when using CloudFormation (trough Serverless) I get
Error: CREATE_FAILED: auroraCluster (AWS::RDS::DBCluster) Resource
handler returned message: "The engine mode serverless you requested is
currently unavailable.
CF Template:
# Database
auroraCluster:
Type: AWS::RDS::DBCluster
Properties:
AutoMinorVersionUpgrade: 'true'
AvailabilityZones:
- eu-north-1a
- eu-north-1b
- eu-north-1c
DatabaseName:
publisher
DeletionProtection: !If [isProd, 'true', 'false']
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: '14.6'
auroraInstance:
Type: AWS::RDS::DBInstance
Properties:
AllowMajorVersionUpgrade: !If [isProd, 'true', 'false']
AutoMinorVersionUpgrade: 'true'
AvailabilityZone: !Sub ${AWS::Region}a
DBClusterIdentifier: !Ref auroraCluster
DBInstanceIdentifier: ${self:service}-rds-${sls:stage}
DeleteAutomatedBackups: !If [isProd, 'false', 'true']
DeletionProtection: !If [isProd, 'true', 'false']
Engine: aurora-postgresql
ManageMasterUserPassword: 'true'
MasterUsername: postgres
MasterUserSecret:
SecretArn: !Ref secretRds

Serverless aurora for postgress 14.6 is only supported for serverless v2. This requires different setup then you have. For example, you have to provide ServerlessV2ScalingConfiguration, delete EngineMode and use DBInstanceClass set to db.serverless. For example:
Resources:
auroraCluster:
Type: AWS::RDS::DBCluster
Properties:
#AutoMinorVersionUpgrade: 'true'
DatabaseName:
publisher
Engine: aurora-postgresql
EngineVersion: '14.6'
MasterUsername: "trdyd"
MasterUserPassword: "gfsdg344231"
ServerlessV2ScalingConfiguration:
MinCapacity: 1
MaxCapacity: 4
auroraInstance:
Type: 'AWS::RDS::DBInstance'
Properties:
Engine: aurora-postgresql
DBInstanceClass: db.serverless
DBClusterIdentifier: !Ref auroraCluster
Obviously you have to adjust the above example, which works, to what exactly you need, taking into account serverless v2 capabilities.

Related

Serverless error - Value of property DBSubnetGroupName must be of type String

I am attempting to geneate an Aurora serverless database for an application using Serverless to build it.
I copied a CloudFormation template I found online for this portion of my setup. I reference the below resource from my Serverless.yml file (which I've excluded because it's very large).
Serverless fails with the message:
An error occurred: AuroraCluster - Value of property DBSubnetGroupName
must be of type String.
I am confused about this because, to my observation, DBSubnetGroupName is a string (which as you can see is set to the value of 'db-subnet-group').
I am new to both Serverless and CloudFormation, and am doing my best to piece a solution together for a product I am building. Any suggestion as to how to fix this problem would be much appreciated.
Below is my Serverless file. Many thanks in advance for any assistance!
Resources:
ServerlessSecurityGroup:
DependsOn: VPC
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SecurityGroup for Serverless Functions
VpcId:
Ref: VPC
PrivateSubnetA:
DependsOn: VPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: VPC
AvailabilityZone: "eu-west-1a"
CidrBlock: "10.0.1.0/24"
PrivateSubnetB:
DependsOn: VPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: VPC
AvailabilityZone: "eu-west-1b"
CidrBlock: "10.0.64.0/19"
AuroraSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Subnet group for Aurora Database
DBSubnetGroupName: "db-subnet-group"
SubnetIds:
- Ref: PrivateSubnetA
- Ref: PrivateSubnetB
AuroraCluster:
Type: AWS::RDS::DBCluster
DeletionPolicy: ${self:custom.deletion_policy}
Properties:
DBClusterIdentifier: ${self:custom.aurora_db_name}
MasterUsername: !Sub ${self:custom.aurora_db_name}
MasterUserPassword: asdfasdfasdf223
DatabaseName: "somename"
Engine: aurora
EngineMode: serverless
DBSubnetGroupName:
- Ref: AuroraSubnetGroup
VpcSecurityGroupIds:
- Ref: ServerlessSecurityGroup
EnableHttpEndpoint: true
ScalingConfiguration:
AutoPause: true
MinCapacity: 1
MaxCapacity: 2
SecondsUntilAutoPause: 3600
Outputs:
AuroraCluster:
Value:
Ref: AuroraCluster
In your AWS::RDS::DBCluster it should be:
DBSubnetGroupName: !Ref AuroraSubnetGroup
In your example, you're passing DBSubnetGroupName as an array, not a string:
DBSubnetGroupName:
- Ref: AuroraSubnetGroup
In order to pass it as a string, you should use this notation:
DBSubnetGroupName: Ref: AuroraSubnetGroup

How can I provide a variable for `AWS::RDS::DBInstance` in cloudformation?

I am using cloudformation to provision RDS aurora to AWS and using AWS::RDS::DBCluster and AWS::RDS::DBInstance resources in the template. I have different environments, e.g. dev, uat and prod. Each environment has different number of db instances under the cluster. How can I set the number of db instances as a variable in the cloudformation template?
Below is my template for AWS::RDS::DBInstance. As you can see there are three instances in the template. It is only for production not dev. How can I use a parameter to indicate the number of instances? For example, deploy 1 instance in dev and 3 for prod.
AuroraDBFirstInstance:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: ${self:provider.postgresqlInstanceClass}
Engine: aurora-postgresql
EngineVersion: ${self:provider.postgresqlEngineVersion}
DBClusterIdentifier: !Ref AuroraDBCluster
PubliclyAccessible: ${self:provider.publiclyAccessible}
AuroraDBSecondInstance:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: ${self:provider.postgresqlInstanceClass}
Engine: aurora-postgresql
EngineVersion: ${self:provider.postgresqlEngineVersion}
DBClusterIdentifier: !Ref AuroraDBCluster
PubliclyAccessible: ${self:provider.publiclyAccessible}
AuroraDBThirdInstance:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: ${self:provider.postgresqlInstanceClass}
Engine: aurora-postgresql
EngineVersion: ${self:provider.postgresqlEngineVersion}
DBClusterIdentifier: !Ref AuroraDBCluster
PubliclyAccessible: ${self:provider.publiclyAccessible}
You can add it as a parameter and pass as you run the stack, you can even make a mappings like this:
Environment:
Type: String
AllowedValues:
- dev
- uat
- prod
Mappings:
EnvironmentToDb
dev:
Cluster: 1
uat:
Cluster: 2
prod:
Cluster: 3
Then you can reference it using:
DBClusterIdentifier: !FindInMap [EnvironmentToDb, !Ref 'Environment', Cluster]

Log Retention for Aurora Serverless in CloudFormation

I have a Aurora Cluster (Serverless - PostgreSQL) setup in CloudFormation and I want to configure the Log Retention to be around 7 days but I haven't been able to find where to set this setting.
This is my CloudFormation definition for the DBCluster:
AuroraDBCluster:
Type: AWS::RDS::DBCluster
DeletionPolicy: Delete
UpdateReplacePolicy: Delete
Properties:
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: 10.7
DatabaseName: test-db
DeletionProtection: false
ScalingConfiguration:
AutoPause: True
MaxCapacity: 8
MinCapacity: 2
SecondsUntilAutoPause: 300
VpcSecurityGroupIds:
- !Ref AuroraClusterSecurityGroup
Port: !Ref DBPort
MasterUsername:
!Join ['', ['{{resolve:secretsmanager:', !Ref AuroraMasterSecret, ':SecretString:username}}' ]]
MasterUserPassword:
!Join ['', ['{{resolve:secretsmanager:', !Ref AuroraMasterSecret, ':SecretString:password}}' ]]
DBSubnetGroupName: !Ref DBSubnetGroup
BackupRetentionPeriod: 35
DBClusterParameterGroupName: !Ref RDSDBClusterParameterGroup
StorageEncrypted: true
KmsKeyId: !Ref AuroraKMSCMK
I have created a different LogGroup like this:
AuroraClusterLogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
RetentionInDays: 7
LogGroupName: !Join ["", ["/aws/rds/cluster/", !Ref AuroraDBCluster, /postgresql]]
But when I deploy the stack, it says:
CREATE_FAILED AWS::Logs::LogGroup AuroraClusterLogGroup /aws/rds/cluster/aurora-serverless-db-ufeihfhef74465/postgresql already exists
Because (I think) the AuroraDBCluster resource creates its own LogGroup with the same name.
I have reviewed the AWS::RDS::DBCluster documentation but I have not found any references for the Log Retention.
What can I do in this case?
Thanks!
If Aurora has already created its own log group, you can't change it from CloudFormation. The only way to do this would be use custom resource in your template.
In the custom resource you could use put-retention-policy to modify the retention time of chosen log group.

Unable to create a CoudFormation RDS template which provision new instance from point in time snapshots

I've trying to write a CloudFormation to provision a new RDS instance from the point in time snapshot an existing RDS DB.
However, I came to know that you can't specify db-name when you provide a snapshot in CloudFormation template and thus it will always restore it to the original DB.
I have got this article for the same on aws blogs, though I'm looking if there is any out of the box solution for the same.
Edit 1
RDS snippet from my Cloud Formation
Resources:
MyDB:
Type: AWS::RDS::DBInstance
Properties:
DBName: Fn::If ["UseDbSnapshot", !Ref AWS:NoValue, !Ref MyDBName]
DBSecurityGroups:
- !Ref MyDbSecurityByEC2SecurityGroup
- !Ref MyDbSecurityByCIDRIPGroup
AllocatedStorage: 20
DBInstanceClass: db.m1.small
Engine: MySQL
MasterUsername: root
MasterUserPassword: password
DBSnapshotIdentifier: Fn::If ["UseDbSnapshot", !Ref DBSnapshotIdentifier, !Ref AWS::NoValue]
DeletionPolicy: Snapshot
What can I try to fix this?
You have to keep in mind, that a couple of RDS properties (such as MasterUsername) are not customizable when you want to restore from snapshot. The AWS documentation says:
If you specify the SourceDBInstanceIdentifier or DBSnapshotIdentifier property, don't specify this property. The value is inherited from the source DB instance or snapshot.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html
So just omit those parameters like this:
Parameters:
DBSnapshotIdentifier:
Description: 'Optional identifier for the DB cluster snapshot from which you want to restore'
Type: String
Default: ''
Conditions:
HasDBSnapshotIdentifier: !Not [!Equals [!Ref DBSnapshotIdentifier, '']]
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
Properties:
...
DBSnapshotIdentifier: !If [HasDBSnapshotIdentifier, !Ref DBSnapshotIdentifier, !Ref 'AWS::NoValue']
KmsKeyId: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !If [HasEncryption, !Ref KmsKey, !Ref 'AWS::NoValue']]
MasterUsername: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !Ref MasterUsername]
MasterUserPassword: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !Ref MasterPassword]
StorageEncrypted: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !If
...

Getting MasterUserPassword for AWS::RDS::DBCluster via SecretsManager in CloudFormation

When creating an AWS::RDS::DBCluster(Aurora-Serverless) in AWS CloudFormation, there is a property MasterUserPassword and its input is specified as string.
So, how to put a stack definition yaml so that RDS definition does not use a string but a random password generated by AWS::SecretsManager::Secret? Is it possible to reference Secrets-manager-generated password with !Ref, !GetAtt or any other means?
Resources:
AuroraMysqlAppCredentialSecretStore:
Type: AWS::SecretsManager::Secret
Properties:
Name: AuroraMysqlAppCredentialSecretStore
GenerateSecretString:
SecretStringTemplate: '{"username": "admin"}'
GenerateStringKey: "password"
PasswordLength: 30
ExcludeCharacters: '"#/\'
ApprovalDbCluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora
EngineMode: serverless
EngineVersion: '5.6'
DatabaseName: MyDatabaseName
MasterUsername: admin
MasterUserPassword: # HOW TO REFERENCE THE PASSWORD HERE??
DBClusterIdentifier: my-cluster-1
BackupRetentionPeriod: 35
DeletionProtection: false
ScalingConfiguration:
AutoPause: true
MaxCapacity: 8
MinCapacity: 2
SecondsUntilAutoPause: 300
DBSubnetGroupName: my-subnet-name
A final note: Docs state that MasterUserPassword should not be specified if SourceDBInstanceIdentifier or DBSnapshotIdentifier property is given, but my configuration has neither, so apparently I should specify MasterUserPassword.
You can use dynamic references in cloudformation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html
ApprovalDbCluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora
EngineMode: serverless
EngineVersion: '5.6'
DatabaseName: MyDatabaseName
MasterUsername: admin
MasterUserPassword: '{{resolve:ssm-secure:MasterPassword:10}}' #See link