Log Retention for Aurora Serverless in CloudFormation - amazon-web-services

I have a Aurora Cluster (Serverless - PostgreSQL) setup in CloudFormation and I want to configure the Log Retention to be around 7 days but I haven't been able to find where to set this setting.
This is my CloudFormation definition for the DBCluster:
AuroraDBCluster:
Type: AWS::RDS::DBCluster
DeletionPolicy: Delete
UpdateReplacePolicy: Delete
Properties:
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: 10.7
DatabaseName: test-db
DeletionProtection: false
ScalingConfiguration:
AutoPause: True
MaxCapacity: 8
MinCapacity: 2
SecondsUntilAutoPause: 300
VpcSecurityGroupIds:
- !Ref AuroraClusterSecurityGroup
Port: !Ref DBPort
MasterUsername:
!Join ['', ['{{resolve:secretsmanager:', !Ref AuroraMasterSecret, ':SecretString:username}}' ]]
MasterUserPassword:
!Join ['', ['{{resolve:secretsmanager:', !Ref AuroraMasterSecret, ':SecretString:password}}' ]]
DBSubnetGroupName: !Ref DBSubnetGroup
BackupRetentionPeriod: 35
DBClusterParameterGroupName: !Ref RDSDBClusterParameterGroup
StorageEncrypted: true
KmsKeyId: !Ref AuroraKMSCMK
I have created a different LogGroup like this:
AuroraClusterLogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
RetentionInDays: 7
LogGroupName: !Join ["", ["/aws/rds/cluster/", !Ref AuroraDBCluster, /postgresql]]
But when I deploy the stack, it says:
CREATE_FAILED AWS::Logs::LogGroup AuroraClusterLogGroup /aws/rds/cluster/aurora-serverless-db-ufeihfhef74465/postgresql already exists
Because (I think) the AuroraDBCluster resource creates its own LogGroup with the same name.
I have reviewed the AWS::RDS::DBCluster documentation but I have not found any references for the Log Retention.
What can I do in this case?
Thanks!

If Aurora has already created its own log group, you can't change it from CloudFormation. The only way to do this would be use custom resource in your template.
In the custom resource you could use put-retention-policy to modify the retention time of chosen log group.

Related

Cannot create PostgreSQL with CloudFormation but works with web interface

I'm trying to create a Serverless V2 Aurora PostgreSQL cluster and an instance with CloudFormation.
It works fine when using the AWS web interface but when using CloudFormation (trough Serverless) I get
Error: CREATE_FAILED: auroraCluster (AWS::RDS::DBCluster) Resource
handler returned message: "The engine mode serverless you requested is
currently unavailable.
CF Template:
# Database
auroraCluster:
Type: AWS::RDS::DBCluster
Properties:
AutoMinorVersionUpgrade: 'true'
AvailabilityZones:
- eu-north-1a
- eu-north-1b
- eu-north-1c
DatabaseName:
publisher
DeletionProtection: !If [isProd, 'true', 'false']
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: '14.6'
auroraInstance:
Type: AWS::RDS::DBInstance
Properties:
AllowMajorVersionUpgrade: !If [isProd, 'true', 'false']
AutoMinorVersionUpgrade: 'true'
AvailabilityZone: !Sub ${AWS::Region}a
DBClusterIdentifier: !Ref auroraCluster
DBInstanceIdentifier: ${self:service}-rds-${sls:stage}
DeleteAutomatedBackups: !If [isProd, 'false', 'true']
DeletionProtection: !If [isProd, 'true', 'false']
Engine: aurora-postgresql
ManageMasterUserPassword: 'true'
MasterUsername: postgres
MasterUserSecret:
SecretArn: !Ref secretRds
Serverless aurora for postgress 14.6 is only supported for serverless v2. This requires different setup then you have. For example, you have to provide ServerlessV2ScalingConfiguration, delete EngineMode and use DBInstanceClass set to db.serverless. For example:
Resources:
auroraCluster:
Type: AWS::RDS::DBCluster
Properties:
#AutoMinorVersionUpgrade: 'true'
DatabaseName:
publisher
Engine: aurora-postgresql
EngineVersion: '14.6'
MasterUsername: "trdyd"
MasterUserPassword: "gfsdg344231"
ServerlessV2ScalingConfiguration:
MinCapacity: 1
MaxCapacity: 4
auroraInstance:
Type: 'AWS::RDS::DBInstance'
Properties:
Engine: aurora-postgresql
DBInstanceClass: db.serverless
DBClusterIdentifier: !Ref auroraCluster
Obviously you have to adjust the above example, which works, to what exactly you need, taking into account serverless v2 capabilities.

Serverless error - Value of property DBSubnetGroupName must be of type String

I am attempting to geneate an Aurora serverless database for an application using Serverless to build it.
I copied a CloudFormation template I found online for this portion of my setup. I reference the below resource from my Serverless.yml file (which I've excluded because it's very large).
Serverless fails with the message:
An error occurred: AuroraCluster - Value of property DBSubnetGroupName
must be of type String.
I am confused about this because, to my observation, DBSubnetGroupName is a string (which as you can see is set to the value of 'db-subnet-group').
I am new to both Serverless and CloudFormation, and am doing my best to piece a solution together for a product I am building. Any suggestion as to how to fix this problem would be much appreciated.
Below is my Serverless file. Many thanks in advance for any assistance!
Resources:
ServerlessSecurityGroup:
DependsOn: VPC
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SecurityGroup for Serverless Functions
VpcId:
Ref: VPC
PrivateSubnetA:
DependsOn: VPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: VPC
AvailabilityZone: "eu-west-1a"
CidrBlock: "10.0.1.0/24"
PrivateSubnetB:
DependsOn: VPC
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: VPC
AvailabilityZone: "eu-west-1b"
CidrBlock: "10.0.64.0/19"
AuroraSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Subnet group for Aurora Database
DBSubnetGroupName: "db-subnet-group"
SubnetIds:
- Ref: PrivateSubnetA
- Ref: PrivateSubnetB
AuroraCluster:
Type: AWS::RDS::DBCluster
DeletionPolicy: ${self:custom.deletion_policy}
Properties:
DBClusterIdentifier: ${self:custom.aurora_db_name}
MasterUsername: !Sub ${self:custom.aurora_db_name}
MasterUserPassword: asdfasdfasdf223
DatabaseName: "somename"
Engine: aurora
EngineMode: serverless
DBSubnetGroupName:
- Ref: AuroraSubnetGroup
VpcSecurityGroupIds:
- Ref: ServerlessSecurityGroup
EnableHttpEndpoint: true
ScalingConfiguration:
AutoPause: true
MinCapacity: 1
MaxCapacity: 2
SecondsUntilAutoPause: 3600
Outputs:
AuroraCluster:
Value:
Ref: AuroraCluster
In your AWS::RDS::DBCluster it should be:
DBSubnetGroupName: !Ref AuroraSubnetGroup
In your example, you're passing DBSubnetGroupName as an array, not a string:
DBSubnetGroupName:
- Ref: AuroraSubnetGroup
In order to pass it as a string, you should use this notation:
DBSubnetGroupName: Ref: AuroraSubnetGroup

Unable to create a CoudFormation RDS template which provision new instance from point in time snapshots

I've trying to write a CloudFormation to provision a new RDS instance from the point in time snapshot an existing RDS DB.
However, I came to know that you can't specify db-name when you provide a snapshot in CloudFormation template and thus it will always restore it to the original DB.
I have got this article for the same on aws blogs, though I'm looking if there is any out of the box solution for the same.
Edit 1
RDS snippet from my Cloud Formation
Resources:
MyDB:
Type: AWS::RDS::DBInstance
Properties:
DBName: Fn::If ["UseDbSnapshot", !Ref AWS:NoValue, !Ref MyDBName]
DBSecurityGroups:
- !Ref MyDbSecurityByEC2SecurityGroup
- !Ref MyDbSecurityByCIDRIPGroup
AllocatedStorage: 20
DBInstanceClass: db.m1.small
Engine: MySQL
MasterUsername: root
MasterUserPassword: password
DBSnapshotIdentifier: Fn::If ["UseDbSnapshot", !Ref DBSnapshotIdentifier, !Ref AWS::NoValue]
DeletionPolicy: Snapshot
What can I try to fix this?
You have to keep in mind, that a couple of RDS properties (such as MasterUsername) are not customizable when you want to restore from snapshot. The AWS documentation says:
If you specify the SourceDBInstanceIdentifier or DBSnapshotIdentifier property, don't specify this property. The value is inherited from the source DB instance or snapshot.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html
So just omit those parameters like this:
Parameters:
DBSnapshotIdentifier:
Description: 'Optional identifier for the DB cluster snapshot from which you want to restore'
Type: String
Default: ''
Conditions:
HasDBSnapshotIdentifier: !Not [!Equals [!Ref DBSnapshotIdentifier, '']]
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
Properties:
...
DBSnapshotIdentifier: !If [HasDBSnapshotIdentifier, !Ref DBSnapshotIdentifier, !Ref 'AWS::NoValue']
KmsKeyId: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !If [HasEncryption, !Ref KmsKey, !Ref 'AWS::NoValue']]
MasterUsername: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !Ref MasterUsername]
MasterUserPassword: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !Ref MasterPassword]
StorageEncrypted: !If [HasDBSnapshotIdentifier, !Ref 'AWS::NoValue', !If
...

CodeDeploy does not deploy to new ASG group because of CloudFormation resources creation order

I am trying to develop an entire AWS architecture by usin CloudFormation only, however I am having some issues with the integration of CodeDeploy with CloudFormation and AutoScaling Group.
The problem is that, since I need to associate the CodeDeploy DeploymentGroup to an AutoScaling Group in order for the auto-deployment to work, CloudFormation recognizes the group as being required before creating the deployment group.
What happens is that the ASG gets created, instances start to spin up BEFORE the deployment group has been created, which means that these instances will never get deployed. I tried to think of a Lambda function to forcefully deploy these instances, however the problems persists because the CodeDeploy Deployment Group will still not be available yet most likely, or if it was, it's not reliable.
This problem only occurs when the stack is created for the first time.
This is my CloudFormation template:
[...]
UpdateApiAutoscalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName:
Fn::Join:
- ''
- - !ImportValue UpdateApiCodeDeployApplication
- -autoscaling-group
- !Ref Environment
DesiredCapacity: !Ref MinimumApiAmount
HealthCheckGracePeriod: 30
HealthCheckType: ELB
LaunchConfigurationName: !Ref UpdateApiAutoscalingLaunchConfiguration
TargetGroupARNs:
- !Ref UpdateApiTargetGroup
MaxSize: !Ref MaximumApiAmount
MinSize: !Ref MinimumApiAmount
VPCZoneIdentifier:
- Fn::Select:
- 0
- !Split
- ","
- Fn::ImportValue:
!Sub "PrivateSubnets-${Environment}"
Tags:
- Key: Environment
Value: !Ref Environment
PropagateAtLaunch: true
- Key: CompanySshAccess
Value: 1
PropagateAtLaunch: true
- Key: Application
Value: update-api
PropagateAtLaunch: true
# Defines how the Update API servers should be provisioned in the scaling group.
UpdateApiAutoscalingLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
AssociatePublicIpAddress: false
IamInstanceProfile: !Ref UpdateApiInstanceRole
ImageId: !FindInMap [Api, Image, !Ref Environment]
InstanceType: !FindInMap [Api, InstanceType, !Ref Environment]
LaunchConfigurationName: !Sub 'update-api-launchconfig-${Environment}'
SecurityGroups:
- Fn::ImportValue: !Sub 'InternalBastionSecurityGroupId-${Environment}'
- !GetAtt LoadBalancerProtectedSecurityGroup.GroupId
UpdateApiCodeDeploymentGroup:
Type: AWS::CodeDeploy::DeploymentGroup
Properties:
DeploymentGroupName: !Ref Environment
DeploymentConfigName: "atleast-one-instance-online"
ServiceRoleArn: !Ref CodeDeployServiceRoleArn # TODO: create CodeDeployServiceRole using CloudFormation
ApplicationName: !ImportValue UpdateApiCodeDeployApplication
LoadBalancerInfo:
ElbInfoList:
- Name: !GetAtt UpdateApiLoadBalancer.LoadBalancerName
DeploymentStyle:
DeploymentOption: !FindInMap [Api, DeploymentStyleOption, !Ref Environment]
DeploymentType: !FindInMap [Api, DeploymentStyleType, !Ref Environment]
AutoScalingGroups:
- !Ref UpdateApiAutoscalingGroup
[...]

AWS CloudFormation RDS Instance fails to create - cannot find DBCluster

I am playing around with cloudformation and running into an issue with DBSubnetGroup that I cannot get around. My goal is to build a simple setup:
a VPC with two subnets
an RDS database subnet group on those subnets
an RDS database cluster in that db subnet
a single RDS instance in that cluster
In cloudformation, I keep getting an error:
Could not find DB Cluster: MyRDSClusterId (Service: AmazonRDS; Status Code:
404; Error Code: DBClusterNotFoundFault; Request ID: ...)
Everything looks right to me, and cloudformation says my DBCluster was created correctly. What am I doing wrong here? Any insights into what I've done wrong would be greatly appreciated.
Here is my cloudformation template:
AWSTemplateFormatVersion: "2010-09-09"
Description: Stack with DBSubnetGroup, DBCluster, and one DBInstance
Resources:
MyAppVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 192.168.0.0/16
EnableDnsSupport: true
EnableDnsHostnames: true
InstanceTenancy: default
MyAppRDSSubnetA:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: us-east-1a
VpcId: !Ref MyAppVPC
CidrBlock: 192.168.0.0/24
MapPublicIpOnLaunch: true
MyAppRDSSubnetB:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: us-east-1b
VpcId: !Ref MyAppVPC
CidrBlock: 192.168.1.0/24
MapPublicIpOnLaunch: true
MyDBSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: My App DBSubnetGroup for RDS
SubnetIds:
- !Ref MyAppRDSSubnetA
- !Ref MyAppRDSSubnetB
MyRDSCluster:
Type: AWS::RDS::DBCluster
Properties:
BackupRetentionPeriod: 1
DatabaseName: MyDB
DBClusterIdentifier: MyRDSClusterId
DBSubnetGroupName: !Ref MyDBSubnetGroup
Engine: aurora
MasterUsername: exampleUsername
MasterUserPassword: examplePassword
MyRDSInstance:
Type: AWS::RDS::DBInstance
Properties:
DBClusterIdentifier: !Ref MyRDSCluster
DBInstanceClass: db.t2.small
DBSubnetGroupName: !Ref MyDBSubnetGroup
Engine: aurora
Some charcter in your DBClusterIdentifier name is in uppercase.
What cloudformation does is, it automatically convert all uppercase character to lowercase.
Now when you try to attach DBInstances under the DBCluster it is not able to find with the DBClusterIdentifier you have mentioned as it contains some uppercase.
MyRDSCluster:
Type: AWS::RDS::DBCluster
Properties:
BackupRetentionPeriod: 1
DatabaseName: MyDB
DBClusterIdentifier: MyRDSClusterId <- here it converts all string to lowercase
DBSubnetGroupName: !Ref MyDBSubnetGroup
Engine: aurora
MasterUsername: exampleUsername
MasterUserPassword: examplePassword
MyRDSInstance:
Type: AWS::RDS::DBInstance
Properties:
DBClusterIdentifier: !Ref MyRDSCluster <- here it does not, so mismatch in name
DBInstanceClass: db.t2.small
DBSubnetGroupName: !Ref MyDBSubnetGroup
Engine: aurora
Solution is : give DBClusterIdentifier all in lowercase.
I hope you got your Answer :)
I removed the "DBClusterIdentifier" property from my cluster definition and suddenly everything worked. Hope this helps someone else one day.
Now it looks like:
MyRDSCluster:
Type: AWS::RDS::DBCluster
Properties:
BackupRetentionPeriod: 1
DatabaseName: My
DBSubnetGroupName: !Ref MyDBSubnetGroup
Engine: aurora
MasterUsername: exampleUsername
MasterUserPassword: examplePassword
Hope this helps someone else one day.
I have to say, I'm not sure I totally understand why this fixed the issue, and an explanation would be helpful (i have been searching for a while with no luck). Why can't I specify my own DBClusterIdentifier in this template?