cloudformation template for migrating data from ec2 to rds - amazon-web-services

Hi I am trying to create a cloudformation template to migrate data from postgres in ec2 to rds.I got this template from this link. https://hands-on.cloud/aws-cloudformation-how-to-create-dms-infrastructure-for-relational-db-migration/.
I modified it but i got a error
"Template format error: Every Parameters member must be an object".
In the link he created vpc and security group.i already have vpc and security to configure.
Can u anybod please
Parameters:
ReplicationInstanceAllocatedStorage:
Description: >
T he amount of storage (in gigabytes) to be initially allocated
for the replication instance.
Type: Number
Default: 256
ReplicationInstanceClass:
Description: >
The compute and memory capacity of the replication instance as specified
by the replication instance class.
Valid Values: dms.t2.micro | dms.t2.small | dms.t2.medium | dms.t2.large |
dms.c4.large | dms.c4.xlarge | dms.c4.2xlarge | dms.c4.4xlarge
Type: String
Default: dms.r5.xlarge
SrcDbName: postgres
Type: String
SrcDbEngine: postgres
Type: String
SrcDbServerName: postgres
Type: String
SrcDbPort: 5432
Type: Number
SrcDbUsername: postgres
Type: String
SrcDbPassword: postgres
Type: String
Resources:
ReplicationInstance:
Type: AWS::DMS::ReplicationInstance
Properties:
AllocatedStorage: !Ref ReplicationInstanceAllocatedStorage
AllowMajorVersionUpgrade: false
AutoMinorVersionUpgrade: false
MultiAZ: false
PubliclyAccessible: false
ReplicationInstanceClass: !Sub '${ReplicationInstanceClass}'
ReplicationInstanceIdentifier: !Sub '${AWS::StackName}-replication-instance'
DmsEndpointSource:
Type: AWS::DMS::Endpoint
Properties:
DatabaseName: !Ref SrcDbName
EndpointType: 'source'
EngineName: !Ref SrcDbEngine
ServerName: !Ref SrcDbServerName
Port: !Ref SrcDbPort
Username: !Ref SrcDbUsername
Password: !Ref SrcDbPassword

There's a couple of errors in your template.
SrcDbName: postgres
Type: String
and all others in that form are not correct. They should be
SrcDbName:
Description: Source Database Name
Type: String
Default: postgres
for example.
You'll also need your replication destination endpoint and a replication task (IIRC).

Related

Review needed to correct this ElastiCache CloudFormation Template

This template is used to create a ElastiCache- Redis cluster.
Its showing me errors like - 1 validation error detected: Value '[AWS::ElastiCache::CacheCluster, AWS::EC2::SecurityGroup::Id]' at 'typeNameList' failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 204, Member must have length greater than or equal to 10, Member must satisfy regular expression pattern: [A-Za-z0-9]{2,64}::[A-Za-z0-9]{2,64}::[A-Za-z0-9]{2,64}(::MODULE){0,1}].
Wanted to know if the parameters are declared rightly or not.
AWSTemplateFormatVersion: 2010-09-09
Description: Create ElastiCache and related resources
Parameters:
VPC:
Description: VPC
Type: AWS::EC2::VPC::Id
Subnet:
Description: Subnet
Type: AWS::EC2::Subnet::Id
ClusterName:
Description: Custom name of the cluster. Auto generated if you
don't supply your own.
Type: String
CacheNodeType:
Description: Cache node instance class, e.g. cache.t2.micro.
Type: String
Default: cache.t2.micro
ConstraintDescription: Node instance class not supported
AllowedValues:
- cache.t2.micro
- cache.t2.small
- cache.t2.medium
- cache.m4.large
- cache.m4.xlarge
- cache.m4.2xlarge
- cache.m4.4xlarge
- cache.m4.10xlarge
- cache.r4.large
- cache.r4.xlarge
- cache.r4.2xlarge
- cache.r4.4xlarge
- cache.r4.8xlarge
- cache.r4.16xlarge
CacheEngine:
Description: The underlying cache engine, either Redis or
Memcached
Type: String
Default: redis
ConstraintDescription: Node instance class not supported
AllowedValues:
- redis
- memcached
CacheNodeCount:
Description: Number of nodes in the cluster. Only used with
memcached engine, for redis this value will be set to 1.
Type: Number
MinValue: 1
MaxValue: 15
ConstraintDescription: Node count must be between 1 and 15
Default: 1
AutoMinorVersionUpgrade:
Description: Whether or not minor version upgrades to the cache
engine should be applied automatically during the maintenance
window.
Type: String
Default: true
AllowedValues:
- true
- false
Resources:
SecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
Properties:
GroupDescription: ElastiCache Security Group
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 6379
ToPort: 6379
- IpProtocol: tcp
FromPort: 11211
ToPort: 11211
Tags:
-
Key: Name
Value: "App-SG"
ElastiCacheCluster:
Type: AWS::ElastiCache::CacheCluster
Properties:
AutoMinorVersionUpgrade: !Ref AutoMinorVersionUpgrade
Engine: !Ref CacheEngine
CacheNodeType: !Ref CacheNodeType
ClusterName : !Ref ClusterName
NumCacheNodes: !Ref CacheNodeCount
CacheSubnetGroupName: !Ref Subnet
VpcSecurityGroupIds: !Ref SecurityGroup
Tags:
- Key: Name
Value: ElastiCache-Redis
Also it will be helpful to review the entire template to avoid more errors.Main issues seem to appear from resource section.
This question is old and I'm sure OP found the resolution but for anyone else who may come across this question - I believe the issue is that ElastiCacheCluster.VpcSecurityGroupIds must be a list, whereas !Ref SecurityGroup will simply return the ID of the security group as a string.
In addition, since both resources are created by the same template, we should make sure that the security group will be created first, otherwise referencing it will cause an error when the Elasticache cluster is created.
Here's the ElastiCache::CacheCluster resource with a list as the value of VpcSecurityGroupIds and a dependency on the security group.
ElastiCacheCluster:
Type: AWS::ElastiCache::CacheCluster
Properties:
AutoMinorVersionUpgrade: !Ref AutoMinorVersionUpgrade
Engine: !Ref CacheEngine
CacheNodeType: !Ref CacheNodeType
ClusterName : !Ref ClusterName
NumCacheNodes: !Ref CacheNodeCount
CacheSubnetGroupName: !Ref Subnet
VpcSecurityGroupIds:
- !Ref SecurityGroup
Tags:
- Key: Name
Value: ElastiCache-Redis
DependsOn: SecurityGroup
I had the same error messages and landed here.
Turns out that in my case, the problem was an incorrect type.
I had AWS::IAM:Role instead of AWS::IAM::Role (1 : vs 2)

Install Aurora PostgresSql using cloudformation

Unable to create aurora postgresSql database using cloudformat yaml template.
Please help me on this.
From AWS::RDS::DBCluster - AWS CloudFormation:
The following example creates an Amazon Aurora PostgreSQL DB cluster that exports logs to Amazon CloudWatch Logs. For more information about exporting Aurora DB cluster logs to Amazon CloudWatch Logs.
AWSTemplateFormatVersion: 2010-09-09
Description: >-
AWS CloudFormation Sample Template for sending Aurora DB cluster logs to
CloudWatch Logs: Sample template showing how to create an Aurora PostgreSQL DB
cluster that exports logs to CloudWatch Logs. **WARNING** This template
enables log exports to CloudWatch Logs. You will be billed for the AWS
resources used if you create a stack from this template.
Parameters:
DBUsername:
NoEcho: 'true'
Description: Username for PostgreSQL database access
Type: String
MinLength: '1'
MaxLength: '16'
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
ConstraintDescription: must begin with a letter and contain only alphanumeric characters.
DBPassword:
NoEcho: 'true'
Description: Password for PostgreSQL database access
Type: String
MinLength: '8'
MaxLength: '41'
AllowedPattern: '[a-zA-Z0-9]*'
ConstraintDescription: must contain only alphanumeric characters.
Resources:
RDSCluster:
Type: 'AWS::RDS::DBCluster'
Properties:
MasterUsername: !Ref DBUsername
MasterUserPassword: !Ref DBPassword
DBClusterIdentifier: aurora-postgresql-cluster
Engine: aurora-postgresql
EngineVersion: '10.7'
DBClusterParameterGroupName: default.aurora-postgresql10
EnableCloudwatchLogsExports:
- postgresql
RDSDBInstance1:
Type: 'AWS::RDS::DBInstance'
Properties:
DBInstanceIdentifier: aurora-postgresql-instance1
Engine: aurora-postgresql
DBClusterIdentifier: !Ref RDSCluster
PubliclyAccessible: 'true'
DBInstanceClass: db.r4.large
RDSDBInstance2:
Type: 'AWS::RDS::DBInstance'
Properties:
DBInstanceIdentifier: aurora-postgresql-instance2
Engine: aurora-postgresql
DBClusterIdentifier: !Ref RDSCluster
PubliclyAccessible: 'true'
DBInstanceClass: db.r4.large

CloudFormation templates for Global Aurora Database

I am trying to write Cloudformation template to get a aws Global Aurora Database. However I am not able to figure out where and how to add the Global database identifier. Can someone help Cloudformation snippet?
below is my code:
Description: RDS Aurora MySQL cluster.
Parameters:
DatabaseName:
Default: "testglobalaurora"
Description: The database name
Type: String
DatabaseInstanceType:
Default: db.r4.large
AllowedValues:
- db.r4.large
- db.r4.xlarge
- db.r4.2xlarge
- db.r4.4xlarge
- db.r4.8xlarge
- db.r4.16xlarge
Description: "The instance type to use for the database."
Type: String
DatabasePassword:
Default: "testglobalaurora"
AllowedPattern: "[a-zA-Z0-9]+"
ConstraintDescription: must contain only alphanumeric characters. Must have length 8-41.
Description: The database admin account password.
MaxLength: '41'
MinLength: '8'
NoEcho: 'true'
Type: String
DatabaseUsername:
Default: "testglobalaurora"
AllowedPattern: "[a-zA-Z0-9]+"
ConstraintDescription: must contain only alphanumeric characters. Must have length 1-16
Description: The database admin account user name.
MaxLength: '16'
MinLength: '1'
Type: String
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Database Configuration
Parameters:
- DatabaseInstanceType
- DatabaseName
- DatabaseUsername
- DatabasePassword
ParameterLabels:
DatabaseName:
default: Database name
DatabaseInstanceType:
default: Database Instance Type
DatabasePassword:
default: Database Password
DatabaseUsername:
default: Database Username
Resources:
ParameterGroup:
Type: "AWS::RDS::DBParameterGroup"
Properties:
Description: testglobalaurora DB parameter group
Family: aurora5.6
Parameters:
max_connections: 300
DatabaseCluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora
EngineMode: global
MasterUsername:
Ref: DatabaseUsername
MasterUserPassword:
Ref: DatabasePassword
BackupRetentionPeriod: 35
PreferredBackupWindow: 02:00-03:00
PreferredMaintenanceWindow: mon:03:00-mon:04:00
VpcSecurityGroupIds:
- Ref: DatabaseSecurityGroup
DatabaseInstance:
Type: AWS::RDS::DBInstance
Properties:
Engine: aurora
EngineVersion : 5.6.10a
DBClusterIdentifier:
Ref: DatabaseCluster
DBInstanceClass:
Ref: DatabaseInstanceType
DBParameterGroupName: !Ref ParameterGroup
PubliclyAccessible: "true"
DBInstanceIdentifier: !Ref DatabaseName
DatabaseSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: vpc-55378f2f
GroupDescription: Access to database
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
FromPort: 3306
ToPort: 3306
IpProtocol: tcp
Tags:
- Key: Name
Value: !Sub ${DatabaseName}-security-group
Outputs:
DatabaseEndpoint:
Description: The database endpoint
Value: !GetAtt DatabaseCluster.Endpoint.Address
DatabasePort:
Description: The database port
Value: !GetAtt DatabaseCluster.Endpoint.Port
My output
"
global-database-1-cluster-1 Regional Aurora MySQL 5.6.10a
global-database-1-instance-1 Writer Aurora MySQL 5.6.10a
"
Actual ouput
"
test-it Global Aurora MySQL 5.6.10a
global-database-1-cluster-1 Primary Aurora MySQL 5.6.10a
global-database-1-instance-1 Writer Aurora MySQL 5.6.10a
"
I recently ran across the need to create a global RDS with Cloudformation. Here is a minimal Cloudformation that got me started.
AWSTemplateFormatVersion: "2010-09-09"
Description: Global RDS database stack
Parameters:
DatabaseInstanceType:
Default: db.r4.large
AllowedValues:
- db.r4.large
- db.r4. # add the other r4 instances
Description: "The instance type to use for the database."
Type: String
DatabasePassword:
Default: SomePassword1
AllowedPattern: "[a-zA-Z0-9]+"
ConstraintDescription: must contain only alphanumeric characters. Must have length 8-41.
Description: The database admin account password.
MaxLength: '41'
MinLength: '8'
NoEcho: 'true'
Type: String
DatabaseUsername:
Default: globaladmin
ConstraintDescription: must contain only alphanumeric characters. Must have length 1-16
Description: The database admin account user name.
MaxLength: '16'
MinLength: '1'
Type: String
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Database Configuration
Parameters:
- DatabaseInstanceType
- DatabaseName
- DatabaseUsername
- DatabasePassword
ParameterLabels:
DatabaseName:
default: Database name
DatabaseInstanceType:
default: Database Instance Type
DatabasePassword:
default: Database Password
DatabaseUsername:
default: Database Username
Resources:
GlobalDbCluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora
EngineMode: global
EngineVersion: 5.6.10a
MasterUsername: !Ref DatabaseUsername
MasterUserPassword: !Ref DatabasePassword
DBClusterParameterGroupName: !Ref GlobalDbParamGroup
GlobalDbParamGroup:
Type: AWS::RDS::DBClusterParameterGroup
Properties:
Description: "parameter group for the global database"
Family: aurora5.6
Parameters:
character_set_database: utf32
InstanceOne:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !Ref DatabaseInstanceType
DBClusterIdentifier: !Ref GlobalDbCluster
Engine: aurora
InstanceTwo:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !Ref DatabaseInstanceType
DBClusterIdentifier: !Ref GlobalDbCluster
Engine: aurora
You need to create a Global Cluster with an identifier, and use that identifier in your DB Cluster. That portion is missing in your CFN template.
Something like:
GlobalCluster:
Type: AWS::RDS::GlobalCluster
Properties:
Engine: aurora
EngineVersion: 5.6.10a
Region: us-east-1
and then use it in your DatabaseCluster properties using GlobalClusterIdentifier: <id>
However, looking at the official docs for the CFN types [1] for RDS, it does not list GlobalCluster. So either its just not documented or this Resource type has not been registered with Cloudformation. If the latter is the case, then you may want to open a support case and put in a feature request.
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_RDS.html

CloudFormation fails to find RDS Subnet Group because of lowercase

I created my RDS Subnet Group via CloudFormation referencing a parameter ProjectName
DB:
Type: AWS::RDS::DBInstance
Properties:
DBSubnetGroupName: !Ref RDSSubnetGroup
Problem now is CloudFormation says it cannot find my subnet group:
DB subnet group 'AbcDef' does not exist because its actually abcdef ... how can I resolve this?
I tried looking for a toLower function but seems like theres none?
The other option appears to be recreate the stack?
Unfortunately everything you do in CloudFormation templates is case-sensitive including property names and parameter values. You may have to recreate the stack.
As you correctly pointed out, there is no Fn::ToLower function. If you really want to achieve what you are trying to, the only way to do it as of now is create Lambda backed custom resource which basically will convert your string to lower case and return it but it is not worth doing it as there are plenty of challenges you will come across when dealing with custom resources.
I have also found that DB Subnet Groups have their name forcibly changed to lowercase when viewed in the RDS console. Very unusual behavior.
However, I have created them in CloudFormation and it has not caused the error you describe. Here are the bits from my CloudFormation template:
###########
# DB Subnet Group
###########
DBSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Lab DB Subnet Group
DBSubnetGroupName: Lab DB Subnet Group
SubnetIds:
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
Tags:
-
Key: Name
Value: DBSubnetGroup
###########
# RDS Database
###########
RDSDatabase:
Type: AWS::RDS::DBInstance
Properties:
DBName: inventory
DBInstanceIdentifier: inventory-db
AllocatedStorage: 5
DBInstanceClass: db.t2.micro
Engine: MySQL
MasterUsername: master
MasterUserPassword: lab-password
MultiAZ: false
DBSubnetGroupName: !Ref DBSubnetGroup
VPCSecurityGroups:
- !Ref DBSecurityGroup
Tags:
-
Key: Name
Value: inventory-db
I would suggest rewriting the function_name and the name of the DBSubnetGroup to dbsubnetgroup
This will fix the issue I suppose.
Had the same issue, tried all possible , the only way to fix , was to create a new DB subnet group with the name lower :
rdssubnetgrouplower:
Type: "AWS::RDS::DBSubnetGroup"
Properties:
DBSubnetGroupDescription: "Private subnet group to keep the cluster private"
DBSubnetGroupName: rdssubnetgrouplower
SubnetIds:
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
Tags:
-
Key: Name
Value: rdssubnetgrouplower
and then in RDS definition used it :
MySQLABC:
Type: "AWS::RDS::DBCluster"
Properties:
DBSubnetGroupName: !Ref rdssubnetgrouplower
...
...
this worked and had the cluster up. in TF there is the lower function :)

Passing Parameter Files for CloudFormation Does Not Work with YML

Solution: Don't use yml.
I'm trying to pass parameters that hold the db name and the db password along with my aws cloudformation stack-create command. I get back an error that doesn't make a whole lot of sense.
Value of property MasterUsername must be of type String
Now, one thing to know is that I have console access so I can literally see the password and username come back in the console as what was set. They both look like strings to me.
YAML Version
I'm defining the stack in a template like this:
Parameters:
DBUser:
Description: db user
Type: String
MinLength: '4'
MaxLength: '16'
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
ConstraintDescription: some description
DBPass:
Description: db password
Type: String
MinLength: '8'
MaxLength: '36'
AllowedPattern: '[a-zA-Z0-9]*'
ConstraintDescription: must contain only alphanumeric characters
Resources:
AppNode:
(...)
DatabaseInstance:
Type: AWS::RDS::DBInstance
Properties:
DBName: app
DBInstanceClass: db.t2.micro
Engine: postgres
EngineVersion: 10
MasterUsername:
- !Ref DBUser
MasterUserPassword:
- !Ref DBPass
DatabaseSG:
Type: AWS::RDS::DBSecurityGroup
Properties:
GroupDescription: Security Group for RDS public access
DBSecurityGroupIngress:
- CIDRIP: 0.0.0.0/0
Inside of my .env file I have this:
[
{
"ParameterKey": "DBUser",
"ParameterValue": "root"
},
{
"ParameterKey": "DBPass",
"ParameterValue": "mydbpassword"
}
]
I run this command:
aws cloudformation create-stack --stack-name app --template-body file://$PWD/stack.json --region us-west-2 --parameters file://$PWD/.env.json
for the file type.
So the first thing that comes to mind is that maybe it's getting those values as blank. That can't be the case though because if I don't pass a template or parameters at all it, throws a different error.
An error occurred (ValidationError) when calling the CreateStack operation: Parameters: [DBUser, DBPass] must have values
Did AWS redefine what a string is or what is going on here?
Correct your template as below.
Resources:
AppNode:
(...)
DatabaseInstance:
Type: AWS::RDS::DBInstance
Properties:
DBName: app
DBInstanceClass: db.t2.micro
Engine: postgres
EngineVersion: 10
MasterUsername:
!Ref DBUser
MasterUserPassword:
!Ref DBPass
When you use - !Ref DBUser, it signifies a LIST in yaml, not String.
MasterUserPassword and MasterUserPassword are both of type String.
AWS supports both .yml and .yaml extensions. So there is no issue with you extensions.
Sorry to say that external parameters are not supported via file for yaml files.
You can pass them in command line as a workaround.
Reference:
https://github.com/aws/aws-cli/issues/2275
There is still in work in progress.
Hope it helps.
EDIT1:
MasterUsername:
- ${DBUser}
MasterUserPassword:
- ${DBPass}
Looks like your YAML syntax is not correct.
Reference:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-general.html
MasterUsername must be of type String
Here you have a list (array), not string:
MasterUsername:
- !Ref DBUser
Use on the same line instead:
MasterUsername: !Ref DBUser