How do you set the Redis ReplicationGroup ID when using a CloudFormation template? All the options in the docs show no way to do that and the CLI you can do this easily. My end goal is to have a Redis replication group with 3 cluster members but I want to choose the name rather than AWS set a unique name for me.
Here's a snippet of my template:
"Resources": {
"mqpReplicationGroup": {
"Type": "AWS::ElastiCache::ReplicationGroup",
"Properties": {
"CacheNodeType": {
"Ref": "CacheNodeType"
},
"CacheSubnetGroupName": {
"Ref": "CacheSubnets"
},
"ReplicationGroupDescription": "Redis Replication Group",
"Engine": "redis",
"EngineVersion": {
"Ref": "RedisVersion"
},
"NumCacheClusters": {
"Ref": "NumberOfCacheNodes"
},
"AutoMinorVersionUpgrade": "true",
"AutomaticFailoverEnabled": "true",
"PreferredMaintenanceWindow": "sat:09:25-sat:22:30",
"SnapshotRetentionLimit": "4",
"SnapshotWindow": "00:05-05:30",
"NotificationTopicArn": {
"Fn::Join" :[":",["arn:aws:sns",{ "Ref" : "AWS::Region" },{ "Ref" : "AWS::AccountId" },"service-aws"]]
},
"SecurityGroupIds": [
{
"Fn::Join": [
",",
{
"Ref": "VpcSecurityGroupIds"
}
]
}
]
}
},
"CacheSubnets": {
"Type": "AWS::ElastiCache::SubnetGroup",
"Properties": {
"Description": "mqp-cache-subnet",
"SubnetIds": {
"Ref": "SubnetIds"
}
}
}
}
As of 2017, this is now possible using ReplicationGroupId property.
Since it is optional, AWS CloudFormation will still generate an unique physical ID when not specified.
Restrictions for Name Type:
Must contain from 1 to 20 alphanumeric characters or hyphens.
First character must be a letter.
Cannot end with a hyphen or contain two consecutive hyphens.
This cannot currently be done with CloudFormation.
Related
I am trying to setup the AWS environment for my SpringBoot application which uses Postgres.
I decided to use CloudFormation to configure all the AWS components needed for my application.
Following is the cloud formation template app.json:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"S3BucketName": {
"Description": "S3 BucketName",
"Type": "String"
},
"S3FileName": {
"Description": "Name of the jar/war file",
"Type": "String"
},
"DBName":{
"Default":"mytododb",
"Description":"The database name",
"Type":"String",
"MinLength":"1",
"MaxLength":"64",
"AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription":"must begin with a letter and contain only alphanumeric characters."
},
"DBUser":{
"Description":"The database admin account username",
"Type":"String",
"MinLength":"1",
"MaxLength":"16",
"AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription":"must begin with a letter and contain only alphanumeric characters."
},
"DBPassword":{
"NoEcho":"true",
"Description":"The database admin account password",
"Type":"String",
"MinLength":"8",
"MaxLength":"41",
"AllowedPattern":"[a-zA-Z0-9]*",
"ConstraintDescription":"must contain only alphanumeric characters."
},
"DBAllocatedStorage":{
"Default":"5",
"Description":"The size of the database (Gb)",
"Type":"Number",
"MinValue":"5",
"MaxValue":"1024",
"ConstraintDescription":"must be between 5 and 1024Gb."
},
"DBInstanceClass":{
"Default":"db.t2.micro",
"Description":"The database instance type",
"Type":"String",
"ConstraintDescription":"must select a valid database instance type."
},
"MultiAZDatabase":{
"Default":"false",
"Description":"Create a multi-AZ RDS database instance",
"Type":"String",
"AllowedValues":[
"true",
"false"
],
"ConstraintDescription":"must be either true or false."
}
},
"Resources": {
"SpringBootApplication": {
"Type": "AWS::ElasticBeanstalk::Application",
"Properties": {
"Description":"Spring boot and elastic beanstalk"
}
},
"SpringBootApplicationVersion": {
"Type": "AWS::ElasticBeanstalk::ApplicationVersion",
"Properties": {
"ApplicationName":{"Ref":"SpringBootApplication"},
"SourceBundle": {
"S3Bucket": {
"Ref": "S3BucketName"
},
"S3Key": {
"Ref": "S3FileName"
}
}
}
},
"SpringBootBeanStalkConfigurationTemplate": {
"Type": "AWS::ElasticBeanstalk::ConfigurationTemplate",
"Properties": {
"ApplicationName": {"Ref":"SpringBootApplication"},
"Description":"A display of speed boot application",
"OptionSettings": [
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBAllocatedStorage",
"Value": {
"Ref": "DBAllocatedStorage"
}
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBDeletionPolicy",
"Value": "Delete"
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBEngine",
"Value": "postgres"
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBEngineVersion",
"Value": "10.4"
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBInstanceClass",
"Value": { "Ref": "DBInstanceClass" }
},
{
"OptionName": "DBPassword",
"Namespace": "aws:rds:dbinstance",
"Value": { "Ref": "DBPassword" }
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBUser",
"Value": { "Ref": "DBUser" }
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "MultiAZDatabase",
"Value": { "Ref": "MultiAZDatabase" }
}
],
"SolutionStackName": "64bit Amazon Linux 2018.03 v2.9.2 running Java 8"
}
},
"SpringBootBeanstalkEnvironment": {
"Type": "AWS::ElasticBeanstalk::Environment",
"Properties": {
"ApplicationName": {"Ref":"SpringBootApplication"},
"EnvironmentName":"TodoAppEnvironment",
"TemplateName": {"Ref":"SpringBootBeanStalkConfigurationTemplate"},
"VersionLabel": {"Ref": "SpringBootApplicationVersion"}
}
}
},
"Outputs": {
"DevURL": {
"Description": "The URL of the DEV Elastic Beanstalk environment",
"Value": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"SpringBootBeanstalkEnvironment",
"EndpointURL"
]
}
]
]
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-EndpointURL"
}
}
}
}
}
And the parameters file parameters.json is:
[
{
"ParameterKey": "DBUser",
"ParameterValue": "myuser"
},
{
"ParameterKey": "DBPassword",
"ParameterValue": "secret"
},
{
"ParameterKey": "S3BucketName",
"ParameterValue": "todoapp"
},
{
"ParameterKey": "S3FileName",
"ParameterValue": "todo-api-spring-boot-0.0.1.jar"
}
]
I am creating the CloudFormation stack using
aws cloudformation create-stack --template-body file://app.json --parameters file://parameters.json --stack-name=todo-stack
Problem:
The stack is getting created, however the database (RDS) instance is not getting created and when I see in ElasticBeanstalk configuration for the app in Management Console there is no Database configuration associated with the application.
My expectation/assumption is by configuring various aws:rds:dbinstance properties in OptionSettings an RDS instance will be provisioned and associates with ElasticBeanstalk application.
Is my understanding wrong or am I missing any other setting?
PS:
The idea is to use integrated RDS so that I can use RDS_HOST, RDS_PORT etc properties in my application to connect to database.
I am able to configure a separate RDS Resource and connect to it by specifying the connection parameters. But I am expecting aws:rds:dbinstance properties in OptionSettings would create the RDS and associate it to Elasticbeanstalk application. If that is not the case then what is the purpose of configuring those parameters in OptionSettings?
Ref:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html#command-options-general-rdsdbinstance
https://github.com/metabase/metabase/blob/master/bin/aws-eb-docker/cloudformation-elasticbeanstalk.json.template
RDS creation is not done due to the Namespace Approach. Basically Namespace is used to override the default properties of the resources which you have created.
So the solution is to include the RDS creation template before your "SpringBootBeanStalkConfigurationTemplate"
"Resources": {
"RDS Creation": {
{
"Type" : "AWS::RDS::DBInstance",
"Properties" : {
"AllocatedStorage" : String,
"AllowMajorVersionUpgrade" : Boolean,
"AssociatedRoles" : [ DBInstanceRole, ... ],
"AutoMinorVersionUpgrade" : Boolean,
"AvailabilityZone" : String,
"BackupRetentionPeriod" : Integer,
"CharacterSetName" : String,
"CopyTagsToSnapshot" : Boolean,
"DBClusterIdentifier" : String,
"DBInstanceClass" : String,
"DBInstanceIdentifier" : String,
"DBName" : String,
"DBParameterGroupName" : String,
"DBSecurityGroups" : [ String, ... ],
"DBSnapshotIdentifier" : String,
"DBSubnetGroupName" : String,
"DeleteAutomatedBackups" : Boolean,
"DeletionProtection" : Boolean,
"Domain" : String,
"DomainIAMRoleName" : String,
"EnableCloudwatchLogsExports" : [ String, ... ],
"EnableIAMDatabaseAuthentication" : Boolean,
"EnablePerformanceInsights" : Boolean,
"Engine" : String,
"EngineVersion" : String,
"Iops" : Integer,
"KmsKeyId" : String,
"LicenseModel" : String,
"MasterUsername" : String,
"MasterUserPassword" : String,
"MonitoringInterval" : Integer,
"MonitoringRoleArn" : String,
"MultiAZ" : Boolean,
"OptionGroupName" : String,
"PerformanceInsightsKMSKeyId" : String,
"PerformanceInsightsRetentionPeriod" : Integer,
"Port" : String,
"PreferredBackupWindow" : String,
"PreferredMaintenanceWindow" : String,
"ProcessorFeatures" : [ ProcessorFeature, ... ],
"PromotionTier" : Integer,
"PubliclyAccessible" : Boolean,
"SourceDBInstanceIdentifier" : String,
"SourceRegion" : String,
"StorageEncrypted" : Boolean,
"StorageType" : String,
"Tags" : [ Tag, ... ],
"Timezone" : String,
"UseDefaultProcessorFeatures" : Boolean,
"VPCSecurityGroups" : [ String, ... ]
}
}
}
}
After this, you can include the RDS parameters in your other Resources as it will be created first.
I was stumbling across the same question and found some sort of an answer here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.db.html
In that document they say:
You can configure your environment's database instance using configuration files. Use the options in the aws:rds:dbinstance namespace. The following example modifies the allocated database storage size to 100 GB.
So the aws:rds:dbinstance properties in OptionSettings are only meant to modify an existing database instance, not to create one in the first place.
I have the following Cloud Formation template to create a VPC. The VPC name is generated based off of the region and the environment that the template was created in. The VPC creates without any issues, and running aws cloud formation validate-template --template-url https://foo.template doesn't complaing about any of the syntax.
I would expect the VPC to be named:
vpc-uw1-d-fs
What happens instead is the VPC is left with an empty name and the Name tag has an empty value. Am I not using the function correctly? If I remove the Fn::FindInMap function usage, I get the name generated - it's just missing the environment mapped value.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "VPC for a new environment to use.",
"Parameters": {
"EnvironmentName": {
"Description": "Name of the environment that this VPC will be used for.",
"Type": "String",
"MinLength": "2",
"MaxLength": "20",
"AllowedPattern": "[a-zA-Z]*",
"AllowedValues": [
"Development",
"QA",
"Test",
"Production"
]
}
},
"Resources": {
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": "10.0.0.0/16",
"EnableDnsSupport": false,
"EnableDnsHostnames": false,
"InstanceTenancy": "default",
"Tags": [ {
"Key": "Name",
"Value": {
"Fn::Join": [
"-",
[
"vpc",
{ "Ref": "AWS::Region" },
{ "Fn::FindInMap": [
"EnvironmentMap", { "Ref": "EnvironmentName" }, "AbbreviatedName"
]},
"fs"
]
]
}
}]
}
}
},
"Mappings": {
"RegionMap": {
"us-east-1": {
"regionName": "ue1"
},
"us-west-1": {
"regionName": "uw1"
}
},
"EnvironmentMap": {
"Development": {
"AbbreviatedName": "d"
},
"QA": {
"AbbreviatedName": "qa"
},
"Test": {
"AbbreviatedName": "t"
},
"Production": {
"AbbreviatedName": "p"
}
}
},
"Outputs": {
}
}
Your template is working perfectly fine for me.
I ran it in the ap-southeast-2 region and it produced the tag:
Name: vpc-ap-southeast-2-d-fs
(The RegionMap is not used in the template given.)
I have the following AWS Cloudformation config, which sets up S3, Repositories.
When I run it via an ansible playbook, on the second time running the playbook this happens
AWS::ECR::Repository Repository CREATE_FAILED: production-app-name already exists
etc
How can I make it so that when this is ran multiple times, it will keep the existing s3 and repository instead of just blowing up? (I had assumed the param "DeletionPolicy": "Retain", would do this)
What I'd like to achieve:
If i run this 100x, I want the same resource state as it was after run #1. I do not want any resources deleted/wiped of any data.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Pre-reqs for Elastic Beanstalk application",
"Parameters": {
"BucketName": {
"Type": "String",
"Description": "S3 Bucket name"
},
"RepositoryName": {
"Type": "String",
"Description": "ECR Repository name"
}
},
"Resources": {
"Bucket": {
"Type": "AWS::S3::Bucket",
"DeletionPolicy": "Retain",
"Properties": {
"BucketName": { "Fn::Join": [ "-", [
{ "Ref": "BucketName" },
{ "Ref": "AWS::Region" }
]]}
}
},
"Repository": {
"Type": "AWS::ECR::Repository",
"DeletionPolicy": "Retain",
"Properties": {
"RepositoryName": { "Ref": "RepositoryName" }
}
}
},
"Outputs": {
"S3Bucket": {
"Description": "Full S3 Bucket name",
"Value": { "Ref": "Bucket" }
},
"Repository": {
"Description": "ECR Repo",
"Value": { "Fn::Join": [ "/", [
{
"Fn::Join": [ ".", [
{ "Ref": "AWS::AccountId" },
"dkr",
"ecr",
{ "Ref": "AWS::Region" },
"amazonaws.com"
]]
},
{ "Ref": "Repository" }
]]}
}
}
}
edit:
DB with similar issue when ran twice
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"DBPassword": {
"MinLength": "8",
"NoEcho": true,
"Type": "String"
},
"Environment": {
"MinLength": "1",
"Type": "String"
},
"DBName": {
"Type": "String",
"Description": "DBName"
},
"DBInstanceIdentifier": {
"Type": "String",
"Description": "DBInstanceIdentifier"
},
"DBPort": {
"Type": "String",
"Description": "DBPort"
},
"DBUsername": {
"Type": "String",
"Description": "DBName"
}
},
"Outputs": {
"Url": {
"Value": {
"Fn::Sub": "postgres://${DBUsername}:${DBPassword}#${Instance.Endpoint.Address}:${Instance.Endpoint.Port}/${DBName}"
}
}
},
"Resources": {
"Instance": {
"Type": "AWS::RDS::DBInstance",
"DeletionPolicy": "Retain",
"Properties": {
"AllocatedStorage": "10",
"DBInstanceClass": "db.t2.micro",
"DBInstanceIdentifier": {"Ref": "DBInstanceIdentifier"},
"DBName": {
"Ref": "DBName"
},
"Engine": "postgres",
"EngineVersion": "9.6.6",
"MasterUsername": {
"Ref": "DBUsername"
},
"MasterUserPassword": {
"Ref": "DBPassword"
},
"MultiAZ": "false",
"Port": {
"Ref": "DBPort"
},
"PubliclyAccessible": "false",
"StorageType": "gp2"
}
}
}
}
The field RepositoryName in AWS::ECR::Repository is actually not required and I would advise against specifying one. By letting CloudFormation dynamically assign a unique name to the repository you'll avoid collision.
If you later want to use the repository name, for exemple: in a task definition, you can use the "Ref" function like so { "Ref": "Repository" } to extract the unique name generated by CloudFormation.
As for the issue with the RDS instance, tt comes down to the same problem of hardcoding resources name.
Using retain will keep the resource alive but it will no longer be managed by CloudFormation which is a big problem.
Just make sure when doing updates to never modify a parameter that require a resource "replacement". The documentation always states what kind of update a parameter change will incur.
Image taken from (here)
If you really need to change a parameter that requires a replacement. Create a new resource with the adapter parameters, migrate whatever data you had in the database or ECR repository, then remove the old resource from the template. If you don't need to migrate anything, make sure you don't have hardcoded names and let CloudFormation perform the replacement.
I have implemented this but it says invalid request. It may totally wrong.
I want to create geolocation for particular country
"Geolocation": {
"Type": "AWS::Route53::RecordSet",
"Properties": {
"HostedZoneName": {
"Ref": "Route53Domain"
},
"Name": {
"Fn::Join": [
"$",
[
{
"Ref": "javadns"
},
{
"Ref": "Route53Domain"
}
]
]
},
"Type": "CNAME",
"TTL": "60",
"GeoLocation": {"CountryCode" : "*"},
"ResourceRecords": [
{ "Fn::Join" : [".", ["geoip", { "Ref" : "javadns" }]]}
]
}
},
To create a Geo Location record you must specify the SetIdentifier property in CloudFormation.
i see you are missing that property in your CloudFormation and probably that is why it is throwing you an error.
Also here are the code countries identifiers that are supported by Route53, you must not specify the wildcard in that parameter, countries and continent codes consist of two letters.
Documentation here.
I'm using ECS-CLI (0.4.5) to launch a CFN template, and now I'm trying to put an Aurora cluster into the CFN template and update the stack with a changeset through the CFN SDK.
I can't figure out why it's upset about my subnets. The subnets are created by the initial 'ecs-cli up' call. They are in the same vpc as the rest of the stack, they already exist before I try to deploy the changeset, and they are in different availability zones (us-west-2b and us-west-2c).
The only info CFN is giving me is that 'some input subnets are invalid'.
CFN Failure:
Subnets:
I can create a DBSubnetGroup through the management console with the exact same subnets with no problems.
Any ideas on what could be going wrong? Is this a bug in CloudFormation? Let me know if more information is needed to solve this... I'm honestly at such a loss
Here's what my initial template boils down to (It's built into ecs-cli):
"PubSubnetAz1": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "Vpc"
},
"CidrBlock": "10.0.0.0/24",
"AvailabilityZone": "us-west-2b"
}
},
"PubSubnetAz2": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "Vpc"
},
"CidrBlock": "10.0.1.0/24",
"AvailabilityZone": "us-west-2c"
}
},
"InternetGateway": {
"Type": "AWS::EC2::InternetGateway"
},
"AttachGateway": {
"Type": "AWS::EC2::VPCGatewayAttachment",
"Properties": {
"VpcId": {
"Ref": "Vpc"
},
"InternetGatewayId": {
"Ref": "InternetGateway"
}
}
},
"RouteViaIgw": {
"Type": "AWS::EC2::RouteTable",
"Properties": {
"VpcId": {
"Ref": "Vpc"
}
}
},
"PublicRouteViaIgw": {
"DependsOn": "AttachGateway",
"Type": "AWS::EC2::Route",
"Properties": {
"RouteTableId": {
"Ref": "RouteViaIgw"
},
"DestinationCidrBlock": "0.0.0.0/0",
"GatewayId": {
"Ref": "InternetGateway"
}
}
},
"PubSubnet1RouteTableAssociation": {
"Type": "AWS::EC2::SubnetRouteTableAssociation",
"Properties": {
"SubnetId": {
"Ref": "PubSubnetAz1"
},
"RouteTableId": {
"Ref": "RouteViaIgw"
}
}
},
"PubSubnet2RouteTableAssociation": {
"Type": "AWS::EC2::SubnetRouteTableAssociation",
"Properties": {
"SubnetId": {
"Ref": "PubSubnetAz2"
},
"RouteTableId": {
"Ref": "RouteViaIgw"
}
}
},
And then when I go to update it, I add this:
"DBSubnetGroup": {
"Type": "AWS::RDS::DBSubnetGroup",
"Properties": {
"DBSubnetGroupDescription": "Aurora Subnet Group using subnets from 2 AZs",
"SubnetIds": {
"Fn::Join": [
",", [{
"Ref": "pubSubnetAz1"
},
{
"Ref": "pubSubnetAz2"
}
]
]
}]
}
}
}
The changeset should be simple enough...
"Changes": [
{
"Type": "Resource",
"ResourceChange": {
"Action": "Add",
"LogicalResourceId": "DBSubnetGroup",
"ResourceType": "AWS::RDS::DBSubnetGroup",
"Scope": [],
"Details": []
}
}
]
I'm using AWSTemplateFormatVersion 2010-09-09 and the JavaScript aws-sdk "^2.7.21"
The issue is that you're concatenating your subnet IDs into a string. Instead, you should pass them in an array. Try this:
"PrivateSubnetGroup": {
"Type": "AWS::RDS::DBSubnetGroup",
"Properties": {
"SubnetIds": [
{
"Ref": "PubSubnetAz1"
},
{
"Ref": "PubSubnetAz2"
}
],
"DBSubnetGroupDescription": "Aurora Subnet Group using subnets from 2 AZs"
}
}
Also, I would highly recommend trying to use yaml instead of json. Cloudformation now supports this natively, along with some shortcut functions to make using references easier, and I think in the long run you'll find it much easier to both read and write.
Here's an example of how you could write equivalent json in yaml:
PrivateSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Subnet group for Aurora Database
SubnetIds:
- !Ref PubSubnetAz1
- !Ref PubSubnetAz2
According to the AWS::RDS::DBSubnetGroup documentation, the SubnetIDs parameter accepts a List of strings, not a CommaDelimitedList which is what you provided in your example. You should pass the subnets in a JSON array directly, without using Fn::Join:
"SubnetIds": [
{"Ref": "pubSubnetAz1"},
{"Ref": "pubSubnetAz2"}
]