ElasticSearch CloudFormation - Property VpcId cannot be empty - amazon-web-services

I created the following CloudFormation Template to kick up an ElasticSearch Domain. I want to use EXISTING VPC and Subnets, however, when I went through and modified this to do that, I get an error when standing up the stack that states:
"Property VpcId cannot be empty."
Yet, in the template parameters, you select the VpcId...
AWSTemplateFormatVersion: 2010-09-09
Description: >-
**NOTE** In order to create Elastisearch Domain in AWS using CloudFormation
verify you have the following Service Role created in IAM!!
-- AWSServiceRoleForAmazonElasticsearchService --
If you do not have this Role, create it using the following CLi Command --
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: "Configure Cluster"
Parameters:
- DomainName
- ElasticsearchVersion
- ZoneAwareness
-
Label:
default: "Data Instances"
Parameters:
- InstanceType
- DataInstanceCount
-
Label:
default: "Dedicated Master Instances"
Parameters:
- DedicatedMaster
- MasterInstanceType
- MasterInstanceCount
-
Label:
default: "Storage Config"
Parameters:
- StorageSize
-
Label:
default: "Network Config"
Parameters:
- VpcId
- SubNet1
- SubNet2
- AvailabilityZone
- AvailabilityZone2
- SecurityGroup
- GroupDescription
-
Label:
default: "IAM User Restriction Policy"
Parameters:
- IamUserArn
ParameterLabels:
DomainName:
default: "Name of the ElasticSearch Domain (lowecase, no spaces) - If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the domain name"
ElasticsearchVersion:
default: "Select the ElasticSearch version desired"
InstanceType:
default: "Instance Size of Data Instances"
DataInstanceCount:
default: "Number of Data Instances Required"
DedicatedMaster:
default: "Select if a Dedicated Master Instance is required"
MasterInstanceType:
default: "Instance Size of Master Instances"
MasterInstanceCount:
default: "How many Dedicated Master Instances are needed? (0, 3 or 5)"
StorageSize:
default: "Storage Size in GB"
VpcId:
default: "Select the VPC to deploy into (must already exist)"
SubNet1:
default: "Select the First Subnet"
SubNet2:
default: "Select the Second Subnet"
GroupDescription:
default: "New Security Group Description"
SecurityGroup:
default: "Name of the New Security Group"
IamUserArn:
default: "Enter the ARN for the IAM User to give initial access to the stack"
ZoneAwareness:
default: "Enable Zone Awareness (Availability Zone Replication) (recommended)"
Parameters:
DomainName:
Type: String
Default: "elasticsearchstack-cf"
MaxLength: '128'
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
ElasticsearchVersion:
Type: String
Default: 7.1
AllowedValues: [7.1, 6.8, 6.7, 6.6, 6.5] # Remove this line for free-form number entry
InstanceType:
Type: String
Default: t2.medium.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
DataInstanceCount:
Type: Number
Default: 2
AllowedValues: [2, 4, 6, 8, 10] # Remove this line for free-form number entry
MasterInstanceType:
Type: String
Default: t2.medium.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
MasterInstanceCount:
Type: Number
Default: 0
AllowedValues: [0, 3, 5] # Remove this line for free-form number entry
VpcId:
Type: AWS::EC2::VPC::Id
ConstraintDescription: "Must be the VPC ID of an existing Virtual Private Cloud."
SubNet1:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SubNet2:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SecurityGroup:
Type: String
Default: "elasticsearchstack-nsg"
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
DedicatedMaster:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
StorageSize:
Type: Number
Default: 20
MinValue: 10 # Remove this line for free-form number entry (suggested to keep this line)
MaxValue: 1000 # Remove this line for free-form number entry
GroupDescription: # Security Group Description
Type: String
Default: "ElasticSearch Stack from CloudFormation"
MaxLength: '128'
ConstraintDescription: "Upper/Lowercase, numbers/letters and/or a dash"
IamUserArn:
Type: String
Default: "arn:aws:iam::<AccountNumber>:user/<username>"
ZoneAwareness:
Description: True or False
Type: String
Default: True
AllowedValues:
- True
- False
Conditions: # Checks to see if Dedicated Master is True
DedicatedMasterYes: !Equals [ !Ref DedicatedMaster, True]
Resources:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref DomainName
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
DedicatedMasterEnabled: !Ref DedicatedMaster
InstanceCount: !Ref DataInstanceCount
ZoneAwarenessEnabled: !Ref ZoneAwareness
InstanceType: !Ref InstanceType
DedicatedMasterType: # If Dedicated Master is True, then use !Ref, if not, use NoValue (NULL)
!If [DedicatedMasterYes, !Ref MasterInstanceType, !Ref "AWS::NoValue"]
DedicatedMasterCount:
!If [DedicatedMasterYes, !Ref MasterInstanceCount, !Ref "AWS::NoValue"]
EBSOptions:
EBSEnabled: True
Iops: 0
VolumeSize: !Ref StorageSize
VolumeType: "gp2"
SnapshotOptions:
AutomatedSnapshotStartHour: "0"
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: !Ref IamUserArn
Action: 'es:*'
Resource: "*"
AdvancedOptions: # BOTH of these settingsd are REQUIRED (regardless of what the documentation states) - Bug filed: https://forums.aws.amazon.com/thread.jspa?messageID=768527
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
VPCOptions:
VpcId: !Ref VpcId
SubnetIds:
- !Ref subnet1
- !Ref subnet2
SecurityGroupIds:
- !Ref mySecurityGroup
subnet1:
Type: AWS::EC2::Subnet
subnet2:
Type: AWS::EC2::Subnet
mySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VpcId
GroupName: !Ref SecurityGroup
GroupDescription: !Ref GroupDescription
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
Outputs:
DomainArn:
Value: !GetAtt ElasticsearchDomain.DomainArn
DomainEndpoint:
Value: !GetAtt ElasticsearchDomain.DomainEndpoint
SecurityGroupId:
Value: !Ref mySecurityGroup
SubnetId:
Value: !Ref subnet1
SubnetId2:
Value: !Ref subnet2
I'm sure I am missing something simple, but my brain is fried. Thanks all!

#WarrenG - Thanks for that, I just managed to get that figured out too...
Here is the working, fixed template:
AWSTemplateFormatVersion: 2010-09-09
Description: >-
**NOTE** In order to create Elastisearch Domain in AWS using CloudFormation
verify you have the following Service Role created in IAM!!
-- AWSServiceRoleForAmazonElasticsearchService --
If you do not have this Role, create it using the following CLi Command --
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: "Configure Cluster"
Parameters:
- DomainName
- ElasticsearchVersion
- ZoneAwareness
- SnapShotHour
-
Label:
default: "Data Instances"
Parameters:
- InstanceType
- DataInstanceCount
-
Label:
default: "Dedicated Master Instances"
Parameters:
- DedicatedMaster
- MasterInstanceType
- MasterInstanceCount
-
Label:
default: "Storage Config"
Parameters:
- StorageSize
-
Label:
default: "Network Config"
Parameters:
- VpcId
- SubNet1
- SubNet2
- SecurityGroup
- GroupDescription
-
Label:
default: "IAM User Restriction Policy"
Parameters:
- IamUserArn
ParameterLabels:
DomainName:
default: "Name of the ElasticSearch Domain (lowecase, no spaces) - If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the domain name"
ElasticsearchVersion:
default: "Select the ElasticSearch version desired"
InstanceType:
default: "Instance Size of Data Instances"
DataInstanceCount:
default: "Number of Data Instances Required"
DedicatedMaster:
default: "Select if a Dedicated Master Instance is required"
MasterInstanceType:
default: "Instance Size of Master Instances"
MasterInstanceCount:
default: "How many Dedicated Master Instances are needed? (0, 3 or 5)"
StorageSize:
default: "Storage Size in GB"
VpcId:
default: "Select the VPC to deploy into (must already exist)"
SubNet1:
default: "Select the First Subnet"
SubNet2:
default: "Select the Second Subnet"
GroupDescription:
default: "New Security Group Description"
SecurityGroup:
default: "Name of the New Security Group"
IamUserArn:
default: "Enter the ARN for the IAM User to give initial access to the stack"
ZoneAwareness:
default: "Enable Zone Awareness (Availability Zone Replication) (recommended)"
SnapShotHour:
default: "Set the hour to run the Automated Snapshot (0-23) (Default: UTC Timezone)"
Parameters:
DomainName:
Type: String
Default: "elasticsearchstack-cf"
MaxLength: '128'
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
SnapShotHour:
Type: Number
Default: 0
MinValue: 0
MaxValue: 23
ElasticsearchVersion:
Type: String
Default: 7.1
AllowedValues: [7.1, 6.8, 6.7, 6.6, 6.5] # Remove this line for free-form number entry
InstanceType:
Type: String
Default: t2.medium.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
DataInstanceCount:
Type: Number
Default: 2
AllowedValues: [2, 4, 6, 8, 10] # Remove this line for free-form number entry
MasterInstanceType:
Type: String
Default: t2.medium.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
MasterInstanceCount:
Type: Number
Default: 0
AllowedValues: [0, 3, 5] # Remove this line for free-form number entry
VpcId:
Type: AWS::EC2::VPC::Id
ConstraintDescription: "Must be the VPC ID of an existing Virtual Private Cloud."
SubNet1:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SubNet2:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SecurityGroup:
Type: String
Default: "elasticsearchstack-nsg"
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
DedicatedMaster:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
StorageSize:
Type: Number
Default: 20
MinValue: 10 # Remove this line for free-form number entry (suggested to keep this line)
MaxValue: 1000 # Remove this line for free-form number entry
GroupDescription: # Security Group Description
Type: String
Default: "ElasticSearch Stack from CloudFormation"
MaxLength: '128'
ConstraintDescription: "Upper/Lowercase, numbers/letters and/or a dash"
IamUserArn:
Type: String
Default: "arn:aws:iam::<AccountNumber>:user/<username>"
ZoneAwareness:
Description: True or False
Type: String
Default: True
AllowedValues:
- True
- False
Conditions: # Checks to see if Dedicated Master is True
DedicatedMasterYes: !Equals [ !Ref DedicatedMaster, True]
Resources:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref DomainName
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
DedicatedMasterEnabled: !Ref DedicatedMaster
InstanceCount: !Ref DataInstanceCount
ZoneAwarenessEnabled: !Ref ZoneAwareness
InstanceType: !Ref InstanceType
DedicatedMasterType: # If Dedicated Master is True, then use !Ref, if not, use NoValue (NULL)
!If [DedicatedMasterYes, !Ref MasterInstanceType, !Ref "AWS::NoValue"]
DedicatedMasterCount:
!If [DedicatedMasterYes, !Ref MasterInstanceCount, !Ref "AWS::NoValue"]
EBSOptions:
EBSEnabled: True
Iops: 0
VolumeSize: !Ref StorageSize
VolumeType: "gp2"
SnapshotOptions:
AutomatedSnapshotStartHour: !Ref SnapShotHour
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: !Ref IamUserArn
Action: 'es:*'
Resource: "*"
AdvancedOptions: # BOTH of these settingsd are REQUIRED (regardless of what the documentation states) - Bug filed: https://forums.aws.amazon.com/thread.jspa?messageID=768527
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
VPCOptions:
SubnetIds:
- !Ref SubNet1
- !Ref SubNet2
SecurityGroupIds:
- !Ref mySecurityGroup
mySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VpcId
GroupName: !Ref SecurityGroup
GroupDescription: !Ref GroupDescription
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
Outputs:
DomainArn:
Value: !GetAtt ElasticsearchDomain.DomainArn
DomainEndpoint:
Value: !GetAtt ElasticsearchDomain.DomainEndpoint
SecurityGroupId:
Value: !Ref mySecurityGroup

Related

!GetAtt parameter error requires 2 arguments but template file doesn't seem to conform to this

I am trying to deploy Tableau to the AWS gov-cloud and I made some edits to the template in order to make it ready for govCloud deployment, however when I upload my YAML file to AWS I keep getting the following error:
Template error: if specifying one argument to Fn::GetAtt, that argument must be a non-empty string in format <LogicalId>.<Attribute>
I didn't edit any of the !GetAtt stuff from the template and the template doesn't seem to follow that order so I'm confused as to why I would keep getting this error on upload. Below is my YAML file snippet I would isolate the error but really not sure what part is causing the issue.
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Template: Single-node Tableau Server running on Windows,
CentOS, or Ubuntu. (qs-1puphiil4)'
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: AWS Environment and Machine Configuration
Parameters:
- KeyPairName
- AvailabilityZones
- VPCCIDR
- PublicSubnet1CIDR
- PublicSubnet2CIDR
- SourceCIDR
- InstanceType
- AMIOS
- Label:
default: Secrets
Parameters:
- Username
- Password
- TableauServerAdminUser
- TableauServerAdminPassword
- Label:
default: Registration
Parameters:
- AcceptEULA
- TableauServerLicenseKey
- RegFirstName
- RegLastName
- RegEmail
- RegCompany
- RegTitle
- RegDepartment
- RegIndustry
- RegPhone
- RegCity
- RegState
- RegZip
- RegCountry
- Label:
default: AWS Quick Start Configuration
Parameters:
- QSS3BucketName
- QSS3KeyPrefix
ParameterLabels:
AcceptEULA:
default: Accept Tableau End User License Agreement
AvailabilityZones:
default: Availability Zones
AMIOS:
default: AMI Operating System
InstanceType:
default: Tableau Amazon EC2 instance type
KeyPairName:
default: Key Pair Name
Password:
default: Tableau Services Manager (TSM) administrator password
PublicSubnet1CIDR:
default: Public Subnet 1 CIDR
PublicSubnet2CIDR:
default: Public Subnet 2 CIDR
QSS3BucketName:
default: Quick Start S3 Bucket Name
QSS3KeyPrefix:
default: Quick Start S3 Key Prefix
RegCity:
default: City
RegCompany:
default: Company
RegCountry:
default: Country
RegDepartment:
default: Department
RegEmail:
default: Email Address
RegFirstName:
default: First Name
RegIndustry:
default: Industry
RegLastName:
default: Last Name
RegPhone:
default: Phone
RegState:
default: State
RegTitle:
default: Title
RegZip:
default: Zip/Postal Code
SourceCIDR:
default: Source CIDR for Access
TableauServerAdminPassword:
default: Tableau Server administrator password
TableauServerAdminUser:
default: Tableau Server administrator username
TableauServerLicenseKey:
default: Tableau Activation Key
Username:
default: Tableau Services Manager (TSM) administrator username
VPCCR:
default: VPC CIDR
Parameters:
AvailabilityZones:
Description: 'List of Availability Zones to use for the subnets in teh VPC, Note: The logical order is preseved and 2 AZs will be used for this deployment'
Type: List<AWS::EC2::AvailabilityZone::Name>
AMIOS:
AllowedValues:
- Windows
- CentOS-7-HVM
- Ubuntu-Server-16.04-LTS-HVM
Default: Ubuntu-Server-16.04-LTS-HVM
Description: Operating System on which Tableau Server will be deployed
Type: String
AcceptEULA:
AllowedPattern: 'yes'
AllowedValues:
- 'yes'
- 'No'
Description: 'View the EULA at the Link: https://www.tableau.com/eula'
Type: String
InstanceType:
AllowedValues:
- m4.2xlarge
- m4.4xlarge
- m4.10xlarge
- m5.4xlarge
- m5.12xlarge
- c5.4xlarge
- c4.4xlarge
- c5d.4xlarge
- r5d.4xlarge
ConstraintDescription: must be a valid EC2 instance type.
Default: m5.4xlarge
Description: Amazon EC2 instance type
Type: String
KeyPairName:
ConstraintDescription: must be the name of an existing EC2 KeyPair.
Description: Name of an existing EC2 KeyPair to enable SSH access to the instances
Type: AWS::EC2::KeyPair::KeyName
Password:
Description: Tableau Services Manager (TSM) administrator password
NoEcho: 'true'
Type: String
PublicSubnet1CIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.128.0/20
Description: CIDR Block for the Public DMZ Subnet located in AZ1
Type: String
PublicSubnet2CIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.144.0/20
Description: CIDR Block for the Public DMZ Subnet located in AZ2
Type: String
QSS3BucketName:
AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
ConstraintDescription: Quick Start bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
(-).
Default: aws-quickstart
Description: S3 bucket name for the Quick Start assets. This string can include
numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
or end with a hyphen (-).
Type: String
QSS3KeyPrefix:
AllowedPattern: ^[0-9a-zA-Z-/]*$
ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
uppercase letters, hyphens (-), and forward slash (/).
Default: quickstart-tableau-server/
Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
can include numbers, lowercase letters, uppercase letters, hyphens (-), and
forward slash (/).
Type: String
RegCity:
Description: City
Type: String
RegCompany:
Description: Company
Type: String
RegCountry:
Description: Country
Type: String
RegDepartment:
Description: Department
Type: String
RegEmail:
Description: Email
MinLength: '1'
Type: String
RegFirstName:
Description: First Name
MinLength: '1'
Type: String
RegIndustry:
Description: Industry
Type: String
RegLastName:
Description: Last Name
MinLength: '1'
Type: String
RegPhone:
Description: Phone
Type: String
RegState:
Description: State
Type: String
RegTitle:
Description: Title
Type: String
RegZip:
Description: ZIP/Postal Code
Type: String
SourceCIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/x
Description: The CIDR address from which you will connect to the instance
Type: String
TableauServerAdminPassword:
Description: The password of the initial administrator for Tableau Server
MinLength: '1'
NoEcho: 'true'
Type: String
TableauServerAdminUser:
Description: The name of the initial administrator for Tableau Server
MinLength: '1'
Type: String
TableauServerLicenseKey:
Description: License Key (leave blank for trial)
Type: String
Username:
AllowedPattern: ^(?!(tableau|tsmagent|admin|root)$)[A-Za-z0-9]*$
Description: Tableau Services Manager (TSM) administrator username (cannot be
'tableau' or 'tsmagent' or 'admin' or 'root')
MaxLength: '30'
Type: String
VPCCIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.0.0/16
Description: CIDR Block for the VPC
Type: String
Conditions:
InfaOnWindows: !Equals
- !Ref 'AMIOS'
- Windows
InfaOnCentos: !Equals
- !Ref 'AMIOS'
- CentOS-7-HVM
Resources:
VPCStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub 'https://${QSS3BucketName}.s3.amazonaws-us-gov.com/${QSS3KeyPrefix}submodules/quickstart-aws-vpc/templates/aws-vpc.template'
Parameters:
AvailabilityZones: !Join
- ','
- !Ref 'AvailabilityZones'
KeyPairName: !Ref 'KeyPairName'
NATInstanceType: t2.small
NumberOfAZs: '2'
PublicSubnet1CIDR: !Ref 'PublicSubnet1CIDR'
PublicSubnet2CIDR: !Ref 'PublicSubnet2CIDR'
CreatePrivateSubnets: 'false'
VPCCIDR: !Ref VPCCIDR
TableauIAMRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws-us-gov:iam::aws:policy/service-role/AmazonEC2RoleforSSM
Policies:
- PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- s3:GetObject
Resource:
- !Sub 'arn:aws-us-gov:s3:::${QSS3BucketName}/${QSS3KeyPrefix}*'
Effect: Allow
PolicyName: aws-quick-start-s3-policy
TableauServerInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref 'TableauIAMRole'
WorkloadStack:
Type: AWS::CloudFormation::Stack
DependsOn: VPCStack
Properties:
TemplateURL: !Sub
- https://${QSS3BucketName}.s3.amazonaws-us-gov.com/${QSS3KeyPrefix}templates/${QSTableauWorkloadTemplate}
- QSTableauWorkloadTemplate: !If
- InfaOnWindows
- tableau-single-server-windows.template
- !If
- InfaOnCentos
- tableau-single-server-centos.template
- tableau-single-server-ubuntu.template
Parameters:
AMIOS: !Ref 'AMIOS'
PublicSubnetId: !Ref 'VPCStack.Outputs.PublicSubnet1ID'
AcceptEULA: !Ref 'AcceptEULA'
InstanceType: !Ref 'InstanceType'
KeyPairName: !Ref 'KeyPairName'
Password: !Ref 'Password'
QSS3BucketName: !Ref 'QSS3BucketName'
QSS3KeyPrefix: !Ref 'QSS3KeyPrefix'
RegCity: !Ref 'RegCity'
RegCompany: !Ref 'RegCompany'
RegCountry: !Ref 'RegCountry'
RegDepartment: !Ref 'RegDepartment'
RegEmail: !Ref 'RegEmail'
RegFirstName: !Ref 'RegFirstName'
RegIndustry: !Ref 'RegIndustry'
RegLastName: !Ref 'RegLastName'
RegPhone: !Ref 'RegPhone'
RegState: !Ref 'RegState'
RegTitle: !Ref 'RegTitle'
RegZip: !Ref 'RegZip'
SourceCIDR: !Ref 'SourceCIDR'
TableauServerAdminPassword: !Ref 'TableauServerAdminPassword'
TableauServerAdminUser: !Ref 'TableauServerAdminUser'
TableauServerLicenseKey: !Ref 'TableauServerLicenseKey'
Username: !Ref 'Username'
VPCId: !GetAtt 'VPCID'
Outputs:
VPCID:
Description: VPC ID
Value: !GetAtt 'VPCStack.Outputs.VPCID'
InstanceID:
Description: EC2 InstanceID of the instance running Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.InstanceID'
PublicIPAddress:
Description: Public IP Address of instance running Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.PublicIPAddress'
TableauServicesManagerURL:
Description: URL for the TSM Web UI
Value: !GetAtt 'WorkloadStack.Outputs.TableauServicesManagerURL'
TableauServerURL:
Description: URL for the Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.TableauServerURL'
The CloudFormation Linter will also catch these errors faster with more informative messages like:
E1010 Invalid GetAtt for Resources/WorkloadStack/Properties/Parameters/VPCId/Fn::GetAtt
template.yaml:333:9
so we know the issue is with line 333:
VPCId: !GetAtt 'VPCID'
The Visual Studio Code extension can also highlight these inline in your templates:
AWS::CloudFormation::Stack return values
Fn::GetAtt documentation
It looks like this line has been updated in the source repository, but I'd recommend opening a Github issue there if you're still experiencing difficulties with their templates

AWS ElasticSearch Service - Set Encryption options from CF template

I am creating a cloud formation template to provision elasticsearch service domain in AWS.
I would like to set this property under Encryption to true
"Require HTTPS for all traffic to the domain" but I am not able to find the way in AWS docs to do so.
Other options for setting encryption properties like
"Enable encryption of data at rest" & "Node-to-node encryption" are well documented.
Does anyone know how to set "Require HTTPS for all traffic to the domain" property from CF template ?
The way I did this was to make sure the security group only allows HTTPS (443) access to the cluster.
Not completely sure if this is what you are looking for, but if it's not, give me some more details and I will see if I can help you out.
mySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VpcId
GroupName: !Ref SecurityGroup
GroupDescription: !Ref GroupDescription
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
Here is what I have put together for Encryption at Rest as well as Node to node encryption:
Now, I had to add/edit plenty of moving parts, so I will post the whole template here so you can see what I did in the different sections so you can pull out what you need/want. I used Conditionals since you can chose to enable it or not. I also made sure to add a few comments here and there. Feel free to strip those if you want, but they do not impact the template at all.
edit: I know that there are several parts of the template that are not NEEDED, but I like to make mine look pretty and organized (when being used). I know I need to work on organization inside the template itself, but it's all there. :P
edit2: The below template assumes you will be using Existing VPC/Subnets/Security Groups, you can check out my repo (in comment below) for other versions of this template.
I did validate this template and was able to successfully build the Domain.
AWSTemplateFormatVersion: 2010-09-09
Description: >-
**NOTE** In order to create Elastisearch Domain in AWS using CloudFormation
verify you have the following Service Role created in IAM!!
-- AWSServiceRoleForAmazonElasticsearchService --
If you do not have this Role, create it using the following CLi Command --
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: "Configure Cluster"
Parameters:
- DomainName
- ElasticsearchVersion
- ZoneAwareness
- SnapShotHour
-
Label:
default: "Data Instances"
Parameters:
- InstanceType
- DataInstanceCount
-
Label:
default: "Dedicated Master Instances"
Parameters:
- DedicatedMaster
- MasterInstanceType
- MasterInstanceCount
-
Label:
default: "Storage Config"
Parameters:
- StorageSize
-
Label:
default: "Network Config"
Parameters:
- VpcId
- SubNet1
- SubNet2
- SecurityGroup
-
Label:
default: "Encryption Settings"
Parameters:
- EncryptionAtRest
- KmsKey
- NodetoNode
-
Label:
default: "IAM User Restriction Policy"
Parameters:
- IamUserArn
ParameterLabels:
DomainName:
default: "Name of the ElasticSearch Domain (lowecase, no spaces) - If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the domain name"
ElasticsearchVersion:
default: "Select the ElasticSearch version desired"
InstanceType:
default: "Instance Size of Data Instances"
DataInstanceCount:
default: "Number of Data Instances Required"
DedicatedMaster:
default: "Select if a Dedicated Master Instance is required"
MasterInstanceType:
default: "Instance Size of Master Instances"
MasterInstanceCount:
default: "How many Dedicated Master Instances are needed? (0, 3 or 5)"
StorageSize:
default: "Storage Size in GB"
VpcId:
default: "Select the VPC to deploy into (must already exist)"
SubNet1:
default: "Select the First Subnet"
SubNet2:
default: "Select the Second Subnet"
SecurityGroup:
default: "Select the Security Group"
IamUserArn:
default: "Enter the ARN for the IAM User to give initial access to the stack"
ZoneAwareness:
default: "Enable Zone Awareness (Availability Zone Replication) (recommended)"
SnapShotHour:
default: "Set the hour to run the Automated Snapshot (0-23) (Default: UTC Timezone)"
EncryptionAtRest:
default: "Enable Encryption at Rest"
KmsKey:
default: "If Encryption at Rest is enabled, supply the KMS Key used for encryption"
NodetoNode:
default: "Enable Node to Node Encryption"
Parameters:
DomainName:
Type: String
Default: "elasticsearchstack-cf"
MaxLength: '128'
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
SnapShotHour:
Type: Number
Default: 0
MinValue: 0
MaxValue: 23
ElasticsearchVersion:
Type: String
Default: 7.1
AllowedValues: [7.1, 6.8, 6.7, 6.6, 6.5] # Remove this line for free-form number entry
InstanceType:
Type: String
Default: r5.large.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
DataInstanceCount:
Type: Number
Default: 2
AllowedValues: [2, 4, 6, 8, 10] # Remove this line for free-form number entry
MasterInstanceType:
Type: String
Default: r5.large.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
MasterInstanceCount:
Type: Number
Default: 0
AllowedValues: [0, 3, 5] # Remove this line for free-form number entry
VpcId:
Type: AWS::EC2::VPC::Id
ConstraintDescription: "Must be the VPC ID of an existing Virtual Private Cloud."
SubNet1:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SubNet2:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
ConstraintDescription: "Must be and existing Security Group."
DedicatedMaster:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
StorageSize:
Type: Number
Default: 20
MinValue: 10 # Remove this line for free-form number entry (suggested to keep this line)
MaxValue: 1000 # Remove this line for free-form number entry
IamUserArn:
Type: String
Default: "arn:aws:iam::<AccountNumber>:user/<username>"
ZoneAwareness:
Description: True or False
Type: String
Default: True
AllowedValues:
- True
- False
EncryptionAtRest:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
KmsKey:
Type: String
NodetoNode:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
Conditions: # Checks to see if Conditional Values are True
DedicatedMasterYes: !Equals [ !Ref DedicatedMaster, True]
EncryptionAtRestYes: !Equals [ !Ref EncryptionAtRest, True]
Resources:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref DomainName
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
DedicatedMasterEnabled: !Ref DedicatedMaster
InstanceCount: !Ref DataInstanceCount
ZoneAwarenessEnabled: !Ref ZoneAwareness
InstanceType: !Ref InstanceType
DedicatedMasterType: # If Dedicated Master is True, then use !Ref, if not, use NoValue (NULL) for both settings below
!If [DedicatedMasterYes, !Ref MasterInstanceType, !Ref "AWS::NoValue"]
DedicatedMasterCount:
!If [DedicatedMasterYes, !Ref MasterInstanceCount, !Ref "AWS::NoValue"]
EBSOptions:
EBSEnabled: True
Iops: 0
VolumeSize: !Ref StorageSize
VolumeType: "gp2"
SnapshotOptions:
AutomatedSnapshotStartHour: !Ref SnapShotHour
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: !Ref IamUserArn
Action: 'es:*'
Resource: "*"
AdvancedOptions: # BOTH of these settingsd are REQUIRED (regardless of what the documentation states) - Bug filed: https://forums.aws.amazon.com/thread.jspa?messageID=768527
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
VPCOptions:
SubnetIds:
- !Ref SubNet1
- !Ref SubNet2
SecurityGroupIds:
- !Ref SecurityGroup
EncryptionAtRestOptions: # If Encryption At Rest is True, then use !Ref, if not, use NoValue (NULL) for both settings below
!If [EncryptionAtRestYes, !Ref EncryptionAtRest, !Ref "AWS::NoValue"]
KmsKeyId:
!If [EncryptionAtRestYes, !Ref KmsKey, !Ref "AWS::NoValue"]
NodeToNodeEncryptionOptions:
Enabled: !Ref NodetoNode
Outputs:
DomainArn:
Value: !GetAtt ElasticsearchDomain.DomainArn
DomainEndpoint:
Value: !GetAtt ElasticsearchDomain.DomainEndpoint
SecurityGroupId:
Value: !Ref SecurityGroup
SubnetId1:
Value: !Ref SubNet1
SubnetId2:
Value: !Ref SubNet2
As feature was recently released, it is not yet available in CloudFormation or Terraform, But we can achieve same by using custom resource in a CF template. This custom resource is nothing but a lambda function which will be executed as part cloud formation stack creation.
complete automation can be divided into 3 section,
Lambda function development
Lambda function creation
Lambda function configuration as custom resource in Cloudformation stack
Lambda function development
As explained here follow below steps to create lambda fuction.
Create an empty folder that you’ll use to place your Lambda source. Then use pip to install crhelper into the folder, and create the lambda_function.py file to put your resource code into.
mkdir sum_function
cd sum_function
pip install -t . crhelper
# on some systems pip may fail with a distutils error, if you run into this, try
running pip with the –system argument
# pip install —system -t . crhelper
touch lambda_function.py
Now open the lambda_function.py file and place the following code into the lambda_function.py file.
from crhelper import CfnResource
import boto3
import logging
logger = logging.getLogger(__name__)
helper = CfnResource()
#helper.create
#helper.update
def update_es_domain_config(event, context):
logger.info("Updating elasticsearch domain config with HTTPS")
es = boto3.client('es')
response = es.update_elasticsearch_domain_config(
DomainName=event['ResourceProperties']['ESDomainName'],
DomainEndpointOptions={
'EnforceHTTPS': True,
'TLSSecurityPolicy': 'Policy-Min-TLS-1-0-2019-07'
}
)
logger.info("results:", response)
#helper.delete
def no_op(_, __):
pass
def handler(event, context):
helper(event, context)
After that zip the whole directory
zip -r ../sum.zip ./
Lambda function creation
Now create the lambda function, I am using CLI for the same. Replace the values according to your environment.
aws lambda create-function \
--function-name "update-es-domain" \
--handler "lambda_function.handler" \
--timeout 900 \
--zip-file fileb://../sum.zip \
--runtime python3.7 \
--role "arn:aws:iam::123412341234:role/lambda-cli-role"
You’ll need to take down the FunctionArn from the output to use in the next step.
Lambda function configuration as custom resource in Cloudformation stack
Now In Resources section of your CF template, after Elasticsearch resource, add custom resource as shown below. Replace the service token with the functionArn you get in last step as a output. Replace ESDomainName with the elasticsearch domain that you want to modify.
ESDomainUpdate:
Type: "Custom::elasticsearch"
Properties:
ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
ESDomainName: "advantagehelp"
PFB Complete Cloudformation template for reference.
AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation templates to create AWS Elasticsearch Service domain.
Parameters:
NodeType:
Description: The node type to be provisioned for the Elasticsearch cluster
Type: String
Default: m4.large.elasticsearch
AllowedValues:
- m4.large.elasticsearch
- m4.xlarge.elasticsearch
- c4.large.elasticsearch
- c4.xlarge.elasticsearch
- r4.large.elasticsearch
- r4.xlarge.elasticsearch
ConstraintDescription: must be a valid Elasticsearch node type.
NodeCount:
Description: The number of nodes in the Elasticsearch cluster.
Type: Number
Default: '1'
AllowedESGetIpList:
Type: CommaDelimitedList
Description: List of ip and cidr for elasticsearch retrieval
AllowedESUpdateIpList:
Type: CommaDelimitedList
Description: List of ip and cidr for elasticsearch update
ESDomainName:
Type: String
Description: Name of Elastic Search Domain
Default: advantagehelp
Resources:
ElasticsearchDomain:
Type: 'AWS::Elasticsearch::Domain'
Metadata:
cfn_nag:
rules_to_suppress:
- id: W28
reason: "Needed to name elasticsearch domain"
Properties:
DomainName: !Ref ESDomainName
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: !Ref NodeCount
ZoneAwarenessEnabled: false
InstanceType: !Ref NodeType
ElasticsearchVersion: '6.4'
AccessPolicies:
Version: 2012-10-17
Statement:
- Action:
- 'es:ESHttpGet'
- 'es:ESHttpHead'
Principal: '*'
Effect: Allow
Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
Condition:
IpAddress:
'aws:SourceIp': !Ref AllowedESGetIpList
- Action:
- 'es:ESHttpGet'
- 'es:ESHttpHead'
- 'es:ESHttpPost'
- 'es:ESHttpPut'
Principal: '*'
Effect: Allow
Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
Condition:
IpAddress:
'aws:SourceIp': !Ref AllowedESUpdateIpList
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 10
VolumeType: gp2
SnapshotOptions:
AutomatedSnapshotStartHour: 1
AdvancedOptions:
rest.action.multi.allow_explicit_index: "true"
ESDomainUpdate:
Type: "Custom::elasticsearch"
Properties:
ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
ESDomainName: "advantagehelp"
DependsOn: ElasticsearchDomain
Outputs:
KibanaURL:
Description: Kibana URL
Value: !Join
- ''
- - !GetAtt
- ElasticsearchDomain
- DomainEndpoint
- /_plugin/kibana/
ElasticsearchEndpoint:
Description: Elasticsearch domain endpoint
Value: !GetAtt
- ElasticsearchDomain
- DomainEndpoint
ElasticsearchDomainARN:
Description: Elasticsearch domain ARN
Value: !GetAtt
- ElasticsearchDomain
- DomainArn

CloudFormation templates for Global Aurora Database

I am trying to write Cloudformation template to get a aws Global Aurora Database. However I am not able to figure out where and how to add the Global database identifier. Can someone help Cloudformation snippet?
below is my code:
Description: RDS Aurora MySQL cluster.
Parameters:
DatabaseName:
Default: "testglobalaurora"
Description: The database name
Type: String
DatabaseInstanceType:
Default: db.r4.large
AllowedValues:
- db.r4.large
- db.r4.xlarge
- db.r4.2xlarge
- db.r4.4xlarge
- db.r4.8xlarge
- db.r4.16xlarge
Description: "The instance type to use for the database."
Type: String
DatabasePassword:
Default: "testglobalaurora"
AllowedPattern: "[a-zA-Z0-9]+"
ConstraintDescription: must contain only alphanumeric characters. Must have length 8-41.
Description: The database admin account password.
MaxLength: '41'
MinLength: '8'
NoEcho: 'true'
Type: String
DatabaseUsername:
Default: "testglobalaurora"
AllowedPattern: "[a-zA-Z0-9]+"
ConstraintDescription: must contain only alphanumeric characters. Must have length 1-16
Description: The database admin account user name.
MaxLength: '16'
MinLength: '1'
Type: String
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Database Configuration
Parameters:
- DatabaseInstanceType
- DatabaseName
- DatabaseUsername
- DatabasePassword
ParameterLabels:
DatabaseName:
default: Database name
DatabaseInstanceType:
default: Database Instance Type
DatabasePassword:
default: Database Password
DatabaseUsername:
default: Database Username
Resources:
ParameterGroup:
Type: "AWS::RDS::DBParameterGroup"
Properties:
Description: testglobalaurora DB parameter group
Family: aurora5.6
Parameters:
max_connections: 300
DatabaseCluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora
EngineMode: global
MasterUsername:
Ref: DatabaseUsername
MasterUserPassword:
Ref: DatabasePassword
BackupRetentionPeriod: 35
PreferredBackupWindow: 02:00-03:00
PreferredMaintenanceWindow: mon:03:00-mon:04:00
VpcSecurityGroupIds:
- Ref: DatabaseSecurityGroup
DatabaseInstance:
Type: AWS::RDS::DBInstance
Properties:
Engine: aurora
EngineVersion : 5.6.10a
DBClusterIdentifier:
Ref: DatabaseCluster
DBInstanceClass:
Ref: DatabaseInstanceType
DBParameterGroupName: !Ref ParameterGroup
PubliclyAccessible: "true"
DBInstanceIdentifier: !Ref DatabaseName
DatabaseSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: vpc-55378f2f
GroupDescription: Access to database
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
FromPort: 3306
ToPort: 3306
IpProtocol: tcp
Tags:
- Key: Name
Value: !Sub ${DatabaseName}-security-group
Outputs:
DatabaseEndpoint:
Description: The database endpoint
Value: !GetAtt DatabaseCluster.Endpoint.Address
DatabasePort:
Description: The database port
Value: !GetAtt DatabaseCluster.Endpoint.Port
My output
"
global-database-1-cluster-1 Regional Aurora MySQL 5.6.10a
global-database-1-instance-1 Writer Aurora MySQL 5.6.10a
"
Actual ouput
"
test-it Global Aurora MySQL 5.6.10a
global-database-1-cluster-1 Primary Aurora MySQL 5.6.10a
global-database-1-instance-1 Writer Aurora MySQL 5.6.10a
"
I recently ran across the need to create a global RDS with Cloudformation. Here is a minimal Cloudformation that got me started.
AWSTemplateFormatVersion: "2010-09-09"
Description: Global RDS database stack
Parameters:
DatabaseInstanceType:
Default: db.r4.large
AllowedValues:
- db.r4.large
- db.r4. # add the other r4 instances
Description: "The instance type to use for the database."
Type: String
DatabasePassword:
Default: SomePassword1
AllowedPattern: "[a-zA-Z0-9]+"
ConstraintDescription: must contain only alphanumeric characters. Must have length 8-41.
Description: The database admin account password.
MaxLength: '41'
MinLength: '8'
NoEcho: 'true'
Type: String
DatabaseUsername:
Default: globaladmin
ConstraintDescription: must contain only alphanumeric characters. Must have length 1-16
Description: The database admin account user name.
MaxLength: '16'
MinLength: '1'
Type: String
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Database Configuration
Parameters:
- DatabaseInstanceType
- DatabaseName
- DatabaseUsername
- DatabasePassword
ParameterLabels:
DatabaseName:
default: Database name
DatabaseInstanceType:
default: Database Instance Type
DatabasePassword:
default: Database Password
DatabaseUsername:
default: Database Username
Resources:
GlobalDbCluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora
EngineMode: global
EngineVersion: 5.6.10a
MasterUsername: !Ref DatabaseUsername
MasterUserPassword: !Ref DatabasePassword
DBClusterParameterGroupName: !Ref GlobalDbParamGroup
GlobalDbParamGroup:
Type: AWS::RDS::DBClusterParameterGroup
Properties:
Description: "parameter group for the global database"
Family: aurora5.6
Parameters:
character_set_database: utf32
InstanceOne:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !Ref DatabaseInstanceType
DBClusterIdentifier: !Ref GlobalDbCluster
Engine: aurora
InstanceTwo:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !Ref DatabaseInstanceType
DBClusterIdentifier: !Ref GlobalDbCluster
Engine: aurora
You need to create a Global Cluster with an identifier, and use that identifier in your DB Cluster. That portion is missing in your CFN template.
Something like:
GlobalCluster:
Type: AWS::RDS::GlobalCluster
Properties:
Engine: aurora
EngineVersion: 5.6.10a
Region: us-east-1
and then use it in your DatabaseCluster properties using GlobalClusterIdentifier: <id>
However, looking at the official docs for the CFN types [1] for RDS, it does not list GlobalCluster. So either its just not documented or this Resource type has not been registered with Cloudformation. If the latter is the case, then you may want to open a support case and put in a feature request.
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_RDS.html

ComputeEnvironment went INVALID with error: The security group 'XXXX' does not exist

Below my cloudformation template:
I have added all the resource code, please excuse the indentation issue(copy paste thing), I assured you the template is running.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Sets up your AWS Batch Environment for running workflows
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Compute Environment Config
Parameters:
- ComputeEnvironmentName
- VpcId
- SubnetIds
- MinvCpus
- MaxvCpus
- DesiredvCpus
- Label:
default: Job Definition
Parameters:
- JobDefinitionName
- DockerImage
- Vcpus
- Memory
- Command
- RetryNumber
- Label:
default: Job Queue
Parameters:
- JobQueueName
Parameters:
VpcId:
Type: 'AWS::EC2::VPC::Id'
Description: >-
VpcId of where the whole batch should be deployed. The VPC should have
2 private subnets.
SubnetIds:
Type: List<AWS::EC2::Subnet::Id>
Description: Subnets you want your batch compute environment to launch in. Recommend private subnets
MinvCpus:
Type: String
Description: Minimum number of CPUs in the compute environment. Default 0.
Default: 0
AllowedPattern: "[0-9]+"
DesiredvCpus:
Type: String
Description: Desired number of CPUs in the compute environment to launch with. Default 0.
Default: 0
AllowedPattern: "[0-9]+"
MaxvCpus:
Type: String
Description: Maximum number of CPUs in the compute environment. Should be >= than MinCpus
Default: 256
AllowedPattern: "[0-9]+"
RetryNumber:
Type: String
Default: "1"
Description: Number of retries for each AWS Batch job. Integer required.
MaxLength: 1
AllowedPattern: "[1-9]"
ConstraintDescription: Value between 1 and 9
DockerImage:
Type: String
Description: Docker image used to run your jobs
Vcpus:
Type: Number
Description: vCPUs available to Jobs. Default is usually fine
Default: 2
Memory:
Type: Number
Description: Memory (in MB) available to Jobs. Default is usually fine
Default: 2000
JobQueueName:
Type: String
Description: Enter job queue Name
JobDefinitionName:
Type: String
Description: Enter JobDefinition Name for the batch
ComputeEnvironmentName:
Type: String
Description: Enter name of the Compute Environment
VPCCidr:
Type: String
Description: 'Cidr Block of the VPC, allows for ssh access internally.'
Default: '10.0.0.0/8'
MinLength: "9"
MaxLength: "18"
AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
ConstraintDescription: "Must be valid CIDR notation (i.e. x.x.x.x/x)."
Command:
Type: CommaDelimitedList
Description: The command that is passed to the container
CreateNewRepository:
Default: false
Description: >-
Set this to true if you want to create a new Repository, else
it will not create a new one
Type: String
AllowedValues:
- true
- false
RepositoryName:
Type: String
Description: Enter name of the new Repository.
Conditions:
CreateRepository: !Equals
- !Ref CreateNewRepository
- true
isCommandPresent: !Not [!Equals [!Ref CreateNewRepository, '']]
Resources:
JobDefinition:
Type: AWS::Batch::JobDefinition
Properties:
Type: container
JobDefinitionName: !Ref JobDefinitionName
ContainerProperties:
Image: !Ref DockerImage
Vcpus: !Ref Vcpus
Memory: !Ref Memory
Command: !Ref Command
ReadonlyRootFilesystem: true
Privileged: true
RetryStrategy:
Attempts: !Ref RetryNumber
JobQueue:
Type: AWS::Batch::JobQueue
Properties:
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment: !Ref MyComputeEnv
State: ENABLED
Priority: 10
JobQueueName: !Ref JobQueueName
myVPCSecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: Security group for batch process.
SecurityGroupEgress:
- CidrIp: 0.0.0.0/0
IpProtocol: '-1'
SecurityGroupIngress:
- CidrIp: !Ref VPCCidr
IpProtocol: tcp
FromPort: '22'
ToPort: '22'
VpcId: !Ref VpcId
MyComputeEnv:
Type: AWS::Batch::ComputeEnvironment
Properties:
Type: MANAGED
ServiceRole: !GetAtt awsBatchServiceRole.Arn
ComputeEnvironmentName: !Ref ComputeEnvironmentName
ComputeResources:
MinvCpus: !Ref MinvCpus
MaxvCpus: !Ref MaxvCpus
DesiredvCpus: !Ref DesiredvCpus
SecurityGroupIds: [!GetAtt myVPCSecurityGroup.GroupId]
Type: EC2
Subnets: !Ref SubnetIds
InstanceRole: !GetAtt InstanceProfile.Arn
InstanceTypes:
- optimal
State: ENABLED
awsBatchServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- "batch.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole
ecsInstanceRole:
Type: AWS::IAM::Role
Properties:
RoleName: InstanceRole
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
- "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: InstanceProfile
Roles:
- !Ref ecsInstanceRole
MyRepository:
Type: AWS::ECR::Repository
Condition: CreateRepository
Properties:
RepositoryName: !Ref RepositoryName
RepositoryPolicyText:
Version: "2012-10-17"
Statement:
-
Sid: AllowPushPull
Effect: Allow
Principal: "*"
Action:
- "ecr:*"
I am getting this error:
Operation failed, ComputeEnvironment went INVALID with error: CLIENT_ERROR - The security group 'sg-d9b85d91' does not exist
I don't know what is wrong with the code but strangely, the SecurityGroupIds created by myVPCSecurityGroup is sg-2869f263 but ComputeEnvironment is trying to find sg-d9b85d91.
taking a stab in the dark here just working for my mobile phone but I think it's because you don't have a V PC to your computer environment possibly
Disabling the Compute Environment in the UI and enabling it back again fixed the issue.

CloudFormation: simple example

I would like to build EC2 behind ELB(Elastic Load Balancer).
What would be the yaml/json code of doing this?
If you are looking for sample templates in json/yaml with Cloud Formation designer, you can use this sample templates provided by AWS.
Below is a sample CF template for simple 1 EC2/1 ELB stack
AWSTemplateFormatVersion: '2010-09-09'
Description: '1 EC2 Instance and 1 ELB'
Parameters:
AppServer:
Description: Hostname of Server
Type: String
Default: ec2instance01
MinLength: '1'
MaxLength: '16'
AllowedPattern: '[0-9a-zA-Z-]*'
ConstraintDescription: 'Must contain valid DNS characters, AD length limit.'
AMI:
Description: AMI to deploy AWSLinux Instances
Type: String
Default: ami-xxxxxxxx
InstanceType:
Description: Application EC2 instance type
Type: String
Default: t2.micro
AllowedValues:
- t2.micro
- t2.2xlarge
- m4.2xlarge
ConstraintDescription: Must be a valid EC2 instance type.
VPCID:
Description: Name of the VPC
Type: 'AWS::EC2::VPC::Id'
Default: vpc-xxxxxxxx
ConstraintDescription: Must be a valid VPC.
KeyName:
Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
Type: 'AWS::EC2::KeyPair::KeyName'
Default: XXXX-key
MinLength: '1'
MaxLength: '255'
AllowedPattern: '[\x20-\x7E]*'
ConstraintDescription: Must contain only ASCII characters.
SubnetIdPrivateEastC:
Description: Private subnet for confidential apps in us-east-1c
Type: 'AWS::EC2::Subnet::Id'
Default: subnet-xxxxxxxx
MinLength: '1'
MaxLength: '255'
ConstraintDescription: Must be a valid Private Subnet.
SubnetIdPrivateEastD:
Description: Private subnet for confidential apps in us-east-1d
Type: 'AWS::EC2::Subnet::Id'
Default: subnet-xxxxxxxx
MinLength: '1'
MaxLength: '255'
ConstraintDescription: Must be a valid Private Subnet.
InstanceProfile:
Description: Instance Profile Name
Type: String
Default: xxxx-role
MinLength: '0'
MaxLength: '255'
AllowedPattern: '[\x20-\x7E]*'
ConstraintDescription: Must contain a vailed instance profile name
RootVolumeSize:
Description: Size (GB) of root EBS volume for application instance
Type: Number
Default: '10'
MinValue: '10'
MaxValue: '1024'
SwapDisk:
Description: Size (GB) of application EBS volume for instance
Type: Number
Default: '2'
MinValue: '2'
MaxValue: '128'
SubnetAvailabilityZone:
Description: Availability Zone for subnet
Type: String
Default: us-east-1d
AllowedValues:
- us-east-1c
- us-east-1d
ConstraintDescription: Must be a valid Availability zone.
PrivateSubnets:
Type: List<AWS::EC2::Subnet::Id>
Description: 'Private subnet for the ELB in us-east-1c and us-east-1d'
Default: "subnet-xxxxxxxx,subnet-xxxxxxxx"
Resources:
ec2instance01:
Type: 'AWS::EC2::Instance'
Properties:
DisableApiTermination: 'true'
AvailabilityZone: us-east-1d
ImageId:
Ref: AMI
InstanceType:
Ref: InstanceType
KeyName:
Ref: KeyName
SecurityGroupIds:
- Ref: WebSG
IamInstanceProfile:
Ref: InstanceProfile
SubnetId:
Ref: SubnetIdPrivateEastD
#EbsOptimized: true
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeSize:
Ref: RootVolumeSize
VolumeType: gp2
- DeviceName: /dev/sds
Ebs:
VolumeSize:
Ref: SwapDisk
VolumeType: gp2
Tags:
- Key: Name
Value:
Ref: AppServer
UserData:
'Fn::Base64': !Sub |-
#!/bin/bash -v
yum update -y aws-cfn-bootstrap
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
date > /home/ec2-user/starttime
date > /home/ec2-user/stoptime
echo END
WebSG:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: Web SG
VpcId:
Ref: VPCID
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp: 10.0.0.0/8
Tags:
- Key: Name
Value: web_sg
ElbSG:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: ELB SG
VpcId:
Ref: VPCID
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '443'
ToPort: '443'
CidrIp: '0.0.0.0/0'
Tags:
- Key: Name
Value: elb_sg
ElasticLoadBalancer:
Type: AWS::ElasticLoadBalancing::LoadBalancer
DependsOn:
- ec2instance01
Properties:
LoadBalancerName: elb_01
SecurityGroups:
- Ref: ElbSG
Subnets: !Ref PrivateSubnets
Scheme: internal
Instances:
- Ref: ec2instance01
Listeners:
- LoadBalancerPort: '80'
InstancePort: '8080'
InstanceProtocol: HTTP
Protocol: HTTP
AccessLoggingPolicy:
EmitInterval: '60'
Enabled: 'False'
S3BucketName: elb-logs
S3BucketPrefix: ELB
HealthCheck:
Target: TCP:8080
HealthyThreshold: '5'
UnhealthyThreshold: '10'
Interval: '30'
Timeout: '5'
ConnectionDrainingPolicy:
Enabled: true
Timeout: '60'
Tags:
- Key: Name
Value: ELB_Name
Outputs:
ElbDNS:
Description: ELB DNS
Value:
'Fn::GetAtt':
- ElasticLoadBalancer
- DNSName
AppServerPrivateIP:
Description: Private IP address of instance ec2instance01
Value:
'Fn::GetAtt':
- ec2instance01
- PrivateIp
It looks like you are using the CloudFormation template designer. When you use the designer, it generates a CloudFormation template for you. You can see this by selecting the Template tab towards the bottom-left of the screen. You are also given the choice of JSON or YAML.