I am trying to deploy Tableau to the AWS gov-cloud and I made some edits to the template in order to make it ready for govCloud deployment, however when I upload my YAML file to AWS I keep getting the following error:
Template error: if specifying one argument to Fn::GetAtt, that argument must be a non-empty string in format <LogicalId>.<Attribute>
I didn't edit any of the !GetAtt stuff from the template and the template doesn't seem to follow that order so I'm confused as to why I would keep getting this error on upload. Below is my YAML file snippet I would isolate the error but really not sure what part is causing the issue.
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Template: Single-node Tableau Server running on Windows,
CentOS, or Ubuntu. (qs-1puphiil4)'
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: AWS Environment and Machine Configuration
Parameters:
- KeyPairName
- AvailabilityZones
- VPCCIDR
- PublicSubnet1CIDR
- PublicSubnet2CIDR
- SourceCIDR
- InstanceType
- AMIOS
- Label:
default: Secrets
Parameters:
- Username
- Password
- TableauServerAdminUser
- TableauServerAdminPassword
- Label:
default: Registration
Parameters:
- AcceptEULA
- TableauServerLicenseKey
- RegFirstName
- RegLastName
- RegEmail
- RegCompany
- RegTitle
- RegDepartment
- RegIndustry
- RegPhone
- RegCity
- RegState
- RegZip
- RegCountry
- Label:
default: AWS Quick Start Configuration
Parameters:
- QSS3BucketName
- QSS3KeyPrefix
ParameterLabels:
AcceptEULA:
default: Accept Tableau End User License Agreement
AvailabilityZones:
default: Availability Zones
AMIOS:
default: AMI Operating System
InstanceType:
default: Tableau Amazon EC2 instance type
KeyPairName:
default: Key Pair Name
Password:
default: Tableau Services Manager (TSM) administrator password
PublicSubnet1CIDR:
default: Public Subnet 1 CIDR
PublicSubnet2CIDR:
default: Public Subnet 2 CIDR
QSS3BucketName:
default: Quick Start S3 Bucket Name
QSS3KeyPrefix:
default: Quick Start S3 Key Prefix
RegCity:
default: City
RegCompany:
default: Company
RegCountry:
default: Country
RegDepartment:
default: Department
RegEmail:
default: Email Address
RegFirstName:
default: First Name
RegIndustry:
default: Industry
RegLastName:
default: Last Name
RegPhone:
default: Phone
RegState:
default: State
RegTitle:
default: Title
RegZip:
default: Zip/Postal Code
SourceCIDR:
default: Source CIDR for Access
TableauServerAdminPassword:
default: Tableau Server administrator password
TableauServerAdminUser:
default: Tableau Server administrator username
TableauServerLicenseKey:
default: Tableau Activation Key
Username:
default: Tableau Services Manager (TSM) administrator username
VPCCR:
default: VPC CIDR
Parameters:
AvailabilityZones:
Description: 'List of Availability Zones to use for the subnets in teh VPC, Note: The logical order is preseved and 2 AZs will be used for this deployment'
Type: List<AWS::EC2::AvailabilityZone::Name>
AMIOS:
AllowedValues:
- Windows
- CentOS-7-HVM
- Ubuntu-Server-16.04-LTS-HVM
Default: Ubuntu-Server-16.04-LTS-HVM
Description: Operating System on which Tableau Server will be deployed
Type: String
AcceptEULA:
AllowedPattern: 'yes'
AllowedValues:
- 'yes'
- 'No'
Description: 'View the EULA at the Link: https://www.tableau.com/eula'
Type: String
InstanceType:
AllowedValues:
- m4.2xlarge
- m4.4xlarge
- m4.10xlarge
- m5.4xlarge
- m5.12xlarge
- c5.4xlarge
- c4.4xlarge
- c5d.4xlarge
- r5d.4xlarge
ConstraintDescription: must be a valid EC2 instance type.
Default: m5.4xlarge
Description: Amazon EC2 instance type
Type: String
KeyPairName:
ConstraintDescription: must be the name of an existing EC2 KeyPair.
Description: Name of an existing EC2 KeyPair to enable SSH access to the instances
Type: AWS::EC2::KeyPair::KeyName
Password:
Description: Tableau Services Manager (TSM) administrator password
NoEcho: 'true'
Type: String
PublicSubnet1CIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.128.0/20
Description: CIDR Block for the Public DMZ Subnet located in AZ1
Type: String
PublicSubnet2CIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.144.0/20
Description: CIDR Block for the Public DMZ Subnet located in AZ2
Type: String
QSS3BucketName:
AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
ConstraintDescription: Quick Start bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
(-).
Default: aws-quickstart
Description: S3 bucket name for the Quick Start assets. This string can include
numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
or end with a hyphen (-).
Type: String
QSS3KeyPrefix:
AllowedPattern: ^[0-9a-zA-Z-/]*$
ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
uppercase letters, hyphens (-), and forward slash (/).
Default: quickstart-tableau-server/
Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
can include numbers, lowercase letters, uppercase letters, hyphens (-), and
forward slash (/).
Type: String
RegCity:
Description: City
Type: String
RegCompany:
Description: Company
Type: String
RegCountry:
Description: Country
Type: String
RegDepartment:
Description: Department
Type: String
RegEmail:
Description: Email
MinLength: '1'
Type: String
RegFirstName:
Description: First Name
MinLength: '1'
Type: String
RegIndustry:
Description: Industry
Type: String
RegLastName:
Description: Last Name
MinLength: '1'
Type: String
RegPhone:
Description: Phone
Type: String
RegState:
Description: State
Type: String
RegTitle:
Description: Title
Type: String
RegZip:
Description: ZIP/Postal Code
Type: String
SourceCIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/x
Description: The CIDR address from which you will connect to the instance
Type: String
TableauServerAdminPassword:
Description: The password of the initial administrator for Tableau Server
MinLength: '1'
NoEcho: 'true'
Type: String
TableauServerAdminUser:
Description: The name of the initial administrator for Tableau Server
MinLength: '1'
Type: String
TableauServerLicenseKey:
Description: License Key (leave blank for trial)
Type: String
Username:
AllowedPattern: ^(?!(tableau|tsmagent|admin|root)$)[A-Za-z0-9]*$
Description: Tableau Services Manager (TSM) administrator username (cannot be
'tableau' or 'tsmagent' or 'admin' or 'root')
MaxLength: '30'
Type: String
VPCCIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.0.0/16
Description: CIDR Block for the VPC
Type: String
Conditions:
InfaOnWindows: !Equals
- !Ref 'AMIOS'
- Windows
InfaOnCentos: !Equals
- !Ref 'AMIOS'
- CentOS-7-HVM
Resources:
VPCStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub 'https://${QSS3BucketName}.s3.amazonaws-us-gov.com/${QSS3KeyPrefix}submodules/quickstart-aws-vpc/templates/aws-vpc.template'
Parameters:
AvailabilityZones: !Join
- ','
- !Ref 'AvailabilityZones'
KeyPairName: !Ref 'KeyPairName'
NATInstanceType: t2.small
NumberOfAZs: '2'
PublicSubnet1CIDR: !Ref 'PublicSubnet1CIDR'
PublicSubnet2CIDR: !Ref 'PublicSubnet2CIDR'
CreatePrivateSubnets: 'false'
VPCCIDR: !Ref VPCCIDR
TableauIAMRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws-us-gov:iam::aws:policy/service-role/AmazonEC2RoleforSSM
Policies:
- PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- s3:GetObject
Resource:
- !Sub 'arn:aws-us-gov:s3:::${QSS3BucketName}/${QSS3KeyPrefix}*'
Effect: Allow
PolicyName: aws-quick-start-s3-policy
TableauServerInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref 'TableauIAMRole'
WorkloadStack:
Type: AWS::CloudFormation::Stack
DependsOn: VPCStack
Properties:
TemplateURL: !Sub
- https://${QSS3BucketName}.s3.amazonaws-us-gov.com/${QSS3KeyPrefix}templates/${QSTableauWorkloadTemplate}
- QSTableauWorkloadTemplate: !If
- InfaOnWindows
- tableau-single-server-windows.template
- !If
- InfaOnCentos
- tableau-single-server-centos.template
- tableau-single-server-ubuntu.template
Parameters:
AMIOS: !Ref 'AMIOS'
PublicSubnetId: !Ref 'VPCStack.Outputs.PublicSubnet1ID'
AcceptEULA: !Ref 'AcceptEULA'
InstanceType: !Ref 'InstanceType'
KeyPairName: !Ref 'KeyPairName'
Password: !Ref 'Password'
QSS3BucketName: !Ref 'QSS3BucketName'
QSS3KeyPrefix: !Ref 'QSS3KeyPrefix'
RegCity: !Ref 'RegCity'
RegCompany: !Ref 'RegCompany'
RegCountry: !Ref 'RegCountry'
RegDepartment: !Ref 'RegDepartment'
RegEmail: !Ref 'RegEmail'
RegFirstName: !Ref 'RegFirstName'
RegIndustry: !Ref 'RegIndustry'
RegLastName: !Ref 'RegLastName'
RegPhone: !Ref 'RegPhone'
RegState: !Ref 'RegState'
RegTitle: !Ref 'RegTitle'
RegZip: !Ref 'RegZip'
SourceCIDR: !Ref 'SourceCIDR'
TableauServerAdminPassword: !Ref 'TableauServerAdminPassword'
TableauServerAdminUser: !Ref 'TableauServerAdminUser'
TableauServerLicenseKey: !Ref 'TableauServerLicenseKey'
Username: !Ref 'Username'
VPCId: !GetAtt 'VPCID'
Outputs:
VPCID:
Description: VPC ID
Value: !GetAtt 'VPCStack.Outputs.VPCID'
InstanceID:
Description: EC2 InstanceID of the instance running Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.InstanceID'
PublicIPAddress:
Description: Public IP Address of instance running Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.PublicIPAddress'
TableauServicesManagerURL:
Description: URL for the TSM Web UI
Value: !GetAtt 'WorkloadStack.Outputs.TableauServicesManagerURL'
TableauServerURL:
Description: URL for the Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.TableauServerURL'
The CloudFormation Linter will also catch these errors faster with more informative messages like:
E1010 Invalid GetAtt for Resources/WorkloadStack/Properties/Parameters/VPCId/Fn::GetAtt
template.yaml:333:9
so we know the issue is with line 333:
VPCId: !GetAtt 'VPCID'
The Visual Studio Code extension can also highlight these inline in your templates:
AWS::CloudFormation::Stack return values
Fn::GetAtt documentation
It looks like this line has been updated in the source repository, but I'd recommend opening a Github issue there if you're still experiencing difficulties with their templates
I am trying to set up a tableau server on my AWS gov cloud account, for starters I created a tableau server on a typical AWS CloudFormation using the Quickstart guide found here
AWS Tableau Server Quickstart guide. I started off using this template template and hoped that I could copy YAML file over to the gov cloud, use the same template YAML and just have the server on the gov cloud however on deployment I got the following error:
Partition "aws" is not valid for resource "arn:aws:s3:::aws-quickstart/quickstart-tableau-server/*"
my guess is that somewhere in the YAML (shown below):
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Template: Single-node Tableau Server running on Windows,
CentOS, or Ubuntu. (qs-1puphiilp)'
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: AWS Environment and Machine Configuration
Parameters:
- KeyPairName
- AvailabilityZones
- PublicSubnet1CIDR
- PublicSubnet2CIDR
- VPCCIDR
- SourceCIDR
- InstanceType
- AMIOS
- Label:
default: Secrets
Parameters:
- Username
- Password
- TableauServerAdminUser
- TableauServerAdminPassword
- Label:
default: Registration
Parameters:
- AcceptEULA
- TableauServerLicenseKey
- RegFirstName
- RegLastName
- RegEmail
- RegCompany
- RegTitle
- RegDepartment
- RegIndustry
- RegPhone
- RegCity
- RegState
- RegZip
- RegCountry
- Label:
default: AWS Quick Start Configuration
Parameters:
- QSS3BucketName
- QSS3KeyPrefix
ParameterLabels:
AvailabilityZones:
default: Availability Zones
AcceptEULA:
default: Accept Tableau End User License Agreement
AMIOS:
default: AMI Operating System
InstanceType:
default: Tableau Amazon EC2 instance type
KeyPairName:
default: Key Pair Name
Password:
default: Tableau Services Manager (TSM) administrator password
PublicSubnet1CIDR:
default: Public Subnet 1 CIDR
PublicSubnet2CIDR:
default: Public Subnet 2 CIDR
QSS3BucketName:
default: Quick Start S3 Bucket Name
QSS3KeyPrefix:
default: Quick Start S3 Key Prefix
RegCity:
default: City
RegCompany:
default: Company
RegCountry:
default: Country
RegDepartment:
default: Department
RegEmail:
default: Email Address
RegFirstName:
default: First Name
RegIndustry:
default: Industry
RegLastName:
default: Last Name
RegPhone:
default: Phone
RegState:
default: State
RegTitle:
default: Title
RegZip:
default: Zip/Postal Code
SourceCIDR:
default: Source CIDR for Access
TableauServerAdminPassword:
default: Tableau Server administrator password
TableauServerAdminUser:
default: Tableau Server administrator username
TableauServerLicenseKey:
default: Tableau Activation Key
Username:
default: Tableau Services Manager (TSM) administrator username
VPCCIDR:
default: VPC CIDR
Parameters:
AvailabilityZones:
Description: 'List of Availability Zones to use for the subnets in the VPC. Note:
The logical order is preserved and 2 AZs will be used for this deployment'
Type: List<AWS::EC2::AvailabilityZone::Name>
AMIOS:
AllowedValues:
- Windows
- CentOS-7-HVM
- Ubuntu-Server-16.04-LTS-HVM
Default: CentOS-7-HVM
Description: Operating System on which Tableau Server will be deployed
Type: String
AcceptEULA:
AllowedPattern: 'yes'
AllowedValues:
- 'yes'
- 'no'
Description: 'View the EULA at the Link: https://www.tableau.com/eula'
Type: String
InstanceType:
AllowedValues:
- m4.2xlarge
- m4.4xlarge
- m4.10xlarge
- m5.4xlarge
- m5.12xlarge
- c5.4xlarge
- c4.4xlarge
- c5d.4xlarge
- r5d.4xlarge
ConstraintDescription: must be a valid EC2 instance type.
Default: m5.4xlarge
Description: Amazon EC2 instance type
Type: String
KeyPairName:
ConstraintDescription: must be the name of an existing EC2 KeyPair.
Description: Name of an existing EC2 KeyPair to enable SSH access to the instances
Type: AWS::EC2::KeyPair::KeyName
Password:
Description: Tableau Services Manager (TSM) administrator password
NoEcho: 'true'
Type: String
PublicSubnet1CIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.128.0/20
Description: CIDR Block for the Public DMZ Subnet located in AZ1
Type: String
PublicSubnet2CIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.144.0/20
Description: CIDR Block for the Public DMZ Subnet located in AZ2
Type: String
QSS3BucketName:
AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
ConstraintDescription: Quick Start bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
(-).
Default: aws-quickstart
Description: S3 bucket name for the Quick Start assets. This string can include
numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
or end with a hyphen (-).
Type: String
QSS3KeyPrefix:
AllowedPattern: ^[0-9a-zA-Z-/]*$
ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
uppercase letters, hyphens (-), and forward slash (/).
Default: quickstart-tableau-server/
Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
can include numbers, lowercase letters, uppercase letters, hyphens (-), and
forward slash (/).
Type: String
RegCity:
Description: City
Type: String
RegCompany:
Description: Company
Type: String
RegCountry:
Description: Country
Type: String
RegDepartment:
Description: Department
Type: String
RegEmail:
Description: Email
MinLength: '1'
Type: String
RegFirstName:
Description: First Name
MinLength: '1'
Type: String
RegIndustry:
Description: Industry
Type: String
RegLastName:
Description: Last Name
MinLength: '1'
Type: String
RegPhone:
Description: Phone
Type: String
RegState:
Description: State
Type: String
RegTitle:
Description: Title
Type: String
RegZip:
Description: ZIP/Postal Code
Type: String
SourceCIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/x
Description: The CIDR address from which you will connect to the instance
Type: String
TableauServerAdminPassword:
Description: The password of the initial administrator for Tableau Server
MinLength: '1'
NoEcho: 'true'
Type: String
TableauServerAdminUser:
Description: The name of the initial administrator for Tableau Server
MinLength: '1'
Type: String
TableauServerLicenseKey:
Description: License Key (leave blank for trial)
Type: String
Username:
AllowedPattern: ^(?!(tableau|tsmagent|admin|root)$)[A-Za-z0-9]*$
Description: Tableau Services Manager (TSM) administrator username (cannot be
'tableau' or 'tsmagent' or 'admin' or 'root')
MaxLength: '30'
Type: String
VPCCIDR:
AllowedPattern: '[a-zA-Z0-9]+\..+'
Default: 10.0.0.0/16
Description: CIDR Block for the VPC
Type: String
Conditions:
InfaOnWindows: !Equals
- !Ref 'AMIOS'
- Windows
InfaOnCentos: !Equals
- !Ref 'AMIOS'
- CentOS-7-HVM
Resources:
VPCStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub 'https://${QSS3BucketName}.s3.amazonaws.com/${QSS3KeyPrefix}submodules/quickstart-aws-vpc/templates/aws-vpc.template'
Parameters:
AvailabilityZones: !Join
- ','
- !Ref 'AvailabilityZones'
KeyPairName: !Ref 'KeyPairName'
NATInstanceType: t2.small
NumberOfAZs: '2'
PublicSubnet1CIDR: !Ref 'PublicSubnet1CIDR'
PublicSubnet2CIDR: !Ref 'PublicSubnet2CIDR'
CreatePrivateSubnets: 'false'
VPCCIDR: !Ref 'VPCCIDR'
WorkloadStack:
Type: AWS::CloudFormation::Stack
DependsOn:
- VPCStack
Properties:
TemplateURL: !Sub 'https://${QSS3BucketName}.s3.amazonaws.com/${QSS3KeyPrefix}templates/tableau-single-server.template'
Parameters:
AcceptEULA: !Ref 'AcceptEULA'
AMIOS: !Ref 'AMIOS'
InstanceType: !Ref 'InstanceType'
KeyPairName: !Ref 'KeyPairName'
Password: !Ref 'Password'
PublicSubnetID: !GetAtt 'VPCStack.Outputs.PublicSubnet1ID'
QSS3BucketName: !Ref 'QSS3BucketName'
QSS3KeyPrefix: !Ref 'QSS3KeyPrefix'
RegCity: !Ref 'RegCity'
RegCompany: !Ref 'RegCompany'
RegCountry: !Ref 'RegCountry'
RegDepartment: !Ref 'RegDepartment'
RegEmail: !Ref 'RegEmail'
RegFirstName: !Ref 'RegFirstName'
RegIndustry: !Ref 'RegIndustry'
RegLastName: !Ref 'RegLastName'
RegPhone: !Ref 'RegPhone'
RegState: !Ref 'RegState'
RegTitle: !Ref 'RegTitle'
RegZip: !Ref 'RegZip'
SourceCIDR: !Ref 'SourceCIDR'
TableauServerAdminPassword: !Ref 'TableauServerAdminPassword'
TableauServerAdminUser: !Ref 'TableauServerAdminUser'
TableauServerLicenseKey: !Ref 'TableauServerLicenseKey'
Username: !Ref 'Username'
VPCID: !GetAtt 'VPCStack.Outputs.VPCID'
Outputs:
VPCID:
Description: VPC ID
Value: !GetAtt 'VPCStack.Outputs.VPCID'
InstanceID:
Description: EC2 InstanceID of the instance running Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.InstanceID'
PublicIPAddress:
Description: Public IP Address of instance running Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.PublicIPAddress'
TableauServicesManagerURL:
Description: URL for the TSM Web UI
Value: !GetAtt 'WorkloadStack.Outputs.TableauServicesManagerURL'
TableauServerURL:
Description: URL for the Tableau Server
Value: !GetAtt 'WorkloadStack.Outputs.TableauServerURL'
there is some sort of hardcoding that is not allowing for govcloud configuration but I have yet to be able to glean where exactly that is
Since that parent template is using nested CloudFormation stacks, the child stack templates also need to be correct:
https://aws-quickstart.s3.amazonaws.com/quickstart-tableau-server/submodules/quickstart-aws-vpc/templates/aws-vpc.template
https://aws-quickstart.s3.amazonaws.com/quickstart-tableau-server/templates/tableau-single-server.template
Line 253 of tableau-single-server.template has that hardcoded ARN partition:
- !Sub 'arn:aws:s3:::${QSS3BucketName}/${QSS3KeyPrefix}*'
Line 245 also has another hardcoded ARN partition:
- arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
These lines should probably use the AWS::Partition psuedoparameter instead
https://github.com/aws-quickstart/quickstart-tableau-server/issues/53
"AWS Partition" (the second element in the ARN id) for US Gov cloud is: aws-us-gov, not aws.
So in all you ARNs, where you see something like arn:aws:s3:::aws-quickstart... you need to use arn:aws-us-gov:s3:::aws-quickstart... instead.
I have been attempting to set up a cloud formation script to create a VPC hosting fragate containers and a aurora DB. When attempting to deploy my aurora script I receive the following.
The DB instance and EC2 security group are in different VPCs. The DB instance is in vpc-f0ec9d98 and the EC2 security group is in vpc-01c5e9bcdb87dc39c (Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterCombination; Request ID: 7aa14530-d73c-4b27-a6d6-fcc8aea61d93)
I do not understand why this is the case as I am using the same security group created by my VPC script, my aurora script is as follows
Aurora
Description: Set up a serverles PostgreSQL cluster with a bastion host (using Aurora)
Parameters:
DatabaseName:
Type: String
EngineVersion:
Type: String
Default: 11.4
MasterUsername:
Type: String
Default: root
MasterUserPassword:
Type: String
Default: root
NoEcho: true
VpcId:
Type: AWS::EC2::VPC::Id
VpcSecurityGroupId:
Type: AWS::EC2::SecurityGroup::Id
BastionImageId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-ebs
BastionKeyName:
Type: AWS::EC2::KeyPair::KeyName
Description: EC2 key used to connect to the bastion host
DeletionProtection:
Type: String
Default: false
AllowedValues:
- true
- false
Resources:
Cluster:
Type: AWS::RDS::DBCluster
Properties:
Engine: aurora-postgresql
EngineVersion: !Ref EngineVersion
DatabaseName: !Ref DatabaseName
MasterUsername: !Ref MasterUsername
MasterUserPassword: !Ref MasterUserPassword
DBClusterIdentifier: !Ref AWS::StackName
BackupRetentionPeriod: 35
DeletionProtection: !Ref DeletionProtection
VpcSecurityGroupIds:
- !Ref VpcSecurityGroupId
BastionSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Sub Bastion for ${AWS::StackName}
SecurityGroupEgress:
- CidrIp: 0.0.0.0/0
FromPort: -1
ToPort: -1
IpProtocol: -1
- DestinationSecurityGroupId: !Ref VpcSecurityGroupId
IpProtocol: tcp
FromPort: 3306
ToPort: 3306
SecurityGroupIngress: []
VpcId: !Ref VpcId
Bastion:
Type: AWS::EC2::Instance
Properties:
DisableApiTermination: true
ImageId: !Ref BastionImageId
InstanceType: t2.nano
KeyName: !Ref BastionKeyName
Monitoring: false
SecurityGroupIds:
- !Ref VpcSecurityGroupId
- !Ref BastionSecurityGroup
UserData: !Base64 'yum install postgresql --assumeyes' # if this script does not work this line broke it
Outputs:
Host:
Value: !GetAtt Cluster.Endpoint.Address
Export:
Name: !Sub ${AWS::StackName}Host
Name:
Value: !Ref DatabaseName
Export:
Name: !Sub ${AWS::StackName}Name
BastionHost:
Value: !GetAtt Bastion.PublicDnsName
Export:
Name: !Sub ${AWS::StackName}BastionHost
BastionIp:
Value: !GetAtt Bastion.PublicIp
Export:
Name: !Sub ${AWS::StackName}BastionIp
BastionSecurityGroupId:
Value: !GetAtt BastionSecurityGroup.GroupId
Export:
Name: !Sub ${AWS::StackName}BastionSecurityGroupId
Without the inclusion of the DBSubnetGroupName property in the AWS::RDS::DBCluster resource, it looks like CloudFormation is attempting to launch the cluster in the default VPC. A DB subnet group allows you to specify a particular VPC when you create DB instances.
Try adding this property and referencing an associated subnet parameter/resource and the issue should be resolved.
Information about creating RDS instances within a VPC can be found in the RDS User Guide.
I am creating a cloud formation template to provision elasticsearch service domain in AWS.
I would like to set this property under Encryption to true
"Require HTTPS for all traffic to the domain" but I am not able to find the way in AWS docs to do so.
Other options for setting encryption properties like
"Enable encryption of data at rest" & "Node-to-node encryption" are well documented.
Does anyone know how to set "Require HTTPS for all traffic to the domain" property from CF template ?
The way I did this was to make sure the security group only allows HTTPS (443) access to the cluster.
Not completely sure if this is what you are looking for, but if it's not, give me some more details and I will see if I can help you out.
mySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VpcId
GroupName: !Ref SecurityGroup
GroupDescription: !Ref GroupDescription
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
Here is what I have put together for Encryption at Rest as well as Node to node encryption:
Now, I had to add/edit plenty of moving parts, so I will post the whole template here so you can see what I did in the different sections so you can pull out what you need/want. I used Conditionals since you can chose to enable it or not. I also made sure to add a few comments here and there. Feel free to strip those if you want, but they do not impact the template at all.
edit: I know that there are several parts of the template that are not NEEDED, but I like to make mine look pretty and organized (when being used). I know I need to work on organization inside the template itself, but it's all there. :P
edit2: The below template assumes you will be using Existing VPC/Subnets/Security Groups, you can check out my repo (in comment below) for other versions of this template.
I did validate this template and was able to successfully build the Domain.
AWSTemplateFormatVersion: 2010-09-09
Description: >-
**NOTE** In order to create Elastisearch Domain in AWS using CloudFormation
verify you have the following Service Role created in IAM!!
-- AWSServiceRoleForAmazonElasticsearchService --
If you do not have this Role, create it using the following CLi Command --
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: "Configure Cluster"
Parameters:
- DomainName
- ElasticsearchVersion
- ZoneAwareness
- SnapShotHour
-
Label:
default: "Data Instances"
Parameters:
- InstanceType
- DataInstanceCount
-
Label:
default: "Dedicated Master Instances"
Parameters:
- DedicatedMaster
- MasterInstanceType
- MasterInstanceCount
-
Label:
default: "Storage Config"
Parameters:
- StorageSize
-
Label:
default: "Network Config"
Parameters:
- VpcId
- SubNet1
- SubNet2
- SecurityGroup
-
Label:
default: "Encryption Settings"
Parameters:
- EncryptionAtRest
- KmsKey
- NodetoNode
-
Label:
default: "IAM User Restriction Policy"
Parameters:
- IamUserArn
ParameterLabels:
DomainName:
default: "Name of the ElasticSearch Domain (lowecase, no spaces) - If you don't specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the domain name"
ElasticsearchVersion:
default: "Select the ElasticSearch version desired"
InstanceType:
default: "Instance Size of Data Instances"
DataInstanceCount:
default: "Number of Data Instances Required"
DedicatedMaster:
default: "Select if a Dedicated Master Instance is required"
MasterInstanceType:
default: "Instance Size of Master Instances"
MasterInstanceCount:
default: "How many Dedicated Master Instances are needed? (0, 3 or 5)"
StorageSize:
default: "Storage Size in GB"
VpcId:
default: "Select the VPC to deploy into (must already exist)"
SubNet1:
default: "Select the First Subnet"
SubNet2:
default: "Select the Second Subnet"
SecurityGroup:
default: "Select the Security Group"
IamUserArn:
default: "Enter the ARN for the IAM User to give initial access to the stack"
ZoneAwareness:
default: "Enable Zone Awareness (Availability Zone Replication) (recommended)"
SnapShotHour:
default: "Set the hour to run the Automated Snapshot (0-23) (Default: UTC Timezone)"
EncryptionAtRest:
default: "Enable Encryption at Rest"
KmsKey:
default: "If Encryption at Rest is enabled, supply the KMS Key used for encryption"
NodetoNode:
default: "Enable Node to Node Encryption"
Parameters:
DomainName:
Type: String
Default: "elasticsearchstack-cf"
MaxLength: '128'
ConstraintDescription: "Must be lowercase, numbers/letters and/or a dash"
SnapShotHour:
Type: Number
Default: 0
MinValue: 0
MaxValue: 23
ElasticsearchVersion:
Type: String
Default: 7.1
AllowedValues: [7.1, 6.8, 6.7, 6.6, 6.5] # Remove this line for free-form number entry
InstanceType:
Type: String
Default: r5.large.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
DataInstanceCount:
Type: Number
Default: 2
AllowedValues: [2, 4, 6, 8, 10] # Remove this line for free-form number entry
MasterInstanceType:
Type: String
Default: r5.large.elasticsearch
AllowedValues: [t2.small.elasticsearch, t2.medium.elasticsearch,
c4.large.elasticsearch, c4.xlarge.elasticsearch, c4.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c4.8xlarge.elasticsearch,
c5.large.elasticsearch, c5.xlarge.elasticsearch, c5.2xlarge.elasticsearch, c5.4xlarge.elasticsearch, c5.9xlarge.elasticsearch, c5.18xlarge.elasticsearch,
m3.medium.elasticsearch, m3.large.elasticsearch, m3.xlarge.elasticsearch, m3.2xlarge.elasticsearch,
m4.large.elasticsearch, m4.xlarge.elasticsearch, m4.2xlarge.elasticsearch, m4.4xlarge.elasticsearch, m4.10xlarge.elasticsearch,
m5.large.elasticsearch, m5.xlarge.elasticsearch, m5.2xlarge.elasticsearch, m5.4xlarge.elasticsearch, m5.12xlarge.elasticsearch,
r3.large.elasticsearch, r3.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r3.8xlarge.elasticsearch,
r4.large.elasticsearch, r4.xlarge.elasticsearch, r4.2xlarge.elasticsearch, r4.4xlarge.elasticsearch, r4.16xlarge.elasticsearch,
r5.large.elasticsearch, r5.xlarge.elasticsearch, r5.2xlarge.elasticsearch, r5.4xlarge.elasticsearch, r5.12xlarge.elasticsearch,
i2.xlarge.elasticsearch, i2.2xlarge.elasticsearch,
i3.large.elasticsearch, i3.xlarge.elasticsearch, i3.2xlarge.elasticsearch, i3.4xlarge.elasticsearch, i3.8xlarge.elasticsearch, i3.16xlarge.elasticsearch]
ConstraintDescription: "Must be a valid EC2 Elasticsearch instance type."
MasterInstanceCount:
Type: Number
Default: 0
AllowedValues: [0, 3, 5] # Remove this line for free-form number entry
VpcId:
Type: AWS::EC2::VPC::Id
ConstraintDescription: "Must be the VPC ID of an existing Virtual Private Cloud."
SubNet1:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SubNet2:
Type: AWS::EC2::Subnet::Id
ConstraintDescription: "Must be the Subnet ID of an existing Subnet."
SecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
ConstraintDescription: "Must be and existing Security Group."
DedicatedMaster:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
StorageSize:
Type: Number
Default: 20
MinValue: 10 # Remove this line for free-form number entry (suggested to keep this line)
MaxValue: 1000 # Remove this line for free-form number entry
IamUserArn:
Type: String
Default: "arn:aws:iam::<AccountNumber>:user/<username>"
ZoneAwareness:
Description: True or False
Type: String
Default: True
AllowedValues:
- True
- False
EncryptionAtRest:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
KmsKey:
Type: String
NodetoNode:
Description: True or False
Type: String
Default: False
AllowedValues:
- True
- False
Conditions: # Checks to see if Conditional Values are True
DedicatedMasterYes: !Equals [ !Ref DedicatedMaster, True]
EncryptionAtRestYes: !Equals [ !Ref EncryptionAtRest, True]
Resources:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref DomainName
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
DedicatedMasterEnabled: !Ref DedicatedMaster
InstanceCount: !Ref DataInstanceCount
ZoneAwarenessEnabled: !Ref ZoneAwareness
InstanceType: !Ref InstanceType
DedicatedMasterType: # If Dedicated Master is True, then use !Ref, if not, use NoValue (NULL) for both settings below
!If [DedicatedMasterYes, !Ref MasterInstanceType, !Ref "AWS::NoValue"]
DedicatedMasterCount:
!If [DedicatedMasterYes, !Ref MasterInstanceCount, !Ref "AWS::NoValue"]
EBSOptions:
EBSEnabled: True
Iops: 0
VolumeSize: !Ref StorageSize
VolumeType: "gp2"
SnapshotOptions:
AutomatedSnapshotStartHour: !Ref SnapShotHour
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: !Ref IamUserArn
Action: 'es:*'
Resource: "*"
AdvancedOptions: # BOTH of these settingsd are REQUIRED (regardless of what the documentation states) - Bug filed: https://forums.aws.amazon.com/thread.jspa?messageID=768527
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
VPCOptions:
SubnetIds:
- !Ref SubNet1
- !Ref SubNet2
SecurityGroupIds:
- !Ref SecurityGroup
EncryptionAtRestOptions: # If Encryption At Rest is True, then use !Ref, if not, use NoValue (NULL) for both settings below
!If [EncryptionAtRestYes, !Ref EncryptionAtRest, !Ref "AWS::NoValue"]
KmsKeyId:
!If [EncryptionAtRestYes, !Ref KmsKey, !Ref "AWS::NoValue"]
NodeToNodeEncryptionOptions:
Enabled: !Ref NodetoNode
Outputs:
DomainArn:
Value: !GetAtt ElasticsearchDomain.DomainArn
DomainEndpoint:
Value: !GetAtt ElasticsearchDomain.DomainEndpoint
SecurityGroupId:
Value: !Ref SecurityGroup
SubnetId1:
Value: !Ref SubNet1
SubnetId2:
Value: !Ref SubNet2
As feature was recently released, it is not yet available in CloudFormation or Terraform, But we can achieve same by using custom resource in a CF template. This custom resource is nothing but a lambda function which will be executed as part cloud formation stack creation.
complete automation can be divided into 3 section,
Lambda function development
Lambda function creation
Lambda function configuration as custom resource in Cloudformation stack
Lambda function development
As explained here follow below steps to create lambda fuction.
Create an empty folder that you’ll use to place your Lambda source. Then use pip to install crhelper into the folder, and create the lambda_function.py file to put your resource code into.
mkdir sum_function
cd sum_function
pip install -t . crhelper
# on some systems pip may fail with a distutils error, if you run into this, try
running pip with the –system argument
# pip install —system -t . crhelper
touch lambda_function.py
Now open the lambda_function.py file and place the following code into the lambda_function.py file.
from crhelper import CfnResource
import boto3
import logging
logger = logging.getLogger(__name__)
helper = CfnResource()
#helper.create
#helper.update
def update_es_domain_config(event, context):
logger.info("Updating elasticsearch domain config with HTTPS")
es = boto3.client('es')
response = es.update_elasticsearch_domain_config(
DomainName=event['ResourceProperties']['ESDomainName'],
DomainEndpointOptions={
'EnforceHTTPS': True,
'TLSSecurityPolicy': 'Policy-Min-TLS-1-0-2019-07'
}
)
logger.info("results:", response)
#helper.delete
def no_op(_, __):
pass
def handler(event, context):
helper(event, context)
After that zip the whole directory
zip -r ../sum.zip ./
Lambda function creation
Now create the lambda function, I am using CLI for the same. Replace the values according to your environment.
aws lambda create-function \
--function-name "update-es-domain" \
--handler "lambda_function.handler" \
--timeout 900 \
--zip-file fileb://../sum.zip \
--runtime python3.7 \
--role "arn:aws:iam::123412341234:role/lambda-cli-role"
You’ll need to take down the FunctionArn from the output to use in the next step.
Lambda function configuration as custom resource in Cloudformation stack
Now In Resources section of your CF template, after Elasticsearch resource, add custom resource as shown below. Replace the service token with the functionArn you get in last step as a output. Replace ESDomainName with the elasticsearch domain that you want to modify.
ESDomainUpdate:
Type: "Custom::elasticsearch"
Properties:
ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
ESDomainName: "advantagehelp"
PFB Complete Cloudformation template for reference.
AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation templates to create AWS Elasticsearch Service domain.
Parameters:
NodeType:
Description: The node type to be provisioned for the Elasticsearch cluster
Type: String
Default: m4.large.elasticsearch
AllowedValues:
- m4.large.elasticsearch
- m4.xlarge.elasticsearch
- c4.large.elasticsearch
- c4.xlarge.elasticsearch
- r4.large.elasticsearch
- r4.xlarge.elasticsearch
ConstraintDescription: must be a valid Elasticsearch node type.
NodeCount:
Description: The number of nodes in the Elasticsearch cluster.
Type: Number
Default: '1'
AllowedESGetIpList:
Type: CommaDelimitedList
Description: List of ip and cidr for elasticsearch retrieval
AllowedESUpdateIpList:
Type: CommaDelimitedList
Description: List of ip and cidr for elasticsearch update
ESDomainName:
Type: String
Description: Name of Elastic Search Domain
Default: advantagehelp
Resources:
ElasticsearchDomain:
Type: 'AWS::Elasticsearch::Domain'
Metadata:
cfn_nag:
rules_to_suppress:
- id: W28
reason: "Needed to name elasticsearch domain"
Properties:
DomainName: !Ref ESDomainName
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: !Ref NodeCount
ZoneAwarenessEnabled: false
InstanceType: !Ref NodeType
ElasticsearchVersion: '6.4'
AccessPolicies:
Version: 2012-10-17
Statement:
- Action:
- 'es:ESHttpGet'
- 'es:ESHttpHead'
Principal: '*'
Effect: Allow
Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
Condition:
IpAddress:
'aws:SourceIp': !Ref AllowedESGetIpList
- Action:
- 'es:ESHttpGet'
- 'es:ESHttpHead'
- 'es:ESHttpPost'
- 'es:ESHttpPut'
Principal: '*'
Effect: Allow
Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ESDomainName}/*
Condition:
IpAddress:
'aws:SourceIp': !Ref AllowedESUpdateIpList
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 10
VolumeType: gp2
SnapshotOptions:
AutomatedSnapshotStartHour: 1
AdvancedOptions:
rest.action.multi.allow_explicit_index: "true"
ESDomainUpdate:
Type: "Custom::elasticsearch"
Properties:
ServiceToken: "arn:aws:lambda:us-east-1:123412341234:function:update-es-domain"
ESDomainName: "advantagehelp"
DependsOn: ElasticsearchDomain
Outputs:
KibanaURL:
Description: Kibana URL
Value: !Join
- ''
- - !GetAtt
- ElasticsearchDomain
- DomainEndpoint
- /_plugin/kibana/
ElasticsearchEndpoint:
Description: Elasticsearch domain endpoint
Value: !GetAtt
- ElasticsearchDomain
- DomainEndpoint
ElasticsearchDomainARN:
Description: Elasticsearch domain ARN
Value: !GetAtt
- ElasticsearchDomain
- DomainArn