Im trying to create an Elasticsearch instance in AWS using cloudformation, but cannot create and shows this error,
Creating Elasticsearch Domain did not stabilize.,
Here is the cloudformation template,
Elasticsearch:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref ElasticsearchDomainName
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: '*'
Action:
- 'es:*'
Condition:
IpAddress:
'aws:SourceIp':
- 12.34.56.78/32
Resource: '*'
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
InstanceCount: !Ref InstanceCount
InstanceType: t2.small.elasticsearch
EBSOptions:
EBSEnabled: true
VolumeSize: 25
NodeToNodeEncryptionOptions:
Enabled: true
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
UpdatePolicy:
EnableVersionUpgrade: true
It seems this issue has been already mentioned here here , but it didn't worked for me
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: ""
even adding that failing for me, What else be the issue,
Thanks in advance.
It worked for me with the following template:
Resources:
Elasticsearch:
Type: AWS::Elasticsearch::Domain
Properties:
AdvancedOptions:
indices.fielddata.cache.size: ''
rest.action.multi.allow_explicit_index: 'true'
DomainEndpointOptions:
CustomEndpoint: 'mycustomdomain.com'
CustomEndpointCertificateArn: '{{resolve:ssm:my-cert-arn:1}}'
CustomEndpointEnabled: true
EnforceHTTPS: true
DomainName: 'logs'
ElasticsearchVersion: 7.10
ElasticsearchClusterConfig:
InstanceCount: 1
InstanceType: t3.small.elasticsearch
EBSOptions:
EBSEnabled: true
VolumeSize: 25
UpdatePolicy:
EnableVersionUpgrade: true
Related
I need to create a VPC + RDS Postgres DB + RDS Proxy with private subnets, I'm setting up everything with CloudFormation using serverless.
I'm able to configure everything except the RDS Proxy, when I run serverless deploy the deployment hangs when creating the Target group and eventually shows a timeout error.
This is what I see in the AWS Console, even if I do the full process manually:
When I run aws rds describe-db-proxy-targets --db-proxy-name so-proxy-rds-db-proxy this is what I get:
{
"Targets": [
{
"Endpoint": "so-proxy-rds-db.csy8ozys6dtv.us-west-2.rds.amazonaws.com",
"RdsResourceId": "so-proxy-rds-db",
"Port": 5432,
"Type": "RDS_INSTANCE",
"Role": "READ_WRITE",
"TargetHealth": {
"State": "UNAVAILABLE",
"Reason": "AUTH_FAILURE",
"Description": "Proxy does not have any registered credentials"
}
}
]
}
However I'm able to see them in the Secrets Manager. This is the reproducible configuration:
service: so-proxy-rds
frameworkVersion: "=2.49.0"
variablesResolutionMode: 20210326
configValidationMode: error
provider:
name: aws
runtime: nodejs14.x
region: us-west-2
versionFunctions: false
memorySize: 1024
timeout: 30
resources:
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Name
Value: vpc
PrivateSubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: VPC
CidrBlock: 10.0.2.0/24
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: ""
Tags:
- Key: Name
Value: private-A
PrivateSubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId:
Ref: VPC
CidrBlock: 10.0.3.0/24
AvailabilityZone:
Fn::Select:
- 1
- Fn::GetAZs: ""
Tags:
- Key: Name
Value: private-B
PrivateRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: private
PrivateSubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId:
Ref: PrivateSubnetA
RouteTableId:
Ref: PrivateRouteTable
PrivateSubnetBRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId:
Ref: PrivateSubnetB
RouteTableId:
Ref: PrivateRouteTable
OpenSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: 'Open firewall'
GroupName: ${self:service}-open
SecurityGroupEgress:
- CidrIp: 0.0.0.0/0
IpProtocol: "-1"
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
IpProtocol: "-1"
VpcId:
Ref: VPC
DBSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: DB subnet group
SubnetIds:
- Ref: PrivateSubnetA
- Ref: PrivateSubnetB
PostgresDB:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceIdentifier: db
DBName: test_db
AllocatedStorage: 20
DBInstanceClass: db.t2.micro
DBSubnetGroupName:
Ref: DBSubnetGroup
Engine: postgres
EngineVersion: 11.12 # only up to version 11 supports proxy
MasterUsername: test_user
MasterUserPassword: test_pass
PubliclyAccessible: false
VPCSecurityGroups:
- Ref: OpenSecurityGroup
DeletionPolicy: Delete
ProxySecretValues:
Type: 'AWS::SecretsManager::Secret'
Properties:
Name: proxy-secrets
SecretString: '{"username":"test_user","password":"test_pass"}'
DBProxy:
Type: AWS::RDS::DBProxy
Properties:
DBProxyName: db-proxy
EngineFamily: POSTGRESQL
RoleArn:
Fn::GetAtt:
- DBProxyRole
- Arn
Auth:
- AuthScheme: SECRETS
IAMAuth: DISABLED
SecretArn:
Ref: ProxySecretValues
VpcSubnetIds:
- Ref: PrivateSubnetA
- Ref: PrivateSubnetB
VpcSecurityGroupIds:
- Ref: OpenSecurityGroup
DBProxyTargetGroup:
Type: "AWS::RDS::DBProxyTargetGroup"
Properties:
DBInstanceIdentifiers:
- Ref: PostgresDB
DBProxyName:
Ref: DBProxy
TargetGroupName: default
DBProxyRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- "rds.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- Ref: DBProxyPolicy
RoleName: db-proxy-role
DBProxyPolicy:
Type: "AWS::IAM::ManagedPolicy"
Properties:
ManagedPolicyName: db-proxy-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "secretsmanager:*"
Resource: '*'
- Effect: Allow
Action:
- "kms:*"
Resource: '*'
Any help is greatly appreciated, thanks.
I dont see the role for roleArn entry declared in your "DBProxy" resource.
See the documentation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbproxy.html?icmpid=docs_cfn_console_designer
Take note of the "RoleArn" in the CF snippet below:
"Resources": {
"TestDBProxy": {
"Type": "AWS::RDS::DBProxy",
"Properties": {
"DebugLogging": true,
"DBProxyName": {
"Ref": "ProxyName"
},
"EngineFamily": "MYSQL",
"IdleClientTimeout": 120,
"RequireTLS": true,
"RoleArn": {
"Ref": "BootstrapSecretReaderRoleArn"
},
"Auth": [
{
"AuthScheme": "SECRETS",
"SecretArn": {
"Ref": "BootstrapProxySecretArn"
},
"IAMAuth": "DISABLED"
}
],
"VpcSubnetIds": {
"Fn::Split": [
",",
{
"Ref": "SubnetIds"
}
]
}
}
}
I'm unable to replicate the same error you had but I had the same error msg and managed to resolve it by going to Cloudwatch service and search for 'rds/proxy' log group.
you will able to get a more granular error message.
The output for aws rds describe-db-proxy-targets --db-proxy-name so-proxy-rds-db-proxy was very misleading.
In my case, I never gave rds permission to assume my proxy role. It can be quickly fixed with
Statement:
- Effect: Allow
Principal:
Service:
- "rds.amazonaws.com"
Action:
- "sts:AssumeRole"
hope this will be helpful.
Everything works perfectly in the code below if run without the 4 lines starting NotificationConfiguration . I thought this might be because of needing the topic policy before setting notification on the bucket. So have tried to do the initial create without the NotificationConfiguration lines and then add these in and update the stack.
But get the error Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; . I've tried things like putting the actual topic arn not using !Ref but no joy. Thanks!
Resources:
DeletionSNSTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName:
!Join [" ",[Data has been deleted from,!Sub '${ServiceName}-${Stage}-${AWS::AccountId}']
]
Subscription:
- Endpoint: !Sub '${DeleteNotifyEmail}'
Protocol: email
TopicName: !Sub 'delete-from-${ServiceName}-bucket'
DataBucket:
Type: AWS::S3::Bucket
DependsOn: DeletionSNSTopic
Description: Create Amazon S3 bucket from CloudFormation
Properties:
BucketName: !Sub '${ServiceName}-${Stage}-${AWS::AccountId}'
AccessControl: Private
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
NotificationConfiguration:
TopicConfigurations:
- Topic: !Ref DeletionSNSTopic
Event: 's3:ObjectRemoved:*'
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: 'deletionTopicPolicy'
Version: '2012-10-17'
Statement:
- Sid: 'deletionTopic-statement-id'
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sns:Publish
Resource: !Ref DeletionSNSTopic
Condition:
StringEquals:
aws:SourceAccount: !Sub '${AWS::AccountId}'
ArnLike:
aws:SourceArn: !Ref DataBucket
Topics:
- !Ref DeletionSNSTopic
You have circular dependency in your code. You create bucket with notifications, before topic policy is applied. Obviously the policy can't be created before the bucket because the bucket must already exist due to !Ref DataBucket.
To solve that the bucket name must be known first, which in your case is possible:
Resources:
DeletionSNSTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName:
!Join [" ",[Data has been deleted from,!Sub '${ServiceName}-${Stage}-${AWS::AccountId}']
]
Subscription:
- Endpoint: !Sub '${DeleteNotifyEmail}'
Protocol: email
TopicName: !Sub 'delete-from-${ServiceName}-bucket'
DataBucket:
Type: AWS::S3::Bucket
DependsOn: BucketToSNSPermission
Description: Create Amazon S3 bucket from CloudFormation
Properties:
BucketName: !Sub '${ServiceName}-${Stage}-${AWS::AccountId}'
AccessControl: Private
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
NotificationConfiguration:
TopicConfigurations:
- Topic: !Ref DeletionSNSTopic
Event: 's3:ObjectRemoved:*'
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: 'deletionTopicPolicy'
Version: '2012-10-17'
Statement:
- Sid: 'deletionTopic-statement-id'
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: SNS:Publish
Resource: !Ref DeletionSNSTopic
Condition:
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
ArnLike:
aws:SourceArn: !Sub "arn:aws:s3:::${ServiceName}-${Stage}-${AWS::AccountId}"
Topics:
- !Ref DeletionSNSTopic
For general case check in:
How do I avoid the "Unable to validate the following destination configurations" error in AWS CloudFormation?
I have a serverless app where I would like to deploy an elasticsearch cluster. I have configured it like this:
PostsSearch:
Type: AWS::Elasticsearch::Domain
Properties:
ElasticsearchVersion: '6.3'
DomainName: images-search-${self:provider.stage}
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: 1
ZoneAwarenessEnabled: false
InstanceType: t2.small.elasticsearch
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 10
VolumeType: 'gp2'
AccessPolicies:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
AWS: '*'
Action: 'es:ESHttp*'
Resource: '*'
But, I get an error:
An error occurred: PostsSearch - Enable fine-grained access control or
apply a restrictive access policy to your domain (Service:
AWSElasticsearch; Status Code: 400; Error Code: ValidationException;
Request ID: be0eca95-23ae-4ac9-be81-67cab37ccd70; Proxy: null).
How should I fix this?
Based on the extra discussion in the comments.
It is not possible to make an ES domain totally public. CloudFormation will not allow for that. Thus, there are three options
to choose from. Below I present three of them with in a sample serverless application. This is just basic hello-world
application, it does not use the ES domain in any capacity, but I use it to verify that each choice works and
can be deployed using serverless framework without errors.
Apply IP-based condition
This will make your domain open for access to only individual IP address or IP CIDR range.
The example below limits access to one, single IP address.
service: estest
provider:
name: aws
runtime: python3.8
stage: ${opt:stage, 'dev'}
region: ${opt:region, 'us-east-1'}
functions:
hello:
handler: handler.hello
resources:
Resources:
PostsSearch:
Type: AWS::Elasticsearch::Domain
Properties:
ElasticsearchVersion: '6.3'
DomainName: images-search-${self:provider.stage}
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: 1
ZoneAwarenessEnabled: false
InstanceType: t2.small.elasticsearch
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 10
VolumeType: 'gp2'
AccessPolicies:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
AWS: '*'
Action: 'es:ESHttp*'
Resource: !Sub "arn:aws:es:${self:provider.region}:${AWS::AccountId}:domain/images-search-${self:provider.stage}/*"
Condition:
IpAddress:
aws:SourceIp: ["12.13.14.15"]
Restrict principal
You can restrict access to your ES domain to selected IAM user or role. This way, only the given
IAM user/role will be able to access the ES domain. In the below I use lambda existing IAM role
as a principle. The function and its role must already exist.
service: estest
provider:
name: aws
runtime: python3.8
stage: ${opt:stage, 'dev'}
region: ${opt:region, 'us-east-1'}
functions:
hello:
handler: handler.hello
resources:
Resources:
PostsSearch:
Type: AWS::Elasticsearch::Domain
Properties:
ElasticsearchVersion: '6.3'
DomainName: images-search-${self:provider.stage}
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: 1
ZoneAwarenessEnabled: false
InstanceType: t2.small.elasticsearch
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 10
VolumeType: 'gp2'
AccessPolicies:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: 'es:ESHttp*'
Principal:
AWS: !Sub "arn:aws:iam::${AWS::AccountId}:role/service-role/lambda-function-es-role-b44mvudf"
Resource: !Sub "arn:aws:es:${self:provider.region}:${AWS::AccountId}:domain/images-search-${self:provider.stage}/*"
Use fine-grained access control
The example here creates publicly accessible ES domain with fine-grained controls that
requires username and password. This does not work in free-tier. I also
hard-coded username and password, which obviously would need to be modified and
provided as a parameter from from SSM Parameter store in real application.
service: estest
provider:
name: aws
runtime: python3.8
stage: ${opt:stage, 'dev'}
region: ${opt:region, 'us-east-1'}
functions:
hello:
handler: handler.hello
resources:
Resources:
PostsSearch:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: images-search-${self:provider.stage}
AccessPolicies: !Sub |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "*"
}
]
}
AdvancedSecurityOptions:
Enabled: true
InternalUserDatabaseEnabled: true
MasterUserOptions:
MasterUserName: admin
MasterUserPassword: fD343sfdf!3rf
EncryptionAtRestOptions:
Enabled: true
NodeToNodeEncryptionOptions:
Enabled: true
DomainEndpointOptions:
EnforceHTTPS: true
EBSOptions:
EBSEnabled: true
VolumeSize: 20
VolumeType: gp2
ElasticsearchClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: 1
InstanceType: c4.large.elasticsearch
ZoneAwarenessEnabled: false
ElasticsearchVersion: 7.7
I am adding route 53 to my DBCluster and keep running into the error: Attribute: ReadEndpoint.Address was not found for resource: <DBCluster-name>
The entire stack is created via cloudformation.
Also, it should be noted that this is for Serverless Aurora in case that matters?
Here is my code:
AWSTemplateFormatVersion: 2010-09-09
Description: RDS Aurora serverless template
Parameters:
CustomFunctionArn:
Default: arn:aws:lambda:us-west-2:123456789:function:vault-secrets-read-lambda-prod
Description: The ARN of the lambda function to retrieve password from Vault
Type: String
DBName:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: Name of the database
Type: String
DBMasterUsername:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: The master user name for the DB instance
Type: String
DBScalingAutoPauseEnabled:
AllowedValues:
- 'true'
- 'false'
Default: 'true'
Description: Pause all DB instances after some inactivity
Type: String
DBScalingMaxCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 8
Description: The maximum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingMinCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 2
Description: The minimum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingSecondsUntilAutoPause:
Default: 300
Description: Auto pause after consecutive seconds of inactivity
MinValue: 300
MaxValue: 86400
Type: Number
Env:
AllowedValues:
- prod
- qa
- dev
Type: String
Description: Environment
VaultPath:
Default: secret/dev/dbPassword
Type: String
SnapshotId:
Description: snapshot ID to restore DB cluster from
Type: String
Conditions:
EnableAutoPause:
!Equals [!Ref DBScalingAutoPauseEnabled, 'true']
DoNotUseSnapshot: !Equals
- !Ref SnapshotId
- ''
Mappings:
Configuration:
prod:
HostedZoneEnv: mydomain.com
HostedZoneId: 'XXX'
SecurityGroup: sg-123321
SubnetGroups:
- subnet-123
- subnet-456
- subnet-789
VPCId: vpc-555
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: prod
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
qa:
HostedZoneEnv: mydomain-qa.com
HostedZoneId: 'XXX'
SecurityGroup: sg-321123
SubnetGroups:
- subnet-098
- subnet-765
- subnet-432
VPCId: vpc-345543
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: qa
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
dev:
HostedZoneEnv: mydomain-dev.com
HostedZoneId: 'XXX'
SecurityGroup: sg-f3453f
SubnetGroups:
- subnet-dsf24327
- subnet-82542gsda
- subnet-casaf2344
VPCId: vpc-23dfsf
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: dev
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
Resources:
AuroraSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allows access to RDS
GroupName: !Sub '${AWS::StackName}-aurora-rds-${Env}'
SecurityGroupIngress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
FromPort: 5432
ToPort: 5432
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcId: !FindInMap [Configuration, !Ref Env, VPCId]
GetValuefromVault:
Type: Custom::CustomResource
Properties:
ServiceToken: !Ref CustomFunctionArn
VaultKeyPath: !Ref VaultPath
DBCluster:
Type: 'AWS::RDS::DBCluster'
DeletionPolicy: Snapshot
UpdateReplacePolicy: Snapshot
Properties:
BackupRetentionPeriod: 7
DBClusterParameterGroupName: default.aurora-postgresql10
DBSubnetGroupName: !Ref DBSubnetGroup
DatabaseName: !Ref DBName
DeletionProtection: false
# EnableHttpEndpoint: true
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: '10.7'
KmsKeyId: !If [DoNotUseSnapshot, !Ref KMSkey, !Ref 'AWS::NoValue']
MasterUserPassword: !If [DoNotUseSnapshot, !GetAtt 'GetValuefromVault.ValueFromVault', !Ref 'AWS::NoValue']
MasterUsername: !If [DoNotUseSnapshot, !Ref DBMasterUsername, !Ref 'AWS::NoValue']
Port: 5432
ScalingConfiguration:
AutoPause: !If [EnableAutoPause, true, false]
MaxCapacity: !Ref DBScalingMaxCapacity
MinCapacity: !Ref DBScalingMinCapacity
SecondsUntilAutoPause: !Ref DBScalingSecondsUntilAutoPause
SnapshotIdentifier: !If [DoNotUseSnapshot, !Ref 'AWS::NoValue', !Ref SnapshotId]
StorageEncrypted: true
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcSecurityGroupIds:
- !GetAtt [AuroraSG, GroupId]
- !FindInMap [Configuration, !Ref Env, SecurityGroup]
DBSubnetGroup:
Type: 'AWS::RDS::DBSubnetGroup'
Properties:
DBSubnetGroupDescription: !Sub '${AWS::StackName}-${Env}'
SubnetIds: !FindInMap [Configuration, !Ref Env, SubnetGroups]
Tags: !FindInMap [Configuration, !Ref Env, Tags]
KmsAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: !Sub 'alias/${AWS::StackName}-${Env}-aurora-rds'
TargetKeyId: !Ref KMSkey
KMSkey:
Type: AWS::KMS::Key
Properties:
KeyPolicy:
Id: key-consolepolicy-3
Version: 2012-10-17
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: 'kms:*'
Resource: '*'
RecordSet:
Type: AWS::Route53::RecordSet
DependsOn: DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -writer-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.Endpoint.Address
TTL: '60'
Type: CNAME
ReadRecordSet:
Type: 'AWS::Route53::RecordSet'
DependsOn:
- DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -reader-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.ReadEndpoint.Address
TTL: '60'
Type: CNAME
Outputs:
AuroraHost:
Value: !GetAtt [DBCluster, Endpoint.Address]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'Host' ]]
AuroraSG:
Value: !GetAtt AuroraSG.GroupId
Export:
Name: !Join [":", [ !Ref "AWS::StackName", AuroraSG ]]
KMS:
Value: !GetAtt [KMSkey, Arn]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'KMS' ]]
DNSName:
Description: 'The connection endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.Endpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-DNSName'
ReadDNSName:
Description: 'The reader endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.ReadEndpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-ReadDNSName'
Some things i have tried:
Create new stack: FAIL
Create new stack without ReadRecordSet: FAIL
Create new stack without RecordSet (old name for read recordset): FAIL
Create new stack without RecordSet (new name for read recordset): FAIL
Add a DependsOn to ReadRecordSet (for first RecordSet): FAIL
Enabling HTTP endpoint on Cluster: FAIL
Update TTL to 60: FAIL Update TTL to 0: FAIL
The RecordSet seems to be creating okay (I tested that by adding a DependsOn: - RecordSet in the ReadRecordSet to allow RecordSet to create first), so it's the ReadRecordSet that is failing and can't find ReadEndpoint.Address
Not sure what I am missing here, been googling like mad and don't see much about this error. Any help is appreciated!
It turns out that Aurora Serverless doesn't require ReadRecordSet, so that entire section is only applicable to provisioned DB, so ReadEndpoint doesn't exist indeed. Unfortunately AWS documentation doesn't mention that explicitly.
I have written the below template to pick an environment based upon the user input. But I am getting error as "An error occurred (ValidationError) when calling the CreateStack operation: Template format error: [/Resources/Type] resource definition is malformed" .Please guide me what need to be changed and whether syntax is in right format.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvironmentValue:
AllowedValues:
- PROD
- TEST
Description: 'Please select an Environment'
Type: String
Mappings:
Environment:
PROD:
VPC: vpc-xxxxxxxx
Subnets: 'subnet-xxxxx,subnet-xxxxx,subnet-xxxx'
Securitygroups: 'sg-xxxx,sg-xxxx'
TEST:
VPC: vpc-xxxxx
Subnets: 'subnet-xxxx,subnet-xxxxx'
Securitygroups: 'sg-xxxx,sg-xxxxx'
#Conditions:
# CreatePRODStack: !Equals [!Ref EnvironmentValue, PROD]
# CreateTESTStack: !Equals [!Ref EnvironmentValue, TEST]
Resources:
Type: 'AWS::Es:Domain'
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions: !FindInMap [Environment, !Ref 'EnvironmentValue', VPC]
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
Type: 'AWS::IAM::Policy'
Properties:
PolicyDocument: YAML
PolicyName: prodtest
When the user gives input as Prod, the stack for Prod should be created in Cloudformation
I'm seeing a few issues here:
1 - You haven't named your resources.
2 - Your indentinging looks incorrect, which is important for yaml
3 - I believe your Type for the Elasticsearch domain is incorrect. You have
Type: 'AWS::Es:Domain'
but I think it should be
Type: AWS::Elasticsearch::Domain
Using your Domain as an example, I think it should be more along the lines of:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions: !FindInMap [Environment, !Ref 'EnvironmentValue', VPC]
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
There may be other issues I'm missing here, but there are definitely syntax errors in here
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticsearch-domain.html
The type tag is expected to be 'AWS::Elasticsearch::Domain' and there are multiple formatting errors as per the yaml declarations. The Properties should be at the same level as Type. Then VPCOptions should have the two properties following it. With the given example the template should look like the following
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvironmentValue:
AllowedValues:
- PROD
- TEST
Description: 'Please select an Environment'
Type: String
Mappings:
Environment:
PROD:
VPC: vpc-xxxxxxxx
Subnets: 'subnet-xxxxx,subnet-xxxxx,subnet-xxxx'
Securitygroups: 'sg-xxxx,sg-xxxx'
TEST:
VPC: vpc-xxxxx
Subnets: 'subnet-xxxx,subnet-xxxxx'
Securitygroups: 'sg-xxxx,sg-xxxxx'
#Conditions:
# CreatePRODStack: !Equals [!Ref EnvironmentValue, PROD]
# CreateTESTStack: !Equals [!Ref EnvironmentValue, TEST]
Resources:
ElasticSearchCluster:
Type: 'AWS::Es:Domain'
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions:
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
IAMPolicyEntry:
Type: 'AWS::IAM::Policy'
Properties:
PolicyDocument: YAML
PolicyName: prodtest