aws cloudformation elasticsearch stack - amazon-web-services

I have written the below template to pick an environment based upon the user input. But I am getting error as "An error occurred (ValidationError) when calling the CreateStack operation: Template format error: [/Resources/Type] resource definition is malformed" .Please guide me what need to be changed and whether syntax is in right format.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvironmentValue:
AllowedValues:
- PROD
- TEST
Description: 'Please select an Environment'
Type: String
Mappings:
Environment:
PROD:
VPC: vpc-xxxxxxxx
Subnets: 'subnet-xxxxx,subnet-xxxxx,subnet-xxxx'
Securitygroups: 'sg-xxxx,sg-xxxx'
TEST:
VPC: vpc-xxxxx
Subnets: 'subnet-xxxx,subnet-xxxxx'
Securitygroups: 'sg-xxxx,sg-xxxxx'
#Conditions:
# CreatePRODStack: !Equals [!Ref EnvironmentValue, PROD]
# CreateTESTStack: !Equals [!Ref EnvironmentValue, TEST]
Resources:
Type: 'AWS::Es:Domain'
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions: !FindInMap [Environment, !Ref 'EnvironmentValue', VPC]
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
Type: 'AWS::IAM::Policy'
Properties:
PolicyDocument: YAML
PolicyName: prodtest
When the user gives input as Prod, the stack for Prod should be created in Cloudformation

I'm seeing a few issues here:
1 - You haven't named your resources.
2 - Your indentinging looks incorrect, which is important for yaml
3 - I believe your Type for the Elasticsearch domain is incorrect. You have
Type: 'AWS::Es:Domain'
but I think it should be
Type: AWS::Elasticsearch::Domain
Using your Domain as an example, I think it should be more along the lines of:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions: !FindInMap [Environment, !Ref 'EnvironmentValue', VPC]
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
There may be other issues I'm missing here, but there are definitely syntax errors in here
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticsearch-domain.html

The type tag is expected to be 'AWS::Elasticsearch::Domain' and there are multiple formatting errors as per the yaml declarations. The Properties should be at the same level as Type. Then VPCOptions should have the two properties following it. With the given example the template should look like the following
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvironmentValue:
AllowedValues:
- PROD
- TEST
Description: 'Please select an Environment'
Type: String
Mappings:
Environment:
PROD:
VPC: vpc-xxxxxxxx
Subnets: 'subnet-xxxxx,subnet-xxxxx,subnet-xxxx'
Securitygroups: 'sg-xxxx,sg-xxxx'
TEST:
VPC: vpc-xxxxx
Subnets: 'subnet-xxxx,subnet-xxxxx'
Securitygroups: 'sg-xxxx,sg-xxxxx'
#Conditions:
# CreatePRODStack: !Equals [!Ref EnvironmentValue, PROD]
# CreateTESTStack: !Equals [!Ref EnvironmentValue, TEST]
Resources:
ElasticSearchCluster:
Type: 'AWS::Es:Domain'
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions:
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
IAMPolicyEntry:
Type: 'AWS::IAM::Policy'
Properties:
PolicyDocument: YAML
PolicyName: prodtest

Related

Aws batch : choose compute environnement on launch

I have a aws cloudformation template for my batch definition. I have 2 compute environnement définition, a on demand and a spot.
DragenComputeEnvironmentSpot:
Type: AWS::Batch::ComputeEnvironment
Properties:
ComputeEnvironmentName: dragen-spot
ServiceRole: !Ref BatchServiceRole
Type: MANAGED
State: ENABLED
ComputeResources:
BidPercentage: !Ref BidPercentage
Ec2KeyPair: !Ref Ec2KeyPair
ImageId: !Ref ImageId
InstanceRole: !Ref DragenInstanceRole
InstanceTypes: [!Ref InstanceType]
MinvCpus: !Ref MinvCpus
DesiredvCpus: !Ref DesiredvCpus
MaxvCpus: !Ref MaxvCpus
SecurityGroupIds:
- !Ref BatchSecurityGroup
SpotIamFleetRole: !Ref SpotFleetRole
Subnets: !Ref SubnetIds
Type: SPOT
LaunchTemplate:
LaunchTemplateId: !Ref DragenLaunchTemplate
DragenComputeEnvironmentOnDemand:
Type: AWS::Batch::ComputeEnvironment
Properties:
ComputeEnvironmentName: dragen-ondemand
ServiceRole: !Ref BatchServiceRole
Type: MANAGED
State: ENABLED
ComputeResources:
Ec2KeyPair: !Ref Ec2KeyPair
ImageId: !Ref ImageId
InstanceRole: !Ref DragenInstanceRole
InstanceTypes: [!Ref InstanceType]
MinvCpus: !Ref MinvCpus
DesiredvCpus: !Ref DesiredvCpus
MaxvCpus: !Ref MaxvCpus
SecurityGroupIds:
- !Ref BatchSecurityGroup
Subnets: !Ref SubnetIds
Type: EC2
LaunchTemplate:
LaunchTemplateId: !Ref DragenLaunchTemplate
Then in the jobqueue définition i have :
DragenJobQueue:
Type: AWS::Batch::JobQueue
Properties:
JobQueueName: dragen-queue
Priority: 100
State: ENABLED
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment: !Ref DragenComputeEnvironmentOnDemand
- Order: 2
ComputeEnvironment: !Ref DragenComputeEnvironmentSpot
So if i understand well the on demand has the priority (so spot will never be use). But i would like to know if there is a way to override during the job launch ? Some job fit well with on demand but others are better on spot.

Cloudformation's condition statement (Glue's subnet)

I need my Glue job to use specific subnet based on environment it is ran in. Below line SubnetId throws syntax error. I read in aws' doc that true/false evaluation can be addressed with !Ref, issue seems to be with syntax for condition.
SubnetId: !If [!Ref UseProdCondition, !Ref PrivateSubnetAz2, !Ref PrivateSubnetAz3]
GlueJDBCConnection:
Type: AWS::Glue::Connection
UseProdCondition: !Equals [!Ref "${AppEnv}", "production"]
Properties:
CatalogId: !Ref AWS::AccountId
ConnectionInput:
ConnectionType: "JDBC"
ConnectionProperties:
USERNAME: !Ref Username
PASSWORD: !Ref Password
JDBC_CONNECTION_URL: !Ref GlueJDBCStringTarget
sslMode: 'REQUIRED'
PhysicalConnectionRequirements:
AvailabilityZone:
Ref: AvailabilityZone2
SecurityGroupIdList:
- Fn::GetAtt: GlueJobSecurityGroup.GroupId
SubnetId: !If [!Ref UseProdCondition, !Ref PrivateSubnetAz2, !Ref PrivateSubnetAz3]
Name: !Ref JDBCConnectionName
Condition needs to be defined as a separate resource, later referenced in specific resource.
Thanks #MisterSmith!
AWSTemplateFormatVersion: 2010-09-09
Description: AWS Glue Spark Job
Conditions:
UseProdCondition: !Equals [!Ref AppEnv, "production"]
GlueJDBCConnection:
Type: AWS::Glue::Connection
Properties:
CatalogId: !Ref AWS::AccountId
ConnectionInput:
ConnectionType: "JDBC"
ConnectionProperties:
USERNAME: !Ref Username
PASSWORD: !Ref Password
JDBC_CONNECTION_URL: !Ref GlueJDBCStringTarget
sslMode: 'REQUIRED'
PhysicalConnectionRequirements:
AvailabilityZone:
Ref: AvailabilityZone2
SecurityGroupIdList:
- Fn::GetAtt: GlueJobSecurityGroup.GroupId
#SubnetId: !Ref PrivateSubnetAz2
SubnetId: !If [UseProdCondition, !Ref PrivateSubnetAz2, !Ref PrivateSubnetAz3]
Name: !Ref RTMIJDBCConnectionName

Cannot create Elasticsearch instance in AWS using Cloudformation

Im trying to create an Elasticsearch instance in AWS using cloudformation, but cannot create and shows this error,
Creating Elasticsearch Domain did not stabilize.,
Here is the cloudformation template,
Elasticsearch:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: !Ref ElasticsearchDomainName
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: '*'
Action:
- 'es:*'
Condition:
IpAddress:
'aws:SourceIp':
- 12.34.56.78/32
Resource: '*'
ElasticsearchVersion: !Ref ElasticsearchVersion
ElasticsearchClusterConfig:
InstanceCount: !Ref InstanceCount
InstanceType: t2.small.elasticsearch
EBSOptions:
EBSEnabled: true
VolumeSize: 25
NodeToNodeEncryptionOptions:
Enabled: true
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: !Ref "AWS::NoValue"
UpdatePolicy:
EnableVersionUpgrade: true
It seems this issue has been already mentioned here here , but it didn't worked for me
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
indices.fielddata.cache.size: ""
even adding that failing for me, What else be the issue,
Thanks in advance.
It worked for me with the following template:
Resources:
Elasticsearch:
Type: AWS::Elasticsearch::Domain
Properties:
AdvancedOptions:
indices.fielddata.cache.size: ''
rest.action.multi.allow_explicit_index: 'true'
DomainEndpointOptions:
CustomEndpoint: 'mycustomdomain.com'
CustomEndpointCertificateArn: '{{resolve:ssm:my-cert-arn:1}}'
CustomEndpointEnabled: true
EnforceHTTPS: true
DomainName: 'logs'
ElasticsearchVersion: 7.10
ElasticsearchClusterConfig:
InstanceCount: 1
InstanceType: t3.small.elasticsearch
EBSOptions:
EBSEnabled: true
VolumeSize: 25
UpdatePolicy:
EnableVersionUpgrade: true

CloudFormation Joining List of Strings with FindInMap

I am facing issue with Cloudformation on joining a list of strings after getting it from a map.
Error caused is Security Group expects a list of string.
"123456":
us-east-1:
#VPC1
VpcId: vpc-xxxx
VpcCidr: 10.78.160.0/20
SecurityGroups: sg-123
AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
"123457":
us-east-1:
#VPC
VpcId: vpc-xxxx
VpcCidr: 10.78.160.0/20
SecurityGroups: sg-123,sg-124
AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
LaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Metadata:
AWS::CloudFormation::Init:
config:
Properties:
InstanceType:
Ref: InstanceType
SecurityGroups: !Join [ ",", [!FindInMap [!Ref "AWS::AccountId", !Ref "AWS::Region", SecurityGroups], !Ref EC2SG ]]
IamInstanceProfile:
please let me know how to get the list for securitygroups and adding it to another resource
Solved by using this question
How do I use nested lists or append to a list in Cloudformation?
Lets take mappings:
Mappings:
"111222333444":
us-east-1:
SecurityGroups:
- sg-abc
- sg-xyz
us-west-2:
SecurityGroups:
- sg-1234
- sg-3456
Security groups can be referred as
SecurityGroupIds:
"Fn::FindInMap":
[!Ref AWS::AccountId, !Ref AWS::Region, SecurityGroups]
Define as array in mappings and pass the full array, this way we can avoid splitting and joining.
Full Template , tested on a Lambda
AWSTemplateFormatVersion: "2010-09-09"
Description: "Test"
Mappings:
"111222333444":
us-east-1:
SecurityGroups:
- sg-abc
- sg-xyz
us-west-2:
SecurityGroups:
- sg-1234
- sg-3456
Resources:
TestLambda:
Type: "AWS::Lambda::Function"
Properties:
Handler: com.test.MainClass::handleRequest
Runtime: java8
FunctionName: "Test-Lambda"
Code:
S3Bucket: code-bucket
S3Key: !Sub "artifacts/test.jar"
Description: "Test Lambda"
MemorySize: 512
Timeout: 60
Role: !Ref my-role-arn
VpcConfig:
SecurityGroupIds:
"Fn::FindInMap":
[!Ref AWS::AccountId, !Ref AWS::Region, SecurityGroups]
SubnetIds:
- subnet-1
- subnet-2

ReadEndpoint.Address was not found for DBCluster

I am adding route 53 to my DBCluster and keep running into the error: Attribute: ReadEndpoint.Address was not found for resource: <DBCluster-name>
The entire stack is created via cloudformation.
Also, it should be noted that this is for Serverless Aurora in case that matters?
Here is my code:
AWSTemplateFormatVersion: 2010-09-09
Description: RDS Aurora serverless template
Parameters:
CustomFunctionArn:
Default: arn:aws:lambda:us-west-2:123456789:function:vault-secrets-read-lambda-prod
Description: The ARN of the lambda function to retrieve password from Vault
Type: String
DBName:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: Name of the database
Type: String
DBMasterUsername:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: The master user name for the DB instance
Type: String
DBScalingAutoPauseEnabled:
AllowedValues:
- 'true'
- 'false'
Default: 'true'
Description: Pause all DB instances after some inactivity
Type: String
DBScalingMaxCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 8
Description: The maximum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingMinCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 2
Description: The minimum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingSecondsUntilAutoPause:
Default: 300
Description: Auto pause after consecutive seconds of inactivity
MinValue: 300
MaxValue: 86400
Type: Number
Env:
AllowedValues:
- prod
- qa
- dev
Type: String
Description: Environment
VaultPath:
Default: secret/dev/dbPassword
Type: String
SnapshotId:
Description: snapshot ID to restore DB cluster from
Type: String
Conditions:
EnableAutoPause:
!Equals [!Ref DBScalingAutoPauseEnabled, 'true']
DoNotUseSnapshot: !Equals
- !Ref SnapshotId
- ''
Mappings:
Configuration:
prod:
HostedZoneEnv: mydomain.com
HostedZoneId: 'XXX'
SecurityGroup: sg-123321
SubnetGroups:
- subnet-123
- subnet-456
- subnet-789
VPCId: vpc-555
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: prod
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
qa:
HostedZoneEnv: mydomain-qa.com
HostedZoneId: 'XXX'
SecurityGroup: sg-321123
SubnetGroups:
- subnet-098
- subnet-765
- subnet-432
VPCId: vpc-345543
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: qa
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
dev:
HostedZoneEnv: mydomain-dev.com
HostedZoneId: 'XXX'
SecurityGroup: sg-f3453f
SubnetGroups:
- subnet-dsf24327
- subnet-82542gsda
- subnet-casaf2344
VPCId: vpc-23dfsf
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: dev
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
Resources:
AuroraSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allows access to RDS
GroupName: !Sub '${AWS::StackName}-aurora-rds-${Env}'
SecurityGroupIngress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
FromPort: 5432
ToPort: 5432
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcId: !FindInMap [Configuration, !Ref Env, VPCId]
GetValuefromVault:
Type: Custom::CustomResource
Properties:
ServiceToken: !Ref CustomFunctionArn
VaultKeyPath: !Ref VaultPath
DBCluster:
Type: 'AWS::RDS::DBCluster'
DeletionPolicy: Snapshot
UpdateReplacePolicy: Snapshot
Properties:
BackupRetentionPeriod: 7
DBClusterParameterGroupName: default.aurora-postgresql10
DBSubnetGroupName: !Ref DBSubnetGroup
DatabaseName: !Ref DBName
DeletionProtection: false
# EnableHttpEndpoint: true
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: '10.7'
KmsKeyId: !If [DoNotUseSnapshot, !Ref KMSkey, !Ref 'AWS::NoValue']
MasterUserPassword: !If [DoNotUseSnapshot, !GetAtt 'GetValuefromVault.ValueFromVault', !Ref 'AWS::NoValue']
MasterUsername: !If [DoNotUseSnapshot, !Ref DBMasterUsername, !Ref 'AWS::NoValue']
Port: 5432
ScalingConfiguration:
AutoPause: !If [EnableAutoPause, true, false]
MaxCapacity: !Ref DBScalingMaxCapacity
MinCapacity: !Ref DBScalingMinCapacity
SecondsUntilAutoPause: !Ref DBScalingSecondsUntilAutoPause
SnapshotIdentifier: !If [DoNotUseSnapshot, !Ref 'AWS::NoValue', !Ref SnapshotId]
StorageEncrypted: true
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcSecurityGroupIds:
- !GetAtt [AuroraSG, GroupId]
- !FindInMap [Configuration, !Ref Env, SecurityGroup]
DBSubnetGroup:
Type: 'AWS::RDS::DBSubnetGroup'
Properties:
DBSubnetGroupDescription: !Sub '${AWS::StackName}-${Env}'
SubnetIds: !FindInMap [Configuration, !Ref Env, SubnetGroups]
Tags: !FindInMap [Configuration, !Ref Env, Tags]
KmsAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: !Sub 'alias/${AWS::StackName}-${Env}-aurora-rds'
TargetKeyId: !Ref KMSkey
KMSkey:
Type: AWS::KMS::Key
Properties:
KeyPolicy:
Id: key-consolepolicy-3
Version: 2012-10-17
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: 'kms:*'
Resource: '*'
RecordSet:
Type: AWS::Route53::RecordSet
DependsOn: DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -writer-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.Endpoint.Address
TTL: '60'
Type: CNAME
ReadRecordSet:
Type: 'AWS::Route53::RecordSet'
DependsOn:
- DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -reader-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.ReadEndpoint.Address
TTL: '60'
Type: CNAME
Outputs:
AuroraHost:
Value: !GetAtt [DBCluster, Endpoint.Address]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'Host' ]]
AuroraSG:
Value: !GetAtt AuroraSG.GroupId
Export:
Name: !Join [":", [ !Ref "AWS::StackName", AuroraSG ]]
KMS:
Value: !GetAtt [KMSkey, Arn]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'KMS' ]]
DNSName:
Description: 'The connection endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.Endpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-DNSName'
ReadDNSName:
Description: 'The reader endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.ReadEndpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-ReadDNSName'
Some things i have tried:
Create new stack: FAIL
Create new stack without ReadRecordSet: FAIL
Create new stack without RecordSet (old name for read recordset): FAIL
Create new stack without RecordSet (new name for read recordset): FAIL
Add a DependsOn to ReadRecordSet (for first RecordSet): FAIL
Enabling HTTP endpoint on Cluster: FAIL
Update TTL to 60: FAIL Update TTL to 0: FAIL
The RecordSet seems to be creating okay (I tested that by adding a DependsOn: - RecordSet in the ReadRecordSet to allow RecordSet to create first), so it's the ReadRecordSet that is failing and can't find ReadEndpoint.Address
Not sure what I am missing here, been googling like mad and don't see much about this error. Any help is appreciated!
It turns out that Aurora Serverless doesn't require ReadRecordSet, so that entire section is only applicable to provisioned DB, so ReadEndpoint doesn't exist indeed. Unfortunately AWS documentation doesn't mention that explicitly.