I need my Glue job to use specific subnet based on environment it is ran in. Below line SubnetId throws syntax error. I read in aws' doc that true/false evaluation can be addressed with !Ref, issue seems to be with syntax for condition.
SubnetId: !If [!Ref UseProdCondition, !Ref PrivateSubnetAz2, !Ref PrivateSubnetAz3]
GlueJDBCConnection:
Type: AWS::Glue::Connection
UseProdCondition: !Equals [!Ref "${AppEnv}", "production"]
Properties:
CatalogId: !Ref AWS::AccountId
ConnectionInput:
ConnectionType: "JDBC"
ConnectionProperties:
USERNAME: !Ref Username
PASSWORD: !Ref Password
JDBC_CONNECTION_URL: !Ref GlueJDBCStringTarget
sslMode: 'REQUIRED'
PhysicalConnectionRequirements:
AvailabilityZone:
Ref: AvailabilityZone2
SecurityGroupIdList:
- Fn::GetAtt: GlueJobSecurityGroup.GroupId
SubnetId: !If [!Ref UseProdCondition, !Ref PrivateSubnetAz2, !Ref PrivateSubnetAz3]
Name: !Ref JDBCConnectionName
Condition needs to be defined as a separate resource, later referenced in specific resource.
Thanks #MisterSmith!
AWSTemplateFormatVersion: 2010-09-09
Description: AWS Glue Spark Job
Conditions:
UseProdCondition: !Equals [!Ref AppEnv, "production"]
GlueJDBCConnection:
Type: AWS::Glue::Connection
Properties:
CatalogId: !Ref AWS::AccountId
ConnectionInput:
ConnectionType: "JDBC"
ConnectionProperties:
USERNAME: !Ref Username
PASSWORD: !Ref Password
JDBC_CONNECTION_URL: !Ref GlueJDBCStringTarget
sslMode: 'REQUIRED'
PhysicalConnectionRequirements:
AvailabilityZone:
Ref: AvailabilityZone2
SecurityGroupIdList:
- Fn::GetAtt: GlueJobSecurityGroup.GroupId
#SubnetId: !Ref PrivateSubnetAz2
SubnetId: !If [UseProdCondition, !Ref PrivateSubnetAz2, !Ref PrivateSubnetAz3]
Name: !Ref RTMIJDBCConnectionName
Related
I have a aws cloudformation template for my batch definition. I have 2 compute environnement définition, a on demand and a spot.
DragenComputeEnvironmentSpot:
Type: AWS::Batch::ComputeEnvironment
Properties:
ComputeEnvironmentName: dragen-spot
ServiceRole: !Ref BatchServiceRole
Type: MANAGED
State: ENABLED
ComputeResources:
BidPercentage: !Ref BidPercentage
Ec2KeyPair: !Ref Ec2KeyPair
ImageId: !Ref ImageId
InstanceRole: !Ref DragenInstanceRole
InstanceTypes: [!Ref InstanceType]
MinvCpus: !Ref MinvCpus
DesiredvCpus: !Ref DesiredvCpus
MaxvCpus: !Ref MaxvCpus
SecurityGroupIds:
- !Ref BatchSecurityGroup
SpotIamFleetRole: !Ref SpotFleetRole
Subnets: !Ref SubnetIds
Type: SPOT
LaunchTemplate:
LaunchTemplateId: !Ref DragenLaunchTemplate
DragenComputeEnvironmentOnDemand:
Type: AWS::Batch::ComputeEnvironment
Properties:
ComputeEnvironmentName: dragen-ondemand
ServiceRole: !Ref BatchServiceRole
Type: MANAGED
State: ENABLED
ComputeResources:
Ec2KeyPair: !Ref Ec2KeyPair
ImageId: !Ref ImageId
InstanceRole: !Ref DragenInstanceRole
InstanceTypes: [!Ref InstanceType]
MinvCpus: !Ref MinvCpus
DesiredvCpus: !Ref DesiredvCpus
MaxvCpus: !Ref MaxvCpus
SecurityGroupIds:
- !Ref BatchSecurityGroup
Subnets: !Ref SubnetIds
Type: EC2
LaunchTemplate:
LaunchTemplateId: !Ref DragenLaunchTemplate
Then in the jobqueue définition i have :
DragenJobQueue:
Type: AWS::Batch::JobQueue
Properties:
JobQueueName: dragen-queue
Priority: 100
State: ENABLED
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment: !Ref DragenComputeEnvironmentOnDemand
- Order: 2
ComputeEnvironment: !Ref DragenComputeEnvironmentSpot
So if i understand well the on demand has the priority (so spot will never be use). But i would like to know if there is a way to override during the job launch ? Some job fit well with on demand but others are better on spot.
All that I am trying to do is follow this guide to get Atlassian services (Jira/JSM/Confluence) up and running on AWS https://aws-quickstart.github.io/quickstart-atlassian-jira/
I have followed all the details precisely and am currently at the point where I need to create the stack on cloudformation. When I paste in the .yaml url I get the error in the title, pointing me to an issue with the resources. I have looked through it and I'm not sure what the problem is, help would be appreciated.
Resources:
VPCStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub
- https://${QSS3BucketName}.${QSS3Region}.amazonaws.com/${QSS3KeyPrefix}submodules/quickstart-atlassian-services/templates/quickstart-vpc-for-atlassian-services.yaml
- QSS3Region: !If
- GovCloudCondition
- s3-us-gov-west-1
- s3
Parameters:
AccessCIDR: !Ref 'AccessCIDR'
AvailabilityZones: !Join
- ','
- !Ref 'AvailabilityZones'
ExportPrefix: !Ref 'ExportPrefix'
KeyPairName: !Ref 'KeyPairName'
PrivateSubnet1CIDR: !Ref 'PrivateSubnet1CIDR'
PrivateSubnet2CIDR: !Ref 'PrivateSubnet2CIDR'
PublicSubnet1CIDR: !Ref 'PublicSubnet1CIDR'
PublicSubnet2CIDR: !Ref 'PublicSubnet2CIDR'
VPCCIDR: !Ref 'VPCCIDR'
BastionHostRequired: !Ref 'BastionHostRequired'
JiraDCStack:
DependsOn: VPCStack
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub
- https://${QSS3BucketName}.${QSS3Region}.amazonaws.com/${QSS3KeyPrefix}templates/quickstart-jira-dc.template.yaml
- QSS3Region: !If ["GovCloudCondition", "s3-us-gov-west-1", "s3"]
Parameters:
CatalinaOpts: !Ref 'CatalinaOpts'
CidrBlock: !Ref 'AccessCIDR'
CloudWatchIntegration: !Ref 'CloudWatchIntegration'
ClusterNodeInstanceType: !Ref 'ClusterNodeInstanceType'
ClusterNodeMax: !Ref 'ClusterNodeMax'
ClusterNodeMin: !Ref 'ClusterNodeMin'
ClusterNodeVolumeSize: !Ref 'ClusterNodeVolumeSize'
CustomDnsName: !Ref 'CustomDnsName'
DBEngine: !Ref DBEngine
DBEngineVersion: !Ref DBEngineVersion
DBInstanceClass: !Ref 'DBInstanceClass'
DBIops: !Ref 'DBIops'
DBMasterUserPassword: !Ref 'DBMasterUserPassword'
DBMaxIdle: !Ref 'DBMaxIdle'
DBMaxWaitMillis: !Ref 'DBMaxWaitMillis'
DBMinEvictableIdleTimeMillis: !Ref 'DBMinEvictableIdleTimeMillis'
DBMinIdle: !Ref 'DBMinIdle'
DBMultiAZ: !Ref 'DBMultiAZ'
DBPassword: !Ref 'DBPassword'
DBPoolMaxSize: !Ref 'DBPoolMaxSize'
DBPoolMinSize: !Ref 'DBPoolMinSize'
DBRemoveAbandoned: !Ref 'DBRemoveAbandoned'
DBRemoveAbandonedTimeout: !Ref 'DBRemoveAbandonedTimeout'
DBStorage: !Ref 'DBStorage'
DBStorageEncrypted: !Ref 'DBStorageEncrypted'
DBStorageType: !Ref 'DBStorageType'
DBTestOnBorrow: !Ref 'DBTestOnBorrow'
DBTestWhileIdle: !Ref 'DBTestWhileIdle'
DBTimeBetweenEvictionRunsMillis: !Ref 'DBTimeBetweenEvictionRunsMillis'
DeploymentAutomationRepository: !Ref 'DeploymentAutomationRepository'
DeploymentAutomationBranch: !Ref 'DeploymentAutomationBranch'
DeploymentAutomationKeyName: !Ref 'DeploymentAutomationKeyName'
DeploymentAutomationPlaybook: !Ref 'DeploymentAutomationPlaybook'
DeploymentAutomationCustomParams: !Ref 'DeploymentAutomationCustomParams'
ExportPrefix: !Ref 'ExportPrefix'
HostedZone: !Ref 'HostedZone'
InternetFacingLoadBalancer: !Ref 'InternetFacingLoadBalancer'
JiraProduct: !Ref 'JiraProduct'
JiraVersion: !Ref 'JiraVersion'
JvmHeapOverride: !Ref 'JvmHeapOverride'
KeyPairName: !Ref 'KeyPairName'
MailEnabled: !Ref 'MailEnabled'
QSS3BucketName: !Ref 'jira-bucket-aws'
QSS3KeyPrefix: !Ref 'QSS3KeyPrefix'
SSLCertificateARN: !Ref 'SSLCertificateARN'
TomcatAcceptCount: !Ref 'TomcatAcceptCount'
TomcatContextPath: !Ref 'TomcatContextPath'
TomcatDefaultConnectorPort: !Ref 'TomcatDefaultConnectorPort'
TomcatEnableLookups: !Ref 'TomcatEnableLookups'
TomcatMaxThreads: !Ref 'TomcatMaxThreads'
TomcatMinSpareThreads: !Ref 'TomcatMinSpareThreads'
TomcatProtocol: !Ref 'TomcatProtocol'
TomcatRedirectPort: !Ref 'TomcatRedirectPort'
BastionHostRequired: !Ref 'BastionHostRequired'
Your missing the template parameters section. If you have the AWS CLI installed you can run aws cloudformation validate-template with either --template-body file://jira-cfn.yaml or --template-url https://path.to/your/cfn/template.yaml to check for template errors. You can also run the validation from the CloudFront console if you go to the editor and load/paste in the template.
Running the below command to validate the template produced this output (i saved your template as jira-cfn.yaml in the current working directory):
aws cloudformation validate-template --template-body file://jira-cfn.yaml
An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: Unresolved resource dependencies
[
TomcatEnableLookups,
DBIops,
SSLCertificateARN,
QSS3KeyPrefix,
JiraProduct,
TomcatMinSpareThreads,
DBMaxWaitMillis,
DBStorageEncrypted,
BastionHostRequired,
DBTestOnBorrow,
DeploymentAutomationPlaybook,
MailEnabled,
DBMultiAZ,
JiraVersion,
QSS3BucketName,
JvmHeapOverride,
DBRemoveAbandoned,
DBInstanceClass,
DBMasterUserPassword,
jira-bucket-aws,
CloudWatchIntegration,
ClusterNodeInstanceType,
DBMinIdle,
DBPassword,
ClusterNodeMin,
DeploymentAutomationCustomParams,
DBStorage,
TomcatMaxThreads,
InternetFacingLoadBalancer,
DeploymentAutomationBranch,
ClusterNodeMax,
DBMaxIdle,
TomcatAcceptCount,
DBStorageType,
DBEngine,
AccessCIDR,
TomcatDefaultConnectorPort,
DBMinEvictableIdleTimeMillis,
DeploymentAutomationKeyName,
TomcatRedirectPort,
DBTestWhileIdle,
HostedZone,
CatalinaOpts,
KeyPairName,
CustomDnsName,
TomcatContextPath,
DBRemoveAbandonedTimeout,
ClusterNodeVolumeSize,
DBPoolMinSize,
DBTimeBetweenEvictionRunsMillis,
DBEngineVersion,
ExportPrefix,
DBPoolMaxSize,
DeploymentAutomationRepository,
TomcatProtocol
] in the Resources block of the template
I'm repeatedly getting an InvalidRouteTableID.NotFound in cloud formation stack when running
aws cloudformation create-stack --stack-name sample --template-body file://aws-network.yml
and I have no idea why.
This is my cloudformation template aws-network.yml. It's pretty standard, it creates VPC, subnets, internet gateway, elastic ip address, nat gateway, route tables and associations.
AWSTemplateFormatVersion: 2010-09-09
# This CloudFormation template deploys a basic VPC / Network.
Resources:
vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsHostnames: true
EnableDnsSupport: false
InstanceTenancy: default
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-vpc"]]
internetGateway:
Type: AWS::EC2::InternetGateway
DependsOn: vpc
Properties:
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-igw"]]
attachGateway:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref vpc
InternetGatewayId: !Ref internetGateway
publicSubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.10.0/24
AvailabilityZone: !Select [ 0, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-public-a"]]
publicSubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.20.0/24
AvailabilityZone: !Select [ 1, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-public-b"]]
privateSubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.30.0/24
AvailabilityZone: !Select [ 0, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-private-a"]]
privateSubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.40.0/24
AvailabilityZone: !Select [ 1, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-private-b"]]
publicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref vpc
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-public"]]
publicRoute1:
Type: AWS::EC2::Route
DependsOn: attachGateway
Properties:
RouteTableId: !Ref publicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref internetGateway
natGateway: # it has a cost https://aws.amazon.com/vpc/pricing/
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt elasticIpAddress.AllocationId # gets the allocation Id from the elasticIpAddress resource
SubnetId: !Ref publicSubnetA # only associated to a public subnet to simplify and reduce costs
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-nat"]]
elasticIpAddress:
Type: AWS::EC2::EIP
Properties:
Domain: vpc
privateRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref vpc
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-private"]]
privateRoute1:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref privateRouteTable
DestinationCidrBlock: 0.0.0.0/0
NateGatewayId: !Ref natGateway
publicSubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref publicSubnetA
RouteTableId: publicRouteTable
publicSubnetBRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref publicSubnetB
RouteTableId: publicRouteTable
privateSubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref privateSubnetA
RouteTableId: privateRouteTable
privateSubnetBRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref privateSubnetB
RouteTableId: privateRouteTable
According to the events, that shouldn't happen, as I can see in order the following:
2021-04-21 17:04:05 UTC+0200 privateRouteTable
CREATE_COMPLETE -
2021-04-21 17:04:05 UTC+0200 publicRouteTable
CREATE_COMPLETE -
2021-04-21 17:04:22 UTC+0200 privateSubnetBRouteTableAssociation
CREATE_FAILED The routeTable ID 'privateRouteTable' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidRouteTableID.NotFound; Request ID: b51b2b9c-af12-4376-b6e4-1698624f7522; Proxy: null)
2021-04-21 17:04:22 UTC+0200 publicSubnetBRouteTableAssociation
CREATE_FAILED The routeTable ID 'publicRouteTable' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidRouteTableID.NotFound; Request ID: 5cb26e14-13ca-4915-9973-109dd44c5b2e; Proxy: null)
2021-04-21 17:04:22 UTC+0200 attachGateway
CREATE_FAILED Resource creation cancelled
2021-04-21 17:04:23 UTC+0200 privateSubnetARouteTableAssociation
CREATE_FAILED Resource creation cancelled
2021-04-21 17:04:23 UTC+0200 publicSubnetARouteTableAssociation
CREATE_FAILED Resource creation cancelled
2021-04-21 17:04:23 UTC+0200 natGateway
CREATE_FAILED Resource creation cancelled
2021-04-21 17:04:24 UTC+0200 rubiko
ROLLBACK_IN_PROGRESS The following resource(s) failed to create: [publicSubnetBRouteTableAssociation, attachGateway, privateSubnetBRouteTableAssociation, natGateway, publicSubnetARouteTableAssociation, privateSubnetARouteTableAssociation]. Rollback requested by user.
Any idea on why some created resources are not being found?
Thanks
Solved, I forgot !Ref (I'll call it a day..)
This is the correct template
AWSTemplateFormatVersion: 2010-09-09
# This CloudFormation template deploys a basic VPC / Network.
Resources:
vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsHostnames: true
EnableDnsSupport: false
InstanceTenancy: default
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-vpc"]]
internetGateway:
Type: AWS::EC2::InternetGateway
DependsOn: vpc
Properties:
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-igw"]]
attachGateway:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref vpc
InternetGatewayId: !Ref internetGateway
publicSubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.10.0/24
AvailabilityZone: !Select [ 0, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-public-a"]]
publicSubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.20.0/24
AvailabilityZone: !Select [ 1, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-public-b"]]
privateSubnetA:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.30.0/24
AvailabilityZone: !Select [ 0, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-private-a"]]
privateSubnetB:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref vpc
CidrBlock: 10.0.40.0/24
AvailabilityZone: !Select [ 1, !GetAZs ]
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-private-b"]]
publicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref vpc
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-public"]]
publicRoute1:
Type: AWS::EC2::Route
DependsOn: attachGateway
Properties:
RouteTableId: !Ref publicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref internetGateway
natGateway: # it has a cost https://aws.amazon.com/vpc/pricing/
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt elasticIpAddress.AllocationId # gets the allocation Id from the elasticIpAddress resource
SubnetId: !Ref publicSubnetA # only associated to a public subnet to simplify and reduce costs
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-nat"]]
elasticIpAddress:
Type: AWS::EC2::EIP
Properties:
Domain: vpc
privateRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref vpc
Tags:
- Key: Name
Value: !Join ['', [!Ref "AWS::StackName", "-private"]]
privateRoute1:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref privateRouteTable
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref natGateway
publicSubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref publicSubnetA
RouteTableId: !Ref publicRouteTable
publicSubnetBRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref publicSubnetB
RouteTableId: !Ref publicRouteTable
privateSubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref privateSubnetA
RouteTableId: !Ref privateRouteTable
privateSubnetBRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref privateSubnetB
RouteTableId: !Ref privateRouteTable
All credit to Mike Atkinson!
I am facing issue with Cloudformation on joining a list of strings after getting it from a map.
Error caused is Security Group expects a list of string.
"123456":
us-east-1:
#VPC1
VpcId: vpc-xxxx
VpcCidr: 10.78.160.0/20
SecurityGroups: sg-123
AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
"123457":
us-east-1:
#VPC
VpcId: vpc-xxxx
VpcCidr: 10.78.160.0/20
SecurityGroups: sg-123,sg-124
AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
LaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Metadata:
AWS::CloudFormation::Init:
config:
Properties:
InstanceType:
Ref: InstanceType
SecurityGroups: !Join [ ",", [!FindInMap [!Ref "AWS::AccountId", !Ref "AWS::Region", SecurityGroups], !Ref EC2SG ]]
IamInstanceProfile:
please let me know how to get the list for securitygroups and adding it to another resource
Solved by using this question
How do I use nested lists or append to a list in Cloudformation?
Lets take mappings:
Mappings:
"111222333444":
us-east-1:
SecurityGroups:
- sg-abc
- sg-xyz
us-west-2:
SecurityGroups:
- sg-1234
- sg-3456
Security groups can be referred as
SecurityGroupIds:
"Fn::FindInMap":
[!Ref AWS::AccountId, !Ref AWS::Region, SecurityGroups]
Define as array in mappings and pass the full array, this way we can avoid splitting and joining.
Full Template , tested on a Lambda
AWSTemplateFormatVersion: "2010-09-09"
Description: "Test"
Mappings:
"111222333444":
us-east-1:
SecurityGroups:
- sg-abc
- sg-xyz
us-west-2:
SecurityGroups:
- sg-1234
- sg-3456
Resources:
TestLambda:
Type: "AWS::Lambda::Function"
Properties:
Handler: com.test.MainClass::handleRequest
Runtime: java8
FunctionName: "Test-Lambda"
Code:
S3Bucket: code-bucket
S3Key: !Sub "artifacts/test.jar"
Description: "Test Lambda"
MemorySize: 512
Timeout: 60
Role: !Ref my-role-arn
VpcConfig:
SecurityGroupIds:
"Fn::FindInMap":
[!Ref AWS::AccountId, !Ref AWS::Region, SecurityGroups]
SubnetIds:
- subnet-1
- subnet-2
I am adding route 53 to my DBCluster and keep running into the error: Attribute: ReadEndpoint.Address was not found for resource: <DBCluster-name>
The entire stack is created via cloudformation.
Also, it should be noted that this is for Serverless Aurora in case that matters?
Here is my code:
AWSTemplateFormatVersion: 2010-09-09
Description: RDS Aurora serverless template
Parameters:
CustomFunctionArn:
Default: arn:aws:lambda:us-west-2:123456789:function:vault-secrets-read-lambda-prod
Description: The ARN of the lambda function to retrieve password from Vault
Type: String
DBName:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: Name of the database
Type: String
DBMasterUsername:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: The master user name for the DB instance
Type: String
DBScalingAutoPauseEnabled:
AllowedValues:
- 'true'
- 'false'
Default: 'true'
Description: Pause all DB instances after some inactivity
Type: String
DBScalingMaxCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 8
Description: The maximum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingMinCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 2
Description: The minimum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingSecondsUntilAutoPause:
Default: 300
Description: Auto pause after consecutive seconds of inactivity
MinValue: 300
MaxValue: 86400
Type: Number
Env:
AllowedValues:
- prod
- qa
- dev
Type: String
Description: Environment
VaultPath:
Default: secret/dev/dbPassword
Type: String
SnapshotId:
Description: snapshot ID to restore DB cluster from
Type: String
Conditions:
EnableAutoPause:
!Equals [!Ref DBScalingAutoPauseEnabled, 'true']
DoNotUseSnapshot: !Equals
- !Ref SnapshotId
- ''
Mappings:
Configuration:
prod:
HostedZoneEnv: mydomain.com
HostedZoneId: 'XXX'
SecurityGroup: sg-123321
SubnetGroups:
- subnet-123
- subnet-456
- subnet-789
VPCId: vpc-555
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: prod
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
qa:
HostedZoneEnv: mydomain-qa.com
HostedZoneId: 'XXX'
SecurityGroup: sg-321123
SubnetGroups:
- subnet-098
- subnet-765
- subnet-432
VPCId: vpc-345543
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: qa
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
dev:
HostedZoneEnv: mydomain-dev.com
HostedZoneId: 'XXX'
SecurityGroup: sg-f3453f
SubnetGroups:
- subnet-dsf24327
- subnet-82542gsda
- subnet-casaf2344
VPCId: vpc-23dfsf
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: dev
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
Resources:
AuroraSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allows access to RDS
GroupName: !Sub '${AWS::StackName}-aurora-rds-${Env}'
SecurityGroupIngress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
FromPort: 5432
ToPort: 5432
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcId: !FindInMap [Configuration, !Ref Env, VPCId]
GetValuefromVault:
Type: Custom::CustomResource
Properties:
ServiceToken: !Ref CustomFunctionArn
VaultKeyPath: !Ref VaultPath
DBCluster:
Type: 'AWS::RDS::DBCluster'
DeletionPolicy: Snapshot
UpdateReplacePolicy: Snapshot
Properties:
BackupRetentionPeriod: 7
DBClusterParameterGroupName: default.aurora-postgresql10
DBSubnetGroupName: !Ref DBSubnetGroup
DatabaseName: !Ref DBName
DeletionProtection: false
# EnableHttpEndpoint: true
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: '10.7'
KmsKeyId: !If [DoNotUseSnapshot, !Ref KMSkey, !Ref 'AWS::NoValue']
MasterUserPassword: !If [DoNotUseSnapshot, !GetAtt 'GetValuefromVault.ValueFromVault', !Ref 'AWS::NoValue']
MasterUsername: !If [DoNotUseSnapshot, !Ref DBMasterUsername, !Ref 'AWS::NoValue']
Port: 5432
ScalingConfiguration:
AutoPause: !If [EnableAutoPause, true, false]
MaxCapacity: !Ref DBScalingMaxCapacity
MinCapacity: !Ref DBScalingMinCapacity
SecondsUntilAutoPause: !Ref DBScalingSecondsUntilAutoPause
SnapshotIdentifier: !If [DoNotUseSnapshot, !Ref 'AWS::NoValue', !Ref SnapshotId]
StorageEncrypted: true
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcSecurityGroupIds:
- !GetAtt [AuroraSG, GroupId]
- !FindInMap [Configuration, !Ref Env, SecurityGroup]
DBSubnetGroup:
Type: 'AWS::RDS::DBSubnetGroup'
Properties:
DBSubnetGroupDescription: !Sub '${AWS::StackName}-${Env}'
SubnetIds: !FindInMap [Configuration, !Ref Env, SubnetGroups]
Tags: !FindInMap [Configuration, !Ref Env, Tags]
KmsAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: !Sub 'alias/${AWS::StackName}-${Env}-aurora-rds'
TargetKeyId: !Ref KMSkey
KMSkey:
Type: AWS::KMS::Key
Properties:
KeyPolicy:
Id: key-consolepolicy-3
Version: 2012-10-17
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: 'kms:*'
Resource: '*'
RecordSet:
Type: AWS::Route53::RecordSet
DependsOn: DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -writer-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.Endpoint.Address
TTL: '60'
Type: CNAME
ReadRecordSet:
Type: 'AWS::Route53::RecordSet'
DependsOn:
- DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -reader-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.ReadEndpoint.Address
TTL: '60'
Type: CNAME
Outputs:
AuroraHost:
Value: !GetAtt [DBCluster, Endpoint.Address]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'Host' ]]
AuroraSG:
Value: !GetAtt AuroraSG.GroupId
Export:
Name: !Join [":", [ !Ref "AWS::StackName", AuroraSG ]]
KMS:
Value: !GetAtt [KMSkey, Arn]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'KMS' ]]
DNSName:
Description: 'The connection endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.Endpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-DNSName'
ReadDNSName:
Description: 'The reader endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.ReadEndpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-ReadDNSName'
Some things i have tried:
Create new stack: FAIL
Create new stack without ReadRecordSet: FAIL
Create new stack without RecordSet (old name for read recordset): FAIL
Create new stack without RecordSet (new name for read recordset): FAIL
Add a DependsOn to ReadRecordSet (for first RecordSet): FAIL
Enabling HTTP endpoint on Cluster: FAIL
Update TTL to 60: FAIL Update TTL to 0: FAIL
The RecordSet seems to be creating okay (I tested that by adding a DependsOn: - RecordSet in the ReadRecordSet to allow RecordSet to create first), so it's the ReadRecordSet that is failing and can't find ReadEndpoint.Address
Not sure what I am missing here, been googling like mad and don't see much about this error. Any help is appreciated!
It turns out that Aurora Serverless doesn't require ReadRecordSet, so that entire section is only applicable to provisioned DB, so ReadEndpoint doesn't exist indeed. Unfortunately AWS documentation doesn't mention that explicitly.