ReadEndpoint.Address was not found for DBCluster - amazon-web-services

I am adding route 53 to my DBCluster and keep running into the error: Attribute: ReadEndpoint.Address was not found for resource: <DBCluster-name>
The entire stack is created via cloudformation.
Also, it should be noted that this is for Serverless Aurora in case that matters?
Here is my code:
AWSTemplateFormatVersion: 2010-09-09
Description: RDS Aurora serverless template
Parameters:
CustomFunctionArn:
Default: arn:aws:lambda:us-west-2:123456789:function:vault-secrets-read-lambda-prod
Description: The ARN of the lambda function to retrieve password from Vault
Type: String
DBName:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: Name of the database
Type: String
DBMasterUsername:
AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
Description: The master user name for the DB instance
Type: String
DBScalingAutoPauseEnabled:
AllowedValues:
- 'true'
- 'false'
Default: 'true'
Description: Pause all DB instances after some inactivity
Type: String
DBScalingMaxCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 8
Description: The maximum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingMinCapacity:
AllowedValues:
- 2
- 4
- 8
- 16
- 32
- 64
- 192
- 384
Default: 2
Description: The minimum capacity for an Aurora DB cluster in serverless DB engine mode
Type: Number
DBScalingSecondsUntilAutoPause:
Default: 300
Description: Auto pause after consecutive seconds of inactivity
MinValue: 300
MaxValue: 86400
Type: Number
Env:
AllowedValues:
- prod
- qa
- dev
Type: String
Description: Environment
VaultPath:
Default: secret/dev/dbPassword
Type: String
SnapshotId:
Description: snapshot ID to restore DB cluster from
Type: String
Conditions:
EnableAutoPause:
!Equals [!Ref DBScalingAutoPauseEnabled, 'true']
DoNotUseSnapshot: !Equals
- !Ref SnapshotId
- ''
Mappings:
Configuration:
prod:
HostedZoneEnv: mydomain.com
HostedZoneId: 'XXX'
SecurityGroup: sg-123321
SubnetGroups:
- subnet-123
- subnet-456
- subnet-789
VPCId: vpc-555
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: prod
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
qa:
HostedZoneEnv: mydomain-qa.com
HostedZoneId: 'XXX'
SecurityGroup: sg-321123
SubnetGroups:
- subnet-098
- subnet-765
- subnet-432
VPCId: vpc-345543
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: qa
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
dev:
HostedZoneEnv: mydomain-dev.com
HostedZoneId: 'XXX'
SecurityGroup: sg-f3453f
SubnetGroups:
- subnet-dsf24327
- subnet-82542gsda
- subnet-casaf2344
VPCId: vpc-23dfsf
Tags:
- Key: Name
Value: my-db
- Key: environment
Value: dev
- Key: component
Value: rds-aurora
- Key: classification
Value: internal
Resources:
AuroraSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allows access to RDS
GroupName: !Sub '${AWS::StackName}-aurora-rds-${Env}'
SecurityGroupIngress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
FromPort: 5432
ToPort: 5432
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcId: !FindInMap [Configuration, !Ref Env, VPCId]
GetValuefromVault:
Type: Custom::CustomResource
Properties:
ServiceToken: !Ref CustomFunctionArn
VaultKeyPath: !Ref VaultPath
DBCluster:
Type: 'AWS::RDS::DBCluster'
DeletionPolicy: Snapshot
UpdateReplacePolicy: Snapshot
Properties:
BackupRetentionPeriod: 7
DBClusterParameterGroupName: default.aurora-postgresql10
DBSubnetGroupName: !Ref DBSubnetGroup
DatabaseName: !Ref DBName
DeletionProtection: false
# EnableHttpEndpoint: true
Engine: aurora-postgresql
EngineMode: serverless
EngineVersion: '10.7'
KmsKeyId: !If [DoNotUseSnapshot, !Ref KMSkey, !Ref 'AWS::NoValue']
MasterUserPassword: !If [DoNotUseSnapshot, !GetAtt 'GetValuefromVault.ValueFromVault', !Ref 'AWS::NoValue']
MasterUsername: !If [DoNotUseSnapshot, !Ref DBMasterUsername, !Ref 'AWS::NoValue']
Port: 5432
ScalingConfiguration:
AutoPause: !If [EnableAutoPause, true, false]
MaxCapacity: !Ref DBScalingMaxCapacity
MinCapacity: !Ref DBScalingMinCapacity
SecondsUntilAutoPause: !Ref DBScalingSecondsUntilAutoPause
SnapshotIdentifier: !If [DoNotUseSnapshot, !Ref 'AWS::NoValue', !Ref SnapshotId]
StorageEncrypted: true
Tags: !FindInMap [Configuration, !Ref Env, Tags]
VpcSecurityGroupIds:
- !GetAtt [AuroraSG, GroupId]
- !FindInMap [Configuration, !Ref Env, SecurityGroup]
DBSubnetGroup:
Type: 'AWS::RDS::DBSubnetGroup'
Properties:
DBSubnetGroupDescription: !Sub '${AWS::StackName}-${Env}'
SubnetIds: !FindInMap [Configuration, !Ref Env, SubnetGroups]
Tags: !FindInMap [Configuration, !Ref Env, Tags]
KmsAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: !Sub 'alias/${AWS::StackName}-${Env}-aurora-rds'
TargetKeyId: !Ref KMSkey
KMSkey:
Type: AWS::KMS::Key
Properties:
KeyPolicy:
Id: key-consolepolicy-3
Version: 2012-10-17
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: 'kms:*'
Resource: '*'
RecordSet:
Type: AWS::Route53::RecordSet
DependsOn: DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -writer-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.Endpoint.Address
TTL: '60'
Type: CNAME
ReadRecordSet:
Type: 'AWS::Route53::RecordSet'
DependsOn:
- DBCluster
Properties:
HostedZoneId: !FindInMap [Configuration, !Ref Env, HostedZoneId]
Name: !Join ['', [!Ref DBName, -reader-db, ., !FindInMap [Configuration, !Ref Env, HostedZoneEnv], .]]
ResourceRecords:
- !GetAtt DBCluster.ReadEndpoint.Address
TTL: '60'
Type: CNAME
Outputs:
AuroraHost:
Value: !GetAtt [DBCluster, Endpoint.Address]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'Host' ]]
AuroraSG:
Value: !GetAtt AuroraSG.GroupId
Export:
Name: !Join [":", [ !Ref "AWS::StackName", AuroraSG ]]
KMS:
Value: !GetAtt [KMSkey, Arn]
Export:
Name: !Join [":", [ !Ref "AWS::StackName", 'KMS' ]]
DNSName:
Description: 'The connection endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.Endpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-DNSName'
ReadDNSName:
Description: 'The reader endpoint for the DB cluster.'
Value: !GetAtt 'DBCluster.ReadEndpoint.Address'
Export:
Name: !Sub '${AWS::StackName}-ReadDNSName'
Some things i have tried:
Create new stack: FAIL
Create new stack without ReadRecordSet: FAIL
Create new stack without RecordSet (old name for read recordset): FAIL
Create new stack without RecordSet (new name for read recordset): FAIL
Add a DependsOn to ReadRecordSet (for first RecordSet): FAIL
Enabling HTTP endpoint on Cluster: FAIL
Update TTL to 60: FAIL Update TTL to 0: FAIL
The RecordSet seems to be creating okay (I tested that by adding a DependsOn: - RecordSet in the ReadRecordSet to allow RecordSet to create first), so it's the ReadRecordSet that is failing and can't find ReadEndpoint.Address
Not sure what I am missing here, been googling like mad and don't see much about this error. Any help is appreciated!

It turns out that Aurora Serverless doesn't require ReadRecordSet, so that entire section is only applicable to provisioned DB, so ReadEndpoint doesn't exist indeed. Unfortunately AWS documentation doesn't mention that explicitly.

Related

AWS Network LoadBalancer listener and the following target groups have incompatible protocols

I have run into an issue trying to setup rtmp on AWS EC2 using CloudFormation. I have two stack's aws.yaml and web.yaml. CF responds with the following error on execution: Internal error reported from downstream service during operation 'The listener and the following target groups have incompatible protocols: [arn:...]. I've tried to specify a protocol directly in the listener but this just results in the error, You cannot specify the protocol version for a target group with the 'TCP_UDP' protocol. Thanks in advance for help!
nlb.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: Deploys a single network load balancer
Parameters:
VpcId:
Type: AWS::EC2::VPC::Id
Description: ID of the VPC
PublicSubnets:
Type: List<AWS::EC2::Subnet::Id>
Description: List of Public subnets to use for the Load Balancer
CertificateArn:
Type: String
Description: Arn of the ACM certificate to be used by the load balancer listener.
Resources:
PublicNlb:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
IpAddressType: ipv4
Name: publicNlb
Scheme: internet-facing
Subnets: !Ref PublicSubnets
Tags:
- Key: Name
Value: wordpress-nlb
Type: network
PublicNlbTargetGroupHttp:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Port: 80
Protocol: TCP
Targets:
- Id: !ImportValue WordPressPublicAlbArn
TargetType: alb
VpcId: !Ref VpcId
PublicNlbTargetGroupHttps:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Port: 443
Protocol: TCP
Targets:
- Id: !ImportValue WordPressPublicAlbArn
TargetType: alb
VpcId: !Ref VpcId
PublicNlbHttpListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn: !Ref PublicNlbTargetGroupHttp
LoadBalancerArn: !Ref PublicNlb
Port: 80
Protocol: TCP
PublicNlbHttpsListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn: !Ref PublicNlbTargetGroupHttps
LoadBalancerArn: !Ref PublicNlb
Port: 443
Protocol: TCP
Outputs:
PublicNlbTargetGroupHttpArn:
Value: !Ref PublicNlbTargetGroupHttp
Export:
Name: WordPressPublicNlbTargetGroupHttpArn
PublicNlbTargetGroupHttpsArn:
Value: !Ref PublicNlbTargetGroupHttps
Export:
Name: WordPressPublicNlbTargetGroupHttpsArn
PublicNlbCanonicalHostedZoneId:
Value: !GetAtt PublicNlb.CanonicalHostedZoneID
PublicNlbDnsName:
Value: !GetAtt PublicNlb.DNSName
PublicNlbFullName:
Value: !GetAtt PublicNlb.LoadBalancerFullName
PublicNlbHostname:
Value:
!Sub https://${PublicNlb.DNSName}
# !If [ NoSslCertificate, !Join [ '', [ 'http://', !GetAtt PublicAlb.DNSName ] ], !Join [ '', [ 'https://', !GetAtt PublicAlb.DNSName ] ] ]
PublicNlbHttpsListenerArn:
Value: !Ref PublicNlbHttpsListener
Export:
Name: WordPressPublicNlbHttpsListenerArn
PublicNlbHttpListenerArn:
Value: !Ref PublicNlbHttpListener
Export:
Name: WordPressPublicNlbHttpListenerArn
web.yaml
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys highly available wordpress application
Parameters:
VpcId:
Type: AWS::EC2::VPC::Id
Description: ID of the VPC
RecipeVersion:
Type: String
Description: The semantic version number you want to give to the recipe (in Major.Minor.Patch format).
AllowedPattern: ^[0-9]+\.[0-9]+\.[0-9]+$
PrivateSubnets:
Type: List<AWS::EC2::Subnet::Id>
Description: List of Private subnets to use for the application
PrivateSubnetsString:
Type: String
Description: List of Private subnets to use for the application as string
Account:
Type: String
Default: non-prod
AllowedValues:
- prod
- non-prod
Environment:
Type: String
Default: dev
AllowedValues:
- dev
- preprod
- prod
InstanceProfileName:
Type: String
Default: EC2RoleForSSM
Description: The name of the InstanceProfile to be used with the instances.
AmazonLinuxAMI:
Type: AWS::EC2::Image::Id
MinSize:
Type: Number
Description: The minimum limit of allowed instances to be deployed.
Default: 1
MaxSize:
Type: Number
Description: The maximum limit of allowed instances to be deployed.
Default: 8
DesiredCapacity:
Type: Number
Description: The average amount of instances to be deployed.
Default: 1
OnDemandBaseCapacity:
Type: Number
Description: The minimum amount of the Auto Scaling group's capacity that must be fulfilled by On-Demand Instances. For production you should always have at least 1 On-Demand instance (set to 1).
Default: 1
OnDemandPercentageAboveBaseCapacity:
Type: Number
Description: Controls the percentages of On-Demand Instances and Spot Instances for your additional capacity beyond OnDemandBaseCapacity. Expressed as a number (for example, 20 specifies 20% On-Demand Instances, 80% Spot Instances). Defaults to 0 if not specified. If set to 100, only On-Demand Instances are provisioned.
Default: 0
Mappings:
LoadBalancerRulePriority:
dev:
"rule": "3"
preprod:
"rule": "2"
prod:
"rule": "1"
LoadBalancerHosts:
dev:
hostOne: "*.example.*"
preprod:
hostOne: "preprod.example.*"
prod:
hostOne: "*.example.*"
SalingServerProperties:
# #link https://aws.amazon.com/ec2/instance-types/
# #link https://instaguide.io/#sortField=vcpu&sortDir=-1&platform=rhel&tenancyCode=dedicated&priceCalc=abs&period=mo
dev:
instanceTypeOne: "c6gd.xlarge"
preprod:
instanceTypeOne: "c6gd.xlarge"
prod:
instanceTypeOne: "c6gd.xlarge"
Conditions:
IsProd: !Equals [!Ref Environment, prod]
IsPreProd: !Equals [!Ref Environment, preprod]
IsDev: !Equals [!Ref Environment, dev]
Resources:
AlbTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub "${Environment}-wordpress"
HealthCheckEnabled: true
HealthCheckIntervalSeconds: 10
HealthCheckPath: /
HealthCheckPort: "80"
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 4
UnhealthyThresholdCount: 2
Port: 80
Protocol: HTTP
ProtocolVersion: HTTP1
Tags:
- Key: Name
Value: !Sub "${Environment}-${RecipeVersion}-wordpress"
TargetGroupAttributes:
- Key: stickiness.enabled
Value: "false"
- Key: stickiness.type
Value: lb_cookie # or app_cookie
TargetType: instance
VpcId: !Ref VpcId
AlbListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref AlbTargetGroup
Conditions:
- Field: host-header
HostHeaderConfig:
Values:
- dropingaming.*
- !FindInMap [ LoadBalancerHosts, !Ref Environment, hostOne ]
ListenerArn: !ImportValue WordPressPublicAlbHttpsListenerArn
Priority: !FindInMap [ LoadBalancerRulePriority, !Ref Environment, rule ]
NlbTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub "${Environment}-wordpress-rtmp"
HealthCheckEnabled: true
HealthCheckIntervalSeconds: 30
HealthCheckPort: "4444"
HealthCheckProtocol: HTTP
Port: 1935
Protocol: TCP_UDP
Tags:
- Key: Name
Value: !Sub "${Environment}-${RecipeVersion}-wordpress"
TargetGroupAttributes:
- Key: stickiness.enabled
Value: "true"
TargetType: instance
VpcId: !Ref VpcId
NlbListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref NlbTargetGroup
Conditions:
# Configures the rule to apply when the host header matches the domain we want to send to the target group
# https://stackoverflow.com/questions/45664638/how-to-make-a-list-item-conditional-in-cloud-formation-template
- Field: host-header
HostHeaderConfig:
Values:
- dropingaming.*
- !FindInMap [ LoadBalancerHosts, !Ref Environment, hostOne ]
ListenerArn: !ImportValue WordPressPublicAlbHttpsListenerArn
Priority: !FindInMap [ LoadBalancerRulePriority, !Ref Environment, rule ]
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: !Sub "${Environment}-wordpress"
LaunchTemplateData:
ImageId: !Ref AmazonLinuxAMI
EbsOptimized: 'true'
IamInstanceProfile:
Name: EC2RoleForSSM
SecurityGroupIds:
- !ImportValue WordPressEc2SecurityGroupId
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
# The user data to make available to the instance. You must provide base64-encoded text.
# User data is limited to 16 KB. For more information, see Running Commands on Your Linux Instance at Launch (Linux) or Adding User Data (Windows).
# logging is sent to :: /var/log/cloud-init-output.log
# this script is stored to :: /var/lib/cloud/instance/scripts/part-001
UserData:
Fn::Base64: !Sub
- |
#!/bin/bash -ex
EC2_INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id || die \"wget instance-id has failed: $?\"`"
test -n "$EC2_INSTANCE_ID" || die 'cannot obtain instance-id'
EC2_AVAIL_ZONE="`wget -q -O - http://instance-data/latest/meta-data/placement/availability-zone || die \"wget availability-zone has failed: $?\"`"
test -n "$EC2_AVAIL_ZONE" || die 'cannot obtain availability-zone'
EC2_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
EC2_PUBLIC_IPV4="`curl http://checkip.amazonaws.com`"
EC2_PRIVATE_IPV4="`curl http://169.254.169.254/latest/meta-data/local-ipv4`"
# https://tldp.org/LDP/abs/html/here-docs.html
sudo cat > '/var/ebs-www-copy/aws.json' <<End-of-message
{
"AmazonLinuxAMI": "${AmazonLinuxAMI}",
"RecipeVersion": "${RecipeVersion}",
"EfsId": "${EfsId}",
"VpcId": "${VpcId}",
"PrivateSubnets": "${PrivateSubnetsString}",
"Account": "${Account}",
"Environment": "${Environment}",
"InstanceProfileName": "${InstanceProfileName}",
"MinSize": "${MinSize}",
"MaxSize": "${MaxSize}",
"DesiredCapacity": "${DesiredCapacity}",
"OnDemandBaseCapacity": "${OnDemandBaseCapacity}",
"OnDemandPercentageAboveBaseCapacity": "${OnDemandPercentageAboveBaseCapacity}",
"EC2_PUBLIC_IPV4": "$EC2_PUBLIC_IPV4",
"EC2_PRIVATE_IPV4": "$EC2_PRIVATE_IPV4",
"EC2_INSTANCE_ID": "$EC2_INSTANCE_ID",
"EC2_AVAIL_ZONE": "$EC2_AVAIL_ZONE",
"EC2_REGION": "$EC2_REGION"
}
End-of-message
cat /var/ebs-www-copy/aws.json
# redacting the overly specific :)
systemctl start httpd
- EfsId: !ImportValue "WordPressEfsId"
# https://github.com/aws-samples/session-manager-without-igw/blob/main/session-manager-without-igw.yml
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName: !Sub "${Environment}-wordpress"
TargetGroupARNs:
- !Ref AlbTargetGroup
- !Ref NlbTargetGroup
MinSize: !Ref MinSize
MaxSize: !Ref MaxSize
DesiredCapacity: !Ref DesiredCapacity
VPCZoneIdentifier: !Ref PrivateSubnets
MixedInstancesPolicy:
LaunchTemplate:
LaunchTemplateSpecification:
# Version: !Ref LaunchTemplateVersionNumber
Version: !GetAtt
- LaunchTemplate
- LatestVersionNumber
LaunchTemplateId: !Ref LaunchTemplate
# https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups-launch-template-overrides.html
# #link https://aws.amazon.com/ec2/instance-types/
# #link https://instaguide.io/#sortField=vcpu&sortDir=-1&platform=rhel&tenancyCode=dedicated&priceCalc=abs&period=mo
Overrides:
# I want to know if predictive load b is bs or nah
- InstanceType: !FindInMap [SalingServerProperties, !Ref Environment, instanceTypeOne]
InstancesDistribution:
OnDemandBaseCapacity: !Ref OnDemandBaseCapacity
OnDemandPercentageAboveBaseCapacity: !Ref OnDemandPercentageAboveBaseCapacity
# https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#allocation-strategies
SpotAllocationStrategy: lowest-price
OnDemandAllocationStrategy: lowest-price
Tags:
- Key: "Name"
PropagateAtLaunch: true
Value: !Sub "${Environment}-wordpress"
- Key: "Version"
PropagateAtLaunch: true
Value: !Sub "${RecipeVersion}"
AutoScalingPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AutoScalingGroupName: !Ref AutoScalingGroup
PolicyType: PredictiveScaling
PredictiveScalingConfiguration:
MetricSpecifications:
- TargetValue: 70
PredefinedMetricPairSpecification:
PredefinedMetricType: ASGCPUUtilization
So based on the error the NlbListenerRule seems to be what is failing; though, the downstream service bit has me confused. Google return near nothing, so I'm hopeful you may know!?
So AWS support along with the comment above lead me straight to the problem. The NlbListenerRule was using an incorrect ListenerArn: !ImportValue WordPressPublicAlbHttpsListenerArn. Also, another issues which is catchable in the provided templates above was caught by the support team "NLB work on Layer-4 of OSI Model , and we cannot create NLB Listener rule based on Host-Header Condition." Thus ListenerRule's may only be applied to HTTP based hosts.
changed in web.yaml
NlbListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn: !Ref NlbTargetGroup
LoadBalancerArn: !ImportValue PublicNlbLoadBalancerArn
Port: 1935
Protocol: TCP_UDP
added in alb.yaml
Outputs:
PublicNlb:
Value: !Ref PublicNlb
Export:
Name: PublicNlbLoadBalancerArn
PublicNlbTargetGroupHttpArn:

Resource handler returned message: "Invalid request provided: Rules are unsupported for Network Load Balancer listeners

I'm creating a CloudFormation stack for ECS automation and I'm using an internal NLB.
I'm having a problem with the ListenerRule that returns the following error in CloudFormation: Resource handler returned message: "Invalid request provided: Rules are unsupported for Network Load Balancer listeners
My script is this:
AWSTemplateFormatVersion: 2010-09-09
Description: ECS Fargate
Parameters:
Name:
Type: String
VPC:
Type: AWS::EC2::VPC::Id
Subnets:
Type: List<AWS::EC2::Subnet::Id>
SecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
CreationVCPEndpoint:
Type: String
AllowedValues: [true, false]
DesiredCount:
Type: String
Conditions:
CreationVCPEndpointSelected: !Equals [!Ref CreationVCPEndpoint, true]
Resources:
# Endpoints necessários para o serviço do ECS funcionar.
EndpointLogs:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .logs
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointS3:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .s3
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointECR:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .ecr.api
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointSSM:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .ssm
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointDKR:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .ecr.dkr
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
# Criação do NLB Privado
LoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Ref Name
Subnets: !Ref Subnets
Type: network
Scheme: internal
Tags:
- Key: Name
Value: !Ref Name
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
LoadBalancerArn: !Ref LoadBalancer
Port: 80
Protocol: TCP
DefaultActions:
- Type: forward
TargetGroupArn: !Ref TargetGroup
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${Name}-tg
VpcId: !Ref VPC
Port: 80
Protocol: TCP
TargetType: ip
ListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn: !Ref LoadBalancerListener
Priority: 1
Conditions:
- Field: source-ip
#Values:
# - /
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward
# Criação da IAM para o ECS
ECSIAM:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "ecs.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: !Ref Name
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
# Criação do ECS Fargate
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Ref Name
ECSService:
Type: AWS::ECS::Service
DependsOn: ListenerRule
Properties:
Cluster: !Ref ECSCluster
Role: !Ref ECSIAM
DesiredCount: !Ref DesiredCount
TaskDefinition: !Ref ECSTaskDefinition
LoadBalancers:
- ContainerName: "website-service"
ContainerPort: 80
TargetGroupArn: !Ref TargetGroup
ECSTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: website-service
NetworkMode: awsvpc
ContainerDefinitions:
- Name: website-service
Essential: true
Image: 489732776903.dkr.ecr.us-west-2.amazonaws.com/daniel:latest
Memory: 128
#Environment:
# - Name: PRODUCT_SERVICE_URL
# Value: !Ref ProductServiceUrl
PortMappings:
- ContainerPort: 80
#LogConfiguration:
# LogDriver: awslogs
# Options:
# awslogs-group: !Ref CloudWatchLogsGroup
# awslogs-region: !Ref AWS::Region
Does anyone have any idea what the correct ListenerRule configuration is?
I managed to get the template right, there is the template ready.
AWSTemplateFormatVersion: 2010-09-09
Description: ECS Fargate
Parameters:
Name:
Type: String
VPC:
Type: AWS::EC2::VPC::Id
Subnets:
Type: List<AWS::EC2::Subnet::Id>
SecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
CreationVCPEndpoint:
Type: String
AllowedValues: [true, false]
DesiredCount:
Type: String
Conditions:
CreationVCPEndpointSelected: !Equals [!Ref CreationVCPEndpoint, true]
Resources:
# Endpoints necessários para o serviço do ECS funcionar.
EndpointLogs:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .logs
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointS3:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .s3
PrivateDnsEnabled: false
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointECR:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .ecr.api
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointSSM:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .ssm
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
EndpointDKR:
Type: AWS::EC2::VPCEndpoint
Condition: CreationVCPEndpointSelected
Properties:
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .ecr.dkr
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
Ref: Subnets
VpcEndpointType: Interface
VpcId:
Ref: VPC
# Criação do NLB Privado
LoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Ref Name
Subnets: !Ref Subnets
Type: network
Scheme: internal
Tags:
- Key: Name
Value: !Ref Name
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
LoadBalancerArn: !Ref LoadBalancer
Port: 80
Protocol: TCP
DefaultActions:
- Type: forward
TargetGroupArn: !Ref TargetGroup
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${Name}-tg
VpcId: !Ref VPC
Port: 80
Protocol: TCP
TargetType: ip
# Criação da IAM para o ECS
ECSIAM:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "ecs.amazonaws.com"
- "ecs-tasks.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: !Ref Name
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
# Criação do ECS Fargate
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Ref Name
ECSService:
Type: AWS::ECS::Service
DependsOn: LoadBalancerListener
Properties:
Cluster: !Ref ECSCluster
LaunchType: FARGATE
DesiredCount: !Ref DesiredCount
TaskDefinition: !Ref ECSTaskDefinition
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- sg-0c2e8f8e84a22bdd4
Subnets:
- subnet-04d79b3e4ac16ba6f
- subnet-07baab102179ee184
LoadBalancers:
- ContainerName: "test-container"
ContainerPort: 80
TargetGroupArn: !Ref TargetGroup
ECSTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: test-container
NetworkMode: awsvpc
Cpu: 512
Memory: 1024
ExecutionRoleArn: !Ref ECSIAM
RequiresCompatibilities:
- FARGATE
ContainerDefinitions:
- Name: test-container
Essential: true
Image: URI
Cpu: 512
Memory: 1024
PortMappings:
- ContainerPort: 80

Troubleshoot Cloudformation script

I have this AWS exercise scenario, where your customer has not used AWS before, and is encountering issues when trying to launch a web application as a proof of concept.
So far your customer has launched an AWS Elastic Load Balancer (ELB) and an Amazon Elastic Compute Cloud (EC2) instance acting as the web server. Both are deployed in a Virtual Private Cloud (VPC) on AWS. While your customer's initial deployment aims to present a static web page to its users (demo.html located in the document root of the web server
Below is the cloud formation script:
AWSTemplateFormatVersion: '2010-09-09'
Description: |
AWS CloudFormation SA Assignment - WARNING: You will be billed for the
AWS resources used if you create a stack from this template and consume all your
promotional credit.
We recommend you create a billing alert. Once you submit your
answers delete the CloudFormation stack and terminate any other resources launched
in relation to this exercise.
Feel free to do that as soon as you have submitted
your document and before your interview.
Parameters:
CandidateName:
Description: 'Please input your first and last name:'
Type: String
MaxLength: '50'
MinLength: '3'
ConstraintDescription: Please input your full name.
Resources:
SAVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
InstanceTenancy: default
EnableDnsSupport: 'true'
EnableDnsHostnames: 'true'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [SAVPC, !Ref 'CandidateName']]
PublicSubnetA:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.0.0/24
AvailabilityZone: eu-west-1a
MapPublicIpOnLaunch: 'True'
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [PublicSubnetA, !Ref 'CandidateName']]
PublicSubnetB:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.1.0/24
AvailabilityZone: eu-west-1b
MapPublicIpOnLaunch: 'True'
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [PublicSubnetB, !Ref 'CandidateName']]
PrivateSubnetA:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.2.0/24
AvailabilityZone: eu-west-1a
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [PrivateSubnetA, !Ref 'CandidateName']]
PrivateSubnetB:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.3.0/24
AvailabilityZone: eu-west-1b
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [PrivateSubnetB, !Ref 'CandidateName']]
SAIGW:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [IGW, !Ref 'CandidateName']]
SANetworkACL:
Type: AWS::EC2::NetworkAcl
Properties:
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [NACL, !Ref 'CandidateName']]
SARoutePublic:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [PublicRoute, !Ref 'CandidateName']]
SARoutePrivate:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [PrivateRoute, !Ref 'CandidateName']]
SAInstance1:
Type: AWS::EC2::Instance
Properties:
DisableApiTermination: 'false'
InstanceInitiatedShutdownBehavior: stop
ImageId: ami-047bb4163c506cd98
InstanceType: t2.micro
Monitoring: 'false'
UserData: IyEvYmluL2Jhc2gNCnl1bSB1cGRhdGUgLXkNCnl1bSBpbnN0YWxsIC15IGh0dHBkMjQNCnNlcnZpY2UgaHR0cGQgc3RhcnQNCmNoa2NvbmZpZyBodHRwZCBvbg0KZ3JvdXBhZGQgd3d3DQp1c2VybW9kIC1hIC1HIHd3dyBlYzItdXNlcg0KY2hvd24gLVIgcm9vdDp3d3cgL3Zhci93d3cNCmNobW9kIDI3NzUgL3Zhci93d3cNCmZpbmQgL3Zhci93d3cgLXR5cGUgZCAtZXhlYyBjaG1vZCAyNzc1IHt9ICsNCmZpbmQgL3Zhci93d3cgLXR5cGUgZiAtZXhlYyBjaG1vZCAwNjY0IHt9ICsNCmVjaG8gJzxodG1sPjxoZWFkPjx0aXRsZT5TdWNjZXNzITwvdGl0bGU+PC9oZWFkPjxib2R5PjxpZnJhbWUgd2lkdGg9IjU2MCIgaGVpZ2h0PSIzMTUiIHNyYz0iaHR0cHM6Ly93d3cueW91dHViZS5jb20vZW1iZWQvSnMyMXhLTUZkd3ciIGZyYW1lYm9yZGVyPSIwIiBhbGxvd2Z1bGxzY3JlZW4+PC9pZnJhbWU+PC9ib2R5PjwvaHRtbD4nID4gL3Zhci93d3cvaHRtbC9kZW1vLmh0bWw=
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [Instance1, !Ref 'CandidateName']]
NetworkInterfaces:
- AssociatePublicIpAddress: 'true'
DeleteOnTermination: 'true'
Description: Primary network interface
DeviceIndex: 0
SubnetId: !Ref 'PublicSubnetA'
GroupSet: [!Ref 'SASGapp']
SAelb:
Type: AWS::ElasticLoadBalancing::LoadBalancer
Properties:
Subnets: [!Ref 'PublicSubnetB']
Instances: [!Ref 'SAInstance1']
SecurityGroups: [!Ref 'SASGELB']
Listeners:
- LoadBalancerPort: '80'
InstancePort: '80'
Protocol: HTTP
HealthCheck:
HealthyThreshold: '2'
Interval: '15'
Target: TCP:443
Timeout: '5'
UnhealthyThreshold: '2'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: !Join ['-', [ELB, !Ref 'CandidateName']]
SASGELB:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SA Assignment - ELB security group
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: ELBSecurityGroup
SASGapp:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SA Assignment - App server security group
VpcId: !Ref 'SAVPC'
Tags:
- Key: environment
Value: sa-assignment
- Key: Name
Value: AppServerSecurityGroup
SANACLEntry1:
Type: AWS::EC2::NetworkAclEntry
Properties:
CidrBlock: 0.0.0.0/0
Egress: 'true'
Protocol: '-1'
RuleAction: allow
RuleNumber: '100'
NetworkAclId: !Ref 'SANetworkACL'
SANACLEntry2:
Type: AWS::EC2::NetworkAclEntry
Properties:
CidrBlock: 0.0.0.0/0
Protocol: '-1'
RuleAction: allow
RuleNumber: '100'
NetworkAclId: !Ref 'SANetworkACL'
subnetacl1:
Type: AWS::EC2::SubnetNetworkAclAssociation
Properties:
NetworkAclId: !Ref 'SANetworkACL'
SubnetId: !Ref 'PublicSubnetA'
subnetacl2:
Type: AWS::EC2::SubnetNetworkAclAssociation
Properties:
NetworkAclId: !Ref 'SANetworkACL'
SubnetId: !Ref 'PublicSubnetB'
subnetacl3:
Type: AWS::EC2::SubnetNetworkAclAssociation
Properties:
NetworkAclId: !Ref 'SANetworkACL'
SubnetId: !Ref 'PrivateSubnetA'
subnetacl4:
Type: AWS::EC2::SubnetNetworkAclAssociation
Properties:
NetworkAclId: !Ref 'SANetworkACL'
SubnetId: !Ref 'PrivateSubnetB'
SAIGWAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref 'SAVPC'
InternetGatewayId: !Ref 'SAIGW'
subnetRoutePublicA:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref 'SARoutePublic'
SubnetId: !Ref 'PublicSubnetA'
subnetRoutePublicB:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref 'SARoutePublic'
SubnetId: !Ref 'PublicSubnetB'
subnetRoutePrivateA:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref 'SARoutePrivate'
SubnetId: !Ref 'PrivateSubnetA'
subnetRoutePrivateB:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref 'SARoutePrivate'
SubnetId: !Ref 'PrivateSubnetB'
publicroute:
Type: AWS::EC2::Route
Properties:
DestinationCidrBlock: 0.0.0.0/0
RouteTableId: !Ref 'SARoutePublic'
GatewayId: !Ref 'SAIGW'
DependsOn: SAIGW
Outputs:
LoadBalancerDNSName:
Description: The DNSName of the load balancer
Value: !GetAtt SAelb.DNSName
what is the problem with this, is ELB configuration all correct?.
I have run your template with CloudFormation Linter and got the following warnings:
E3012 Property Resources/SAVPC/Properties/EnableDnsSupport should be of type Boolean
logesh.template:33:9
E3012 Property Resources/SAVPC/Properties/EnableDnsHostnames should be of type Boolean
logesh.template:34:9
W3010 Don't hardcode eu-west-1a for AvailabilityZones
logesh.template:45:9
E3012 Property Resources/PublicSubnetA/Properties/MapPublicIpOnLaunch should be of type Boolean
logesh.template:46:9
W3010 Don't hardcode eu-west-1b for AvailabilityZones
logesh.template:58:9
E3012 Property Resources/PublicSubnetB/Properties/MapPublicIpOnLaunch should be of type Boolean
logesh.template:59:9
W3010 Don't hardcode eu-west-1a for AvailabilityZones
logesh.template:71:9
W3010 Don't hardcode eu-west-1b for AvailabilityZones
logesh.template:83:9
E3012 Property Resources/SAInstance1/Properties/DisableApiTermination should be of type Boolean
logesh.template:133:9
E3012 Property Resources/SAInstance1/Properties/Monitoring should be of type Boolean
logesh.template:137:9
E3012 Property Resources/SAInstance1/Properties/NetworkInterfaces/0/AssociatePublicIpAddress should be of type Boolean
logesh.template:145:11
E3012 Property Resources/SAInstance1/Properties/NetworkInterfaces/0/DeleteOnTermination should be of type Boolean
logesh.template:146:11
E3012 Property Resources/SAInstance1/Properties/NetworkInterfaces/0/DeviceIndex should be of type String
logesh.template:148:11
E3012 Property Resources/SANACLEntry1/Properties/Egress should be of type Boolean
logesh.template:200:9
E3012 Property Resources/SANACLEntry1/Properties/Protocol should be of type Integer
logesh.template:201:9
E3012 Property Resources/SANACLEntry1/Properties/RuleNumber should be of type Integer
logesh.template:203:9
E3012 Property Resources/SANACLEntry2/Properties/Protocol should be of type Integer
logesh.template:210:9
E3012 Property Resources/SANACLEntry2/Properties/RuleNumber should be of type Integer
logesh.template:212:9
W3005 Obsolete DependsOn on resource (SAIGW), dependency already enforced by a "Ref" at Resources/publicroute/Properties/GatewayId/Ref
logesh.template:275:7
You might want to correct these problems before proceeding.
ELB is not configured to receive public internet traffic.
ELB is not configured for the right availability zone where the EC2 instance is running
ELB health check is not configured to access HTTP 80 to EC2
EC2 is not configured to receive traffic from the ELB

error CloudFormation template removing instead of creating new elements

I am new to AWS Cloud Formation, well I am reusing 2 templates, the first one works totally fine, it creates a Network Stack for AWS Fargate, please see template #1 below, but the second one (which is failing) supposed to creates the services, instead it is trying to delete most of the elements of the Network Stack, please see template #2 below.
I can see in the "Changes Preview" how it is marking to "remove" almost everything that I created before with the Network Stack template, please see image below #3.
Does somebody can advise what is wrong with the second template?, thank you.
1) Network Stack
AWSTemplateFormatVersion: '2010-09-09'
Description: Create a network stack with a public vpc, fargate cluster and load balancer as a parent stack.
Mappings:
SubnetConfig:
VPC:
CIDR: '10.0.0.0/16'
PublicOne:
CIDR: '10.0.0.0/24'
PublicTwo:
CIDR: '10.0.1.0/24'
Resources:
FargateVpc:
Type: AWS::EC2::VPC
Properties:
EnableDnsSupport: true
EnableDnsHostnames: true
CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR']
PublicSubnetOne:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref FargateVpc
CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR']
MapPublicIpOnLaunch: true
PublicSubnetTwo:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 1
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref FargateVpc
CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR']
MapPublicIpOnLaunch: true
InternetGateway:
Type: AWS::EC2::InternetGateway
GatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref FargateVpc
InternetGatewayId: !Ref InternetGateway
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref FargateVpc
PublicRoute:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: '0.0.0.0/0'
GatewayId: !Ref InternetGateway
PublicSubnetOneRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetOne
RouteTableId: !Ref PublicRouteTable
PublicSubnetTwoRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetTwo
RouteTableId: !Ref PublicRouteTable
# ECS Cluster
ECSCluster:
Type: AWS::ECS::Cluster
# ECS Roles
# ECS Roles
# This role is used by the ECS tasks themselves.
ECSTaskExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [ecs-tasks.amazonaws.com]
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: AmazonECSTaskExecutionRolePolicy
PolicyDocument:
Statement:
- Effect: Allow
Action:
# Allow the ECS Tasks to download images from ECR
- 'ecr:GetAuthorizationToken'
- 'ecr:BatchCheckLayerAvailability'
- 'ecr:GetDownloadUrlForLayer'
- 'ecr:BatchGetImage'
# Allow the ECS tasks to upload logs to CloudWatch
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
# This is an IAM role which authorizes ECS to manage resources on our
# account on our behalf, such as updating our load balancer with the
# details of where our containers are, so that traffic can reach your
# containers.
ECSRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [ecs.amazonaws.com]
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: ecs-service
PolicyDocument:
Statement:
- Effect: Allow
Action:
# Rules which allow ECS to attach network interfaces to instances
# on our behalf in order for awsvpc networking mode to work right
- 'ec2:AttachNetworkInterface'
- 'ec2:CreateNetworkInterface'
- 'ec2:CreateNetworkInterfacePermission'
- 'ec2:DeleteNetworkInterface'
- 'ec2:DeleteNetworkInterfacePermission'
- 'ec2:Describe*'
- 'ec2:DetachNetworkInterface'
# Rules which allow ECS to update load balancers on our behalf
# with the information about how to send traffic to our containers
- 'elasticloadbalancing:DeregisterInstancesFromLoadBalancer'
- 'elasticloadbalancing:DeregisterTargets'
- 'elasticloadbalancing:Describe*'
- 'elasticloadbalancing:RegisterInstancesWithLoadBalancer'
- 'elasticloadbalancing:RegisterTargets'
Resource: '*'
# Load Balancer Security group
PublicLoadBalancerSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to the public facing load balancer from entire internet range
VpcId: !Ref FargateVpc
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
IpProtocol: -1
# Fargate Container Security Group
FargateContainerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to fargate containers
VpcId: !Ref FargateVpc
EcsSecurityGroupIngressFromPublicALB:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Ingress from the public ALB
GroupId: !Ref FargateContainerSecurityGroup
IpProtocol: -1
SourceSecurityGroupId: !Ref PublicLoadBalancerSG
EcsSecurityGroupIngressFromSelf:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Ingress from other containers in the same security group
GroupId: !Ref FargateContainerSecurityGroup
IpProtocol: -1
SourceSecurityGroupId: !Ref FargateContainerSecurityGroup
# Load Balancer
PublicLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Scheme: internet-facing
LoadBalancerAttributes:
- Key: idle_timeout.timeout_seconds
Value: '30'
Subnets:
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
SecurityGroups: [!Ref 'PublicLoadBalancerSG']
# Target Group
DummyTargetGroupPublic:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 6
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
Name: !Join ['-', [!Ref 'AWS::StackName', 'drop-1']]
Port: 80
Protocol: HTTP
UnhealthyThresholdCount: 2
VpcId: !Ref 'FargateVpc'
# Listener
PublicLoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
DependsOn:
- PublicLoadBalancer
Properties:
DefaultActions:
- TargetGroupArn: !Ref 'DummyTargetGroupPublic'
Type: 'forward'
LoadBalancerArn: !Ref 'PublicLoadBalancer'
Port: 80
Protocol: HTTP
Outputs:
VPCId:
Description: The ID of the vpc that this stack is deployed on
Value: !Ref FargateVpc
Export:
Name: !Join [':', [!Ref 'AWS::StackName', 'VPCId']]
PublicSubnetOne:
Description: Public subnet one
Value: !Ref 'PublicSubnetOne'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnetOne' ] ]
PublicSubnetTwo:
Description: Public subnet two
Value: !Ref 'PublicSubnetTwo'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnetTwo' ] ]
FargateContainerSecurityGroup:
Description: A security group used to allow Fargate containers to receive traffic
Value: !Ref 'FargateContainerSecurityGroup'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'FargateContainerSecurityGroup' ] ]
# ECS Outputs
ClusterName:
Description: The name of the ECS cluster
Value: !Ref 'ECSCluster'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ClusterName' ] ]
ECSRole:
Description: The ARN of the ECS role
Value: !GetAtt 'ECSRole.Arn'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ECSRole' ] ]
ECSTaskExecutionRole:
Description: The ARN of the ECS role
Value: !GetAtt 'ECSTaskExecutionRole.Arn'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ECSTaskExecutionRole' ] ]
PublicListener:
Description: The ARN of the public load balancer's Listener
Value: !Ref PublicLoadBalancerListener
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicListener' ] ]
ExternalUrl:
Description: The url of the external load balancer
Value: !Join ['', ['http://', !GetAtt 'PublicLoadBalancer.DNSName']]
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ExternalUrl' ] ]
2) Service Stack
AWSTemplateFormatVersion: '2010-09-09'
Description: Deploy a service on AWS Fargate, hosted in a public subnet of a VPC, and accessible via a public load balancer
# Input Paramters
Parameters:
StackName:
Type: String
Default: test-fargate
Description: The name of the parent fargate networking stack
ServiceName:
Type: String
Default: nginx
Description: Name of the ECS service
ImageUrl:
Type: String
Default: nginx
Description: The url of a docker image that contains the application process that
will handle the traffic for this service
ContainerPort:
Type: Number
Default: 80
Description: What port number the application inside the docker container is binding to
ContainerCpu:
Type: Number
Default: 256
Description: How much CPU to give the container. 1024 is 1 CPU
ContainerMemory:
Type: Number
Default: 512
Description: How much memory in megabytes to give the container
Path:
Type: String
Default: "*"
Description: A path on the public load balancer that this service
should be connected to. Use * to send all load balancer
traffic to this service.
Priority:
Type: Number
Default: 1
Description: The priority for the routing rule added to the load balancer.
This only applies if your have multiple services which have been
assigned to different paths on the load balancer.
DesiredCount:
Type: Number
Default: 2
Description: How many copies of the service task to run
Role:
Type: String
Default: ""
Description: (Optional) An IAM role to give the service's containers if the code within needs to
access other AWS resources like S3 buckets, DynamoDB tables, etc
Conditions:
HasCustomRole: !Not [!Equals [!Ref 'Role', '']]
# Task Definition
Resources:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Ref 'ServiceName'
Cpu: !Ref 'ContainerCpu'
Memory: !Ref 'ContainerMemory'
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn:
Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'ECSTaskExecutionRole']]
TaskRoleArn:
Fn::If:
- 'HasCustomRole'
- !Ref 'Role'
- !Ref "AWS::NoValue"
ContainerDefinitions:
- Name: !Ref 'ServiceName'
Cpu: !Ref 'ContainerCpu'
Memory: !Ref 'ContainerMemory'
Image: !Ref 'ImageUrl'
PortMappings:
- ContainerPort: !Ref 'ContainerPort'
# ALB Target Group
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 6
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
TargetType: ip
Name: !Ref 'ServiceName'
Port: !Ref 'ContainerPort'
Protocol: HTTP
UnhealthyThresholdCount: 2
VpcId:
Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'VPCId']]
# ALB Rule
LoadBalancerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
Actions:
- TargetGroupArn: !Ref 'TargetGroup'
Type: 'forward'
Conditions:
- Field: path-pattern
Values: [!Ref 'Path']
ListenerArn:
Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'PublicListener']]
Priority: !Ref 'Priority'
# ECS or Fargate Service
Service:
Type: AWS::ECS::Service
DependsOn: LoadBalancerRule
Properties:
ServiceName: !Ref 'ServiceName'
Cluster:
Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'ClusterName']]
LaunchType: FARGATE
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 75
DesiredCount: !Ref 'DesiredCount'
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'FargateContainerSecurityGroup']]
Subnets:
- Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'PublicSubnetOne']]
- Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'PublicSubnetTwo']]
TaskDefinition: !Ref TaskDefinition
LoadBalancers:
- ContainerName: !Ref 'ServiceName'
ContainerPort: !Ref 'ContainerPort'
TargetGroupArn: !Ref 'TargetGroup'
Based on the comments.
The issue was that the first and second templates were being deployed into one stack. The solution was to deploy the second template as a separate stack.

ComputeEnvironment went INVALID with error: The security group 'XXXX' does not exist

Below my cloudformation template:
I have added all the resource code, please excuse the indentation issue(copy paste thing), I assured you the template is running.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Sets up your AWS Batch Environment for running workflows
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Compute Environment Config
Parameters:
- ComputeEnvironmentName
- VpcId
- SubnetIds
- MinvCpus
- MaxvCpus
- DesiredvCpus
- Label:
default: Job Definition
Parameters:
- JobDefinitionName
- DockerImage
- Vcpus
- Memory
- Command
- RetryNumber
- Label:
default: Job Queue
Parameters:
- JobQueueName
Parameters:
VpcId:
Type: 'AWS::EC2::VPC::Id'
Description: >-
VpcId of where the whole batch should be deployed. The VPC should have
2 private subnets.
SubnetIds:
Type: List<AWS::EC2::Subnet::Id>
Description: Subnets you want your batch compute environment to launch in. Recommend private subnets
MinvCpus:
Type: String
Description: Minimum number of CPUs in the compute environment. Default 0.
Default: 0
AllowedPattern: "[0-9]+"
DesiredvCpus:
Type: String
Description: Desired number of CPUs in the compute environment to launch with. Default 0.
Default: 0
AllowedPattern: "[0-9]+"
MaxvCpus:
Type: String
Description: Maximum number of CPUs in the compute environment. Should be >= than MinCpus
Default: 256
AllowedPattern: "[0-9]+"
RetryNumber:
Type: String
Default: "1"
Description: Number of retries for each AWS Batch job. Integer required.
MaxLength: 1
AllowedPattern: "[1-9]"
ConstraintDescription: Value between 1 and 9
DockerImage:
Type: String
Description: Docker image used to run your jobs
Vcpus:
Type: Number
Description: vCPUs available to Jobs. Default is usually fine
Default: 2
Memory:
Type: Number
Description: Memory (in MB) available to Jobs. Default is usually fine
Default: 2000
JobQueueName:
Type: String
Description: Enter job queue Name
JobDefinitionName:
Type: String
Description: Enter JobDefinition Name for the batch
ComputeEnvironmentName:
Type: String
Description: Enter name of the Compute Environment
VPCCidr:
Type: String
Description: 'Cidr Block of the VPC, allows for ssh access internally.'
Default: '10.0.0.0/8'
MinLength: "9"
MaxLength: "18"
AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
ConstraintDescription: "Must be valid CIDR notation (i.e. x.x.x.x/x)."
Command:
Type: CommaDelimitedList
Description: The command that is passed to the container
CreateNewRepository:
Default: false
Description: >-
Set this to true if you want to create a new Repository, else
it will not create a new one
Type: String
AllowedValues:
- true
- false
RepositoryName:
Type: String
Description: Enter name of the new Repository.
Conditions:
CreateRepository: !Equals
- !Ref CreateNewRepository
- true
isCommandPresent: !Not [!Equals [!Ref CreateNewRepository, '']]
Resources:
JobDefinition:
Type: AWS::Batch::JobDefinition
Properties:
Type: container
JobDefinitionName: !Ref JobDefinitionName
ContainerProperties:
Image: !Ref DockerImage
Vcpus: !Ref Vcpus
Memory: !Ref Memory
Command: !Ref Command
ReadonlyRootFilesystem: true
Privileged: true
RetryStrategy:
Attempts: !Ref RetryNumber
JobQueue:
Type: AWS::Batch::JobQueue
Properties:
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment: !Ref MyComputeEnv
State: ENABLED
Priority: 10
JobQueueName: !Ref JobQueueName
myVPCSecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: Security group for batch process.
SecurityGroupEgress:
- CidrIp: 0.0.0.0/0
IpProtocol: '-1'
SecurityGroupIngress:
- CidrIp: !Ref VPCCidr
IpProtocol: tcp
FromPort: '22'
ToPort: '22'
VpcId: !Ref VpcId
MyComputeEnv:
Type: AWS::Batch::ComputeEnvironment
Properties:
Type: MANAGED
ServiceRole: !GetAtt awsBatchServiceRole.Arn
ComputeEnvironmentName: !Ref ComputeEnvironmentName
ComputeResources:
MinvCpus: !Ref MinvCpus
MaxvCpus: !Ref MaxvCpus
DesiredvCpus: !Ref DesiredvCpus
SecurityGroupIds: [!GetAtt myVPCSecurityGroup.GroupId]
Type: EC2
Subnets: !Ref SubnetIds
InstanceRole: !GetAtt InstanceProfile.Arn
InstanceTypes:
- optimal
State: ENABLED
awsBatchServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- "batch.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole
ecsInstanceRole:
Type: AWS::IAM::Role
Properties:
RoleName: InstanceRole
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
- "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: InstanceProfile
Roles:
- !Ref ecsInstanceRole
MyRepository:
Type: AWS::ECR::Repository
Condition: CreateRepository
Properties:
RepositoryName: !Ref RepositoryName
RepositoryPolicyText:
Version: "2012-10-17"
Statement:
-
Sid: AllowPushPull
Effect: Allow
Principal: "*"
Action:
- "ecr:*"
I am getting this error:
Operation failed, ComputeEnvironment went INVALID with error: CLIENT_ERROR - The security group 'sg-d9b85d91' does not exist
I don't know what is wrong with the code but strangely, the SecurityGroupIds created by myVPCSecurityGroup is sg-2869f263 but ComputeEnvironment is trying to find sg-d9b85d91.
taking a stab in the dark here just working for my mobile phone but I think it's because you don't have a V PC to your computer environment possibly
Disabling the Compute Environment in the UI and enabling it back again fixed the issue.