AWS ELB: 503 Service Temporarily Unavailable - amazon-web-services

Question

I am trying to deploy microservices to AWS ECS following this example repo. Here, a load balancer is utilized for networking between different docker services. I have adjusted the cloudformation templates to my needs and have only deployed one service yet, the webserver. I want public access to the web interface and add other services this service is talking to subsequently. However, I currently struggle to get the webserver going using the URL of the load balancer. If I go directly to the EC2 instance using the public IP, I get access to the interface. However, if I go to the load balancer DNS, I get 503 Service Temporarily Unavailable. I have checked the AWS docs, but the webservice target group shows a registered target (the EC2 instance) with status healthy. What am I missing?
Parameters:
EnvironmentName:
Description: An environment name that will be prefixed to resource names
Type: String
VPC:
Type: AWS::EC2::VPC::Id
Description: Choose which VPC the Application Load Balancer should be deployed to
Subnets:
Description: Choose which subnets the Application Load Balancer should be deployed to
Type: AWS::EC2::Subnet::Id
PublicSubnet:
Description: Choose which public subnet the EC2 instance should be deployed to
Type: AWS::EC2::Subnet::Id
Resources:
LoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${EnvironmentName}-loadbalancer
GroupDescription: Access to the load balancer that sits in front of ECS
VpcId: !Ref VPC
SecurityGroupIngress:
# Allow access from anywhere to our ECS services
- CidrIp: 0.0.0.0/0
IpProtocol: -1
Tags:
- Key: Name
Value: !Sub ${EnvironmentName}-LoadBalancers
LoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Ref EnvironmentName
Subnets:
- !Ref Subnets
SecurityGroups:
- !Ref LoadBalancerSecurityGroup
Tags:
- Key: Name
Value: !Ref EnvironmentName
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
LoadBalancerArn: !Ref LoadBalancer
Port: 80
Protocol: HTTP
DefaultActions:
- Type: forward
TargetGroupArn: !Ref DefaultTargetGroup
DefaultTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${EnvironmentName}-default
VpcId: !Ref VPC
Port: 80
Protocol: HTTP
ECSHostSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${EnvironmentName}-ecs-hosts
GroupDescription: Access to the ECS hosts and the tasks/containers that run on them
VpcId: !Ref VPC
SecurityGroupIngress:
# Only allow inbound access to ECS from the ELB
- SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup
IpProtocol: -1
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
Tags:
- Key: Name
Value: !Sub ${EnvironmentName}-ECS-Hosts
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Ref EnvironmentName
ECSRole:
Type: AWS::IAM::Role
Properties:
Path: /
RoleName: !Sub ${EnvironmentName}-ecs-role
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role'
- 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM'
- 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy'
ECSInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref ECSRole
EC2Webserver:
Type: AWS::EC2::Instance
Properties:
AvailabilityZone: eu-central-1a
ImageId: !Ref ECSAMI
InstanceType: !Ref InstanceType
IamInstanceProfile: !Ref ECSInstanceProfile
UserData:
Fn::Base64:
!Sub |
#!/bin/bash
# Add to cluster:
echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
SecurityGroupIds:
- !Ref ECSHostSecurityGroup
SubnetId: !Ref PublicSubnet
Tags:
- Key: Name
Value: !Sub ${EnvironmentName}
Service:
Type: AWS::ECS::Service
DependsOn: ListenerRule
Properties:
Cluster: !Ref Cluster
Role: !Ref ServiceRole
DesiredCount: !Ref DesiredCount
TaskDefinition: !Ref TaskDefinitionWebserver
LoadBalancers:
- ContainerName: !Sub ${EnvironmentName}-webserver
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroup
TaskDefinitionWebserver:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Sub ${EnvironmentName}-webserver
ContainerDefinitions:
- Name: !Sub ${EnvironmentName}-webserver
Essential: true
Image: !Ref Image
Memory: 512
PortMappings:
- ContainerPort: 8080
HostPort: 80
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${EnvironmentName}-webserver
VpcId: !Ref VPC
Port: 80
Protocol: HTTP
Matcher:
HttpCode: 200-299
HealthCheckIntervalSeconds: 30
HealthCheckPath: /health
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 10
HealthyThresholdCount: 5
ListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn: !Ref LoadBalancerListener
Priority: 1
Conditions:
- Field: path-pattern
Values:
- /
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward
ServiceRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ecs-service-${AWS::StackName}
Path: /
AssumeRolePolicyDocument: |
{
"Statement": [{
"Effect": "Allow",
"Principal": { "Service": [ "ecs.amazonaws.com" ]},
"Action": [ "sts:AssumeRole" ]
}]
}
Policies:
- PolicyName: !Sub ecs-service-${AWS::StackName}
PolicyDocument:
{
"Version": "2012-10-17",
"Statement":
[
{
"Effect": "Allow",
"Action":
[
"ec2:AuthorizeSecurityGroupIngress",
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
}
]
}
Outputs:
WebsiteServiceUrl:
Description: The URL endpoint for the website service
Value: !Join ["", [!GetAtt LoadBalancer.DNSName, "/"]]

Answer 1

Thanks everyone! I finally figured it out, what I had to do was adjusting the path as my service redirects on /. So I only changed the listener rule using a wildcard:
ListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn: !Ref LoadBalancerListener
Priority: 1
Conditions:
- Field: path-pattern
Values:
- [/*]
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward

Answer 2

SG egress rules
It looks the Security Groups (SG) has no egress rule defined. Both ALB and EC2.
LoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${EnvironmentName}-loadbalancer
GroupDescription: Access to the load balancer that sits in front of ECS
VpcId: !Ref VPC
SecurityGroupIngress:
# Allow access from anywhere to our ECS services
- CidrIp: 0.0.0.0/0
IpProtocol: -1
I believe when you look at the outbound rules of the SG in the EC2 console, there would be no rule. If this is true, then the cause would be that traffic can come in to port 80 of ALB, but it cannot go out from ALB,
This is my theory. So please add a egress rule to verify?
LoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${EnvironmentName}-loadbalancer
GroupDescription: Access to the load balancer that sits in front of ECS
VpcId: !Ref VPC
SecurityGroupIngress:
# Allow access from anywhere to our ECS services
- CidrIp: 0.0.0.0/0
IpProtocol: -1
SecurityGroupEgress:
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0 # <--- Maybe better change to VPC CIDR or ECS/EC2 subnet CIDR rather than any IP.
Regarding EC2, because SG is stateful, traffic can go through port 80 to reach the port 8080 of the docker container, and the response can go back through the SG because the SG knows it is the response of the incoming connection.
Whereas for ALB, the incoming connection from the Internet is terminated at ALB port 80, then a new outbound connection needs to be established to EC2 instance(s) port 80, hence needs an egress rule defined, if I am correct.
ECSHostSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${EnvironmentName}-ecs-hosts
GroupDescription: Access to the ECS hosts and the tasks/containers that run on them
VpcId: !Ref VPC
SecurityGroupIngress:
# Only allow inbound access to ECS from the ELB
- SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup
IpProtocol: -1
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
However, if the EC2 instances need to install packages or create an outbound connection, then the EC2 SG needs egress rules too.
IAM
Regarding the IAM role for the ECS service, there are pre-defined AWS managed role, so I suppose better to use them?
AWS::ECS::Service
"Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"]
}
}
As AWS has introduced Service Linked Role now, it should be even better to use it.
Service-Linked Role for Amazon ECS
Prior to the introduction of a service-linked role for Amazon ECS, you were required to create an IAM role for your Amazon ECS services which granted Amazon ECS the permission it needed. This role is no longer required, however it is available if needed. For more information, see Legacy IAM Roles for Amazon ECS.
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"