AWS Fargate: unable to pull secrets or registry auth - amazon-web-services

I am trying to launch a containerized grpc application on aws fargate. I've tested the image locally, and have pushed it to ecr. I've created a task with a role that has permission to reach ecr, yet I am still getting an error pulling the container (error message shown below). I even tried launching the container in a public subnet with internet gateway/route table association that auto assigns public ips and the security group allows all outbound traffic.
The full cloudformation template is given below:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
TicketingAppTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Essential: true
Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/ticketing-app:latest"
Name: ticketing-app
PortMappings:
- ContainerPort: 8080
Cpu: "1 vCPU"
ExecutionRoleArn: !Ref ExecutionRole
Memory: "2 GB" #this is smallest for 1 vcpu .... could maybe decrease
NetworkMode: awsvpc
RuntimePlatform:
CpuArchitecture: X86_64
OperatingSystemFamily: LINUX
ExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: [ sts:AssumeRole ]
Effect: Allow
Principal:
Service: [ ecs-tasks.amazonaws.com ]
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
TicketingEcsService:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref TicketingEcsCluster
LaunchType: FARGATE
#TODO I think we eventually need to specify load balancers here
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED #TODO remove this when done, just seeing if this let's us grab image from ecr?
SecurityGroups: [ !GetAtt TicketingServiceSecurityGroup.GroupId ]
Subnets:
- !Ref TicketingServicePrivateSubnet01
- !Ref TicketingServicePrivateSubnet02
TaskDefinition: !Ref TicketingAppTaskDefinition
TicketingEcsCluster:
Type: AWS::ECS::Cluster
TicketingServiceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: "A security group used for the ticketing app"
VpcId: !Ref TicketingServiceVpc
TicketingServiceVpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
TicketingServicePrivateSubnet01:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Sub "${AWS::Region}a"
VpcId: !Ref TicketingServiceVpc
CidrBlock: 10.0.0.0/18
TicketingServicePrivateSubnet02:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Sub "${AWS::Region}b"
VpcId: !Ref TicketingServiceVpc
CidrBlock: 10.0.64.0/18
#TODO public subnets and NAT gateway?
TicketingSecurityGroupHttpIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
#TODO I would probably use load balancer security group name here once we make one instead of opening up to any ip
GroupId: !GetAtt TicketingServiceSecurityGroup.GroupId
CidrIpv6: "::/0"
FromPort: 8080
IpProtocol: tcp
ToPort: 8080
TicketingSecurityGroupAllTrafficEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !GetAtt TicketingServiceSecurityGroup.GroupId
IpProtocol: "-1" #-1 indicates all -- like a wildcard
CidrIp: "0.0.0.0/0"
TicketingServiceInternetGateway:
Type: AWS::EC2::InternetGateway
DependsOn: TicketingServiceVpc
AttachGateway:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref TicketingServiceVpc
InternetGatewayId: !Ref TicketingServiceInternetGateway
TicketingAppRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref TicketingServiceVpc
TicketingVPCRouteAllTrafficToInternetGateway:
Type: AWS::EC2::Route
DependsOn: AttachGateway
Properties:
RouteTableId: !Ref TicketingAppRouteTable
DestinationCidrBlock: "0.0.0.0/0"
GatewayId: !Ref TicketingServiceInternetGateway
Would anyone be able to point out a simple way to get this working (even if it uses not best practices, such as public subnet instead of private subnet and nat gateway) -- it is just for a poc.
Thanks

I had pushed my image to ecr in a different region..... changed the region and it worked (face palm).
So check your region if anyone else is in the same spot as me.

Related

AWS CloudFormation service was unable to place a task because no container instance met all of its requirements

I am trying to create an ECS using a ci/cd pipeline. I have defined TaskDefination and ECSService along with VPC.
The cloudformation created the cluster and got stuck ECSService creation.
I went to the ECSService event, I found the error 'service my-service-name was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster. For more information, see the Troubleshooting section.'
Am I missing someting in my pipeline?
Here is my TaskDefination and ECSService
AWSTemplateFormatVersion: 2010-09-09
Description: The CloudFormation template for the Fargate ECS Cluster.
Parameters:
Stage:
Type: String
ContainerPort:
Type: Number
ImageURI:
Type: String
Resources:
# Create an ECS Cluster
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Join ['-', [!Ref Stage, !Ref 'AWS::AccountId', 'Cluster']]
# Create a VPC
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsHostnames: True
EnableDnsSupport: True
# Create a Subnet
SubnetA:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.0.0/24
VpcId: !Ref VPC
AvailabilityZone: !Join ['', [!Ref "AWS::Region", 'a']]
# Create a Subnet
SubnetB:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.1.0/24
VpcId: !Ref VPC
AvailabilityZone: !Join ['', [!Ref "AWS::Region", 'b']]
# Create Access Role for ECS-Tasks
ExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['-', [!Ref Stage, !Ref 'AWS::AccountId', 'ExecutionRole']]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
# Create a TaskDefinition with container details
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
NetworkMode: awsvpc
RequiresCompatibilities:
- 'EC2'
TaskRoleArn: !Ref ExecutionRole
ExecutionRoleArn: !Ref ExecutionRole
ContainerDefinitions:
- Name: !Join ['-', [!Ref Stage, !Ref 'AWS::AccountId', 'Container']]
Image: !Ref ImageURI
Cpu: 1024
Memory: 1024
PortMappings:
- ContainerPort: !Ref ContainerPort
HostPort: !Ref ContainerPort
# Create an ECS Service and add created Cluster, TaskDefintion, Subnets, TargetGroup and SecurityGroup
ECSService:
Type: AWS::ECS::Service
Properties:
ServiceName: !Join ['-', [!Ref Stage, !Ref 'AWS::AccountId', 'ECSService']]
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDefinition
DesiredCount: 1
LaunchType: EC2
NetworkConfiguration:
AwsvpcConfiguration:
Subnets:
- !Ref SubnetA
- !Ref SubnetB
I have tried answers of already posted questions. In most of cases people get this error on AWS web interface. For me ECS is working using Web interface. I am not able to get it working using my pipeline.
You have to explicitly provision EC2 container instances for your ECS tasks. Your current TF code does not create any EC2 instances for used by your ECS cluster and tasks.
'No container instances found' is the error. You have created an empty cluster with no instances in the cluster
You can manually do it as per this doc
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html

Translate ECS service to cloudformation

I am trying to translate a manually created service to a cloudformation template but I keep getting errors.
Task definition is already created with UI because it needs some specific roles
This template gives me: Classic Load Balancers are not supported with Fargate
ServicesSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for cluster services
VpcId: !Ref 'VPC'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 0
ToPort: 65535
SourceSecurityGroupId: !Ref "PublicLoadBalancerSG"
ServiceStaging:
Type: AWS::ECS::Service
Properties:
ServiceName: pouch-svc-staging
TaskDefinition: pouch-td-staging:4
Cluster: !Ref 'ClusterECS'
DesiredCount: 2
SchedulingStrategy: REPLICA
LaunchType: FARGATE
EnableECSManagedTags: true
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DeploymentCircuitBreaker:
Enable: false
Rollback: false
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
SecurityGroups:
- !Ref ServicesSG
LoadBalancers:
- ContainerName: pouch-image-staging
LoadBalancerName: !Ref 'LoadBalancerName'
ContainerPort: 3100
Update: Here is the modified full cloudformation template without a load balancer name as explicited by some comments
AWSTemplateFormatVersion: '2010-09-09'
Description: VPC, subnets and external, public facing load balancer, for forwarding public traffic to containers
Parameters:
LoadBalancerName:
Type: String
Default: pouch-api-elb
ClusterName:
Type: String
Default: pouch-api-cluster
Mappings:
SubnetConfig:
VPC:
CIDR: '172.16.0.0/16'
PublicOne:
CIDR: '172.16.0.0/24'
PublicTwo:
CIDR: '172.16.1.0/24'
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
EnableDnsSupport: true
EnableDnsHostnames: true
CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR']
PublicSubnetOne:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR']
MapPublicIpOnLaunch: true
PublicSubnetTwo:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 1
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR']
MapPublicIpOnLaunch: true
InternetGateway:
Type: AWS::EC2::InternetGateway
GatewayAttachement:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref 'VPC'
InternetGatewayId: !Ref 'InternetGateway'
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
PublicRoute:
Type: AWS::EC2::Route
DependsOn: GatewayAttachement
Properties:
RouteTableId: !Ref 'PublicRouteTable'
DestinationCidrBlock: '0.0.0.0/0'
GatewayId: !Ref 'InternetGateway'
PublicSubnetOneRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetOne
RouteTableId: !Ref PublicRouteTable
PublicSubnetTwoRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetTwo
RouteTableId: !Ref PublicRouteTable
PublicLoadBalancerSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to the public facing load balancer
VpcId: !Ref 'VPC'
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
IpProtocol: -1
PublicLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Ref 'LoadBalancerName'
Scheme: internet-facing
LoadBalancerAttributes:
- Key: idle_timeout.timeout_seconds
Value: "30"
Subnets:
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
SecurityGroups: [!Ref "PublicLoadBalancerSG"]
TargetGroupStaging:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 6
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
Port: 80
Protocol: HTTP
TargetType: ip
UnhealthyThresholdCount: 2
VpcId: !Ref 'VPC'
TargetGroupProduction:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 6
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
Port: 80
Protocol: HTTP
TargetType: ip
UnhealthyThresholdCount: 2
VpcId: !Ref 'VPC'
ListenerRuleProduction:
Type: "AWS::ElasticLoadBalancingV2::ListenerRule"
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref "TargetGroupProduction"
Conditions:
- Field: path-pattern
PathPatternConfig:
Values:
- /production/*
ListenerArn: !Ref PublicLoadBalancerListener
Priority: 100
ListenerRuleStaging:
Type: "AWS::ElasticLoadBalancingV2::ListenerRule"
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref "TargetGroupStaging"
Conditions:
- Field: path-pattern
PathPatternConfig:
Values:
- /staging/*
ListenerArn: !Ref PublicLoadBalancerListener
Priority: 50
PublicLoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: "redirect"
RedirectConfig:
Protocol: "#{protocol}"
Port: "#{port}"
Host: "#{host}"
Path: "/production/"
StatusCode: "HTTP_301"
LoadBalancerArn: !Ref "PublicLoadBalancer"
Port: 80
Protocol: HTTP
ClusterECS:
Type: AWS::ECS::Cluster
DependsOn: PublicLoadBalancerListener
Properties:
ClusterName: !Ref 'ClusterName'
CapacityProviders:
- FARGATE
ServicesSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for cluster services
VpcId: !Ref 'VPC'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 0
ToPort: 65535
SourceSecurityGroupId: !Ref "PublicLoadBalancerSG"
ServiceStaging:
Type: AWS::ECS::Service
Properties:
ServiceName: pouch-svc-staging
TaskDefinition: pouch-td-staging:4
Cluster: !Ref 'ClusterECS'
DesiredCount: 2
SchedulingStrategy: REPLICA
LaunchType: FARGATE
EnableECSManagedTags: true
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DeploymentCircuitBreaker:
Enable: false
Rollback: false
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
SecurityGroups:
- !Ref ServicesSG
LoadBalancers:
- ContainerName: pouch-image-staging
TargetGroupArn: !Ref 'TargetGroupStaging'
ContainerPort: 3100
ServiceProduction:
Type: AWS::ECS::Service
Properties:
ServiceName: pouch-svc-production
TaskDefinition: pouch-td-production:4
Cluster: !Ref 'ClusterECS'
DesiredCount: 2
SchedulingStrategy: REPLICA
LaunchType: FARGATE
EnableECSManagedTags: true
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DeploymentCircuitBreaker:
Enable: false
Rollback: false
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
SecurityGroups:
- !Ref ServicesSG
LoadBalancers:
- ContainerName: pouch-image-production
TargetGroupArn: !Ref 'TargetGroupProduction'
ContainerPort: 3100
Outputs:
VpcId:
Description: The ID of the VPC that this stack is deployed in
Value: !Ref 'VPC'
PublicSubnetOne:
Description: Public subnet one
Value: !Ref 'PublicSubnetOne'
PublicSubnetTwo:
Description: Public subnet two
Value: !Ref 'PublicSubnetTwo'
ExternalUrl:
Description: The url of the external load balancer
Value: !Sub http://${PublicLoadBalancer.DNSName}
And now I am getting "The target group with targetGroupArn arn:aws:elasticloadbalancing: ... :targetgroup/pouch-Targe-XFJ4AI7HCF6G/f2a665925da27326 does not have an associated load balancer"
From official AWS documentation:
LoadBalancerName
The name of the load balancer to associate with the Amazon ECS service or task set.
A load balancer name is only specified when using a Classic Load Balancer. If you are using an Application Load Balancer or a Network Load Balancer the load balancer name parameter should be omitted.
That means that you should not specify the name of the load balancer in the CloudFormation template, since you're using Fargate, and consequently, don't use the classic load balancer.
Also, in your place, I would consider using something like Former2. It's a great tool that can generate CloudFormation template for you. You can scan your account, choose the services for which you want a template created. In order to use it though, you need an Access key and secret key, so consider creating an IAM user with read-only privilege.
Edit to cover 2nd error about target group:
The Amazon ECS service requires an explicit dependency on the
Application Load Balancer listener rule and the Application Load
Balancer listener. This prevents the service from starting before the
listener is ready.
It's possible that AWS::ECS::Service is trying to attach to the target group before the target group is added to the load balancer. In order to fix that, you should add a dependency in your Service, like this:
Type: AWS::ECS::Service
DependsOn: Listener # Add exact listener name, depending on the service
Properties:
That should help fix the issue.
I cannot add a comment yet. I think the following link could help with the issue Classic Load Balancers are not supported with Fargate
AWS Load Balancing multiple ports for an ECS Service with Fargate

AWS ECS Scheduled task not running when released by CI/CD

I'm experiencing a very annoying problem. I created a CI/CD pipelines using AWS CodePipeline and CloudFormation.
This is the template.yml used by CloudFormation to create a ScheduledTask on ECS.
AWSTemplateFormatVersion: "2010-09-09"
Description: Template for deploying a ECR image on ECS
Resources:
VPC:
Type: "AWS::EC2::VPC"
Properties:
CidrBlock: "10.0.0.0/16"
EnableDnsSupport: true
EnableDnsHostnames: true
InstanceTenancy: default
Subnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [0, !GetAZs ""]
CidrBlock: !Sub "10.0.0.0/20"
MapPublicIpOnLaunch: true
Subnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [1, !GetAZs ""]
CidrBlock: !Sub "10.0.32.0/20"
MapPublicIpOnLaunch: true
InternetGateway:
Type: "AWS::EC2::InternetGateway"
VPCGatewayAttachment:
Type: "AWS::EC2::VPCGatewayAttachment"
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref VPC
RouteTable:
Type: "AWS::EC2::RouteTable"
Properties:
VpcId: !Ref VPC
RouteTableAssociation1:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Properties:
SubnetId: !Ref Subnet1
RouteTableId: !Ref RouteTable
RouteTableAssociation2:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Properties:
SubnetId: !Ref Subnet2
RouteTableId: !Ref RouteTable
InternetRoute:
Type: "AWS::EC2::Route"
DependsOn: VPCGatewayAttachment
Properties:
GatewayId: !Ref InternetGateway
RouteTableId: !Ref RouteTable
DestinationCidrBlock: "0.0.0.0/0"
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: "SLAComputation"
LoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: ecs-services
Subnets:
- !Ref "Subnet1"
- !Ref "Subnet2"
SecurityGroups:
- !Ref LoadBalancerSecurityGroup
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
LoadBalancerArn: !Ref LoadBalancer
Protocol: HTTP
Port: 80
DefaultActions:
- Type: forward
TargetGroupArn: !Ref DefaultTargetGroup
LoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for loadbalancer to services on ECS
VpcId: !Ref "VPC"
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
IpProtocol: -1
DefaultTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: default
VpcId: !Ref "VPC"
Protocol: "HTTP"
Port: "80"
CloudWatchLogsGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: "sla_computation"
RetentionInDays: 1
ContainerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref "VPC"
GroupDescription: for ecs containers
SecurityGroupIngress:
- SourceSecurityGroupId: !Ref "LoadBalancerSecurityGroup"
IpProtocol: -1
Task:
Type: AWS::ECS::TaskDefinition
Properties:
Family: apis
Cpu: 1024
Memory: 2048
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn: !Ref ECSTaskExecutionRole
ContainerDefinitions:
- Name: ass001
Image: !Sub 649905970782.dkr.ecr.eu-west-1.amazonaws.com/ass001:latest
Cpu: 1024
Memory: 2048
HealthCheck:
Command: [ "CMD-SHELL", "exit 0" ]
Interval: 30
Retries: 5
Timeout: 10
StartPeriod: 30
PortMappings:
- ContainerPort: 8080
Protocol: tcp
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: "sla_computation"
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: "ass001"
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: employee-tg
VpcId: !Ref VPC
Port: 80
Protocol: HTTP
Matcher:
HttpCode: 200-299
TargetType: ip
ListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn: !Ref LoadBalancerListener
Priority: 2
Conditions:
- Field: path-pattern
Values:
- /*
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward
ECSTaskExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [ecs-tasks.amazonaws.com]
Action: ["sts:AssumeRole"]
Path: /
Policies:
- PolicyName: AmazonECSTaskExecutionRolePolicy
PolicyDocument:
Statement:
- Effect: Allow
Action:
# ECS Tasks to download images from ECR
- "ecr:GetAuthorizationToken"
- "ecr:BatchCheckLayerAvailability"
- "ecr:GetDownloadUrlForLayer"
- "ecr:BatchGetImage"
# ECS tasks to upload logs to CloudWatch
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Resource: "*"
TaskSchedule:
Type: AWS::Events::Rule
Properties:
Description: SLA rule ass001
Name: ass001
ScheduleExpression: cron(0/5 * * * ? *)
State: ENABLED
Targets:
- Arn:
!GetAtt ECSCluster.Arn
Id: dump-data-ecs-task
RoleArn:
!GetAtt ECSTaskExecutionRole.Arn
EcsParameters:
TaskDefinitionArn:
!Ref Task
TaskCount: 1
LaunchType: FARGATE
PlatformVersion: LATEST
NetworkConfiguration:
AwsVpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- sg-07db5ae6616a8c5fc
Subnets:
- subnet-031d0787ad492c1c4
TaskSchedule:
Type: AWS::Events::Rule
Properties:
Description: SLA rule ass002
Name: ass002
ScheduleExpression: cron(0/5 * * * ? *)
State: ENABLED
Targets:
- Arn:
!GetAtt ECSCluster.Arn
Id: dump-data-ecs-task
RoleArn:
!GetAtt ECSTaskExecutionRole.Arn
EcsParameters:
TaskDefinitionArn:
!Ref Task
TaskCount: 1
LaunchType: FARGATE
PlatformVersion: LATEST
NetworkConfiguration:
AwsVpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- sg-07db5ae6616a8c5fc
Subnets:
- subnet-031d0787ad492c1c4
Outputs:
ApiEndpoint:
Description: Employee API Endpoint
Value: !Join ["", ["http://", !GetAtt LoadBalancer.DNSName, "/employees"]]
Export:
Name: "EmployeeApiEndpoint"
The ScheduledTask is created successfully but it is not running actually. Very strange. But the strangest thing is that the ScheduledTask starts working when I click on "Edit" from the AWS console and (without making any change) I save.
The main issue I see is that you are using wrong role for your scheduled rule. It can't be !GetAtt ECSTaskExecutionRole.Arn. Instead you should create new role (or edit existing one) which has AmazonEC2ContainerServiceEventsRole AWS Managed policy.
It works after you edit in console, because AWS console will probably create the correct role in the background and use it instead of yours.

NodeGroup is not joining EKS cluster when using CloudFormation

I have been following this guide to create a Kubernetes cluster via CloudFormation, but the NodeGroup never joins the cluster, and I never get an error or explanation about why is not joining.
I can see the autoscaling group and the EC2 machines are created, but EKS reports that there is not node groups.
If I create a new node group manually through the web admin tool, it works, but it assigns different security groups. It has a launch template instead of a launch configuration.
Same AMI, same IAM role, same machine type...
I am very new in both CloudFormation and EKS, and I don't know how to proceed now to find out what the problem is.
Here is the template:
Description: >
Kubernetes cluster
Parameters:
EnvironmentName:
Description: An environment name that will be prefixed to resource names
Type: String
KeyName:
Description: The EC2 Key Pair to allow SSH access to the instances
Type: AWS::EC2::KeyPair::KeyName
VpcBlock:
Type: String
Default: 192.168.0.0/16
Description: The CIDR range for the VPC. This should be a valid private (RFC 1918) CIDR range.
Subnet01Block:
Type: String
Default: 192.168.64.0/18
Description: CidrBlock for subnet 01 within the VPC
Subnet02Block:
Type: String
Default: 192.168.128.0/18
Description: CidrBlock for subnet 02 within the VPC
Subnet03Block:
Type: String
Default: 192.168.192.0/18
Description: CidrBlock for subnet 03 within the VPC. This is used only if the region has more than 2 AZs.
NodeInstanceType:
Description: EC2 instance type for the node instances
Type: String
NodeImageId:
Type: AWS::EC2::Image::Id
Description: AMI id for the node instances.
NodeAutoScalingGroupMinSize:
Type: Number
Description: Minimum size of Node Group ASG.
Default: 1
NodeAutoScalingGroupMaxSize:
Type: Number
Description: Maximum size of Node Group ASG. Set to at least 1 greater than NodeAutoScalingGroupDesiredCapacity.
Default: 3
NodeAutoScalingGroupDesiredCapacity:
Type: Number
Description: Desired capacity of Node Group ASG.
Default: 3
BootstrapArguments:
Description: Arguments to pass to the bootstrap script. See files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami
Default: ""
Type: String
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcBlock
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Environment
Value: !Ref EnvironmentName
InternetGateway:
Type: "AWS::EC2::InternetGateway"
Properties:
Tags:
- Key: Environment
Value: !Ref EnvironmentName
VPCGatewayAttachment:
Type: "AWS::EC2::VPCGatewayAttachment"
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref VPC
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Environment
Value: !Ref EnvironmentName
Route:
DependsOn: VPCGatewayAttachment
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref RouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
Subnet01:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: !Ref Subnet01Block
VpcId: !Ref VPC
MapPublicIpOnLaunch: true
Tags:
- Key: Environment
Value: !Ref EnvironmentName
Subnet02:
Type: AWS::EC2::Subnet
Metadata:
Comment: Subnet 02
Properties:
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: !Ref Subnet02Block
VpcId: !Ref VPC
MapPublicIpOnLaunch: true
Tags:
- Key: Environment
Value: !Ref EnvironmentName
Subnet03:
Type: AWS::EC2::Subnet
Metadata:
Comment: Subnet 03
Properties:
AvailabilityZone: !Select [ 2, !GetAZs '' ]
CidrBlock: !Ref Subnet03Block
VpcId: !Ref VPC
MapPublicIpOnLaunch: true
Tags:
- Key: Environment
Value: !Ref EnvironmentName
Subnet01RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref Subnet01
RouteTableId: !Ref RouteTable
Subnet02RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref Subnet02
RouteTableId: !Ref RouteTable
Subnet03RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref Subnet03
RouteTableId: !Ref RouteTable
ControlPlaneSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Cluster communication with worker nodes
VpcId: !Ref VPC
ClusterRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${EnvironmentName}KubernetesClusterRole
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: eks.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEKSServicePolicy
- arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
Tags:
- Key: Environment
Value: !Ref EnvironmentName
Cluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Sub ${EnvironmentName}KubernetesCluster
RoleArn: !GetAtt ClusterRole.Arn
ResourcesVpcConfig:
SecurityGroupIds:
- !Ref ControlPlaneSecurityGroup
SubnetIds:
- !Ref Subnet01
- !Ref Subnet02
- !Ref Subnet03
NodeRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${EnvironmentName}KubernetesNodeRole
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
Path: /
Tags:
- Key: Environment
Value: !Ref EnvironmentName
NodeInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles:
- !Ref NodeRole
NodeSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for all nodes in the cluster
VpcId: !Ref VPC
Tags:
- Key: !Sub "kubernetes.io/cluster/${EnvironmentName}KubernetesCluster"
Value: 'owned'
- Key: Environment
Value: !Ref EnvironmentName
NodeSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
DependsOn: NodeSecurityGroup
Properties:
Description: Allow node to communicate with each other
GroupId: !Ref NodeSecurityGroup
SourceSecurityGroupId: !Ref NodeSecurityGroup
IpProtocol: '-1'
FromPort: 0
ToPort: 65535
NodeSecurityGroupFromControlPlaneIngress:
Type: AWS::EC2::SecurityGroupIngress
DependsOn: NodeSecurityGroup
Properties:
Description: Allow worker Kubelets and pods to receive communication from the cluster control plane
GroupId: !Ref NodeSecurityGroup
SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup
IpProtocol: tcp
FromPort: 1025
ToPort: 65535
ControlPlaneEgressToNodeSecurityGroup:
Type: AWS::EC2::SecurityGroupEgress
DependsOn: NodeSecurityGroup
Properties:
Description: Allow the cluster control plane to communicate with worker Kubelet and pods
GroupId: !Ref ControlPlaneSecurityGroup
DestinationSecurityGroupId: !Ref NodeSecurityGroup
IpProtocol: tcp
FromPort: 1025
ToPort: 65535
NodeSecurityGroupFromControlPlaneOn443Ingress:
Type: AWS::EC2::SecurityGroupIngress
DependsOn: NodeSecurityGroup
Properties:
Description: Allow pods running extension API servers on port 443 to receive communication from cluster control plane
GroupId: !Ref NodeSecurityGroup
SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup
IpProtocol: tcp
FromPort: 443
ToPort: 443
ControlPlaneEgressToNodeSecurityGroupOn443:
Type: AWS::EC2::SecurityGroupEgress
DependsOn: NodeSecurityGroup
Properties:
Description: Allow the cluster control plane to communicate with pods running extension API servers on port 443
GroupId: !Ref ControlPlaneSecurityGroup
DestinationSecurityGroupId: !Ref NodeSecurityGroup
IpProtocol: tcp
FromPort: 443
ToPort: 443
ClusterControlPlaneSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
DependsOn: NodeSecurityGroup
Properties:
Description: Allow pods to communicate with the cluster API Server
GroupId: !Ref ControlPlaneSecurityGroup
SourceSecurityGroupId: !Ref NodeSecurityGroup
IpProtocol: tcp
ToPort: 443
FromPort: 443
NodeGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
DesiredCapacity: !Ref NodeAutoScalingGroupDesiredCapacity
LaunchConfigurationName: !Ref NodeLaunchConfig
MinSize: !Ref NodeAutoScalingGroupMinSize
MaxSize: !Ref NodeAutoScalingGroupMaxSize
VPCZoneIdentifier:
- !Ref Subnet01
- !Ref Subnet02
- !Ref Subnet03
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}KubernetesCluster-Node"
PropagateAtLaunch: 'true'
- Key: !Sub 'kubernetes.io/cluster/${EnvironmentName}KubernetesCluster'
Value: 'owned'
PropagateAtLaunch: 'true'
UpdatePolicy:
AutoScalingRollingUpdate:
MaxBatchSize: '1'
MinInstancesInService: !Ref NodeAutoScalingGroupDesiredCapacity
PauseTime: 'PT5M'
NodeLaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
AssociatePublicIpAddress: 'true'
IamInstanceProfile: !Ref NodeInstanceProfile
ImageId: !Ref NodeImageId
InstanceType: !Ref NodeInstanceType
KeyName: !Ref KeyName
SecurityGroups:
- !Ref NodeSecurityGroup
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeSize: 20
VolumeType: gp2
DeleteOnTermination: true
UserData:
Fn::Base64:
!Sub |
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh ${EnvironmentName}KubernetesCluster ${BootstrapArguments}
/opt/aws/bin/cfn-signal --exit-code $? \
--stack ${AWS::StackName} \
--resource NodeGroup \
--region ${AWS::Region}
Outputs:
KubernetesClusterName:
Description: Cluster name
Value: !Ref Cluster
Export:
Name: KubernetesClusterName
KubernetesClusterEndpoint:
Description: Cluster endpoint
Value: !GetAtt Cluster.Endpoint
Export:
Name: KubernetesClusterEndpoint
KubernetesNodeInstanceProfile:
Description: The name of the IAM profile for K8
Value: !GetAtt NodeInstanceProfile.Arn
Export:
Name: KubernetesNodeInstanceProfileArn
There are two ways of adding Worker nodes to your EKS cluster:
Launch and register workers on your own (https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html)
Use managed node groups (https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)
As I can see from your template, you are using the first approach by now. Important when doing this is, that you need to wait until the EKS Cluster is ready and in state active, before launching the worker nodes. You can achieve this by using the DependsOn Attribute. If this does not resolve your issues, have a look at the cloud init logs (/var/log/cloud-init-output.log) to check what is happening while joining the cluster.
If you would like to use Managed Node Groups, just remove the AutoScaling Group and LaunchConfiguration and use this type instead: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-nodegroup.html
The benefit is, that AWS takes care of creating the required resources (AutoScaling Group and LaunchTemplate) in your account for you and you can see the Node Group in the AWS Console.
I approached with managed node groups (https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) option. It was working. But how to define the autoscaling policy. It just allows giving max and min node counts not even name.

Creating a publicly accessible RDS instance in AWS Cloudformation

I'm totally throwing my hands up with this one. I've been trying to create a publicly accessible RDS instance using CloudFormation. I want to be able to connect to my instance via a mysql client. When I deploy this stack it says that the instance is publicly accessible in the RDS console, but I can't connect to via the endpoint provided in the RDS console. I'm guessing that I messed up/missed something with the VPC pieces. He's my stack.yaml file:
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsHostnames: true
Tags:
- Key: Name
Value: 'VPC created by cf'
InternetGateway:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
- Key: Name
Value: Created By CF
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref Vpc
InternetGatewayId: !Ref InternetGateway
DataSourceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Open database for access
VpcId: !Ref Vpc
DSSGIngressRule:
Type: AWS::EC2::SecurityGroupIngress
Properties:
FromPort: "3306"
ToPort: "3306"
GroupId: !Ref DataSourceSecurityGroup
IpProtocol: tcp
SourceSecurityGroupId: !Ref DataSourceSecurityGroup
PublicSubnet1:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: us-east-1a
CidrBlock: 10.0.0.0/20
MapPublicIpOnLaunch: true
VpcId: !Ref Vpc
PublicSubnet2:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: us-east-1b
CidrBlock: 10.0.16.0/20
MapPublicIpOnLaunch: true
VpcId: !Ref Vpc
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref Vpc
Tags:
- Key: Name
Value: 'RouteTable created by CF'
RouteTable1Association:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet1
RouteTableId: !Ref RouteTable
RouteTable2Association:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet2
RouteTableId: !Ref RouteTable
InternetRouteRule:
Type: AWS::EC2::Route
DependsOn: VPCGatewayAttachment
Properties:
RouteTableId: !Ref RouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
DataSourceSubtNetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Created by CF
SubnetIds:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
DataSource:
Type: AWS::RDS::DBInstance
Properties:
AllocatedStorage: '5'
DBInstanceClass: db.m1.small
DBName: MyDb
DBSubnetGroupName: !Ref DataSourceSubtNetGroup
Engine: MySQL
MasterUsername: AdminUser
MasterUserPassword: AdminPassword
PubliclyAccessible: true
VPCSecurityGroups:
- !Ref DataSourceSecurityGroup
DeletionPolicy: Snapshot
Thanks
Your DataSourceSecurityGroup security group is currently configured as:
Permit inbounded connections on Port 3306 from Security Group DataSourceSecurityGroup
That is, it will allow inbound connections from any Amazon EC2 instance that is itself a member of the DataSourceSecurityGroup security group.
If you wanted to allow access from anywhere on the Internet, then change your template to permit inbound access from 0.0.0.0/0:
DSSGIngressRule:
Type: AWS::EC2::SecurityGroupIngress
Properties:
FromPort: "3306"
ToPort: "3306"
GroupId: !Ref DataSourceSecurityGroup
IpProtocol: tcp
CidrIp: 0.0.0.0/0
I made this change, tested your template and it worked fine.
For future reference: You can debug this type of thing by creating the stack and then examining the Security Group in the management console.