AWS EKS managed nodegroup created with CloudFormation fails - amazon-web-services

Question

I've created a VPC and EKS cluster using CloudFormation, and when I try to create an AWS managed nodegroup through CloudFormation, it fails to create with the error message: Nodegroup test-ng failed to stabilize: [{Code: NodeCreationFailure,Message: Unhealthy nodes in the kubernetes cluster. I haven't been able to pinpoint the exact issue, but my setup is based on the AWS docs. For reference, I want a VPC with 3 public and 3 private subnets, with the managed nodegroup deployed to the private subnets. Here are the templates I've used to deploy everything:
VPC Template:
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EKS VPC - Private and Public Subnets'
Parameters:
VpcName:
Type: String
Default: EKS-VPC
Description: The name of the VPC
VpcBlock:
Type: String
Default: 10.0.0.0/16
Description: The CIDR range for the VPC. This should be a valid private (RFC 1918) CIDR range.
PrivateSubnet01Block:
Type: String
Default: 10.0.0.0/19
Description: CidrBlock for private subnet 01 within the VPC
PrivateSubnet02Block:
Type: String
Default: 10.0.32.0/19
Description: CidrBlock for private subnet 02 within the VPC
PrivateSubnet03Block:
Type: String
Default: 10.0.64.0/19
Description: CidrBlock for private subnet 03 within the VPC
PublicSubnet01Block:
Type: String
Default: 10.0.128.0/20
Description: CidrBlock for public subnet 01 within the VPC
PublicSubnet02Block:
Type: String
Default: 10.0.144.0/20
Description: CidrBlock for public subnet 02 within the VPC
PublicSubnet03Block:
Type: String
Default: 10.0.160.0/20
Description: CidrBlock for public subnet 02 within the VPC
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: "Main"
Parameters:
- VpcName
-
Label:
default: "Network Configuration"
Parameters:
- VpcBlock
- PublicSubnet01Block
- PublicSubnet02Block
- PublicSubnet03Block
- PrivateSubnet01Block
- PrivateSubnet02Block
- PrivateSubnet03Block
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcBlock
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Name
Value: !Ref VpcName
InternetGateway:
Type: "AWS::EC2::InternetGateway"
VPCGatewayAttachment:
Type: "AWS::EC2::VPCGatewayAttachment"
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref VPC
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: Public Subnets RT
- Key: Network
Value: Public
PrivateRouteTable01:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: Private Subnet 01 RT
- Key: Network
Value: Private
PrivateRouteTable02:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: Private Subnet 02 RT
- Key: Network
Value: Private
PrivateRouteTable03:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: Private Subnet 03 RT
- Key: Network
Value: Private
PublicRoute:
DependsOn: VPCGatewayAttachment
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
PrivateRoute01:
DependsOn:
- VPCGatewayAttachment
- NatGateway01
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTable01
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NatGateway01
PrivateRoute02:
DependsOn:
- VPCGatewayAttachment
- NatGateway02
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTable02
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NatGateway02
PrivateRoute03:
DependsOn:
- VPCGatewayAttachment
- NatGateway03
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTable03
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NatGateway03
NatGateway01:
DependsOn:
- NatGatewayEIP1
- PublicSubnet01
- VPCGatewayAttachment
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt 'NatGatewayEIP1.AllocationId'
SubnetId: !Ref PublicSubnet01
Tags:
- Key: Name
Value: !Sub '${VpcName}-NatGateway01'
NatGateway02:
DependsOn:
- NatGatewayEIP2
- PublicSubnet02
- VPCGatewayAttachment
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt 'NatGatewayEIP2.AllocationId'
SubnetId: !Ref PublicSubnet02
Tags:
- Key: Name
Value: !Sub '${VpcName}-NatGateway02'
NatGateway03:
DependsOn:
- NatGatewayEIP3
- PublicSubnet03
- VPCGatewayAttachment
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt 'NatGatewayEIP3.AllocationId'
SubnetId: !Ref PublicSubnet03
Tags:
- Key: Name
Value: !Sub '${VpcName}-NatGateway03'
NatGatewayEIP1:
DependsOn:
- VPCGatewayAttachment
Type: 'AWS::EC2::EIP'
Properties:
Domain: vpc
NatGatewayEIP2:
DependsOn:
- VPCGatewayAttachment
Type: 'AWS::EC2::EIP'
Properties:
Domain: vpc
NatGatewayEIP3:
DependsOn:
- VPCGatewayAttachment
Type: 'AWS::EC2::EIP'
Properties:
Domain: vpc
PublicSubnet01:
Type: AWS::EC2::Subnet
Metadata:
Comment: Public Subnet 01
Properties:
MapPublicIpOnLaunch: true
AvailabilityZone:
Fn::Select:
- '0'
- Fn::GetAZs:
Ref: AWS::Region
CidrBlock:
Ref: PublicSubnet01Block
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: !Sub "${VpcName}-PublicSubnet01"
- Key: kubernetes.io/role/elb
Value: 1
PublicSubnet02:
Type: AWS::EC2::Subnet
Metadata:
Comment: Public Subnet 02
Properties:
MapPublicIpOnLaunch: true
AvailabilityZone:
Fn::Select:
- '1'
- Fn::GetAZs:
Ref: AWS::Region
CidrBlock:
Ref: PublicSubnet02Block
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: !Sub "${VpcName}-PublicSubnet02"
- Key: kubernetes.io/role/elb
Value: 1
PublicSubnet03:
Type: AWS::EC2::Subnet
Metadata:
Comment: Public Subnet 03
Properties:
MapPublicIpOnLaunch: true
AvailabilityZone:
Fn::Select:
- '2'
- Fn::GetAZs:
Ref: AWS::Region
CidrBlock:
Ref: PublicSubnet03Block
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: !Sub "${VpcName}-PublicSubnet03"
- Key: kubernetes.io/role/elb
Value: 1
PrivateSubnet01:
Type: AWS::EC2::Subnet
Metadata:
Comment: Private Subnet 01
Properties:
AvailabilityZone:
Fn::Select:
- '0'
- Fn::GetAZs:
Ref: AWS::Region
CidrBlock:
Ref: PrivateSubnet01Block
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: !Sub "${VpcName}-PrivateSubnet01"
- Key: kubernetes.io/role/internal-elb
Value: 1
PrivateSubnet02:
Type: AWS::EC2::Subnet
Metadata:
Comment: Private Subnet 02
Properties:
AvailabilityZone:
Fn::Select:
- '1'
- Fn::GetAZs:
Ref: AWS::Region
CidrBlock:
Ref: PrivateSubnet02Block
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: !Sub "${VpcName}-PrivateSubnet02"
- Key: kubernetes.io/role/internal-elb
Value: 1
PrivateSubnet03:
Type: AWS::EC2::Subnet
Metadata:
Comment: Private Subnet 03
Properties:
AvailabilityZone:
Fn::Select:
- '2'
- Fn::GetAZs:
Ref: AWS::Region
CidrBlock:
Ref: PrivateSubnet03Block
VpcId:
Ref: VPC
Tags:
- Key: Name
Value: !Sub "${VpcName}-PrivateSubnet03"
- Key: kubernetes.io/role/internal-elb
Value: 1
PublicSubnet01RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet01
RouteTableId: !Ref PublicRouteTable
PublicSubnet02RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet02
RouteTableId: !Ref PublicRouteTable
PublicSubnet02RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet03
RouteTableId: !Ref PublicRouteTable
PrivateSubnet01RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet01
RouteTableId: !Ref PrivateRouteTable01
PrivateSubnet02RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet02
RouteTableId: !Ref PrivateRouteTable02
PrivateSubnet03RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet03
RouteTableId: !Ref PrivateRouteTable03
ControlPlaneSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Cluster communication with worker nodes
VpcId: !Ref VPC
WorkerNodeSshSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SG for ssh access to worker nodes in managed nodegroup
VpcId: !Ref VPC
Outputs:
PublicSubnetIds:
Description: Public Subnets IDs in the VPC
Value: !Join [ ", ", [ !Ref PublicSubnet01, !Ref PublicSubnet02, !Ref PublicSubnet03 ] ]
PrivateSubnetIds:
Description: Private Subnets IDs in the VPC
Value: !Join [ ", ", [ !Ref PrivateSubnet01, !Ref PrivateSubnet02, !Ref PrivateSubnet03 ] ]
ControlPlaneSecurityGroups:
Description: Security group for the cluster control plane communication with worker nodes
Value: !Join [ ",", [ !Ref ControlPlaneSecurityGroup ] ]
WorkerNodeSshSecurityGroup:
Description: SG for ssh access to worker nodes in managed nodegroup
Value: !Ref WorkerNodeSshSecurityGroup
VpcId:
Description: The VPC Id
Value: !Ref VPC
IAM Roles Template:
Mappings:
ServicePrincipals:
aws-cn:
ec2: ec2.amazonaws.com.cn
aws-us-gov:
ec2: ec2.amazonaws.com
aws:
ec2: ec2.amazonaws.com
Resources:
eksClusterRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- eks.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
NodeInstanceRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- !FindInMap [ServicePrincipals, !Ref "AWS::Partition", ec2]
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy"
- !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
Path: /
Outputs:
eksClusterRoleArn:
Description: The role that Amazon EKS will use to create AWS resources for Kubernetes clusters
Value: !GetAtt eksClusterRole.Arn
NodeInstanceRole:
Description: The node instance role
Value: !GetAtt NodeInstanceRole.Arn
EKS Cluster Template:
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EKS Production-Grade Cluster'
Parameters:
KubernetesVersion:
Type: String
Description: The EKS supported Kubernetes version for your cluster
Default: 1.19
AllowedValues:
- 1.19
- 1.18
- 1.17
ClusterName:
Type: String
Description: The name of the cluster
ControlPlaneClusterRoleArn:
Type: String
Description: The eksClusterRole Arn to use for the eks cluster (control plane)
ControlPlaneSubnetIds:
Type: List<AWS::EC2::Subnet::Id>
Description: Register your public and private subnets with your EKS managed control plane
SecurityGroupIds:
Type: List<AWS::EC2::SecurityGroup::Id>
Description: The security group(s) for the cross-account elastic network interfaces that Amazon EKS creates to use to allow communication between your nodes and the Kubernetes control plane.
Resources:
EKSManagedControlPlane:
Type: AWS::EKS::Cluster
Properties:
KubernetesNetworkConfig:
ServiceIpv4Cidr: 172.16.0.0/12
Name: !Ref ClusterName
ResourcesVpcConfig:
SecurityGroupIds: !Ref SecurityGroupIds
SubnetIds: !Ref ControlPlaneSubnetIds
RoleArn: !Ref ControlPlaneClusterRoleArn
Version: !Ref KubernetesVersion
Managed Nodegroup Template:
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'EKS Production-Grade Nodegroup'
Parameters:
KubernetesVersion:
Type: String
Description: The EKS supported Kubernetes version for your cluster
Default: 1.19
AllowedValues:
- 1.19
- 1.18
- 1.17
ClusterName:
Type: String
Description: The name of the cluster
NodeInstanceRoleArn:
Type: String
Description: The NodeInstanceRole Arn to use for the eks nodegroup (managed data plane)
DataPlanePrivateSubnetIds:
Type: List<AWS::EC2::Subnet::Id>
Description: Private subnets for your Amazon EKS data plane nodes
WorkerNodeGroupName:
Type: String
Description: The name of the node group for the worker nodes in the data plane. Right now we are only supporting 1 node group per cluster.
WorkerNodesInstanceType:
Type: String
Description: The instance type for the worker nodes in the data plane. Right now we are only supporting 1 instance type for all worker nodes.
Default: t3.medium
AllowedValues:
- t3.small
- t3.medium
- t3.large
- m5.large
- m5.xlarge
- c5.large
- c5.xlarge
WorkerNodesEc2SshKey:
Type: AWS::EC2::KeyPair::KeyName
Description: The Amazon EC2 SSH key that provides access for SSH communication with the nodes in the managed node group
SourceSecurityGroupsForWorkerNodes:
Type: List<AWS::EC2::SecurityGroup::Id>
Description: The security groups that are allowed SSH access (port 22) to the nodes. If you specify an Amazon EC2 SSH key but do not specify a source security group when you create a managed node group, then port 22 on the nodes is opened to the internet.
Resources:
EKSManagedDataPlane:
Type: AWS::EKS::Nodegroup
Properties:
AmiType: AL2_x86_64
CapacityType: ON_DEMAND
ClusterName: !Ref ClusterName
ForceUpdateEnabled: false
InstanceTypes:
- !Ref WorkerNodesInstanceType
NodegroupName: !Ref WorkerNodeGroupName
NodeRole: !Ref NodeInstanceRoleArn
RemoteAccess:
Ec2SshKey: !Ref WorkerNodesEc2SshKey
SourceSecurityGroups: !Ref SourceSecurityGroupsForWorkerNodes
ScalingConfig:
DesiredSize: 3
MaxSize: 4
MinSize: 3
Subnets: !Ref DataPlanePrivateSubnetIds
Version: !Ref KubernetesVersion
For the EKS Cluster template, I use the eksClusterRole ARN for the EKS cluster parameter, and I pass in all 6 subnet ids (public and private) from the VPC template output when creating the cluster. The SG id I pass in from the VPC ControlPlaneSecurityGroups output field as well.
For the Managed Nodegroup template I only pass it the private subnet ids from the VPC output, the ssh security group id from the VPC output, and the NodeInstanceRole ARN. I have ensured that the cluster name matches what I have given for the EKS Cluster template when creating that.
I plan on configuring the VPC CNI plugin to use IAM Roles for Service accounts once the cluster and managed nodegroup are setup, which is why I have left the policy for the CNI off of that IAM role.