how to update dynamically kubeconfig during cluster creation on aws? - amazon-web-services

Question

I use a template cloud formation to create my cluster on aws :
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys an EKS cluster in a new VPC (qs-1p7nknoht)
Metadata:
LintSpellExclude:
- Kubernetes
- ARNs
- Resource Names
- autoscaler
- IOPS
- EfsStorageClass
- Lambda
- maxIO
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: VPC network configuration
Parameters:
- NumberOfAZs
- AvailabilityZones
- VPCCIDR
- PrivateSubnet1CIDR
- PrivateSubnet2CIDR
- PrivateSubnet3CIDR
- PublicSubnet1CIDR
- PublicSubnet2CIDR
- PublicSubnet3CIDR
- RemoteAccessCIDR
- ProvisionBastionHost
- Label:
default: Amazon EC2 configuration
Parameters:
- KeyPairName
- Label:
default: Amazon EKS configuration
Parameters:
- NodeInstanceType
- NumberOfNodes
- MaxNumberOfNodes
- NodeGroupName
- NodeVolumeSize
- ManagedNodeGroup
- ManagedNodeGroupAMIType
- AdditionalEKSAdminArns
- KubernetesVersion
- Label:
default: Optional Kubernetes add-ins
Parameters:
- ClusterAutoScaler
- EfsStorageClass
- EfsPerformanceMode
- EfsThroughputMode
- EfsProvisionedThroughputInMibps
- MonitoringStack
- Label:
default: AWS Quick Start configuration
Parameters:
- QSS3BucketName
- QSS3KeyPrefix
- QSS3BucketRegion
- LambdaZipsBucketName
ParameterLabels:
AvailabilityZones:
default: Availability Zones
KeyPairName:
default: SSH key name
PrivateSubnet1CIDR:
default: Private subnet 1 CIDR
PrivateSubnet2CIDR:
default: Private subnet 2 CIDR
PrivateSubnet3CIDR:
default: Private subnet 3 CIDR
PublicSubnet1CIDR:
default: Public subnet 1 CIDR
PublicSubnet2CIDR:
default: Public subnet 2 CIDR
PublicSubnet3CIDR:
default: Public subnet 3 CIDR
QSS3BucketName:
default: Quick Start S3 bucket name
QSS3KeyPrefix:
default: Quick Start S3 key prefix
QSS3BucketRegion:
default: Quick Start S3 bucket region
RemoteAccessCIDR:
default: Allowed external access CIDR
VPCCIDR:
default: VPC CIDR
NodeInstanceType:
default: Nodes instance type
NumberOfNodes:
default: Number of nodes
MaxNumberOfNodes:
default: Maximum number of nodes
NodeGroupName:
default: Node group name
NodeVolumeSize:
default: Node volume size
ManagedNodeGroup:
default: Managed node group
ManagedNodeGroupAMIType:
default: Managed node group AMI type
AdditionalEKSAdminArns:
default: Additional EKS admin ARNs
KubernetesVersion:
default: Kubernetes version
LambdaZipsBucketName:
default: Lambda zips bucket name
ClusterAutoScaler:
default: Cluster autoscaler
EfsStorageClass:
default: EFS storage class
EfsPerformanceMode:
default: EFS performance mode
EfsThroughputMode:
default: EFS throughput mode
EfsProvisionedThroughputInMibps:
default: EFS provisioned throughput in Mibps
MonitoringStack:
default: Monitoring Stack
NumberOfAZs:
default: Number of Availability Zones
ProvisionBastionHost:
default: Provision Bastion Host
Parameters:
AvailabilityZones:
Description: The list of Availability Zones to use for the subnets in the VPC. Three
Availability Zones are used for this deployment, and the logical order of your
selections is preserved.
Type: List<AWS::EC2::AvailabilityZone::Name>
KeyPairName:
Description: The name of an existing public/private key pair, which allows you
to securely connect to your instance after it launches
Type: AWS::EC2::KeyPair::KeyName
PrivateSubnet1CIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.0.0/19
Description: The CIDR block for private subnet 1 located in Availability Zone 1
Type: String
PrivateSubnet2CIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.32.0/19
Description: The CIDR block for private subnet 2 located in Availability Zone 2
Type: String
PrivateSubnet3CIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.64.0/19
Description: The CIDR block for private subnet 3 located in Availability Zone 3
Type: String
PublicSubnet1CIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.128.0/20
Description: CIDR block for the public (DMZ) subnet 1 located in Availability
Zone 1
Type: String
PublicSubnet2CIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.144.0/20
Description: The CIDR block for the public (DMZ) subnet 2 located in Availability
Zone 2
Type: String
PublicSubnet3CIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.160.0/20
Description: The CIDR block for the public (DMZ) subnet 3 located in Availability
Zone 3
Type: String
QSS3BucketName:
AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
ConstraintDescription: Quick Start bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
(-).
Default: aws-quickstart
Description: S3 bucket name for the Quick Start assets. This string can include
numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
or end with a hyphen (-).
Type: String
QSS3KeyPrefix:
AllowedPattern: ^[0-9a-zA-Z-/.]*$
ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
uppercase letters, hyphens (-), dots(.) and forward slash (/).
Default: quickstart-amazon-eks/
Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
can include numbers, lowercase letters, uppercase letters, hyphens (-), dots(.) and
forward slash (/).
Type: String
QSS3BucketRegion:
Default: 'us-east-1'
Description: The AWS Region where the Quick Start S3 bucket (QSS3BucketName) is
hosted. When using your own bucket, you must specify this value.
Type: String
RemoteAccessCIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/x
Description: The CIDR IP range that is permitted to access the instances. We recommend
that you set this value to a trusted IP range.
Type: String
VPCCIDR:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28
Default: 10.0.0.0/16
Description: The CIDR block for the VPC
Type: String
AdditionalEKSAdminArns:
Default: ""
Description: "[OPTIONAL] Comma separated list of IAM user/role Amazon Resource Names (ARNs) to be granted admin access to the EKS cluster"
Type: CommaDelimitedList
NodeInstanceType:
Default: t3.medium
AllowedValues:
- t2.small
- t2.medium
- t2.large
- t2.xlarge
- t2.2xlarge
- t3.nano
- t3.micro
- t3.small
- t3.medium
- t3.large
- t3.xlarge
- t3.2xlarge
- m3.medium
- m3.large
- m3.xlarge
- m3.2xlarge
- m4.large
- m4.xlarge
- m4.2xlarge
- m4.4xlarge
- m4.10xlarge
- m5.large
- m5.xlarge
- m5.2xlarge
- m5.4xlarge
- m5.12xlarge
- m5.24xlarge
- c4.large
- c4.xlarge
- c4.2xlarge
- c4.4xlarge
- c4.8xlarge
- c5.large
- c5.xlarge
- c5.2xlarge
- c5.4xlarge
- c5.9xlarge
- c5.18xlarge
- i3.large
- i3.xlarge
- i3.2xlarge
- i3.4xlarge
- i3.8xlarge
- i3.16xlarge
- r3.xlarge
- r3.2xlarge
- r3.4xlarge
- r3.8xlarge
- r4.large
- r4.xlarge
- r4.2xlarge
- r4.4xlarge
- r4.8xlarge
- r4.16xlarge
- x1.16xlarge
- x1.32xlarge
- p2.xlarge
- p2.8xlarge
- p2.16xlarge
- p3.2xlarge
- p3.8xlarge
- p3.16xlarge
- r5.large
- r5.xlarge
- r5.2xlarge
- r5.4xlarge
- r5.12xlarge
- r5.24xlarge
- r5d.large
- r5d.xlarge
- r5d.2xlarge
- r5d.4xlarge
- r5d.12xlarge
- r5d.24xlarge
- z1d.large
- z1d.xlarge
- z1d.2xlarge
- z1d.3xlarge
- z1d.6xlarge
- z1d.12xlarge
ConstraintDescription: Must be a valid EC2 instance type
Description: The type of EC2 instance for the node instances.
Type: String
NumberOfNodes:
Default: 3
Description: The number of Amazon EKS node instances. The default is one for each of the three Availability Zones.
Type: Number
MaxNumberOfNodes:
Default: 3
Description: The maximum number of Amazon EKS node instances. The default is three node.
Type: Number
NodeGroupName:
Default: Default
Description: The name for EKS node group.
Type: String
NodeVolumeSize:
Default: 20
Description: "The size for the node's root EBS volumes."
Type: String
ManagedNodeGroup:
AllowedValues: [ "yes", "no" ]
Default: "no"
Description: Choose if you want to use a managed node group. If you select "yes", you must select Kubernetes Version 1.14 or higher.
Type: String
ManagedNodeGroupAMIType:
Description: Select one of the two AMI types for your managed node group (only applies if you chose "yes" for ManagedNodeGroup). GPU instance types should use the AL2_x86_64_GPU AMI type, which uses the Amazon EKS-optimized Linux AMI with GPU support. Non-GPU instances should use the AL2_x86_64 AMI type, which uses the Amazon EKS-optimized Linux AMI.
AllowedValues: [ "AL2_x86_64", "AL2_x86_64_GPU", ""]
Default: "AL2_x86_64"
Type: String
KubernetesVersion:
Type: String
AllowedValues: [ "1.13", "1.14", "1.15" ]
Default: "1.15"
Description: The Kubernetes control plane version.
LambdaZipsBucketName:
Description: '[OPTIONAL] The name of the S3 bucket where the Lambda zip files should be placed. If you leave this parameter blank, an S3 bucket will be created.'
Type: String
Default: ''
ClusterAutoScaler:
Type: String
AllowedValues: [ Enabled, Disabled ]
Default: Disabled
Description: Choose Enabled to enable Kubernetes cluster autoscaler.
EfsStorageClass:
Type: String
AllowedValues: [ Enabled, Disabled ]
Default: Disabled
Description: Choose Enabled to enable EFS storage class, which will create the required EFS volume.
EfsPerformanceMode:
Type: String
AllowedValues: [ generalPurpose, maxIO ]
Default: generalPurpose
Description: Choose maxIO mode to provide greater IOPS with an increased latency. Only has an effect when EfsStorageClass is enabled.
EfsThroughputMode:
Type: String
AllowedValues: [ bursting, provisioned ]
Default: bursting
Description: Choose provisioned for throughput that is not dependent on the amount of data stored in the file system. Only has an effect when EfsStorageClass is enabled.
EfsProvisionedThroughputInMibps:
Type: Number
MinValue: 0
Default: 0
Description: Set to 0 if EfsThroughputMode is set to bursting. Only has an effect when EfsStorageClass is enabled.
MonitoringStack:
Type: String
AllowedValues: [ "Prometheus + Grafana", "None" ]
Default: "None"
Description: Enable Monitoring stack with "Prometheus+Grafana"
NumberOfAZs:
Type: String
AllowedValues: ["2", "3"]
Default: "3"
Description: Number of Availability Zones to use in the VPC. This must match your selections in the list of Availability Zones parameter.
ProvisionBastionHost:
Type: String
AllowedValues: [ "Enabled", "Disabled" ]
Default: "Enabled"
Description: "Skip creating a bastion host by setting this is set to Disabled."
Rules:
EKSSupport:
Assertions:
- AssertDescription: Your AWS Region does *NOT* yet support Amazon EKS
Assert: !Contains
- - us-west-2
- us-east-1
- us-east-2
- sa-east-1
- eu-west-1
- eu-west-2
- eu-west-3
- eu-north-1
- eu-central-1
- ap-southeast-1
- ap-southeast-2
- ap-northeast-1
- ap-northeast-2
- ap-south-1
- ca-central-1
- !Ref 'AWS::Region'
EKSVersion_ManagedNodeGroup:
RuleCondition: !Equals [ !Ref 'ManagedNodeGroup', 'yes' ]
Assertions:
- AssertDescription: To use Managed Node Groups you must use EKS version 1.14 or higher
Assert: !Contains
- [ '1.15', '1.14' ]
- !Ref 'KubernetesVersion'
ClusterAutoScalerVerification:
RuleCondition: !Equals [ !Ref 'ManagedNodeGroup', 'yes' ]
Assertions:
- AssertDescription: To use Cluster AutoScaler you should not use Managed Node Groups
Assert: !Contains
- - 'Disabled'
- !Ref 'ClusterAutoScaler'
Conditions:
3AZDeployment: !Equals [!Ref NumberOfAZs, "3"]
2AZDeployment: !Or
- !Equals [!Ref NumberOfAZs, "2"]
- !Equals [!Ref NumberOfAZs, "3"]
UsingDefaultBucket: !Equals [!Ref QSS3BucketName, 'aws-quickstart']
Resources:
VPCStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub
- 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}submodules/quickstart-aws-vpc/templates/aws-vpc.template'
- S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion]
S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
Parameters:
AvailabilityZones: !Join [ ',', !Ref 'AvailabilityZones' ]
KeyPairName: !Ref 'KeyPairName'
NumberOfAZs: !Ref 'NumberOfAZs'
PrivateSubnet1ACIDR: !Ref 'PrivateSubnet1CIDR'
PrivateSubnet2ACIDR: !Ref 'PrivateSubnet2CIDR'
PrivateSubnet3ACIDR: !Ref 'PrivateSubnet3CIDR'
PrivateSubnetATag2: "kubernetes.io/role/internal-elb="
PublicSubnet1CIDR: !Ref 'PublicSubnet1CIDR'
PublicSubnet2CIDR: !Ref 'PublicSubnet2CIDR'
PublicSubnet3CIDR: !Ref 'PublicSubnet3CIDR'
PublicSubnetTag2: "kubernetes.io/role/elb="
VPCCIDR: !Ref 'VPCCIDR'
EKSStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub
- 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}templates/amazon-eks.template.yaml'
- S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion]
S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
Parameters:
PublicSubnet1ID: !GetAtt VPCStack.Outputs.PublicSubnet1ID
PublicSubnet2ID: !If
- 2AZDeployment
- !GetAtt VPCStack.Outputs.PublicSubnet2ID
- !Ref AWS::NoValue
PublicSubnet3ID: !If
- 3AZDeployment
- !GetAtt VPCStack.Outputs.PublicSubnet3ID
- !Ref AWS::NoValue
KeyPairName: !Ref KeyPairName
QSS3BucketName: !Ref QSS3BucketName
QSS3KeyPrefix: !Ref QSS3KeyPrefix
QSS3BucketRegion: !Ref QSS3BucketRegion
PrivateSubnet1ID: !GetAtt VPCStack.Outputs.PrivateSubnet1AID
PrivateSubnet2ID: !If
- 2AZDeployment
- !GetAtt VPCStack.Outputs.PrivateSubnet2AID
- !Ref AWS::NoValue
PrivateSubnet3ID: !If
- 3AZDeployment
- !GetAtt VPCStack.Outputs.PrivateSubnet3AID
- !Ref AWS::NoValue
NumberOfNodes: !Ref NumberOfNodes
MaxNumberOfNodes: !Ref MaxNumberOfNodes
NodeGroupName: !Ref NodeGroupName
NodeVolumeSize: !Ref NodeVolumeSize
ManagedNodeGroup: !Ref ManagedNodeGroup
ManagedNodeGroupAMIType: !Ref ManagedNodeGroupAMIType
LambdaZipsBucketName: !Ref LambdaZipsBucketName
NodeInstanceType: !Ref NodeInstanceType
RemoteAccessCIDR: !Ref RemoteAccessCIDR
AdditionalEKSAdminArns: !Join [ ",", !Ref AdditionalEKSAdminArns ]
VPCID: !GetAtt VPCStack.Outputs.VPCID
KubernetesVersion: !Ref KubernetesVersion
ProvisionClusterAutoScaler: !Ref ClusterAutoScaler
EfsStorageClass: !Ref EfsStorageClass
EfsPerformanceMode: !Ref EfsPerformanceMode
EfsThroughputMode: !Ref EfsThroughputMode
EfsProvisionedThroughputInMibps: !Ref EfsProvisionedThroughputInMibps
ProvisionMonitoringStack: !Ref MonitoringStack
ProvisionBastionHost: !Ref ProvisionBastionHost
Outputs:
KubeConfigPath:
Value: !GetAtt EKSStack.Outputs.KubeConfigPath
HelmLambdaArn:
Value: !GetAtt EKSStack.Outputs.HelmLambdaArn
KubeManifestLambdaArn:
Value: !GetAtt EKSStack.Outputs.KubeManifestLambdaArn
KubeGetLambdaArn:
Value: !GetAtt EKSStack.Outputs.KubeGetLambdaArn
EKSClusterName:
Value: !GetAtt EKSStack.Outputs.EKSClusterName
BastionIP:
Value: !GetAtt EKSStack.Outputs.BastionIP
BastionSecurityGroup:
Value: !GetAtt EKSStack.Outputs.BastionSecurityGroup
NodeGroupSecurityGroup:
Value: !GetAtt EKSStack.Outputs.NodeGroupSecurityGroup
I created also an IAM user to access my cluster whith this ARN : arn:aws:iam::XXXXXXXXXXXX:role/testrole
In order to access my cluster with this role aws suggested to update the config map aws-auth and to add something like this :
$ kubectl edit configmap aws-auth -n kube-system
mapRoles: |
- rolearn: arn:aws:iam::XXXXXXXXXXXX:role/testrole
username: testrole
groups:
- system:masters
link od recommandation here : https://aws.amazon.com/fr/premiumsupport/knowledge-center/eks-api-server-unauthorized-error/
Each day, when I leave work I have to delete my cluster and recreate it on monday (because of control plane cost). But I don't want each day, when I create my cluster with my cloud dormation template, to update manually my config map aws-auth to add the previous code.
How could I add this role in my cloudfourmation template in order to create my cluster with this default role added automatically ?
Thanks in advance

Answer 1

During the creation of the cluster with the template, there is a step asking you to add additionnal arn. You can specify here your arn then it will be added by default in your cluster automatically. You can set also this in your cloud formation file under :
AdditionalEKSAdminArns:
default: Additional EKS admin ARNs