I have created kops cluster and getting below error when logging to the cluster.
Error log :
*****INFO! KUBECONFIG env var set to /home/user/scripts/kube/kubeconfig.yaml
INFO! Testing kubectl connection....
error: You must be logged in to the server (Unauthorized)
ERROR! Test Failed, AWS role might not be recongized by cluster*****
Using script for iam-authentication and logged in to server with proper role before connecting.
I am able to login to other server which is in the same environment. tried with diff k8s version and diff configuration.
KUBECONFIG doesn't have any problem and same entry and token details like other cluster.
I can see the token with 'aws-iam-authenticator' command
Went through most of the articles and didn't helped
with kops vs1.19 you need to add --admin or --user to update your kubernetes cluster and each time you log out of your server you have to export the cluster name and the storage bucket and then update the cluster again. this will work.
It seems as a AWS authorization issue. At cluster creation only the IAM user who created the cluster has admin rights on it, so you may need to add your own IAM User first.
1- Start by verifying the IAM user identity used implicitly in all commands: aws sts get-caller-identity
If your aws-cli is set correctly you will have an output similar to this:
{
"UserId": "ABCDEFGHIJK",
"Account": "12344455555",
"Arn": "arn:aws:iam::1234577777:user/Toto"
}
we will refer to the value in Account as YOUR_AWS_ACCOUNT_ID in step 3. (in this example YOUR_AWS_ACCOUNT_ID="12344455555"
2- Once you have this identity you have to add it to AWS role binding to get EKS permissions.
3- You will need to edit the ConfigMap file used by kubectl to add your user kubectl edit -n kube-system configmap/aws-auth
In the editor opened, create a username you want to use to refer to yourself using the cluster YOUR_USER_NAME (for simplicity you may use the same as your aws user name, example Toto in step 2) , you will need it in step 4, and use the aws account id (don't forget to keep the quotes ""),you found it in your identity info at step 1 YOUR_AWS_ACCOUNT_ID, as follows in sections mapUsers and mapAccounts.
mapUsers: |
- userarn: arn:aws:iam::111122223333:user/ops-user
username: YOUR_USER_NAME
groups:
- system:masters
mapAccounts: |
- "YOUR_AWS_ACCOUNT_ID"
4- Finally you need to create a role binding on the kubernetes cluster for the user specified in the ConfigMap
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user YOUR_USER_NAME
I am doing a lab setup of EKS/Kubectl and after the completion cluster build, I run the following:
> kubectl get node
And I get the following error:
Unable to connect to the server: getting credentials: exec: exit status 2
Moreover, I am sure it is a configuration issue for,
kubectl version
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws: error: argument operation: Invalid choice, valid choices are:
create-cluster | delete-cluster
describe-cluster | describe-update
list-clusters | list-updates
update-cluster-config | update-cluster-version
update-kubeconfig | wait
help
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: getting credentials: exec: exit status 2
Please advise next steps for troubleshooting.
Please delete the cache folder folder present in
~/.aws/cli/cache
For me running kubectl get nodes or kubectl cluster-info gives me the following error.
Unable to connect to the server: getting credentials: exec: executable kubelogin not found
It looks like you are trying to use a client-go credential plugin that is not installed.
To learn more about this feature, consult the documentation available at:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
I did the following to resolve this.
Deleted all of the contents inside ~/.kube/. In my case, its a windows machine, so its C:\Users\nis.kube. Here nis is the user name that I logged into.
Ran the get credentials command as follows.
az aks get-credentials --resource-group terraform-aks-dev --name terraform-aks-dev-aks-cluster --admin
Note --admin in the end. Without it, its giving me the same error.
Now the above two commands are working.
Reference: https://blog.baeke.info/2021/06/03/a-quick-look-at-azure-kubelogin/
Did you have the kubectl configuration file ready?
Normally we put it under ~/.kube/config and the file includes the cluster endpoint, ceritifcate, contexts, admin users, and so on.
Furtherly, read this document: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
In my case, as I am using azure (not aws), I had to install "kubelogin" which resolved the issue.
"kubelogin" is a client-go credential (exec) plugin implementing azure authentication. This plugin provides features that are not available in kubectl. It is supported on kubectl v1.11+
Can you check your ~/.kube/config file?
Assume if you have start local cluster using minikube for that if your config is available, you should not be getting the error for server.
Sample config file
apiVersion: v1
clusters:
- cluster:
certificate-authority: /Users/singhvi/.minikube/ca.crt
server: https://127.0.0.1:32772
name: minikube
contexts:
- context:
cluster: minikube
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
client-certificate: /Users/singhvi/.minikube/profiles/minikube/client.crt
client-key: /Users/singhvi/.minikube/profiles/minikube/client.key
You need to update/recreate your local kubeconfig. In my case I deleted the whole ~/.kube/config and followed this tutorial:
https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
Make sure you have installed AWS CLI.
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
I had the same problem, the issue was that in my .aws/credentials file there was multiple users, and the user that had the permissions on the cluster of EKS (admin_test) wasn't the default user. So in my case, i made the "admin_test" user as my default user in the CLI using environment variables:
export $AWS_PROFILE='admin_test'
After that, i checked the default user with the command:
aws sts get-caller-identity
Finally, i was able to get the nodes with the kubectl get nodes command.
Reference: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
In EKS you can retrieve your kubectl credentials using the following command:
% aws eks update-kubeconfig --name cluster_name
Updated context arn:aws:eks:eu-west-1:xxx:cluster/cluster_name in /Users/theofpa/.kube/config
You can retrieve your cluster name using:
% aws eks list-clusters
{
"clusters": [
"cluster_name"
]
}
I had the same error and solved it by upgrading my awscli to the latest version.
Removing and adding the ~/.aws/credentials file worked to resolve this issue for me.
rm ~/.aws/credentials
touch ~/.aws/credentials
I'm having an extremely hard time setting up EKS on AWS. I've followed this tutorial: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-launch-workers
I got up to the ~/.kube/config file and when I try to run kubectl get svc I'm prompted with the below.
▶ kubectl get svc
Please enter Username: Alex
Please enter Password: ********
Error from server (Forbidden): services is forbidden: User
"system:anonymous" cannot list services in the namespace "default"
I'm unsure where to find the username and password for this entry. Please point me to the exact place where I can find this information.
I think this also has to do with EKS RBAC. I'm not sure how to get around this without having access to the server.
This issue occurs if your user configuration isn't working in your kubeconfig, or if you are on a version of kubectl less than v1.10
I was getting the same error.
I created the EKS cluster via the aws console, however when I followed the steps in the docs to configure my kubeconfig, I got the same error:
$ kubectl get svc
Please enter Username: JessicaG
Please enter Password: ****************
Error from server (Forbidden): services is forbidden: User "system:anonymous" cannot list services in the namespace "default"
This is what ended up being my problem:
In the AWS Getting Started guide in the section "Step 1: Create Your Amazon EKS Cluster: To create your cluster with the console", it says this:
"You must use IAM user credentials for this step, not root credentials. If you create your Amazon EKS cluster using root credentials, you cannot authenticate to the cluster."
It turned out that I had created the EKS cluster with my root credentials, however I was trying to authenticate with my admin user JessicaG.
My solution:
I re-created the cluster with the admin IAM user JessicaG. To do so here are the steps I took:
1) I configured the default user in my local file ~/.aws/credentials with the user's access keys
$ cat ~/.aws/credentials
[default]
aws_access_key_id = <JessicaG access key>
aws_secret_access_key = <JessicaG secret key>
2) Created an eks cluster from the command line:
aws eks create-cluster --name eksdemo --role-arn <eksRole> --resources-vpc-config subnetIds=<subnets>,securityGroupIds=<securityGrps>
3) Configured kubeconfig:
apiVersion: v1
clusters:
- cluster:
server: REDACTED
certificate-authority-data: REDACTED
name: eksdemo
contexts:
- context:
cluster: eksdemo
user: aws-jessicag
name: eksdemo
current-context: eksdemo
kind: Config
preferences: {}
users:
- name: aws-jessicag
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: heptio-authenticator-aws
args:
- "token"
- "-i"
- "eksdemo"
That solved this problem for me.
Make sure you have stable version of kubectl install
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
Also if you getting access denied error then make sure you are using the same IAM user access for kubectl which you used for creating EKS cluster.
When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the
cluster is added to the Kubernetes RBAC authorization table as the administrator
(with system:master permissions. Initially, only that IAM user can make calls to the
Kubernetes API server using kubectl.
If you use the console to create the cluster, you must ensure that the same IAM user
credentials are in the AWS SDK credential chain when you are running kubectl commands
on your cluster.
As pointed out rightly in #monokrome's answer, the issue happens either when your kubectl version is outdated or your user configuration is not updated in kubeconfig file (Default Location: ~/.kube/config).
In my case, the configuration for the cluster was not updated in kubeconfig and the following steps helped me update the same:
Ensure your AWS IAM user to have (at least READ) access to the EKS cluster through proper IAM roles and policies.
Upgrade aws cli to version greater than 1.16.156 and run aws sts get-caller-identity. Ensure that the AWS account number and the user ARN are mentioned correctly in the output.
Run aws eks update-kubeconfig --region region-code --name EKS-cluster-name which automatically updates the kubeconfig.
After successfully executing the above steps you should be able to run kubectl commands without any issues. You can try kubectl get svc to verify the same.
I have been trying to follow the getting started guide to EKS.
When I tried to call kubectl get service I got the message: error: You must be logged in to the server (Unauthorized)
Here is what I did:
1. Created the EKS cluster.
2. Created the config file as follows:
apiVersion: v1
clusters:
- cluster:
server: https://*********.yl4.us-west-2.eks.amazonaws.com
certificate-authority-data: *********
name: *********
contexts:
- context:
cluster: *********
user: aws
name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: heptio-authenticator-aws
args:
- "token"
- "-i"
- "*********"
- "-r"
- "arn:aws:iam::*****:role/******"
Downloaded and installed latest aws cli
Ran aws configure and set the credentials for my IAM user and the region as us-west-2
Added a policy to the IAM user for sts:AssumeRole for the EKS role and set it up as a trusted relationship
Setup kubectl to use the config file
I can get a token when I run heptio-authenticator-aws token -r arn:aws:iam::**********:role/********* -i my-cluster-ame
However when I try to access the cluster I keep receiving error: You must be logged in to the server (Unauthorized)
Any idea how to fix this issue?
When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl.
eks-docs
So to add access to other aws users, first
you must edit ConfigMap to add an IAM user or role to an Amazon EKS cluster.
You can edit the ConfigMap file by executing:
kubectl edit -n kube-system configmap/aws-auth, after which you will be granted with editor with which you map new users.
apiVersion: v1
data:
mapRoles: |
- rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
mapUsers: |
- userarn: arn:aws:iam::111122223333:user/ops-user
username: ops-user
groups:
- system:masters
mapAccounts: |
- "111122223333"
Pay close attention to the mapUsers where you're adding ops-user together with mapAccounts label which maps the AWS user account with a username on Kubernetes cluster.
However, no permissions are provided in RBAC by this action alone; you must still create role bindings in your cluster to provide these entities permissions.
As the amazon documentation(iam-docs) states you need to create a role binding on the kubernetes cluster for the user specified in the ConfigMap. You can do that by executing following command (kub-docs):
kubectl create clusterrolebinding ops-user-cluster-admin-binding --clusterrole=cluster-admin --user=ops-user
which grants the cluster-admin ClusterRole to a user named ops-user across the entire cluster.
I am sure issue is resolved but I will be putting more information here so if any other people are still facing the issue related to any of the below setup then they can use the steps below.
When we create the EKS cluster by any method via CloudFormation/CLI/EKSCTL the IAM role/user who created the cluster will automatically binded to the default kubernetes RBAC API group system:masters (https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) and in this way creator of the cluster will get the admin access to the cluster. Although we can always give the access to other IAM user/role using the aws-auth file but for that we must have to use the IAM user/role who created the cluster.
To verify the role/user for the EKS cluster we can search for the CreateCluster" Api call on cloudtrail and it will tell us the creator of the cluster in the sessionIssuer section for field arn (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).
When we create the cluster using the IAM role or IAM user, setting up the access for the EKS cluster will become little tricky when we created the cluster using the role compare to user.
I will put the steps we can follow for each different method while setting up the access to EKS cluster.
Scenario-1: Cluster was Created using the IAM user (For example "eks-user")
Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity
$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}
After that update the kubeconfig file using the below command
aws eks --region region-code update-kubeconfig --name cluster_name
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: CERT
server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- eks-cluster
command: aws
Once above setup is done you should be able to run the kubectl command.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP xxx.xx.x.x <none> 443/TCP 12d
Scenario-2: Cluster was Created using the IAM Role (For example "eks-role")
Mainly there are four different way to setup the access via cli when cluster was created via IAM role.
1. Setting up the role directly in kubeconfig file.
In this case we do not have to make any assume role api call via cli manually, before running kubectl command because that will be automatically done by aws/aws-iam-authenticator set in the kube config file.
Lets say now we are trying to setup the access for the user eks-user the first make sure that user does have permission to assume the role eks-role
Add the assume role permission to the eks-user
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::xxxxxxxxxxx:role/eks-role"
}
]
}
Edit the trust relationship on the role so that it will allow the eks-user to assume the role.
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
},
"Action": "sts:AssumeRole"
}
]
}
Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity. Important thing to remember it should show us the IAM user ARN not the IAM assumed ROLE ARN.
$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}
After that update the kubeconfig file using the below command
aws eks --region region-code update-kubeconfig --name cluster_name --role-arn arn:aws:iam::xxxxxxxxxxx:user/eks-role
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: CERT
server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- eks-cluster
- --role
- arn:aws:iam::xxxxxxx:role/eks-role
command: aws
Once above setup is done you should be able to run the kubectl command.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP xxx.xx.x.x <none> 443/TCP 12d
2. If you have setup the AWS profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) on CLI and if you want to use that with the kube config.
Confirm that profile is set properly so that it can use the credentials for the eks-user
$ cat ~/.aws/config
[default]
output = json
region = us-east-1
[eks]
output = json
region = us-east-1
[profile adminrole]
role_arn = arn:aws:iam::############:role/eks-role
source_profile = eks
$ cat ~/.aws/credentials
[default]
aws_access_key_id = xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[eks]
aws_access_key_id = xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Once this profile configuration is done please confirm that profile configuration is fine by running the command aws sts get-caller-identity --profile eks
$ aws sts get-caller-identity --profile eks
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}
After that update the kubeconfig file using the below command with the profile and please make sure we are not using the role here.
aws eks update-kubeconfig --name devel --profile eks
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: CERT
server: https://xxxxx.sk1.us-east-1.eks.amazonaws.com
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- eks-cluster
command: aws
env:
- name: AWS_PROFILE
value: eks
Once above setup is done you should be able to run the kubectl command.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP xxx.xx.x.x <none> 443/TCP 12d
3. Assume the role by any other way, For example we can attach the IAM role to the instance directly.
If role is directly attached to the instance profile then we can follow the similar steps as we followed while setting up the access for IAM user in Scenario-1
Verify that we have attached the correct role to EC2 instance and as this instance profile will come into least precedence, this step will also verify that there are no any other credentials setup on the instnace.
[ec2-user#ip-xx-xxx-xx-252 ~]$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx:i-xxxxxxxxxxx",
"Arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/eks-role/i-xxxxxxxxxxx"
}
After that update the kubeconfig file using the below command
aws eks --region region-code update-kubeconfig --name cluster_name
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: CERT
server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- eks-cluster
command: aws
Once above setup is done you should be able to run the kubectl command.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP xxx.xx.x.x <none> 443/TCP 12d
4. Manually assuming the IAM role via aws sts assume-role command.
Assume the role eks-role manually by running the cli command.
aws sts assume-role --role-arn arn:aws:iam::xxxxxxxxxxx:role/eks-role --role-session-name test
{
"AssumedRoleUser": {
"AssumedRoleId": "xxxxxxxxxxxxxxxxxxxx:test",
"Arn": "arn:aws:sts::xxxxxxxxxxx:assumed-role/eks-role/test"
},
"Credentials": {
"SecretAccessKey": "xxxxxxxxxx",
"SessionToken": xxxxxxxxxxx",
"Expiration": "xxxxxxxxx",
"AccessKeyId": "xxxxxxxxxx"
}
}
After that set the required environment variable using the value from above output so that we can use the correct credentials generated from the session.
export AWS_ACCESS_KEY_ID=xxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx
export AWS_SESSION_TOKEN=xxxxxxxxxx
After that verify that we assumed the IAM role by running the command aws sts get-caller-identity.
aws sts get-caller-identity
{
"Account": "xxxxxxxxxx",
"UserId": "xxxxxxxxxx:test",
"Arn": "arn:aws:sts::xxxxxxxxxx:assumed-role/eks-role/test"
}
After that update the kubeconfig file using the below command
aws eks --region region-code update-kubeconfig --name cluster_name
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: CERT
server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- us-east-1
- eks
- get-token
- --cluster-name
- eks-cluster
command: aws
Once above setup is done you should be able to run the kubectl command.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP xxx.xx.x.x <none> 443/TCP 12d
NOTE:
I have try to cover major use case here but there might be other use case too where we need to setup the access to the cluster.
Also the above tests are mainly aiming at the first time setup of the EKS cluster and none of the above method is touching the aws-auth configmap yet.
But once you have given access to other IAM user/role to EKS cluster via aws-auth (https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) file you can use the same set of commands for those users too as mentioned in above answer.
Update: If you are using the SSO then setup will be preety much same but one thing we have to consider is either in case of SSO or while using the role directly if we are trying to update path based role in ConfigMap then we have to get rid of the paths in role for example instead arn:aws:iam::xxxxxxxxxxx:role/path-1/subpath-1/eks-role of this use arn:aws:iam::xxxxxxxxxxx:role/eks-role so basically we are getting rid of the /path-1/subpath-1 becuase when we run kubectl command it will first make AssumeRole api call and if we see assumed role ARN then it will not contians the path so if we include the path then it EKS will deny those requests.
If you are using eksctl to manage your aws eks deployments you can add the user to the config map with one command:
eksctl create iamidentitymapping --cluster <cluster-name> --arn arn:aws:iam::<id>:user/<user-name> --group system:masters --username ops-user
I commented out the last two lines of the config file
# - "-r"
# - "arn:aws:iam::**********:role/**********"
and it worked though I have no idea why
You need to create the cluster under the same IAM profile that you are accessing it from via AWS cli.
Said in another way, inside ~/.aws/credentials, the profile that is accessing kubectl must match exactly the same IAM that was used to create the cluster.
My recommendation is to use AWS cli to create your clusters as creating from the GUI may be more confusing than helpful. The Getting Started guide is your best bet to get up and running.
Also, make sure your users are in the aws-auth k8s ConfigMap:
https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
I had the same problem . It's likely that you are using a root account. It appears root accounts are blocked from assuming the required roles. This error can sometimes be cloaked if you are using expired keys.
I had the same problem, my AWS credentials for CLI change frequently. These steps fixed the problem:
export AWS_ACCESS_KEY_ID="***************"
export AWS_SECRET_ACCESS_KEY="*************"
export AWS_SESSION_TOKEN="************************"
If you have created the EKS cluster with kops, Then all you need to do is update your kubecfig file with following kops command
kops export kubecfg --admin
If you have exhausted all of the above solutions and are still getting the same error. Make sure kubectl is actually using the AWS credentials you think you are.
I made a very stupid mistake. I have environment variables setup for different accounts and kubectl seems to always pick up the one from AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. So precedence of options on this page may help https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
This happens also to me with local environment on minikube, independently of EKS. My problem is related to this issue: https://github.com/kubernetes/kubernetes/issues/76774
The solution i adopted is to remove the cache directories of kubectl: rm -rf ~/.kube/{cache,http-cache}. I guess is the only workaround at the time of writing.
In my case it is the AWS profile issue, be sure to use aws sts get-caller-identity to verify the IAM user.
I just debugged this issue. I have a question. Are you running this on a corporate wifi network? If yes, could you create an EC2 instance and then test if you are able to do kubectl get svc?
Also, try if this command works
kubectl get svc --insecure-skip-tls-verify
The issue for me was that I had set up environment variables with different, invalid, AWS credentials (maybe a long time ago, and forgot). I realised this after running aws configure list and seeing that the credentials are different from what I expected with aws configure list --profile default. Finding and deleting those invalid environment variables fixed the issue, now I can run kubectl get svc.
The issue is with the policy added for the roles created. We must have AWSEKSCNI policy.
Better to create eks using the command:
eksctl create cluster --name ekscluster --version 1.19 --with-oidc
--vpc-public-subnets=subnet-08c6b0b0166abc1d1,subnet-02822a142bb5a802a
--vpc-private-subnets=subnet-09bbf4871902ee64c,subnet-0926c224909b5a811
This will create and assign the policy automatically using cloudformation.
In my case the scenario was, that I replaced a person who worked as a devops and created all the infrastructure on aws. He had his own IAM user (so yes, no root access case, however could be also applicable) with all AWS permission and I had my own IAM user also with all available AWS permissions.
So, he left the company and that was the problem, since I didn't get any access to cluster which by defauly is shared to the creator only ... and in fact all of my approaches to get the access to cluster were failed, even despite a fact that I had all permissions.
The good thing was, guy whom I replaced was still had his IAM user available (not removed). And what I did, I simply generated new AWS access pair under his account and set them as my default aws credentials on my ubuntu host (from which I've tried to access the cluster). Important part was to make sure that after running aws sts get-caller-identity command it meant to be HIS account to appear on the output. In that case I've been able to run all the kubectl commands that I wanted without error You must be logged in to the server (Unauthorized) message.
So the solution is - be lucky to find cluster creator credentials and use them! (sounds like a crime, however ...)
I had the same issue.
Refer the answers:
Cannot create namespaces in AWS Elastic Kubernetes Service - Forbidden
User cannot log into EKS Cluster using kubectl
The correct way is to:
Create an IAM Group with the following permissions.
1. AmazonEKSClusterPolicy
2. AmazonEKSWorkerNodePolicy
3. AmazonEC2ContainerRegistryReadOnly
4. AmazonEKS_CNI_Policy
5. AmazonElasticContainerRegistryPublicReadOnly
6. EC2InstanceProfileForImageBuilderECRContainerBuilds
7. AmazonElasticContainerRegistryPublicFullAccess
8. AWSAppRunnerServicePolicyForECRAccess
9. AmazonElasticContainerRegistryPublicPowerUser
10. SecretsManagerReadWrite
Create an IAM user (AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID will be provided) and add the user to the IAM Group created above.
Next, login to AWS Console as the IAM user and create the EKS Cluster.
Next, use the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID to setup the AWS CLI in local machine.
Next, run the following commands in the local machine:
1. aws sts get-caller-identity
2. aws eks describe-cluster --name [cluster-name] --region [aws-region] --query cluster.status (To check the status of the Cluster)
3. aws eks update-kubeconfig --name [cluster-name] --region [aws-region]
After this, you will be able to run the kubectl commands.
If you need to add additional users to the EKS Cluster, create the additional IAM user, add the user to the IAM Group in AWS. Next, log into the EKS cluster as the original IAM user and run: kubectl edit -n kube-system configmap/aws-auth.
Add the following block of code to the existing configuration:
(Make sure to add changes to the ARN and username)
mapUsers: |
- userarn: arn:aws:iam::111122223333:user/admin
username: admin
groups:
- system:masters
Refer the following link to understand this better: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
Hope this become useful.
Happy learning!
In addition to the great answers that have already been given, I would like to add a good way to troubleshoot issues. In my case, I was trying to run kubectl in an ECS task as part of an AWS pipeline, and kubectl version was failing with the "You must be logged in to the server" message.
What was happening was that the ECS task was assuming a service role, and then aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_DEFAULT_REGION was being run. It turns out the service role, CodeBuildServiceRole, was not mapped to an RBAC user via a clusterrolebinding in the EKS cluster, and the aws-iam-authenticator EKS service was denying access to the AWS service account (or something like that).
What helped me figure out what was going on in this case was to go to CloudWatch → Log Insights and run the following query against the /aws/eks/<cluster-name>/cluster log group:
fields #timestamp, #message
| sort #timestamp desc
| filter #logStream like /authenticator/
| filter #message like /access denied/
| limit 50
It was clear from the log output that the authenticator service was expecting an ARN in configmap/aws-auth that was lowercase: arn:aws:iam::123456789:role/codebuildservicerole. Once I fixed the case of the ARN in the configmap, kubectl version started working in the ECS task.
$ kubectl edit -n kube-system configmap/aws-auth
I had the same problem and the fix was to remove an alias I had added when setting up minikube. Their docs say to add this alias:
alias kubectl="minikube kubectl --"
Which breaks some things like Azure integrations, as you would expect.
I got this error when I created the eks cluster using the root from the eks console. I recreated the eks cluster using an IAM user and use the access keys to update the aws configure. It worked. Now you can add additional IAM users to issue kubectl commands.
I was trying to create an EKS cluster with a private endpoint. Read this thread several times and the thing that worked for me:
Created a user: admin (with access to the console)
Logged into the console using that user
Created the cluster using the admin user
Created my kube/config file using aws eks (aws eks --region update-kubeconfig --name )
Done (kubectl get ns)
The last 2 commands were executed from an ec2 instance part of the same VPC.
I faced the same issue. I tried to configure AWS CLI directly with access key and secret key and it worked.
It should be bug to not able to assume role. Try setup cli and test.
I followed these docs. It took some time to understand things.
But finally implemented the things easily with full understanding.
https://kubernetes.io/docs/reference/access-authn-authz/rbac/
https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
When invoking eksctl with an assumed role through sso, the following steps got me access to the cluster.
eksctl.exe create cluster --name test --without-nodegroup --profile <your_profile>.
Update the kubeconfig :
eksctl utils write-kubeconfig --cluster=test --profile<your_profile>.
Enable Cloudwatch to get the iam role arn:
eksctl utils update-cluster-logging --enable-types=all --cluster=test --approve --profile=<your_profile>.
run "kubectl get pods -A" . server responds with "error: You must be logged in to the server (Unauthorized)" message.
Switch to the console and get the role arn from the cloudwatch group audit log.
Update the aws-auth config map by mapping the derived arn to system:masters group. The username is users:name field in your current kubeconfig context as extracted it below
user_id=` k config view --minify -o json | jq -r '.users[0].name'`
eksctl create iamidentitymapping --cluster test --group system:masters --username ${user_id} --arn arn:aws:iam::<account_id>:role/<assumed_role> --profile <your_profile>
For me adding the user in a single line like below worked
kubectl edit -n kube-system configmap/aws-auth
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::******:role/eksctl-atoa-microservices-nodegro-NodeInstanceRole-346C48Q1W7OB
username: system:node:{{EC2PrivateDNSName}}
mapUsers: "- groups:\n - system:masters\n userarn: arn:aws:iam::*****:user/<username>
\ \n username: <username>\n"