I'm receiving the following error for kubectl cluster-info :
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Error from server (Forbidden): services is forbidden: User "admin" cannot list resource "services" in API group "" in the namespace "kube-system"
You can check permissions in the GCP Console under IAM -> Members -> click on the right the Over granted permissions tab and it will display a list of permissions. Look out for services.list permissions.
Related
I want to get resource ids of resources in google cloud platform.
I have enough access to google cloud console to access those resources.
However I can't access through gcloud shell utility and need additional permissions.
$ gcloud alpha monitoring policies list --project kbc-env --filter="userLabels.alerting_db_type:*" --format json > ~/Downloads/kbc-env.json
ERROR: (gcloud.alpha.monitoring.policies.list) User [sjain#kbc.com] does not have permission to access projects instance [kbc-env] (or it may not exist): Caller does not have required permission to use project kbc-env. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=kbc-env and then retry. Propagation of the new permission may take a few minutes.
- '#type': type.googleapis.com/google.rpc.Help
links:
- description: Google developer console IAM admin
url: https://console.developers.google.com/iam-admin/iam/project?project=kbc-env
- '#type': type.googleapis.com/google.rpc.ErrorInfo
domain: googleapis.com
metadata:
consumer: projects/kbc-env
service: monitoring.googleapis.com
reason: USER_PROJECT_DENIED
How can i get resource ids without gcloud permissions ?
To get resource id without gcloud
Step 1 :
Go to GCP console and open the resource directly
For e.g. You can go to Cloud Monitoring then click on Alerting and then click to one of alerts.
Step 2 :
We can get resource id from the URL like shown on the screenshot
I was trying to add permission to view nodes to my admin IAM using information in this article (https://aws.amazon.com/premiumsupport/knowledge-center/eks-kubernetes-object-access-error/) and ended up saving the configmap with a malformed mapUsers section (didn't include the username at all)
Now every kubectl command return an error like this: Error from server (Forbidden): nodes is forbidden: User "" cannot list resource "nodes" in API group "" at the cluster scope
How can I circumvent corrupted configmap and regain access to the cluster? I found two questions at Stackoverflow but as I am very new to kubernetes and still buffled as to exactly I need to do.
Mistakenly updated configmap aws-auth with rbac & lost access to the cluster
I have an access to root user but kubectl doesn't work for this user, too.
Is there another way to authenticate to the cluster?
Update 1
Yesterday I recreated this problem on a new cluster: I still got this error even if I am the root user.
The structure of the configmap goes like this:
apiVersion: v1
data:
mapRoles: <default options>
mapUsers: |
- userarn: arn:aws:iam::<root id>:root
username: #there should be a username value on this line, but it's missing in my configmap; presumable this is the cause
groups:
- system:bootstrappers
- system:nodes
Update 2
Tried to use serviceAccount token, got an error:
Error from server (Forbidden): configmaps "aws-auth" is forbidden: User "system:serviceaccount:kube-system:aws-node" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
How did you create your cluster? The IAM user, or IAM role, that you used to actually create it, is grandfathered in as a sysadmin. So long as you use the same credentials that you used for
aws eks create-cluster
You can do
aws eks update-kubeconfig, followed by using kubectl to modify the configmap and give other entites the permissions.
You haven't said what you actually tried. Lets do some more troubleshooting-
system:serviceaccount:kube-system:aws-node this is saying that THIS kubernetes user does not have permission to modify configmaps. But, that is completely correct- it SHOULDNT. What command did you run to get that error? What were the contents of your kubeconfig context to generate that message? Did you run the command from a worker node maybe?
You said "I have access to the root user". Access in what way? Through the console? With an AWS_SECRET_ACCESS_KEY? You'll need the second - assuming thats the case run aws iam get-caller-identity and post the results.
Root user or not, the only user that has guaranteed access to the cluster is the one that created it. Are there ANY OTHER IAM users or roles in your account? Do you have cloudtrail enabled? If so, you could go back and check the logs and verify that it was the root user that issued the create cluster command.
After running get-caller-identity, remove your .kube/config file and run aws eks update-kubeconfig. Tell us the output from the command, and the contents of the new config file.
Run kubectl auth can-i '*' '*' with the new config and let us know the result.
I have created a EKS cluster using the the command line eksctl and verified that the application is working fine.
But noticing a strange issue, when i try yo access the nodes in the cluster in the web browser i see the following error
Error loading Namespaces
Unauthorized: Verify you have access to the Kubernetes cluster
I am able to see the nodes using kubectl get nodes
I am logged in as the admin user. Any help on how to workaround this would be really great. Thanks.
You will need to add your IAM role/user to your cluster's aws-auth config map
Basic steps to follow taken from https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
kubectl edit -n kube-system configmap/aws-auth
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
mapRoles: |
- rolearn: <arn:aws:iam::111122223333:role/eksctl-my-cluster-nodegroup-standard-wo-NodeInstanceRole-1WP3NUE3O6UCF>
username: <system:node:{{EC2PrivateDNSName}}>
groups:
- <system:bootstrappers>
- <system:nodes>
mapUsers: |
- userarn: <arn:aws:iam::111122223333:user/admin>
username: <admin>
groups:
- <system:masters>
- userarn: <arn:aws:iam::111122223333:user/ops-user>
username: <ops-user>
groups:
- <system:masters>
Also seeing this error and it got introduced by the latest addition to EKS, see https://aws.amazon.com/blogs/containers/introducing-the-new-amazon-eks-console/
Since then, the console makes requests to EKS in behalf of the user or role you are logged in.
So make sure the kube-system:aws-auth configmap has that user or role added.
This user/role might not be the same you are using locally with AWS CLI, hence kubectl might work while you still see that error !
Amazon added recently (2020.12) new feature that allows you to browse workloads inside cluster from Aws Console.
If you miss permissions you will get that error.
What permissions are needed is described here
https://docs.aws.amazon.com/eks/latest/userguide/security_iam_id-based-policy-examples.html#policy_example3
This might as well be because you created the AWS EKS cluster using a different IAM user than the one currently logged into the AWS Management Console hence the IAM user currently logged into the AWS Management Console does not have permissions to view the namespaces on the AWS EKS cluster.
Try logging in to the AWS Management Console using the IAM user credentials of the user who created the AWS EKS cluster, the issue should be fixed.
I am trying to setup aws EKS cluster and want to connect that cluster from my local windows workstation. Not able to connect that. Here are the steps i did;
Create a aws service role (aws console -> IAM -> Roles -> click "Create role" -> Select AWS service role "EKS" -> give role name "eks-role-1"
Create another user in IAM named "eks" for programmatic access. this will help me to connect my EKS cluster from my local windows workstation. Policy i added into it is "AmazonEKSClusterPolicy", "AmazonEKSWorkerNodePolicy", "AmazonEKSServicePolicy", "AmazonEKS_CNI_Policy".
Next EKS cluster has been created with roleARN, which has been created in Step#1. Finally EKS cluster has been created in aws console.
In my local windows workstation, i have download "kubectl.exe" & "aws-iam-authenticator.exe" and did 'aws configure' using accesskey and token from step#2 for the user "eks". After configuring "~/.kube/config"; i ran below command and get error like this:
Command:kubectl.exe get svc
output:
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Unable to connect to the server: getting credentials: exec: exit status 1
Not sure what wrong setup here. Can someone pls help? I know some of the places its saying you have to use same aws user to connect cluster (EKS). But how can i get accesskey and token for aws assign-role (step#2: eks-role-1)?
For people got into this, may be you provision eks with profile.
EKS does not add profile inside kubeconfig.
Solution:
export AWS credential
$ export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxx
$ export AWS_SECRET_ACCESS_KEY=ssssssssss
If you've already config AWS credential. Try export AWS_PROFILE
$ export AWS_PROFILE=ppppp
Similar to 2, but you just need to do one time. Edit your kubeconfig
users:
- name: eks # This depends on your config.
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "general"
env:
- name: AWS_PROFILE
value: "<YOUR_PROFILE_HERE>" #
I think i got the answer for this issue; want to write down here so people will be benefit out of it.
When you first time creating EKS cluster; check from which you are (check your aws web console user setting) creating. Even you are creating from CFN script, also assign different role to create the cluster. You have to get CLI access for the user to start access your cluster from kubectl tool. Once you get first time access (that user will have admin access by default); you may need to add another IAM user into cluster admin (or other role) using congifMap; then only you can switch or use alternative IAM user to access cluster from kubectl command line.
Make sure the file ~/.aws/credentials has a AWS key and secret key for an IAM account that can manage the cluster.
Alternatively you can set the AWS env parameters:
export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=ssssssssss
Adding another option.
Instead of working with aws-iam-authenticator you can change the command to aws and replace the args as below:
- name: my-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args: #<--- Change the args
- --region
- <YOUR_REGION>
- eks
- get-token
- --cluster-name
- my-cluster
command: aws #<--- Change to command to aws
env:
- name: AWS_PROFILE
value: <YOUR_PROFILE_HERE>
I am trying to provision a k8s cluster on Google Cloud Platform using gcloud
$ gcloud container clusters create my-cluster
ERROR: (gcloud.container.clusters.create) ResponseError: code=400,
message=service account "xxxxxx-compute#developer.gserviceaccount.com" does not exist.
Any idea as to why this is error is occurring? I have been able to provision in the past w/o any issue until now.
Verify that the active account is your own account, and not a service account that lacks permission to create clusters.
Example command to check this and output:
$ gcloud auth list
Credentialed Accounts:
- xxxxx#developer.gserviceaccount.com
- user#example.com ACTIVE
If you don't see ACTIVE next to the expected account, then set it to the right account with:
$ gcloud config set account <correctaccountname>
Maybe you switched to the service account a while back and forgot about it?
I got the same error while creating a cluster using GKE. Earlier I was getting this error because I did not attach the service account (which needs to be created and assigned with a role having Kubernetes specific permissions).
Now, the further question you might have where to attach the service account, for this you need to go the Create Cluster --> Node Pool Section --> More Options --> (a new window will popup) --> security section --> In-Service account dropdown Choose the Service account.
Once you do it, then save and say create a cluster, hopefully you will not get the same error.