I am attempting to use a gitlab runner to build an image in GKE and then push that image to GCR. This always fails and outputs an AWS error at first.
My understanding is that the building pod is created within the same namespace as my GCR appropriate service account should push. The pod seems to not be using workload identity or failing for some other reason.
Gitlab Yaml
buildmygcr:
stage: build
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
tags:
- cluster
- kubernetes
script:
- /kaniko/executor --verbosity debug --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination gcr.io/myproject/test:test --cache=false
Build Error
DEBU[0000] Copying file /builds/user/test/Dockerfile to /kaniko/Dockerfile
E0403 20:18:16.646782 21 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/myproject/test:test": creating push check transport for gcr.io failed: GET https://gcr.io/v2/token?scope=repository%myproject%2Ftest%3Apush%2Cpull&service=gcr.io: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1
Workload Identity Test shows the proper account for the namespace
root#workload-identity-test:/# gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* mysvcaccount#myproject.iam.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
Related
I use a workflow to create a batch job using a docker image hosted in a docker registry.
All of this happens within the same google cloud project.
My batch job fails with this error :
"docker: Error response from daemon: Head "https://us-west1-docker.pkg.dev/v2/entity/docker-registry/image-name/manifests/latest": denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/project-id/locations/us-west1/repositories/docker-registry" (or it may not exist).
See 'docker run --help'.
From google documentation I understand that Compute Engine's service account doesn't have the roles/artifactregistry.admin : Jobs default to using the Compute Engine default service account
I get the same error after giving the role to the service account :
gcloud projects add-iam-policy-binding project-id \
--member=serviceAccount:compute#developer.gserviceaccount.com \
--role=roles/artifactregistry.admin
While digging service accounts I found another service another service account and also gave it the role : service-xxxx#gcp-sa-cloudbatch.iam.gserviceaccount.com.
It does not solve the problem.
How can I see which service account is used ?
Can I see logs about denied permissions ?
The error occurs when you are trying to push an image on a repository in which a specific hostname associated with its repository location is not yet authenticated and specified in the credential helper.You may refer to this Setting up authentication for Docker .You may check and confirm the service account to make sure you are still impersonating the correct one ,run below as mentioned in document
gcloud auth list
This command will show the active account, along with the other
accounts that are authorized to access your Google Cloud project. The
active account will be marked with an asterisk (*).
Try to run the authentication using a command specifying the location of your repository.You may try to run the configure-docker command against the auth group and see.
gcloud auth configure-docker <location>-docker.pkg.dev
And then try pulling the Docker image again.
Refer Authenticating to a repository for more information and you can see these logs permission denied logs in Cloud logging for more details.
I've followed these instructions to the letter to allow me to use the short lived token authentication method to access gcloud resources from our github actions workflow.
I've created the service account, workload identity pool and github provider pool using the exact instructions above, but it doesn't appear that the auth step is getting the correct token (or any token at all). The GCP service account has the correct IAM permissions.
On the gcloud compute instances list step, I'm receiving the error:
ERROR: (gcloud.compute.instances.list) There was a problem refreshing your current auth tokens: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n "error": {\n "code": 403,\n "message": "The caller does not have permission",\n "status": "PERMISSION_DENIED"\n }\n}\n')
Please run:
$ gcloud auth login
to obtain new credentials.
My github actions file is as follows:
jobs:
backup:
permissions:
contents: 'read'
id-token: 'write'
runs-on: ubuntu-latest
steps:
- name: 'Checkout code'
uses: actions/checkout#v3
- id: 'auth'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth#v0'
with:
workload_identity_provider: 'projects/*******/locations/global/workloadIdentityPools/**REDACTED**/providers/github-provider'
service_account: '**REDACTED**#**REDACTED**.iam.gserviceaccount.com'
# Install gcloud, `setup-gcloud` automatically picks up authentication from `auth`.
- name: 'Set up Cloud SDK'
uses: 'google-github-actions/setup-gcloud#v0'
- name: 'Use gcloud CLI'
run: gcloud compute instances list
I enabled logging for the token exchange and I can see it occurring (with no obvious errors) in GCP logs either. So I'm completely stumped.
Any ideas?
So I later found out what this was. Despite running:
gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_EMAIL \
--role=roles/iam.workloadIdentityUser \
--member="MEMBER_EXPRESSION"
As per the docs, it had not granted permission - I went into the console and checked the workload identity pool under "connected service accounts" menu (to the left) and the service account wasn't in there, so I added it manually.
In addition to the OP's own answer of the service account not being connected (bound) at all, this can result from the service account binding being constrained using attribute mappings.
In the default setup for GitHub Actions discussed here on the google blog for WIF, the provider is set up with a set of attribute mappings:
--attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository"
Those attributes can be used to restrict access when you connect (bind) the service account... in the following, the member uses the repository attribute to constrain it so that only actions executing in my-org/my-repo on GitHub will be permitted.
gcloud iam service-accounts add-iam-policy-binding "my-service-account#${PROJECT_ID}.iam.gserviceaccount.com" \
--project="${PROJECT_ID}" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/attribute.repository/my-org/my-repo"
A cross-repository provider
Of course, you want to use strong restrictions here. Restricting to a repository, or even a particular branch (so only main branch actions have privilege to deploy to production, for example). No restrictions allows anything, which is absolutely not what you want!!!
In my case, I set up my WIF provider, then tried to reuse it from another repository, resulting in the error experienced by the OP.
I chose to add the repository_owner attribute mapping from the list of all possible attributes that are in GitHub's OIDC token (the attribute mappings are editable in the google cloud console), then bind my service account to that rather than the repository-specific principal:
--attribute-mapping="google.subject=assertion.sub,attribute.repository_owner=assertion.repository_owner"
and
gcloud iam service-accounts add-iam-policy-binding "my-service-account#${PROJECT_ID}.iam.gserviceaccount.com" \
--project="${PROJECT_ID}" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/attribute.repository_owner/my-org"
Bingo, it works a charm now.
Take care to think about your attack surface though, loosening this constraint too widely creates real vulnerability.
I was trying to build a simple serverless web app in AWS. So, I put my static files in codecommit and tried to host it through AWS Amplify.The AWS builds the amplify.yml for us by default, but it is failing in the build stage. I understood that there is something wrong in amplify.yml and am not getting how to configure it.
amplify.yml :
version: 1
frontend:
phases:
# IMPORTANT - Please verify your build commands
build:
commands: []
artifacts:
# IMPORTANT - Please verify your build output directory
baseDirectory: /
files:
- '**/*'
cache:
paths: []
error I was shown:
2021-01-23T15:01:54.840Z [INFO]: # Cloning repository: https://git-codecommit.us-east-
2.amazonaws.com/v1/repos/wildrydes-site
2021-01-23T15:01:55.176Z [INFO]: Cloning into 'wildrydes-site'...
2021-01-23T15:02:02.660Z [INFO]: fatal: unable to access 'https://git-codecommit.us-east-
2.amazonaws.com/v1/repos/wildrydes-site/': The requested URL returned error: 403
With the limited amount of information provided, the best I can offer is the issue is somewhere in your IAM privileges for your CodeCommit user. AWS components generally need to have the appropriate Group, Role, and/or Policy to complete the action.
Put another way: CodeCommit is giving a 403 Forbidden error because whatever is attempting doing the cloning doesn't have the right permissions. Give it the right permissions.
You can find the execution role that Amplify uses to run its builds by navigating to AWS Amplify > General (in the left-hand column). Under the Settings section, you'll see a value for the "Service role" field. It will look something like this: "arn:aws:iam:::role/service-role/AWSAmplifyExecutionRole-xxxx".
Once you find that, navigate to IAM > Roles and search for the service role that you just found (AWSAmplifyExecutionRole-xxxx) and click on it. Then click "Attach policies". Search for the "AWSCodeCommitReadOnly" policy and attach it to the role.
You should be able to clone your CodeCommit repository in your Amplify build now.
I am trying to deploy the new changes to kubernetes cluster using the Google Cloud Provider CloudBuild. Whenever I make some changes the trigger is working fine and its starting a new build but here is the issue I am getting with this cloudbuild.yaml.
cloudbuild.yaml
steps:
#step1
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/cloudbuildtest-image', '.' ]
#step 2
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/cloudbuildtest-image']
#step 3 for testing
name: 'gcr.io/cloud-builders/kubectl'
args: ['get', 'pods']
env:
- 'CLOUDSDK_COMPUTE_ZONE=us-central1-a'
- 'CLOUDSDK_CONTAINER_CLUSTER=cloudbuild-test'
#STEP-4
images:
- 'gcr.io/$PROJECT_ID/cloudbuildtest-image'
Step 1 and 2 are working fine but the issue is with the step3 where for testing purpose I simply ran the get pods command to test if it will work or not. Here is the issue I am getting in the logs.
Running: gcloud container clusters get-credentials --project="journeyfoods-io" --zone="us-central1-a" "cloudbuild-test"
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required "container.clusters.get" permission(s) for "projects/XXXX/zones/us-central1-a/clusters/cloudbuild-test".
What permissions is it looking for? Do I need to do some
authentication before running the steps or What exactly am I missing?
The steps of a Cloud Build build are executed using with the [PROJECT_NUMBER]#cloudbuild.gserviceaccount.com service account. From the Cloud Build documentation page about this topic:
When you enable the Cloud Build API, the service account is
automatically created and granted the Cloud Build Service Account role
for your project. This role is sufficient for several tasks,
including:
Fetching code from your project's Cloud Source Repository
Downloading files from any Cloud Storage bucket owned by your project
Saving build logs in Cloud Logging
Pushing Docker images to Container Registry
Pulling base images from Container Registry
But this service account does not have permissions for certain actions by default ( in particular the container.clusters.get permission is not grated by default). So you need to grant it with a proper IAM role. In your case the Kubernetes Engine Developer role contains the container.clusters.get permission as you can see in this page.
I am trying to setup aws EKS cluster and want to connect that cluster from my local windows workstation. Not able to connect that. Here are the steps i did;
Create a aws service role (aws console -> IAM -> Roles -> click "Create role" -> Select AWS service role "EKS" -> give role name "eks-role-1"
Create another user in IAM named "eks" for programmatic access. this will help me to connect my EKS cluster from my local windows workstation. Policy i added into it is "AmazonEKSClusterPolicy", "AmazonEKSWorkerNodePolicy", "AmazonEKSServicePolicy", "AmazonEKS_CNI_Policy".
Next EKS cluster has been created with roleARN, which has been created in Step#1. Finally EKS cluster has been created in aws console.
In my local windows workstation, i have download "kubectl.exe" & "aws-iam-authenticator.exe" and did 'aws configure' using accesskey and token from step#2 for the user "eks". After configuring "~/.kube/config"; i ran below command and get error like this:
Command:kubectl.exe get svc
output:
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Unable to connect to the server: getting credentials: exec: exit status 1
Not sure what wrong setup here. Can someone pls help? I know some of the places its saying you have to use same aws user to connect cluster (EKS). But how can i get accesskey and token for aws assign-role (step#2: eks-role-1)?
For people got into this, may be you provision eks with profile.
EKS does not add profile inside kubeconfig.
Solution:
export AWS credential
$ export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxx
$ export AWS_SECRET_ACCESS_KEY=ssssssssss
If you've already config AWS credential. Try export AWS_PROFILE
$ export AWS_PROFILE=ppppp
Similar to 2, but you just need to do one time. Edit your kubeconfig
users:
- name: eks # This depends on your config.
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "general"
env:
- name: AWS_PROFILE
value: "<YOUR_PROFILE_HERE>" #
I think i got the answer for this issue; want to write down here so people will be benefit out of it.
When you first time creating EKS cluster; check from which you are (check your aws web console user setting) creating. Even you are creating from CFN script, also assign different role to create the cluster. You have to get CLI access for the user to start access your cluster from kubectl tool. Once you get first time access (that user will have admin access by default); you may need to add another IAM user into cluster admin (or other role) using congifMap; then only you can switch or use alternative IAM user to access cluster from kubectl command line.
Make sure the file ~/.aws/credentials has a AWS key and secret key for an IAM account that can manage the cluster.
Alternatively you can set the AWS env parameters:
export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=ssssssssss
Adding another option.
Instead of working with aws-iam-authenticator you can change the command to aws and replace the args as below:
- name: my-cluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args: #<--- Change the args
- --region
- <YOUR_REGION>
- eks
- get-token
- --cluster-name
- my-cluster
command: aws #<--- Change to command to aws
env:
- name: AWS_PROFILE
value: <YOUR_PROFILE_HERE>