We have a Vertex AI model that was created using a custom image.
We are trying to access a bucket on startup but we are getting the following error:
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/...?projection=noAcl&prettyPrint=false: {service account name} does not have storage.buckets.get access to the Google Cloud Storage bucket.
The problem is that I can't find the service account that is mentioned in the error to give it the right access permissions..
Solved it by giving the endpoint the right service account
I am dealing with the same general issue. Vertex AI can't access my Cloud Storage bucket from a custom prediction container.
After digging through a complex tree of GCP Vertex docs, I found this:
https://cloud.google.com/vertex-ai/docs/general/custom-service-account#setup
And
https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements#artifacts
UPDATE
After working through this, I was able to get GCS access to work.
Create a new service account and give it proper roles/permissions. You need your PROJECT_ID and PROJECT_NUMBER.
NEW_SA_EMAIL=new-service-account#$PROJECT_ID.iam.gserviceaccount.com
AI_PLATFORM_SERVICE_AGENT=service-$PROJECT_NUMBER#gcp-sa-aiplatform.iam.gserviceaccount.com
gcloud iam service-accounts create $NEW_SA_NAME \
--display-name="New Vertex AI Service Account" \
--quiet
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$NEW_SA_EMAIL" \
--role="roles/storage.admin" \
--quiet
gcloud iam service-accounts add-iam-policy-binding \
--role=roles/iam.serviceAccountAdmin \
--member=serviceAccount:$AI_PLATFORM_SERVICE_AGENT \
--quiet \
$NEW_SA_EMAIL
Upload your model and create the endpoint using the gcloud vertex ai commands...
Associate your new SA with the Vertex AI Endpoint when you deploy the model. You will need the GCP_REGION, ENDPOINT_ID, MODEL_ID, DEPLOYED_MODEL_NAME, and the NEW_SA_EMAIL from the first step.
gcloud ai endpoints deploy-model $ENDPOINT_ID \
--region=$GCP_REGION \
--model=$MODEL_ID \
--display-name=$DEPLOYED_MODEL_NAME \
--machine-type=n1-standard-4 \
--service-account=$NEW_SA_EMAIL
Related
I'm trying to query the GCP IAM recommender API (API documentation here) and fetch role revision recommendations for my project. I'm looking for ACTIVE recommendations only. However, the input filter stateInfo.state filter (listed in the above documentation) is not working for me. It returns the error Invalid Filter. Can someone please let me know what am I doing wrong here? Thanks.
Here's my API query: https://recommender.googleapis.com/v1/projects/my-demo-project/locations/global/recommenders/google.iam.policy.Recommender/recommendations?filter=stateInfo.state:ACTIVE
Please include a minimally reproducible example in questions:
PROJECT=[YOUR-PROJECT-ID]
LOCATION="global"
SERVICES=(
"cloudresourcemanager"
"recommender"
)
for SERVICE in ${SERVICES[#]}
do
gcloud services enable ${SERVICE}.googleapis.com \
--project=${PROJECT}
done
ACCOUNT="tester"
EMAIL=${ACCOUNT}#${PROJECT}.iam.gserviceaccount.com
gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}
# Minimal role for Recommender for IAM
gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/recommender.iamViewer
gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json \
--iam-account=${EMAIL} \
--project=${PROJECT}
# Be careful this overwrites the default gcloud auth account
# Remember to revert this to your e.g. me#gmail.com afterwards
gcloud auth activate-service-account --key-file=${PWD}/${ACCOUNT}.json
TOKEN=$(gcloud auth print-access-token ${EMAIL})
RECOMMENDER="google.iam.policy.Recommender"
PARENT="projects/${PROJECT}/locations/${LOCATION}/recommenders/${RECOMMENDER}"
FILTER="stateInfo.state=ACTIVE"
curl \
--header "Authorization: Bearer ${TOKEN}" \
https://recommender.googleapis.com/v1/${PARENT}/recommendations?filter=${FILTER}
Yields (HTTP 200):
{}
I've a Dataproc cluster with image version - 2.0.39-ubuntu18, which seems to be putting all logs into Cloud Logging, this is increasing our costs a lot.
Here is the command used to create the cluster, i've added the following - spark:spark.eventLog.dir=gs://dataproc-spark-logs/joblogs,spark:spark.history.fs.logDirectory=gs://dataproc-spark-logs/joblogs
to stop using the Cloud Logging, however that is not working .. Logs are being re-directed to Cloud Logging as well.
Here is the command used to create the Dataproc cluster :
REGION=us-east1
ZONE=us-east1-b
IMG_VERSION=2.0-ubuntu18
NUM_WORKER=3
# in versa-sml-googl
gcloud beta dataproc clusters create $CNAME \
--enable-component-gateway \
--bucket $BUCKET \
--region $REGION \
--zone $ZONE \
--no-address --master-machine-type $TYPE \
--master-boot-disk-size 100 \
--master-boot-disk-type pd-ssd \
--num-workers $NUM_WORKER \
--worker-machine-type $TYPE \
--worker-boot-disk-type pd-ssd \
--worker-boot-disk-size 500 \
--image-version $IMG_VERSION \
--autoscaling-policy versa-dataproc-autoscaling \
--scopes 'https://www.googleapis.com/auth/cloud-platform' \
--project $PROJECT \
--initialization-actions 'gs://dataproc-spark-configs/pip_install.sh','gs://dataproc-spark-configs/connectors-feb1.sh' \
--metadata 'gcs-connector-version=2.0.0' \
--metadata 'bigquery-connector-version=1.2.0' \
--properties 'dataproc:dataproc.logging.stackdriver.job.driver.enable=true,dataproc:job.history.to-gcs.enabled=true,spark:spark.dynamicAllocation.enabled=false,spark:spark.executor.instances=6,spark:spark.executor.cores=2,spark:spark.eventLog.dir=gs://dataproc-spark-logs/joblogs,spark:spark.history.fs.logDirectory=gs://dataproc-spark-logs/joblogs,spark:spark.jars.packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2'
We have another Dataproc cluster (image version 1.4.37-ubuntu18, similar configuration as the image version 2.0-ubuntu18), which has similar configuration but does not seem to using Cloud Logging as much.
Attached is screenshot properties of both the clusters.
What do i need to change to ensure the Dataproc jobs(pyspark) donot use the Cloud Logging ?
tia!
[
I saw dataproc:dataproc.logging.stackdriver.job.driver.enable is set to true. By default, the value is false, which means driver logs will be saved to GCS and streamed back to the client for viewing, but it won't be saved to Cloud Logging. You can try disabling it. BTW, when it is enabled, the job driver logs will be available in Cloud Logging under the job resource (instead of the cluster resource).
If you want to disable Cloud Logging completely for a cluster, you can either add dataproc:dataproc.logging.stackdriver.enable=false when creating the cluster or write an init action with systemctl stop google-fluentd.service. Both will stop Cloud Logging on the cluster's side, but using property is recommended.
See Dataproc cluster properties for the property.
Here is the update on this (based on discussions with GCP Support) :
In the GCP Logging, we need to create a Log Routing sink with inclusion filter - this will write the logs to BigQuery or Cloud Storage depending upon the target you specify.
Additionally, the _Default sink needs to be modified to add exclusion filters so specific logs will NOT be re-directed to GCP Logging
Attached are screenshots of the _Default log sink and the Inclusion sink for Dataproc.
In order to implement CI pipeline from github to gcp, I have configured workload identity.
SERVICE_ACCOUNT="xyz"
PROJECT_ID="ABC"
Service account created by the command:
gcloud iam service-accounts create "${SERVICE_ACCOUNT}" \
--description="${SERVICE_ACCOUNT}" \
--display-name="${SERVICE_ACCOUNT}"
Added principalSet by the following command:
gcloud iam service-accounts add-iam-policy-binding "${SERVICE_ACCOUNT}#${PROJECT_ID}.iam.gserviceaccount.com" \
--project="${PROJECT_ID}" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_NAME}/attribute.repository/${ORG_NAME}/${REPOSITORY}"
Upto this point was working fine.
But using this account I want to provision infrastructure and deploy applications as well.
So I have used following command:
gcloud iam service-accounts add-iam-policy-binding "${SERVICE_ACCOUNT}#${PROJECT_ID}.iam.gserviceaccount.com" \
--member "serviceAccount:${SERVICE_ACCOUNT}#${PROJECT_ID}.iam.gserviceaccount.com" \
--role "roles/container.clusterAdmin"
Likewise some more roles to be added. But I have following error:
ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) INVALID_ARGUMENT: Role roles/container.clusterAdmin is not supported for this resource.
Any feedback how to obtain the rights?
Reference: https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions
Add the IAM policy to the project and not to the service account.
gcloud iam projects add-iam-policy-binding "${PROJECT_ID} \
--member "serviceAccount:${SERVICE_ACCOUNT}#${PROJECT_ID}.iam.gserviceaccount.com" \
--role "roles/container.clusterAdmin"
am trying to create Dataproc cluster with a service account via cloud sdk. It's throwing an error that compute.projects.get is denied. The service account has compute viewer access, compute instance admin, dataproc editor access. Unable to understand why this error. In the IAM policy troubleshooter, I checked dataproc.cluster.create is assigned to the service account
The command is:
gcloud dataproc clusters create cluster-dqm01 \
--region europe-west-2 \
--zone europe-west2-b \
--subnet dataproc-standalone-paasonly-europe-west2 \
--master-machine-typne n1-standard-4 \
--master-boot-disk-size 500 \
--num-workers 2 \
--worker-machine-type n1-standard-4 \
--worker-boot-disk-size 500 \
--image-version 1.3-deb9 \
--project xxxxxx \
--service-account xxxx.iam.gserviceaccount.com
ERROR: (gcloud.dataproc.clusters.create) PERMISSION_DENIED: Required 'compute.projects.get' permission for 'projects/xxxxxx'
The project is correct as I have tried to create from the console getting the same error, generated the gcloud command from the console to run with a service account. This is the first time dataproc cluster is being created for the project
If you had assigned the various permissions to the same service account you're specifying with --service-account, the issue is that you probably meant to specify --impersonate-service-account instead.
There are three identities that are relevant here:
The identity issuing the CreateCluster command - this is often a human identity, but if you're automating things, using --impersonate-service-account, or running the command from inside another GCE VM, it may be a service account itself.
The "Control plane" identity - this is what the Dataproc backend service uses to actually create VMs
The "Data plane" identity - this is what the Dataproc workers behave as when processing data.
Typically, #1 and #2 need the various "compute" permissions and some minimal GCS permissions. #3 typically just needs GCS and optionally BigQuery, CloudSQL, Bigtable, etc. permissions depending on what you're actually processing.
See https://cloud.google.com/dataproc/docs/concepts/iam/dataproc-principals for more in-depth explanation of these identities.
It also lists the pre-existing curated roles to make this all easy (and typically, "default" project settings will automatically have the correct roles already so that you don't have to worry about it). Basically, the "human identity" or the service account you use with --impersonate-service-account needs Dataproc Editor or Project Editor roles, the "control plane identity" needs Dataproc Service Agent, and the "data plane identity" needs Dataproc Worker.
I'm trying to automate creation of service accounts for use with GKE via the gcloud command-line tool. I've figured out a flow that appears to mirror the process used by the Google Cloud Console, but my users don't see to receive the appropriate access.
Here's the commands I'm executing in order:
# Environment:
# - uname=<username>
# - email=<user's email address>
# - GCLOUD_PROJECT_ID=<project identifier>
# - serviceAccount="${uname}#${GCLOUD_PROJECT_ID}.iam.gserviceaccount.com"
$ gcloud iam service-accounts \
create "${uname}" --display-name "email:${email}" --format json
$ gcloud projects \
add-iam-policy-binding "${GCLOUD_PROJECT_ID}" \
--member "serviceAccount:${serviceAccount}" \
--role=roles/container.developer --format=json
$ gcloud iam service-accounts keys \
create "${GCLOUD_PROJECT_ID}-${uname}.json" \
--iam-account="${serviceAccount}"
When this executes, it creates a new service account and generates a key file locally. I then try to use this key to get credentials for my Kubernetes cluster.
$ gcloud config configurations create devcluster --activate
$ gcloud config set project devnet-166017
$ gcloud config set compute/zone us-central1-b
$ gcloud auth activate-service-account \
--key-file="${GCLOUD_PROJECT_ID}-${uname}.json"
$ gcloud container clusters get-credentials devcluster
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: \
code=403, message=Required "container.clusters.get" permission for \
"projects/${GCLOUD_PROJECT_ID}/zones/us-central1-b/clusters/devcluster".
It appears that for some reason my service account doesn't have one of the permissions it needs to get credentials, but based on what I've read and what I've observed in the Console, I believe this permission should be part of the roles/container.developer role.
Thanks!
I assume by service account, you mean the Service Account for Google Cloud. Here are the IAM roles related to GKE: https://cloud.google.com/container-engine/docs/iam-integration (search for container.).
First create a service account:
gcloud iam service-accounts create --display-name "GKE cluster access" gke-test
Then create a key:
gcloud iam service-accounts keys create key.json --iam-account=gke-test#[PROJECT_ID].iam.gserviceaccount.com
Now you need to assign some roles to this service account, your options are:
roles/container.admin Full management of Container Clusters and their Kubernetes API objects.
roles/container.clusterAdmin Management of Container Clusters.
roles/container.developer Full access to Kubernetes API objects inside Container Clusters.
roles/container.viewer Read-only access to Container Engine resources.
Again look at https://cloud.google.com/container-engine/docs/iam-integration page for details.
I assign roles/container.viewer (a read-only role, minimum you can assign to get-credentials) to this service account:
gcloud projects add-iam-policy-binding [PROJECT_ID] --role=roles/container.viewer --member=serviceAccount:gke-test#[PROJECT_ID].iam.gserviceaccount.com
Logout on gcloud from your current account:
gcloud auth revoke
Login to gcloud with the service account key:
gcloud auth activate-service-account --key-file=key.json
Try get-credentials:
$ gcloud container clusters get-credentials test --zone us-west1-a
Fetching cluster endpoint and auth data.
kubeconfig entry generated for test.
It works. I tried it with roles/container.developer, which also works.
You can try other permissions and see what works and what doesn't, although you made it clear that the documentation doesn't make it clear which roles have access to container.clusters.getCredentials.