Trying connect SQL instance to Cloud Run Service, using Fully Managed cloud run works fine but when I try to connect service via Anthos (which is required as we need to use websockets on services) I just get ENOENT (No Entry), update IAM for GKE with correct permissions, recreated cluster with all services enabled/
Here's the deploy command I am doing
gcloud run deploy \
--project ${GOOGLE_PROJECT_ID} \
--platform gke \
--cluster dev \
--cluster-location ${GOOGLE_COMPUTE_ZONE} \
--image gcr.io/${GOOGLE_PROJECT_ID}/${PROJECT_NAME} \
--set-cloudsql-instances "${GOOGLE_PROJECT_ID}:europe-west1:dev" \
--set-env-vars "$(tr '\n' ',' < "${ENV_KEY_PRODUCTION}")" \
--set-env-vars "SERVICE=${1}" \
--set-env-vars "DB_HOST=/cloudsql/${GOOGLE_PROJECT_ID}:europe-west1:dev" \
"${1}"
If I use the private IP from SQL and remove --set-cloudsql-instances and set DB_HOST as private IP it works.
But adding --set-cloudsql-instances should make a sidecar for service in GKE cluster and allow it to connect to SQL?
The documentation isn't clear... the parameter '--set-cloudsql-instances' is only available for Cloud Run Managed version. The first sentence of the section is important. And the limitation is not clear in the doc
Only applicable if connecting to Cloud Run (fully managed). Specify --platform=managed to use:
--[no-]allow-unauthenticated
Whether to enable allowing unauthenticated access to the service. This may take a few moments to take effect. Use --allow-unauthenticated to enable and --no-allow-unauthenticated to disable.
--clear-vpc-connector
Remove the VPC connector for this Service.
--revision-suffix=REVISION_SUFFIX
Specify the suffix of the revision name. Revision names always start with the service name automatically. For example, specifying [--revision-suffix=v1] for a service named 'helloworld', would lead to a revision named 'helloworld-v1'.
--vpc-connector=VPC_CONNECTOR
Set a VPC connector for this Service.
These flags modify the Cloud SQL instances this Service connects to. You can specify a name of a Cloud SQL instance if it's in the same project and region as your Cloud Run service; otherwise specify :: for the instance. At most one of these may be specified:
--add-cloudsql-instances=[CLOUDSQL-INSTANCES,…]
Append the given values to the current Cloud SQL instances.
--clear-cloudsql-instances
Empty the current Cloud SQL instances.
--remove-cloudsql-instances=[CLOUDSQL-INSTANCES,…]
Remove the given values from the current Cloud SQL instances.
--set-cloudsql-instances=[CLOUDSQL-INSTANCES,…]
Completely replace the current Cloud SQL instances with the given values.
Related
I have a cloud run service which is trying to connect to a cloud SQL (postgres) instance. There is a timeout connecting:
File "/usr/local/lib/python3.10/site-packages/google/cloud/sql/connector/connector.py", line 261, in connect_async
return await asyncio.wait_for(get_connection(), timeout)
File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc"
Checked that:
The instance is added in the cloud run config under "Cloud SQL connections"
This code works fine locally against the same instance with the same service account. To ensure it uses the service account, not my own login, checked:
gcloud auth revoke
Check that application doesn't work
export GOOGLE_APPLICATION_CREDENTIALS=/<serviceaccountcreds>.json
Check that application works.
Where do I dig?
The connection is gotten with the sample code, nothing fancy.
You can connect to Cloud SQL using TCP to private db instance IP by the VPC Connector. Or by the Cloud SQL Proxy (auto SSL/TSL) to the Cloud SQL Proxy Server available by default in the Cloud SQL instance (auth by IAM).
To test from your local dev PC you do:
gcloud components install cloud_sql_proxy
create an instance (you have one already..)
gcloud sql instances create sql-db ...
create a sample db:
gcloud sql databases create my_db --instance sql-db
connect with:
gcloud beta sql connect sql-db --user root
remove the default root user and configure a new one that can connect only passing by the Cloud SQL Proxy (if you want that kind of security)
gcloud sql users delete root --host % --instance sql-db
gcloud sql users create root --host "cloudsqlproxy~%" --instance sql-db
# network "cloudsqlproxy" is shared between
# cloud sql instance and the proxy server
to deploy the service with access to cloudsql:
DB_INSTANCE=$PROJECT\:$REGION\:sql-db
gcloud run deploy <SERVICE_NAME> \
--add-cloudsql-instances $DB_INSTANCE \
--set-env-vars DB="mysql://root#unix(/cloudsql/$DB_INSTANCE)/my_db"
# all other flags...
# DB is alternative to SOCKET_PATH, both for the Cloud SQL client init
the role for the service account to work with cloudsql:
gcloud run services add-iam-policy-binding $SERVICE_NAME \
--member serviceAccount:$SERVICE_ACCOUNT \
--role roles/cloudsql.client
I am currently trying to provision a GCE instance that will execute a Docker container in order to retrieve some information from the web and push them to BigQuery.
Now, the newly created service account (screenshot below) doesn't affect the api scopes whatsoever. This obviously makes the container fail when authenthicating to BQ. Funny thing is, when I use the GCE default service account and select auth scopes manually from the GUI everything works like a charm.
I am failing to understand why the following service account doesn't open api auth scopes to the machine. I might be overlooking something really simple on this one.
Context
The virtual machine is created and run with the following gcloud command:
#!/bin/sh
gcloud compute instances create-with-container gcp-scrape \
--machine-type="e2-micro" \
--boot-disk-size=10 \
--container-image="gcr.io/my_project/gcp_scrape:latest" \
--container-restart-policy="on-failure" \
--zone="us-west1-a" \
--service-account gcp-scrape#my_project.iam.gserviceaccount.com \
--preemptible
This is how bigquery errors out when using my custom service account:
Access Denied: BigQuery BigQuery: Missing required OAuth scope. Need BigQuery or Cloud Platform read scope.
You haven't specified a --scopes flag, so the instance uses the default scope which doesn't include BigQuery.
To let the instance access all services that the service account can access, add --scopes https://www.googleapis.com/auth/cloud-platform to your command line.
am trying to create Dataproc cluster with a service account via cloud sdk. It's throwing an error that compute.projects.get is denied. The service account has compute viewer access, compute instance admin, dataproc editor access. Unable to understand why this error. In the IAM policy troubleshooter, I checked dataproc.cluster.create is assigned to the service account
The command is:
gcloud dataproc clusters create cluster-dqm01 \
--region europe-west-2 \
--zone europe-west2-b \
--subnet dataproc-standalone-paasonly-europe-west2 \
--master-machine-typne n1-standard-4 \
--master-boot-disk-size 500 \
--num-workers 2 \
--worker-machine-type n1-standard-4 \
--worker-boot-disk-size 500 \
--image-version 1.3-deb9 \
--project xxxxxx \
--service-account xxxx.iam.gserviceaccount.com
ERROR: (gcloud.dataproc.clusters.create) PERMISSION_DENIED: Required 'compute.projects.get' permission for 'projects/xxxxxx'
The project is correct as I have tried to create from the console getting the same error, generated the gcloud command from the console to run with a service account. This is the first time dataproc cluster is being created for the project
If you had assigned the various permissions to the same service account you're specifying with --service-account, the issue is that you probably meant to specify --impersonate-service-account instead.
There are three identities that are relevant here:
The identity issuing the CreateCluster command - this is often a human identity, but if you're automating things, using --impersonate-service-account, or running the command from inside another GCE VM, it may be a service account itself.
The "Control plane" identity - this is what the Dataproc backend service uses to actually create VMs
The "Data plane" identity - this is what the Dataproc workers behave as when processing data.
Typically, #1 and #2 need the various "compute" permissions and some minimal GCS permissions. #3 typically just needs GCS and optionally BigQuery, CloudSQL, Bigtable, etc. permissions depending on what you're actually processing.
See https://cloud.google.com/dataproc/docs/concepts/iam/dataproc-principals for more in-depth explanation of these identities.
It also lists the pre-existing curated roles to make this all easy (and typically, "default" project settings will automatically have the correct roles already so that you don't have to worry about it). Basically, the "human identity" or the service account you use with --impersonate-service-account needs Dataproc Editor or Project Editor roles, the "control plane identity" needs Dataproc Service Agent, and the "data plane identity" needs Dataproc Worker.
I am trying to create a cluster with GKE. I have a project I have been using already.
When I run
gcloud container clusters create cluster1
I get the following:
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Google Compute Engine: Required 'compute.networks.get' permission for 'projects//global/networks/default'.
The same thing happens when I use the web UI. Both my service account and my user have owner roles.
I have tried the following to get the cluster create command to work:
I tried adding a policy binding for the project for my existing service account:
gcloud projects add-iam-policy-binding <my-project> \
--member serviceAccount:<my-user>#<my-project>.iam.gserviceaccount.com \
--role roles/compute.admin
I read enabling the container api service was required
gcloud services enable container.googleapis.com
Started over. I deleted the service account, created a new one and activated the creds with:
gcloud auth activate-service-account <my-user>#<my-project>.iam.gserviceaccount.com --key-file ${GOOGLE_APPLICATION_CREDENTIALS}
I also tried authenticating with my account user:
gcloud auth login
None of these work and I can't create a cluster
I think I will answer my own question here. From service account docs
When you create a new Cloud project using GCP Console and if Compute Engine API is enabled for your project, a Compute Engine Service account is created for you by default. It is identifiable using the email:
PROJECT_NUMBER-compute#developer.gserviceaccount.com
I had delete the default created service accounts somehow and possible the associated roles. I think this is why I couldn't create a cluster under my project anymore. Rather than try to figure out how to recreate, I decided it was best to just start a new project. Afterwords, the cluster create API and console work just fine.
Debug:
gcloud container subnets list-usable --project service-project --network-project shared-vpc-project
If you get warning in output:
WARNING: Failed to get metadata from network project. GCE_PERMISSION_DENIED:
Google Compute Engine: Required 'compute.projects.get' permission for
'projects/shared-vpc-project'
It means your google managed gke service account in host project doesn't exist.
To solve go to host project apis and enable Kubernetes Engine API. If it's enabled, disable it and enable again back.
I think you should set the compute engine service account permission:
gcloud projects add-iam-policy-binding <my-project> \
--member [PROJECT_NUMBER]-compute#developer.gserviceaccount.com \
--role roles/compute.admin
I'm running a pyspark application through Google's dataproc on a cluster I created. In one stage, the application needs to access a directory in an Amazon S3 directory. At that stage, I get the error:
AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
I logged onto the headnode of the cluster and set the /etc/boto.cfg with my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY information, but that didn't solve the access issue.
(1) Any other suggestions for how to access AWS S3 from a dataproc cluster?
(2) Also, what is the name of the user that dataproc uses to access the cluster? If I knew that, I could set the ~/.aws directory on the cluster for that user.
Thanks.
Since you're using the Hadoop/Spark interfaces (like sc.textFile), everything should indeed be done through the fs.s3.* or fs.s3n.* or fs.s3a.* keys rather than trying to wire through any ~/.aws or /etc/boto.cfg settings. There are a few ways you can plumb those settings through to your Dataproc cluster:
At cluster creation time:
gcloud dataproc clusters create --properties \
core:fs.s3.awsAccessKeyId=<s3AccessKey>,core:fs.s3.awsSecretAccessKey=<s3SecretKey> \
--num-workers ...
The core prefix here indicates you want the settings to be placed in the core-site.xml file, as explained in the Cluster Properties documentation.
Alternatively, at job-submission time, if you're using Dataproc's APIs:
gcloud dataproc jobs submit pyspark --cluster <your-cluster> \
--properties spark.hadoop.fs.s3.awsAccessKeyId=<s3AccessKey>,spark.hadoop.fs.s3.awsSecretAccessKey=<s3SecretKey> \
...
In this case, we're passing the properties through as Spark properties, and Spark provides a handy mechanism to define "hadoop" conf properties as a subset of Spark conf, simply using the spark.hadoop.* prefix. If you're submitting at the command line over SSH, this is equivalent to:
spark-submit --conf spark.hadoop.fs.s3.awsAccessKeyId=<s3AccessKey> \
--conf spark.hadoop.fs.s3.awsSecretAccessKey=<s3SecretKey>
Finally, if you want to set it up at cluster creation time but prefer not to have your access keys explicitly set in your Dataproc metadata, you might opt to use an initialization action instead. There's a handy tool called bdconfig that should be present on the path with which you can modify XML settings easily:
#!/bin/bash
# Create this shell script, name it something like init-aws.sh
bdconfig set_property \
--configuration_file /etc/hadoop/conf/core-site.xml \
--name 'fs.s3.awsAccessKeyId' \
--value '<s3AccessKey>' \
--clobber
bdconfig set_property \
--configuration_file /etc/hadoop/conf/core-site.xml \
--name 'fs.s3.awsSecretAccessKey' \
--value '<s3SecretKey>' \
--clobber
Upload that to a GCS bucket somewhere, and use it at cluster creation time:
gsutil cp init-aws.sh gs://<your-bucket>/init-aws.sh
gcloud dataproc clustres create --initialization-actions \
gs://<your-bucket>/init-aws.sh
While Dataproc metadata is indeed encrypted at rest and heavily secured just like any other user data, using the init action instead helps prevent inadvertently showing your access key/secret for example to someone standing behind your screen when viewing your Dataproc cluster properties.
You can try with setting the AWS config, while initialization of sparkContext.
conf = < your SparkConf()>
sc = SparkContext(conf=conf)
sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", <s3AccessKey>)
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", <s3SecretKey>)