In the billing section for one of my projects the costs for Compute Engine - E2 Instance Core of 12 hours are listed every day. But there are no instances in the Compute Engine section. The project actually only contains special Google Maps API keys that cannot be transferred.
I have also tried to disable the Compute Engine API. Unfortunately this fails with the following error: Hook call/poll resulted in failed op for service 'compute.googleapis.com': Could not turn off service, as it still has resources in use] with failed services [compute.googleapis.com]
Any idea?
Based on the error message: ‘Could not turn off service, as it still has resources in use] with failed services [compute.googleapis.com]’
That means that there are resources under Compute Engine API, so, you can either run a gcloud command to list the current instances or run a gcloud command to view the Asset Inventory, I suggest you to open your GCP project in a Chrome incognito window and use cloud shell.
List instances
gcloud compute instances list
List Asset Inventory
gcloud asset search-all-resources
NOTE: The Asset Inventory API is not enabled by default, so, after you run the command you’ll receive this message:
user#cloudshell:~ (project-id)$ gcloud asset search-all-resources
API [cloudasset.googleapis.com] not enabled on project [project-id].
Would you like to enable and retry (this will take a few minutes)?
(y/N)?
Please type y, to enable the API and be able to see the output of the command.
Having said that, when you see the results on the screen you’ll be able to identify the resources under the Compute Engine API and all its components, e.g.
---
additionalAttributes:
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/project-id/global/networks/default
networkIP: 1.18.0.5
assetType: compute.googleapis.com/Instance
displayName: linux-instance
location: us-central1-a
name: //compute.googleapis.com/projects/project-id/zones/us-central1-a/instances/linux-instance
project: projects/12345678910
---
additionalAttributes: {}
assetType: compute.googleapis.com/Disk
displayName: linux-instance
location: us-central1-a
name: //compute.googleapis.com/projects/project-id/zones/us-central1-a/disks/linux-instance
project: projects/12345678910
---
As you can see the 2 above lines describe the instance ‘linux-instance’ and its components (disk and ip address), all of them are under the API -> compute.googleapis.com
If you need further assistance, please send the output of the command to a TXT file and remove the sensitive information like: project-id, external IPs, internal IPs and share the output with me so I can take a look at it.
Alternatively, you can sanitize the output of the command just like I did it by replacing the instance name, project ID, project number and IP address with fake data.
Please keep in mind that since this is a billing concern the GCP billing team is open to hear you.
Curious.
There are some services that require Compute Engine resources, e.g. Kubernetes Engine, but I thought that, if used, the resources are always exposed.
One way to surface the user of this resource may be to enumerate the project's services and eyeball the result for a service that may be consuming VMs:
gcloud services list --enabled --project=[[YOUR-PROJECT]]
Related
I'd like to measure the number of times a Docker image has been downloaded from a Google Artifact registry repository in my GCP project.
Is this possible?
Interesting question.
I think this would be useful too.
I think there aren't any Monitoring metrics (no artifactregistry resource type is listed nor metrics are listed)
However, you can use Artifact Registry audit logs and you'll need to explicitly enable Data Access logs see e.g. Docker-GetManifest.
NOTE I'm unsure whether this can be achieved from gcloud.
Monitoring Developer tools, I learned that Audit Logs are configured in Project Policies using AuditConfig's. I still don't know whether this functionality is available through gcloud (anyone?) but evidently, you can effect these changes directly using API calls e.g. projects.setIamPolicy:
gcloud projects get-iam-policy ${PROJECT}
auditConfigs:
- auditLogConfigs:
- logType: DATA_READ
- logType: DATA_WRITE
service: artifactregistry.googleapis.com
bindings:
- members:
- user:me
role: roles/owner
etag: BwXanQS_YWg=
Then, pull something from the repo and query the logs:
PROJECT=[[YOUR-PROJECT]]
REGION=[[YOUR-REGION]]
REPO=[[YOUR-REPO]]
FILTER="
logName=\"projects/${PROJECT}/logs/cloudaudit.googleapis.com%2Fdata_access\"
protoPayload.methodName=\"Docker-GetManifest\"
"
gcloud logging read "${FILTER}" \
--project=${PROJECT} \
--format="value(timestamp,protoPayload.methodName)"
Yields:
2022-03-20T01:57:16.537400441Z Docker-GetManifest
You ought to be able to create a logs-based metrics for these too.
We do not yet have platform logs for Artifact Registry unfortunately, so using the CALs is the only way to do this today. You can also turn the CALs into log-based metrics and get graphs and metrics that way too.
The recommendation to filter by 'Docker-GetManifest' is also correct - it's the only request type for which a Docker Pull always has exactly one. There will be a lot of other requests that are related but don't match 1:1. The logs will have all requests (Docker-Token, 0 or more layer pulls), including API requests like ListRepositories which is called by the UI in every AR region when you load the page.
Unfortunately, the theory about public requests not appearing is correct. CALs are about logging authentication events, and when a request has no authentication whatsover, CALs are not generated.
I was happily deploying to Kubernetes Engine for a while, but while working on an integrated cloud container builder pipeline, I started getting into trouble.
I don't know what changed. I can not deploy to kubernetes anymore, even in ways I did before without cloud builder.
The pods rollout process gives an error indicating that it is unable to pull from the registry. Which seems weird because the images exist (I can pull them using cli) and I granted all possibly related permissions to my user and the cloud builder service account.
I get the error ImagePullBackOff and see this in the pod events:
Failed to pull image
"gcr.io/my-project/backend:f4711979-eaab-4de1-afd8-d2e37eaeb988":
rpc error: code = Unknown desc = unauthorized: authentication required
What's going on? Who needs authorization, and for what?
In my case, my cluster didn't have the Storage read permission, which is necessary for GKE to pull an image from GCR.
My cluster didn't have proper permissions because I created the cluster through terraform and didn't include the node_config.oauth_scopes block. When creating a cluster through the console, the Storage read permission is added by default.
The credentials in my project somehow got messed up. I solved the problem by re-initializing a few APIs including Kubernetes Engine, Deployment Manager and Container Builder.
First time I tried this I didn't succeed, because to disable something you have to disable first all the APIs that depend on it. If you do this via the GCloud web UI then you'll likely see a list of services that are not all available for disabling in the UI.
I learned that using the gcloud CLI you can list all APIs of your project and disable everything properly.
Things worked after that.
The reason I knew things were messed up, is because I had a copy of the same things as a production environment, and there these problems did not exist. The development environment had a lot of iterations and messing around with credentials, so somewhere things got corrupted.
These are some examples of useful commands:
gcloud projects get-iam-policy $PROJECT_ID
gcloud services disable container.googleapis.com --verbosity=debug
gcloud services enable container.googleapis.com
More info here, including how to restore service account credentials.
I want to create GKE clusters by gcloud command. But I cannot solve this error:
$ gcloud container clusters create myproject --machine-type=n1-standard1# --zone=asia-northeast1-a
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Google
Compute Engine: Required 'compute.networks.get' permission for
'projects/myproject/global/networks/default'
cloud account linked to my gmail is owner of the project and relative powers, so I anticipate that there is no problem about permissions.
When you create a cluster though $ gcloud container clusters create command you should keep in mind that there are hundreds of operations hidden.
When you have the owner rights then you are able to give the initial "Kick" to the process to make everything start. At this point Service accounts starts to enter in the process and they taking care of creating all the resource for you, automatically.
These service account have different powers and permissions (that can be customised) in order to limit the attack surface in case of one of them is compromise and to keep a sort of order, you will have for example ****-compute#developer.gservuceaccount.com that is a Default compute engine service account.
When you enable different the API some of these service accounts can be created in order to make the components work as expected, but if one of them is deleted or modified you might face one of the error that you are experiencing.
Usually the easiest way to solve the issue is recreate the service account for example deleting it and disabling an enabling the corresponting API.
For example when you enable Kubernetes engine service-****#container-engine-robot-iam-gaservice account is created
In my test project for example I modified them removing the "Kubernetes Engine service Agent" permission and I modified as well the Google APIs service account setting it as a "project viewer" and I am facing permission issues both creating and deleting clusters.
You can navigate through IAM&Amin-->admin to check the status and which service accounts are at the moment authorised in your project.
Here you can find a more deep explanation of some default service accounts.
Here you can find a small guide regarding how to re-enable Kubernetes Engine's default service account:
"If you remove this role binding from the service account, the default service account becomes unbound from the project, which can prevent you from deploying applications and performing other cluster operations."
While trying for the first time to use Google Kubernetes Cloud solution, and according to the tutorial... I am trying to create new cluster.
But after pressing Create i receive
The request contains invalid arguments: "EXTERNAL: service account
"****#developer.gserviceaccount.com" does not exist.". Error code: "7"
in a red circle near the Kubernetes cluster name.
After some investigations it's looks like the default service account which google generated for my account.
I've looked over the create cluster options, but there isn't any option to change the service account.
Do I need to change Google Compute Engine default service account? how i can do it?
How I can overcome this issue?
Thank you
Default Compute Engine Service Account is essential for functions related to Compute Engine and is being generated automatically. Kubernetes Engine utilizes Compute Engine VM Instances as Nodes used for the cluster. GKE uses the Compute Engine Service Account to authorize the creation of these nodes.
In order to regenerate default service there are two options:
Regenerate by Disabling and Re-enabling the Google Compute Engine API. In the "API's & Services" dashboard. If for some reason performing this option encountering errors when disabling the API, then try option 2.
run command gcloud services enable compute.googleapis.com in Cloud SDK or Cloud Shell which is in the header of the page.
Looks like you either do not have any default service account or have more than one.
Simply go to the "Service Accounts" section "IAM & Admin" and select the app engine default service account, and provide this as an argument while creating cluster from gcloud or gshell as below:
gcloud container clusters create my-cluster --zone=us-west1-b --machine-type=n1-standard-1 --disk-size=100 --service-account=abc#appspot.gserviceaccount.com
To initialize GKE, go to the GCP Console. Wait for the "Kubernetes Engine is getting ready. This may take a minute or more" message to disappear.
Please open the page and wait for a while
From a VM in GCE, I did the following
gcloud auth activate-service-account --key-file <blah>
# "blah" is a service account key file (JSON) I generated from the web interface
gcloud config set project <project-name>
gcloud config set compute/zone <zone-name>
gcloud set container/cluster <cluster-name>
Then when I tried to run
gcloud container clusters get-credentials <cluster-name>
and it failed with the error message:
Error message: "ERROR: (gcloud.container.clusters.get-credentials)
ResponseError: code=403, message=Request had insufficient
authentication scopes."
The VM is on the same network as the GKE cluster. I tried the same thing, with the same service account key file from a machine outside GCE, against a GKE cluster on the "default" network and it succeeded...
To use the Google Kubernetes Engine API from a GCE virtual machine you need to add the cloud platform scope ("https://www.googleapis.com/auth/cloud-platform") to your VM when it is created.
If you are using The Google Kubernetes Engine API from a VM in GCP. You first need to add required scope at vm level https://www.googleapis.com/auth/projecthosting,cloud-platform. This can be done by GCP console as well. Stop the VM instance then go to edit option and at the end you will find Cloud API access scopes.
There is now a solution (in beta and alpha only) to set scope on an existing GCE VM. All it needs to be successful is to stop the VM before executing the command.
First, you should be aware (and copy) current scopes of your VM so you can set them along with your new scopes, use:
gcloud compute instances describe your-instance
At the bottom you should see a list of scopes, copy them.
Then, read documentation for this command in beta (available to everyone but to be used at your own risk): https://cloud.google.com/sdk/gcloud/reference/beta/compute/instances/set-scopes
Before you execute this command, stop the instance from the GCE page and wait for it to be shut down. A scary warning will appear, be aware that if the VM does not shut down gracefully in 90 seconds (= all processes and services successfully turned off) the file system might get corrupt when force shutting down the VM. Take good note and backup important files if you feel unsafe about this.
For me, with the existing scopes plus the new one (cloud-platform) the resulting set-scopes command was:
gcloud beta compute instances set-scopes my-instance --zone=us-central1-a --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append,https://www.googleapis.com/auth/cloud-platform
Step 1 : gcloud init
Step 2 : Select [2] Create a new configuration
Step 3 : Enter configuration name. Names start with a lower case letter and contain only lower case letters a-z, digits 0-9, and hyphens '-': kubernetes-service-account
Step 4 : Choose the account you would like to use to perform operations for this configuration:[2] Log in with a new account
Step 5 : Do you want to continue (Y/n)? y
Step 6 : Copy paste the link to brwoser and login with the ID which is used to create your google Cloud Account
Step 7 : Copy the verification code provided by google after login and paste it in to the console.
Step 8 : Pick cloud project to use:
Step 9: Do you want to configure a default Compute Region and Zone? (Y/n)? y
Step 10 : Please enter numeric choice or text value (must exactly match list item): 8
Your Google Cloud SDK is configured and ready to use!
Once this is done, make sure the service account configured for the VM has permissions to do the required tasks.