I am trying to make an automatic migration of workloads between two node pools in a GKE cluster. I am running Terraform in GitLab pipeline. When new node pool is created the local-exec runs and I want to cordon and drain the old node so that the pods are rescheduled on the new one. I am using this registry.gitlab.com/gitlab-org/terraform-images/releases/1.1:v0.43.0 image for my Gitlab jobs. Also, python3 is installed with apk add as well as gcloud cli - downloading the tar and using the gcloud binary executable from google-cloud-sdk/bin directory.
I am able to use commands like ./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file=<key here>.
The problem is that I am not able to use kubectl against my cluster.
Although I have installed the gke-gcloud-auth-plugin with ./google-cloud-sdk/bin/gcloud components install gke-gcloud-auth-plugin --quiet once in the CI job and second time in the local-exec script in HCL code I get the following errors:
module.create_gke_app_cluster.null_resource.node_pool_provisioner (local-exec): E0112 16:52:04.854219 259 memcache.go:238] couldn't get current server API group list: Get "https://<IP>/api?timeout=32s": getting credentials: exec: executable <hidden>/google-cloud-sdk/bin/gke-gcloud-auth-plugin failed with exit code 1
290module.create_gke_app_cluster.null_resource.node_pool_provisioner (local-exec): Unable to connect to the server: getting credentials: exec: executable <hidden>/google-cloud-sdk/bin/gke-gcloud-auth-plugin failed with exit code 1
When I check the version of the gke-gcloud-auth-plugin with gke-gcloud-auth-plugin --version
I am getting the following error:
174/bin/sh: eval: line 253: gke-gcloud-auth-plugin: not found
Which clearly means that the plugin is not installed.
The image that I am using is based on alpine for which there is no way to install the plugin via package manager, unfortunately.
Edit: gcloud components list shows gke-gcloud-auth-plugin as installed too.
The solution was to use google/cloud-sdk image in which I have installed terraform and used this image for the job in question.
trying to follow the Getting Started instructions for Deploying a Cloud Run service with Cloud Code in Intellij (deploying HelloWorld Flask app container with Cloud Run: Deploy) but getting the following error, any idea why this might be happening
it worked initially i.e. deployed the app on Cloud Run service using the same steps, and then started throwing this error after a week or so when trying to redeploy, there was no change in project settings.
intellij and docker versions are the latest.
authenticated to google cloud project with gcloud auth login --update-adc
The local run works fine (Cloud Run: Run Locally),
but running the Cloud Run: Deploy throws this "code 89" error
Preparing Google Cloud SDK (this may take several minutes for first time setup)...
Creating skaffold file: /var/.../skaffold8013155926954225609.tmp
Configuring image push settings in /var/.../skaffold8013155926954225609.tmp
../Library/Application Support/cloud-code/bin/versions/../
skaffold build --filename /var/.../skaffold8013155926954225609.tmp --tag latest --skip-tests=true
invalid skaffold config: getting minikube env:
running [/Users/USER/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/
minikube docker-env --shell none -p minikube --user=skaffold]
- stdout: "false exit code 89"
- stderr: ""
- cause: exit status 89
Failed to build and push Cloud Run container image.
Please ensure your builder settings are correct, network is available, you are logged in to a valid GCP project, and try again.
Edit: I see minikube error code 89: ExGuestUnavailable and it's an error code specific to the guest host, still unclear what might be causing this
Looks like an issue with skaffold attempting to communicate with minikube (which could be used for building images as well). Please try cleaning minikube
minikube stop
minikube delete --all --purge
and try again.
ok, i still don't know why it fails to deploy to cloud run from intellij but i got it to deploy from command line
cd my-flask-app
#step 1: build container image from Dockerfile and submit to container registry
gcloud builds submit --tag gcr.io/GCP_PROJECT_ID/my-flask-app
#step 2: deploy the image on cloud run (reference)
gcloud run deploy --image gcr.io/GCP_PROJECT_ID/my-flask-app
references:
https://cloud.google.com/build/docs/building/build-containers
https://cloud.google.com/container-registry/docs/quickstart
Edit: the answer above did the trick : minikube delete --all --purge
I like to re-run or run a DAG from composer, and below command is what i have used, but i got some exceptions like this
kubeconfig entry generated for europe-west1-leo-stage-bi-db7ea92f-gke.
Executing within the following Kubernetes cluster namespace: composer-1-7-7-airflow-1-10-1-db7ea92f
command terminated with exit code 2
[2020-07-14 12:44:34,472] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2020-07-14 12:44:35,624] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2020-07-14 12:44:35,628] {__init__.py:51} INFO - Using executor CeleryExecutor
[2020-07-14 12:44:35,860] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2020-07-14 12:44:35,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2020-07-14 12:44:35,895] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
usage: airflow [-h]
{backfill,list_tasks,clear,pause,unpause,trigger_dag,delete_dag,pool,variables,kerberos,render,run,initdb,list_dags,dag_state,task_failed_deps,task_state,serve_logs,test,webserver,resetdb,upgradedb,scheduler,worker,flower,version,connections,create_user}
...
airflow: error: unrecognized arguments: --yes
ERROR: (gcloud.composer.environments.run) kubectl returned non-zero status code.
This is my command, the second line I have specified the parameters, can anyone help with this?
Thank you
gcloud composer environments run leo-stage-bi --location=europe-west1 backfill -- regulatory_spain_monthly -s 20190701 -e 20190702 -t "regulatory_spain_rud_monthly_materialization" --reset_dagruns
gcloud composer environments run project-name --location=europe-west1 backfill -- DAG name -s start date -e end date -t task in the DAG --reset_dagruns
I've checked Airflow backfill sub-command functionality within gcloud util from Google Cloud SDK 300.0.0 tools-set and I've finished my test attempts running backfill action with the same error:
airflow: error: unrecognized arguments: --yes
Digging into this issue and launching --verbosity=debug for gcloud composer environments run command, I've realized the cause of this lag:
gcloud composer environments run <ENVIRONMENT> --location=<LOCATION> --verbosity=debug backfill -- <DAG> -s <start_date> -e <end_date> -t "task_id" --reset_dagruns
DEBUG: Executing command: ['/usr/bin/kubectl', '--namespace',
'', 'exec', 'airflow-worker-*', '--stdin', '--tty',
'--container', 'airflow-worker', '--', 'airflow', 'backfill', '',
'-s', '<start_date>', '-e', '<end_date>', '-t', 'task_id',
'--reset_dagruns', '--yes']
The above output reflects a way how gcloud decouples command line arguments, dispatching them to kubectl command inheritor. Saying this, I assume that --yes argument for unknown reason was propagated and even more wrongly positioned out the rest of parameters.
Looking for a workaround I was on my way composing relevant kubectl command call to particular Airflow worker Pod, manually dispatching Airflow command line parameters:
kubectl -it exec $(kubectl get po -l run=airflow-worker -o jsonpath='{.items[0].metadata.name}' \
-n $(kubectl get ns| grep composer*| awk '{print $1}')) -n $(kubectl get ns| grep composer*| awk '{print $1}') \
-c airflow-worker -- airflow backfill <DAG> -s <start_date> -e <end_date> -t "task_id" --reset_dagruns
By now, airflow backfill command successes without throwing any error.
To trigger a manual run you can use the trigger_dag parameter:
gcloud composer environments run <COMPOSER_INSTANCE_NAME> --location <LOCATION> trigger_dag -- <DAG_NAME>
Both gcloud auth login and gcloud compute --project XXX ssh --zone YYY ZZZ printed the error message in the title.
ERROR: gcloud crashed (TypeError): setproxy() takes at most 7 arguments (8 given)
The gcloud version is
Google Cloud SDK 186.0.0
bq 2.0.28
core 2018.01.22
gsutil 4.28
gcloud auth login has worked before the upgrade to 186.0.0 (from 182.0.0). All of gcloud has worked some time previously. To double-check, the proxy has been set:
gcloud config set proxy/address 127.0.0.1
gcloud config set proxy/port 8080
gcloud config set proxy/type http
but this has not changed anything (proxy is set via environment variables anyways).
https_proxy=http://127.0.0.1:8080
http_proxy=http://127.0.0.1:8080
UPDATE: The error has persisted even after removing the config directory ~/.config/gcloud. There's an issue at https://issuetracker.google.com/issues/72407295.
How can this be fixed?
As a workaround, downgrading to 185 solved the issue:
Go to https://storage.cloud.google.com/cloud-sdk-release, type "google-cloud-sdk-185" into the search bar, download the file for your OS, unzip, etc. You need to use a new shell to unset the change to the PATH that gcloud did.
It has been fixed in 187:
We've found a fix; expect it in the 187.0.0 Cloud SDK release next week.
Update Version 187 has been released: Download at https://cloud.google.com/sdk/downloads#versioned. This also gives the message
To revert your SDK to the previously installed version, you may run:
$ gcloud components update --version 185.0.0
after upgrade, so that could be used to revert as well, if similar problems occur.
After running: gcloud components update beta
It seems I lost compute commands:
gcloud -h
Usage: gcloud [optional flags] <group | command>
group may be auth | beta | components | config | dns | preview |
topic
command may be docker | help | info | init | version
How do I get compute back in order to run simple things like: gcloud compute images list --project google-containers | grep container-vm?
I followed migration path from: https://cloud.google.com/deployment-manager/latest/migration-guide
This is my gcloud -v:
Google Cloud SDK 0.9.67
beta 2015.06.30
bq 2.0.18
bq-nix 2.0.18
core 2015.06.30
core-nix 2015.06.02
gcloud 2015.06.30
gcutil-msg 2015.06.09
gsutil 4.13
gsutil-nix 4.12
preview 2015.06.30
Everytime I run the compute command, console gets in loop until I kill with ctrl+c:
[gcloud _19:33:01 $]]^_^ gcloud compute -h
You do not currently have this command group installed. Using it
requires the installation of components: [compute]
WARNING: Component [compute] no longer exists.
All components are up to date.
Restarting gcloud command:
$ gcloud compute -h
You do not currently have this command group installed. Using it
requires the installation of components: [compute]
WARNING: Component [compute] no longer exists.
All components are up to date.
Restarting gcloud command:
$ gcloud compute -h
^C
Is there anything I missed?
cheers
Leo
I had same error after updating my gcloud tools. I had to replace the current one by re-installing the gcloud kit to make it work.