I’m trying to create a docker container that will execute a BigQuery query. I started with the Google provided image that had gcloud already and I add my bash script that has my query. I'm passing my service account key as an environment file.
Dockerfile
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:latest
COPY main.sh main.sh
main.sh
gcloud auth activate-service-account X#Y.iam.gserviceaccount.com --key-file=/etc/secrets/service_account_key.json
bq query --use_legacy_sql=false
The gcloud command successfully authenticates but can't save to /.config/gcloud saying it is read-only. I've tried modifying that folders permissions during build and struggling to get it right.
Is this the right approach or is there a better way? If this is the right approach, how can I get ensure gcloud can write to the necessary folder?
See the example at the bottom of the Usage section.
You ought to be able to combine this into a single docker run command:
KEY="service_account_key.json"
echo "
[auth]
credential_file_override = /certs/${KEY}
" > ${PWD}/config
docker run \
--detach \
-env=CLOUDSDK_CONFIG=/config \
--volume=${PWD}/config:/config \
--volume=/etc/secrets/${KEY}:/certs/${KEY} \
gcr.io/google.com/cloudsdktool/cloud-sdk:latest \
bq query \
--use_legacy_sql=false
Where:
--env set the container's value for CLOUDSDK_CONFIG which depends on the first --volume flag which maps the host's config that we created in ${PWD} to the container's /config.
The second --volume flag maps the host's /etc/secrets/${KEY} (per your question) to the container's /certs/${KEY}. Change as you wish.
Suitably configured (🤞), you can run bq
I've not tried this but that should work :-)
I like to re-run or run a DAG from composer, and below command is what i have used, but i got some exceptions like this
kubeconfig entry generated for europe-west1-leo-stage-bi-db7ea92f-gke.
Executing within the following Kubernetes cluster namespace: composer-1-7-7-airflow-1-10-1-db7ea92f
command terminated with exit code 2
[2020-07-14 12:44:34,472] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2020-07-14 12:44:35,624] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2020-07-14 12:44:35,628] {__init__.py:51} INFO - Using executor CeleryExecutor
[2020-07-14 12:44:35,860] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2020-07-14 12:44:35,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2020-07-14 12:44:35,895] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
usage: airflow [-h]
{backfill,list_tasks,clear,pause,unpause,trigger_dag,delete_dag,pool,variables,kerberos,render,run,initdb,list_dags,dag_state,task_failed_deps,task_state,serve_logs,test,webserver,resetdb,upgradedb,scheduler,worker,flower,version,connections,create_user}
...
airflow: error: unrecognized arguments: --yes
ERROR: (gcloud.composer.environments.run) kubectl returned non-zero status code.
This is my command, the second line I have specified the parameters, can anyone help with this?
Thank you
gcloud composer environments run leo-stage-bi --location=europe-west1 backfill -- regulatory_spain_monthly -s 20190701 -e 20190702 -t "regulatory_spain_rud_monthly_materialization" --reset_dagruns
gcloud composer environments run project-name --location=europe-west1 backfill -- DAG name -s start date -e end date -t task in the DAG --reset_dagruns
I've checked Airflow backfill sub-command functionality within gcloud util from Google Cloud SDK 300.0.0 tools-set and I've finished my test attempts running backfill action with the same error:
airflow: error: unrecognized arguments: --yes
Digging into this issue and launching --verbosity=debug for gcloud composer environments run command, I've realized the cause of this lag:
gcloud composer environments run <ENVIRONMENT> --location=<LOCATION> --verbosity=debug backfill -- <DAG> -s <start_date> -e <end_date> -t "task_id" --reset_dagruns
DEBUG: Executing command: ['/usr/bin/kubectl', '--namespace',
'', 'exec', 'airflow-worker-*', '--stdin', '--tty',
'--container', 'airflow-worker', '--', 'airflow', 'backfill', '',
'-s', '<start_date>', '-e', '<end_date>', '-t', 'task_id',
'--reset_dagruns', '--yes']
The above output reflects a way how gcloud decouples command line arguments, dispatching them to kubectl command inheritor. Saying this, I assume that --yes argument for unknown reason was propagated and even more wrongly positioned out the rest of parameters.
Looking for a workaround I was on my way composing relevant kubectl command call to particular Airflow worker Pod, manually dispatching Airflow command line parameters:
kubectl -it exec $(kubectl get po -l run=airflow-worker -o jsonpath='{.items[0].metadata.name}' \
-n $(kubectl get ns| grep composer*| awk '{print $1}')) -n $(kubectl get ns| grep composer*| awk '{print $1}') \
-c airflow-worker -- airflow backfill <DAG> -s <start_date> -e <end_date> -t "task_id" --reset_dagruns
By now, airflow backfill command successes without throwing any error.
To trigger a manual run you can use the trigger_dag parameter:
gcloud composer environments run <COMPOSER_INSTANCE_NAME> --location <LOCATION> trigger_dag -- <DAG_NAME>
I am writing a script which involves kubectl delete , it is of course existential to run in against correct context-cluster.
The problem is that from what I observe, if you open two terminals and do:
kubectl config use-context bla
in one window, then the other one will switch as well. Therefore, concern if something switches context during script execution my delete operation will start deleting resources in the wrong cluster.
I understand that I could use labels on my pods or different namespaces, but in my case namespaces are the same and there are no labels.
So is there a way to specify for each command individually which context it should execute against? Something like:
kubectl get po use-context bla
Use the --context flag:
kubectl get po --context bla
If you run any kubectl command, you'll also see it says you can run kubectl options to see a list of global options that can be applied to any command. --context is one such global option.
I have a cloud_build.yaml script for my CI/CD pipeline on GP using Cloud Build. In command line I can pass a subtitution variable which will include the actual timestamp: "notebook-instance-$(date +%Y-%m-%d-%H-%M)-v05". This is working fine.
When I add github trigger on the Cloud Build webpage, then I didn't find the way to get the timestamp extracted in the same way that I was using in cli $(date +%Y-%m-%d-%H-%M)-v05:
Any idea idea how to do that on the Triggers Cloud Build page ?
I aslo tried to do it inside the cloud_build.yaml script but without success for now.
- name: 'gcr.io/cloud-builders/gcloud'
id: Deploy the AI Platform Notebook instance
args:
- 'deployment-manager'
- 'deployments'
- 'create'
- '$(date -u +%Y-%m-%d-%H-%M)-${_NAME_INSTANCE}'
Any idea how to extract and create a variable using the actual timestamp in the .yaml CloudBuild script ?
A third option is to extract the timestamp in my .jinja deployment script. Here I get the same issue as well that I don't find the way to to extract the actual timestampt to build my variable name.
One of the solution is to do the following:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: sh
args:
- '-c'
- |
gcloud \
deployment-manager \
deployments \
create \
xxxx
The issue is that you cannot use it in another step later. Another option is is to write te variable in a file on the workspace. This can be access later during the build stackoverflow
I have Ansible playbooks ready, they includes several encrypted vars. With normal process, I can feed a vault password file to decrypt them with --vault-password-file ~/.vault_pass.txt and deploy the change to remote EC2 instance. So I needn't expose the password file.
But my request is different here. I need include ansible-playbook change in user-data script when create a new EC2 instance. Ideally I should automatically have all setting ready after the instance is running.
I deploy the instances with Terraform by below simple user-data script:
#!/usr/bin/bash
yum -y update
/usr/local/bin/aws s3 cp s3://<BUCKET>/ansible.tar.gz ansible.tar.gz
gtar zxvf ansible.tar.gz
cd ansible
ansible-playbook -i inventory/ec2.py -c local ROLE.yml
So I have to upload my password file into user-data script as well, if in the playbook, there are some encrypted vars.
Anything I can do to avoid it? Will Ansible Tower help for this request?
I did test with CredStash, but still a chicken and egg issue.
If you want your instances to configure themselves they are going to either need all the credentials or another way to get the credentials, ideally with some form of one time pass.
The best I can think of off the top of my head is to use Hashicorp's Vault to store the credentials (potentially all of our secrets or maybe just the Ansible Vault password that then can be used to un-vault your Ansible variables) and have your deploy process create a one time use token that is injected into the user-data script via Terraform's templating.
To do this you'll probably want to wrap your Terraform apply command with some form of helper script that might look like this (untested):
#!/bin/bash
vault_host="10.0.0.3"
vault_port="8200"
response=`curl \
-X POST \
-H "X-Vault-Token:$VAULT_TOKEN" \
-d '{"num_uses":"1"}' \
http://${vault_host}:${vault_port}/auth/token/create/ansible_vault_read`
vault_token=`echo ${response} | jq '.auth.client_token' --raw-output`
terraform apply \
-var 'vault_host=${vault_host}'
-var 'vault_port=${vault_port}'
-var 'vault_token=${vault_token}'
And then your user data script will want to be templated in Terraform with something like this (also untested):
template.tf:
resource "template_file" "init" {
template = "${file("${path.module}/init.tpl")}"
vars {
vault_host = "${var.vault_host}"
vault_port = "${var.vault_port}"
vault_token = "${var.vault_token}"
}
}
init.tpl:
#!/usr/bin/bash
yum -y update
response=`curl \
-H "X-Vault-Token: ${vault_token}" \
-X GET \
http://${vault_host}:${vault_port}/v1/secret/ansible_vault_pass`
ansible_vault_password=`echo ${response} | jq '.data.ansible_vault_pass' --raw-output`
echo ${ansible_vault_password} > ~/.vault_pass.txt
/usr/local/bin/aws s3 cp s3://<BUCKET>/ansible.tar.gz ansible.tar.gz
gtar zxvf ansible.tar.gz
cd ansible
ansible-playbook -i inventory/ec2.py -c local ROLE.yml --vault-password-file ~/.vault_pass.txt
Alternatively you could simply have the instances call something such as Ansible Tower to trigger the playbook to be run against it. This allows you to keep the secrets on the central box doing the configuration rather than having to distribute them to every instance you are deploying.
With Ansible Tower this is done using callbacks and you will need to set up job templates and then have your user data script curl the Tower to trigger the configuration run. You could change your user data script to something like this instead:
template.tf:
resource "template_file" "init" {
template = "${file("${path.module}/init.tpl")}"
vars {
ansible_tower_host = "${var.ansible_tower_host}"
ansible_host_config_key = "${var.ansible_host_config_key}"
}
}
init.tpl:
#!/usr/bin/bash
curl \
-X POST
--data "host_config_key=${ansible_host_config_key}" \
http://{${ansible_tower_host}/v1/job_templates/1/callback/
The host_config_key may seem to be a secret at first glance but it's a shared key that can be used for multiple hosts to access a job template and Ansible Tower will still only run if the host is either defined in a static inventory for the job template or if you are using dynamic inventories then if the host is found in that lookup.