Why is Rsync failing to sync a specific folder? - google-cloud-platform

We are using rsync when syncing to our dev server on GCP. Recently ive noticed that it does not sync some of our files in a specific folder. The others work well like before.
This is how our trigger looks like:
First one is working
rsync -aqe ssh \
--no-g \
--no-p \
--delete \
--force \
--dirs \
--perms \
--no-owner \
--no-group \
--exclude-from="/opt/projects/puppet/devtools/sync_script/support_files/sync_web.exclude" \
--log-file="/tmp/rsync.log" \
/opt/projects/web/src/ \
"${DEV_IP}":/var/www/src/
Not working example:
rsync -aqe ssh \
--no-g \
--no-p \
--delete \
--force \
--dirs \
--perms \
--no-owner \
--no-group \
--exclude-from="/opt/projects/puppet/devtools/sync_script/support_files/sync_web.exclude" \
--log-file="/tmp/rsync.log" \
/opt/projects/web/ \
"${DEV_IP}":/var/www/
So when we specifinc that folder it works. But if use one level up from it thoose files will not be synced.

In my opinion, it is best to turn off the -q option, which would indicate if there is an error. In case there is no error you can always try to debug with
-v, --verbose increase verbosity
--info=FLAGS fine-grained informational verbosity
--debug=FLAGS fine-grained debug verbosity
you can also check the rsync.log log file.

Related

Google cloud compute VM startup script got interrupted and does not finish

I followed this guide in order to create self deleting virtual machine after 60 seconds with the following script calling it from a python script. Bellow you can find the startup script:
#!/bin/bash
echo Start the startup script
sleep 60s
echo BEFORE Deleting the VMs after max running time
export NAME="$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/name -H 'Metadata-Flavor: Google')"
export ZONE="$(curl -X GET http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google')"
echo AFTER Deleting the VMs after max running time
gcloud --quiet compute instances delete $NAME --zone=$ZONE
Here is how it was triggered from the python code:
cmd = """gcloud compute instances create-with-container \
{0} \
--project={1} \
--zone=us-central1-c \
--container-image=gcr.io/project/image \
--machine-type={2} \
--scopes "bigquery","gke-default","storage-full","compute-rw" \
--boot-disk-size {3} \
--boot-disk-type "pd-ssd" \
--container-env YAML={4},DATE={5},BUCKET={6} \
--service-account "{7}" \
--metadata-from-file=startup-script=startup.sh \
--description="{8}"
""".format(vm,
gcp,
machine,
disk_size,
yamlup,
self.partition,
bucket_name,
serviceaccount,
description
)
On the google cloud compute engine, I can see the first echo appear in the logs: "Start the startup script" but after the sleep nothing happens. I am also not even sure if the sleep command works. Is there anything missing?

Why is Dataflow Runner v2 experiments flag not appearing in pipeline options?

I've added to this command so that Dataflow Runner v2 is used:
mvn -Pdataflow-runner compile exec:java \
-Dexec.mainClass=org.apache.beam.examples.WordCount \
-Dexec.args="--project=PROJECT_ID \
--gcpTempLocation=gs://BUCKET_NAME/temp/ \
--output=gs://BUCKET_NAME/output \
--runner=DataflowRunner \
--region=REGION \
--experiments=use_runner_v2"
Note --experiments=use_runner_v2 is added at the end. But experiments flag is not shown in the pipeline options in the gcp ui:
The userAgent is Apache_Beam_SDK_for_Java/2.39.0(JRE_17_environment) and region is europe-west1. This might be relevant information about why my setup isn't working.

using gcloud beta builds triggers create cloud-source-repositories doesn't working with --dockerfile-image

I'm working on a auto devops workflow only based on the dockerfile using Cloud Build on GCP, when I try to use the following command it seems is not using the flag: --dockerfile-image
gcloud beta builds triggers create cloud-source-repositories \
--name="test-trigger-2" \
--repo="projects/nodrize-dev/repos/b722166a-56e0-46af-bd0d-42af8d37c570/bf11672f-34d5-4d8c-80cb-31120f39251a/quirino-backend" \
--branch-pattern="^master$" \
--dockerfile="Dockerfile" \
--dockerfile-dir="" \
--dockerfile-image="gcr.io/nodrize-dev/test-backend"
Created [https://cloudbuild.googleapis.com/v1/projects/nodrize-dev/triggers/896f8ac8-397c-464a-84f7-43e69f1bc6cb].
NAME CREATE_TIME STATUS
test-trigger-2 2021-06-02T21:06:54+00:00
I want to create trigger to run it later but the last flag isnt working I asume is using the default or fallback, because as you can see in the image name is:
gcr.io/nodrize-dev/b722166a-56e0-46af-bd0d-42af8d37c570/bf11672f-34d5-4d8c-80cb-31120f39251a/quirino-backend:$COMMIT_SHA:
dockerimage-name in gcp concole:
I hope someone can help me or at least know what is happening.
This works for me.
I suspect perhaps that the trigger is incorrect or is not being triggered and|or the image is not what was generated by the trigger.
PROJECT=...
REPO=...
gcloud source repos create ${REPO} \
--project=${PROJECT}
gcloud beta builds triggers create cloud-source-repositories \
--name="trigger" \
--project=${PROJECT} \
--repo=${REPO} \
--branch-pattern="^master$" \
--dockerfile="Dockerfile" \
--dockerfile-dir="." \
--dockerfile-image="gcr.io/${PROJECT}/freddie-01"
NAME CREATE_TIME STATUS
trigger 2021-06-03T15:24:27+00:00
git push google master
gcloud builds list \
--project=${PROJECT} \
--format="value(images)"
gcr.io/${PROJECT}/freddie-01:7dcf74e126af711d24bb2b652d86f0d28bbe3bd9
gcloud container images list \
--project=${PROJECT}
NAME
gcr.io/${PROJECT}/freddie-01

How to change the scope of existing VM in GCP?

I created a windows VM where I have the BERT master, SQUAD, and BERT-large model. I tried to run the squad using this:
python run_squad.py \
--vocab_file=$BERT_LARGE_DIR/vocab.txt \
--bert_config_file=$BERT_LARGE_DIR/bert_config.json \
--init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \
--do_train=True \
--train_file=$SQUAD_DIR/train-v2.0.json \
--do_predict=True \
--predict_file=$SQUAD_DIR/dev-v2.0.json \
--train_batch_size=24 \
--learning_rate=3e-5 \
--num_train_epochs=2.0 \
--max_seq_length=384 \
--doc_stride=128 \
--output_dir=gs://some_bucket/squad_large/ \
--use_tpu=True \
--tpu_name=$TPU_NAME \
--version_2_with_negative=True
It threw an error: googleapiclient.errors.HttpError: <HttpError 403 when requesting https://tpu.googleapis.com/v1alpha1/projects/projectname/locations/us-central1-a/nodes/testnode?alt=json returned "Request had insufficient authentication scopes.">
Is there a way to change the scope of existing VM to cloud-platform after VM is created?
Is there a way to change the scope of existing VM to cloud-platform
after VM is created?
Yes you can. Go to the Google Cloud Console. Select your instance and stop it. Then edit your instance and change the scopes, etc. The restart your instance.

Adding custom dependancy wont work in ML-Engine submit training

I have a .sh script that lunches a submit training job as following:
now=$(date +"%Y%m%d_%H%M%S")
JOB_NAME="campign_retention_model__$now"
JOB_DIR="gs://machine_learning_datasets/campaign_retention"
REGION="us-east1"
PYTHON_VERSION='3.5'
RUNTIME_VERSION='1.12'
TRAINER_PACKAGE_PATH="./trainer/"
PACKAGE_STAGING_PATH="gs://machine_learning_datasets/campaign_retention"
CLOUDSDK_PYTHON="/usr/bin/python"
MAIN_TRAINER_MODULE="trainer.task"
gcloud ml-engine jobs submit training $JOB_NAME \
--job-dir $JOB_DIR \
--package-path $TRAINER_PACKAGE_PATH \
--module-name $MAIN_TRAINER_MODULE \
--region $REGION \
--runtime-version=$RUNTIME_VERSION \
--python-version=$PYTHON_VERSION \
Which works great (Notice that the .sh is located next to the trainer dir).
Due to external infra requirements, i was forced to save the content of my project within a bucket named:
"gs://campign_retention_code/camp_ret"
And hand out a stand alone sh, So I've just changed it to (just changed the path of TRAINER_PACKAGE_PATH):
now=$(date +"%Y%m%d_%H%M%S")
JOB_NAME="campign_retention_model__$now"
JOB_DIR="gs://machine_learning_datasets/campaign_retention"
REGION="us-east1"
PYTHON_VERSION='3.5'
RUNTIME_VERSION='1.12'
TRAINER_PACKAGE_PATH="gs://campign_retention_code/camp_ret/trainer"
PACKAGE_STAGING_PATH="gs://machine_learning_datasets/campaign_retention"
CLOUDSDK_PYTHON="/usr/bin/python"
MAIN_TRAINER_MODULE="trainer.task"
gcloud ml-engine jobs submit training $JOB_NAME \
--job-dir $JOB_DIR \
--package-path $TRAINER_PACKAGE_PATH \
--module-name $MAIN_TRAINER_MODULE \
--region $REGION \
--runtime-version=$RUNTIME_VERSION \
--python-version=$PYTHON_VERSION \
Now when i'm running it (I moved it to a different location on the desktop to /Users/yehoshaphatschellekens/Desktop, to make sure its not close to my project) i'm getting the following error:
ERROR: (gcloud.ml-engine.jobs.submit.training) Source directory [/Users/yehoshaphatschellekens/Desktop/camp_ret] is not a valid directory.
Looking at the docs packaging-trainer i noticed that there are two examples, one that works like my original script, which as i said, works perfectly, and another example that uses a packaged dependancy.
Why the submit job won't recognise my dependancies on gs, can't i just point to --package-path a directory from gs instead of my local dir?
Thanks in Advance!!!
I believe what you are trying to do requires using
--packages gs://path/to/packages
INSTEAD of --package-path