I want to implement an automated code to shut down VM Instances in my Google Cloud account via Crontab. I use Ubuntu 18.
I tried this code:
#!/bin/bash
source/etc/profile
echo Y | gcloud compute instances stop my-vm-name --zone us-central1-a
sleep 120s
echo "completed"
It's work mannually but when i add this code in cron
20 17 * * * /bin/sh /home/user/Desktop/script.sh >>/home/user/cron.log
The VM doesn't stop in my GC compute engine. I made the answers to this question but, the rest does not work.
Google cloud SDK code to execute via cron
Can anyone help?
Related
Is there a GCP command for creating OR replacing a cloud run job? I'm using github-actions to create cloud run and scheduler jobs, and need to keep switching the commands between:
gcloud alpha run jobs create
and
gcloud alpha run jobs update
Is there a way to create the job and overwrite it if it already exists?
gcloud beta run jobs deploy was recently added to gcloud which does what you're looking for. Documentation is here
To create a Cloud Run new job :
gcloud beta run jobs create JOB_NAME --image IMAGE_URL OPTIONS
To update existing job :
gcloud beta run jobs update JOB_NAME
If you want a single command to handle the creation and update at the same time, you can develop your own Shell script, example :
#!/usr/bin/env bash
set -e
set -o pipefail
set -u
export JOB_NAME=my_job
res=$(gcloud beta run jobs describe $JOB_NAME --region=europe-west1 || echo "NOT_EXIST")
echo "#######Result : $res"
if [ "$res" = "NOT_EXIST" ]; then
echo "Creating your job..."
gcloud beta run jobs create $JOB_NAME
else
echo "Updating your job..."
gcloud beta run jobs update $JOB_NAME
fi
I am trying to do a backfill in Google Composer using a gcloud composer command but I am struggling to pass a -y or --yes for a corresponding --reset_dagruns argument.
The response I get is airflow: error: unrecognized arguments: -y.
Command:
gcloud composer environments run my_env --project my_project --location us-east1 backfill -- --reset_dagruns -y -s 2020-01-01 -e 2020-01-31 my_dag
How can I supply this argument?
Before answering your specific question I have a suggestion:
If your DAG (my_dag) already specifies the start & end date over which you need backfill, then just clear the status of runs you need backfilled/redone. The scheduler then picks them up again.
For your question specifically:
The AirFlow CLI documentation has unfortunately not proved directly usable with Google Cloud Composer, this is what works on Composer version 1.10.2
gcloud --project=my_project composer environments run my_env --location us-east1 backfill -- my_dag --reset_dagruns -s 2020-01-01 -e 2020-01-31
Note that this will hold on to your CLI session and will stop if you ctrl-c out, if your backfill is long you are better off defining a start date on the dag and setting catchup=True
In case anyone else has this issue, I was not able to find a way to pass in the argument using gcloud composer.
Instead I used gcloud container clusters get-credentials to do a kubectl exec to get a bash prompt on the airflow scheduler. From there I was able to use the native airflow command to move things along.
I want to test my Python library in GPU machine once a day.
I decided to use AWS EC2 for testing.
However, the fee of gpu machine is very high, so I want to stop the instance after the test ends.
Thus, I want to do the followings once a day automatically
Start EC2 instance (which is setup manually)
Execute command (test -> push logs to S3)
Stop EC2 (not remove)
How to do this?
It is very simple...
Run script on startup
To run a script automatically when the instance starts (every time it starts, not just the first time), put your script in this directory:
/var/lib/cloud/scripts/per-boot/
Stop instance when test has finished
Simply issue a shutdown command to the operating system at the end of your script:
sudo shutdown now -h
You can push script logs to custom coudwatch namespaces. Like when the process ends publish a state to cloudwatch. In cloudwatch create alarms based on the state of process, so if it has a completed state trigger an AWS lambda function that will stop instance after completion of your job.
Also if you want to start and stop on specific time you can use ec2 instance scheduler to start/stop instances. It just works like a cron job at specific intervals.
You can use the aws cli
To start an instance you would do the following
aws ec2 start-instances --instance-ids i-1234567890abcdef0
and to stop the instance you would do the following
aws ec2 stop-instances --instance-ids i-1234567890abcdef0
To execute commands inside the machine, you will need to ssh into it and run the commands that you need, then you can use the aws cli to upload files to s3
aws s3 cp test.txt s3://mybucket/test2.txt
I suggest reading the aws cli documentation, you will find most if not all what you need to automate aws commands there.
I created a shell script to start an EC2 instance -if not already running,- connect via SSH and, if you want, run a command.
https://gist.github.com/jotaelesalinas/396812f821785f76e5e36cf928777a12
You can use it in three different ways:
./ec2-start-and-ssh.sh -i <instance id> -s
will show status information about your instance: running state and private and public IP addresses.
./ec2-start-and-ssh.sh -i <instance id>
will connect and leave you inside the default shell.
./ec2-start-and-ssh.sh -i <instance id> <command>
will run whatever command you specify, e.g.:
./ec2-start-and-ssh.sh -i <instance id> ./run.sh
./ec2-start-and-ssh.sh -i <instance id> sudo poweroff
I use the last two commands to run periodic jobs minimizing billing costs.
I hope this helps!
How can I know how long has an instance been running (machine) on Google Cloud Platform?
If I run
gcloud compute instances describe <instance-name> --zone <zone>
I can know if it's running, but i want to know how long it has been running.
If you SSH into the instance you can run the command uptime -p and it will show you how long the instance has been running.
You can execute this command:
gcloud compute --project <project_id> ssh --zone <zone> <instance-name> -- command 'uptime'
Similar to uptime you can use "createdTimstamp" as something to measure against. Check out the compute engine's REST API: cloud.google.com/compute/docs/reference/rest/v1/instances/get. In python:
import googleapiclient.discovery
import google.auth
credentials, project_id = google.auth.default()
compute = googleapiclient.discovery.build('compute', 'v1', credentials=credentials)
zone = "us-central1-a"
result = compute.instances().list(project=project_id, zone=zone).execute()
for instance in result['items']:
print(instance['name'], instance['creationTimestamp'])
Example output:
my-awesome-instance 2020-10-27T03:35:45.814-07:00
The timestamp is in ISO 8601 format and can be easily parsed: How to convert a timezone aware string to datetime in Python without dateutil?
gcloud compute instances describe lists lastStartTimestamp, but another way to get it is:
gcloud compute instances list --format="table(name,lastStartTimestamp)"
Probably the best way is a combination of a few gcloud commands.
FIRST
gcloud compute intances describe <instance-name> --format="get(status)"
will return the current status: GCloud Status Info
PROVISIONING: resources are allocated for the VM. The VM is not running yet.
STAGING: resources are acquired, and the VM is preparing for first boot.
RUNNING: the VM is booting up or running.
STOPPING: the VM is being stopped. You requested a stop, or a failure occurred.
This is a temporary status after which the VM enters the TERMINATED status.
REPAIRING: the VM is being repaired. Repairing occurs when the VM encounters an internal error or the underlying machine is unavailable due to maintenance. During this time, the VM is unusable. If repair succeeds, the VM returns to one of the above states.
TERMINATED: the VM is stopped. You stopped the VM, or the VM encountered a failure. You can restart or delete the VM.
SUSPENDING: The VM is in the process of being suspended. You suspended the VM.
SUSPENDED: The VM is in a suspended state. You can resume the VM or delete it.
SECOND
IF status == 'RUNNING' THEN...
gcloud compute instances describe total-server --format="get(lastStartTimestamp)"
Which will return a UTC Timestamp in ISO 8601 Format as discussed above.
Example: 2022-05-18T02:18:26.086-07:00
From here you just need to get UTC NOW in whatever programming language you're using and subtract lastStartTimestamp to get your 'Running Time'
OPTIONAL THIRD
If status == 'TERMINATED' Then you'll do the same as above but you'll use
gcloud compute instances describe total-server --format="get(lastStopTimestamp)"
From there you'll again do the same as above and do a UTC NOW - lastStopTimestamp to get the 'Time since last start' or 'Stopped Time' or whatever you want to call it.
How can i run periodic job in background on EMR cluster?
I have script.sh with cron job and application.py in s3 and want to run cluster with this command:
aws emr create-cluster
--name "Test cluster"
–-release-label emr-5.12.0
--applications Name=Hive Name=Pig Name=Ganglia Name=Spark
--use-default-roles
--ec2-attributes KeyName=myKey
--instance-type m3.xlarge
--instance-count 3
--steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,
Jar=s3://region.elasticmapreduce/libs/script-runner/script-runner.jar,
Args["s3://mybucket/script-path/script.sh"]
Finally, i want that cron job from script.sh execute application.py
Now i don't understand how to install cron on master node, python file need some libraries, they should be installed to.
You need to SSH into the master node, and then perform the crontab setup from there, not on your local machine:
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-ssh.html
Connect to the Master Node Using SSH
Secure Shell (SSH) is a network protocol you can use to create a
secure connection to a remote computer. After you make a connection,
the terminal on your local computer behaves as if it is running on the
remote computer. Commands you issue locally run on the remote
computer, and the command output from the remote computer appears in
your terminal window.
When you use SSH with AWS, you are connecting to an EC2 instance,
which is a virtual server running in the cloud. When working with
Amazon EMR, the most common use of SSH is to connect to the EC2
instance that is acting as the master node of the cluster.
Using SSH to connect to the master node gives you the ability to
monitor and interact with the cluster. You can issue Linux commands on
the master node, run applications such as Hive and Pig interactively,
browse directories, read log files, and so on. You can also create a
tunnel in your SSH connection to view the web interfaces hosted on the
master node. For more information, see View Web Interfaces Hosted on
Amazon EMR Clusters.
By default crontab is installed in linux system, you don't need to install manually.
To add spark job scheduling in cron, please follow below steps
Login into master node (SSH into master).
run command
crontab -e
Add the below line in crontab and save it ( :w)
*/15 0 * * * /script-path/script.sh
Now cron will schedule the jobs for every 15 minutes.
Please refer this link to know about cron.
Hope this helps.
Thanks
Ravi