Control GPU machine to start and stop from one function? - google-cloud-platform

Question

Thanks to Google cloud we get free credits for running GPU on cloud, but we getting stuck at a very beginning.
We use to get images daily for processing through machine learning model, but somehow GPU System are not getting used through out the day is there any way we can control this system to start and stop once all the images are processed through one function? Which we can call through cron at specific day and timing.
I have heard about aws lambda but am not sure of what google cloud can provide for this problem.
Thanks in Advance.

Answer 1

If you are willing to spend effort, you can achieve this using Google Kubernetes Engine. As far is I know this is the only way right now to have self-starting and stopping GPU instances on GCP. To achieve this, you have to add a GPU node pool with auto-scaling to your Kubernetes cluster.
gcloud container node-pools create gpu_pool \
--cluster=${GKE_CLUSTER_NAME} \
--machine-type=n1-highmem-96 \
--accelerator=nvidia-tesla-v100,8 \
--node-taints=reserved-pool=true:NoSchedule \
--enable-autoscaling \
--min-nodes=0 \
--max-nodes=4 \
--zone=${GCP_ZONE} \
--project=${PROJECT_ID}
Make sure to sub the env variables with your actual project ID etc. and also make sure to use a GCP zone that actually has the GPU types available that you want (not all zones have all GPUs types). Make sure to specifiy the zone like europe-west1-b not europe-west1.
This command will start all the nodes at once, but they will be automatically shut down after whatever the default timeout for autoscaling nodes is in your default cluster configuration (for me I think it was 5minutes). However, you can change that setting.
You can then start a Kubernetes Job (NOT deployment) from CLI or using any of the available Kubernetes API Client libraries which explicitly request GPU resources.
Here is some example job.yaml with the main necessary components, however, you would need to tweak that according to your cluster config:
apiVersion: batch/v1
kind: Job
metadata:
name: some-job
spec:
parallelism: 1
template:
metadata:
name: some-job
labels:
app: some-app
spec:
containers:
- name: some-image
image: gcr.io/<project-id>/some-image:latest
resources:
limits:
cpu: 3500m
nvidia.com/gpu: 1
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- some-app
topologyKey: "kubernetes.io/hostname"
tolerations:
- key: reserved-pool
operator: Equal
value: "true"
effect: NoSchedule
- key: nvidia.com/gpu
operator: Equal
value: "present"
effect: NoSchedule
restartPolicy: OnFailure
It is vital that the tolerations are set up like this and that the actual resource limit is set to how many GPUs you want. Otherwise it won't work.
The nodes will then be started (if none are available) and the job will be computed. Idle nodes will once again be shutdown after the specified autoscaling timeout.
I got the idea from here.

Answer 2

You can use Cloud Scheduler for this use cases or you can trigger Cloud Function when images are available and process it.
However, the free $300 quota is for training and innovation purpose not for actual production application.

Answer 3

You can try and optimize the GPU usage of the instances by following the guide over here, however, you would need to manage it through a cron or something in the instance.
Also, watch out for your credit usage when using GPU on free trial. Free trial gives you only $300 USD in credits, however, as seen over here, GPU usage is expensive and you may spend all your credits in 1 or 2 weeks if you are not careful.
Hope you find this useful!