Can ml-engine HP optimization be run locally? - google-cloud-ml

Question

I am attempting to tune HPs for my model using the ml-engine on a local server. In my case the model trains a single pass, but no HP trials are performed. Is this a configuration issue, or is HP optimization not supported in local mode?
My local command:
gcloud ml-engine local train --package-path $PWD --module-name example.train --configuration example/hpconfig.yaml -- --param1 16 --param2 2
My config file:
trainingInput:
workerCount: 1
hyperparameters:
goal: MINIMIZE
hyperparameterMetricTag: val_loss
maxTrials: 10
maxParallelTrials: 1
enableTrialEarlyStopping: True
params:
- parameterName: param1
type: INTEGER
minValue: 4
maxValue: 128
scaleType: UNIT_LINEAR_SCALE
- parameterName: param2
type: INTEGER
minValue: 1
maxValue: 4
scaleType: UNIT_LINEAR_SCALE

Answer 1

Unfortunately, HP Tuning cannot be run in local mode. I would recommend a workflow like so:
Run locally with small data, etc. to ensure everything is working (I recommend using GCS paths).
Run a small test on cloud (single job) to ensure dependencies are correct, data files properly point to GCS instead of locally, etc.
Run an HP Tuning job.
Once 1 and 2 are working, 3 generally will, too.
Also, as a side note. Kubeflow supports Katib for running HP tuning jobs from any kubernetes deployment, including Minikube (for local development).