To submit a training job from gcloud ai-platform (ex gcloud ml-engine) you use the following command from the gcloud SDK:
gcloud ai-platform jobs submit COMMAND [GCLOUD_WIDE_FLAG …]
I want to do this programmatically, i.e. from python code (or any other language). Something like
import gcloud-ai-platform as gap
gap.submit_job(COMMAND)
Is there such a command? And if it does not exist, how can I build a workaround? (using gcloud sdk programmatically)
I found the docs a bit confusing with regards to custom images. Trick is to specify --master-image-uri via masterConfig/imageUri:
training_inputs = {
'scaleTier': 'BASIC',
'packageUris': [
],
'masterConfig': {
'imageUri': settings["AI_SERVER_URI"]
},
'args': [
"java", "-cp", "MY.jar:jars/*", "io.manycore.Test",
"jar positional argument"
],
'region': 'us-central1',
'pythonVersion': '3.7',
'scheduling': {
'maxRunningTime': '3600s'
},
}
job_spec = {'jobId': jobid, 'trainingInput': training_inputs}
project_name = settings["PROJECT_ID"]
project_id = 'projects/{}'.format(project_name)
cloudml = discovery.build('ml', 'v1', credentials=self.credentials)
request = cloudml.projects().jobs().create(body=job_spec, parent=project_id)
For submitting a training job, here you have an example that you can follow.
It has both methods, using gcloud and the equivalent python code.
Related
I am looking for the best pattern to be able to execute and export a BigQuery query result to a cloud storage bucket. I would like this to be executed when the BigQuery table is written to or modified.
I think I would traditionally setup a pubsub topic that would be written to when the table is modified, which would trigger a GCP function that is responsible for executing the query and writing the result to a GCP bucket. I just am not too confident that there isn't a better approach (more straight forward) to do this in GCP.
Thanks in advance.
I propose you an approach based on Eventarc.
The goal is to launch a Cloud Function or Cloud Run action when the data is inserted or updated in a BigQuery table, example with Cloud Run :
SERVICE=bq-cloud-run
PROJECT=$(gcloud config get-value project)
CONTAINER="gcr.io/${PROJECT}/${SERVICE}"
gcloud builds submit --tag ${CONTAINER}
gcloud run deploy ${SERVICE} --image $CONTAINER --platform managed
gcloud eventarc triggers create ${SERVICE}-trigger \
--location ${REGION} --service-account ${SVC_ACCOUNT} \
--destination-run-service ${SERVICE} \
--event-filters type=google.cloud.audit.log.v1.written \
--event-filters methodName=google.cloud.bigquery.v2.JobService.InsertJob \
--event-filters serviceName=bigquery.googleapis.com
When a BigQuery job was executed, the Cloud Run action will be triggered.
Example of Cloud Run action :
#app.route('/', methods=['POST'])
def index():
# Gets the Payload data from the Audit Log
content = request.json
try:
ds = content['resource']['labels']['dataset_id']
proj = content['resource']['labels']['project_id']
tbl = content['protoPayload']['resourceName']
rows = int(content['protoPayload']['metadata']
['tableDataChange']['insertedRowsCount'])
if ds == 'cloud_run_tmp' and \
tbl.endswith('tables/cloud_run_trigger') and rows > 0:
query = create_agg()
return "table created", 200
except:
# if these fields are not in the JSON, ignore
pass
return "ok", 200
You can apply logic based on the current dataset, table or other elements existing in the current payload.
For a DataFlow Job, I need to extract Job_ID from JOB_NAME. I have the below command and the corresponding o/p. Can you please guide how to extract JOB_ID from the below response
$ gcloud dataflow jobs list --region=us-central1 --status=active --filter="name=sample-job"
JOB_ID NAME TYPE CREATION_TIME STATE REGION
2020-10-07_10_11_20-15879763245819496196 sample-job Streaming 2020-10-07 17:11:21 Running us-central1
If we can use Python script to achieve it, even that will be fine.
gcloud dataflow jobs list --region=us-central1 --status=active --filter="name=sample-job" --format="value(JOB_ID)"
You can use standard command line tools to parse the response of that command, for example
gcloud dataflow jobs list --region=us-central1 --status=active --filter="name=sample-job" | tail -n 1 | cut -f 1 -d " "
Alternatively, if this is from a Python program already, you can use the Dataflow API directly instead of using the gcloud tool, like in How to list down all the dataflow jobs using python API
With python, you can retrieve the jobs' list with a REST request to the Dataflow's method https://dataflow.googleapis.com/v1b3/projects/{projectId}/jobs
Then, the json response can be parsed to filter the job name you are searching for by using a if clause:
if job["name"] == 'sample-job'
I tested this approached and it worked:
import requests
import json
base_url = 'https://dataflow.googleapis.com/v1b3/projects/'
project_id = '<MY_PROJECT_ID>'
location = '<REGION>'
response = requests.get(f'{base_url}{project_id}/locations/{location}/jobs', headers = {'Authorization':'Bearer <BEARER_TOKEN_HERE>'})
# <BEARER_TOKEN_HERE> can be retrieved with 'gcloud auth print-access-token' obtained with an account that has access to Dataflow jobs.
# Another authentication mechanism can be found in the link provided by danielm
jobslist = response.json()
for key,jobs in jobslist.items():
for job in jobs:
if job["name"] == 'beamapp-0907191546-413196':
print(job["name"]," Found, job ID:",job["id"])
else:
print(job["name"]," Not matched")
# Output:
# windowedwordcount-0908012420-bd342f98 Not matched
# beamapp-0907200305-106040 Not matched
# beamapp-0907192915-394932 Not matched
# beamapp-0907191546-413196 Found, job ID: 2020-09-07...154989572
Created my GIST with Python script to achieve it.
I currently have a docker container on the gcloud container repository which I want to launch an instance of in order to run a calculation, save a result and close:
gcloud compute instances create-with-container test-1 \
--machine-type=n1-standard-4 \
--boot-disk-size=20GB \
--container-image=eu.gcr.io/<container-link> \
--container-env=GCLOUD_INPUT_FILENAME=file.txt \
--container-env=GCLOUD_PROJECT=project-name
However, I want to be able to launch these instances using a web-interface (flask) which implies I want to use the googleapiclient (python) in order to create and manage these instances:
It looks like while you can create a instance creation order using the discovery api:
compute = googleclientapi.discovery.build('compute', 'v1')
compute.instances().insert(...).execute()
but it doesn't look like it is possible to emulate create-with-container gcloud sdk command, although you can pass 'machineImage' as part of the creation request.
Can one create a compute instance 'with-container' without using subprocess to call the gcloud sdk
OR
Can I convert my create-with-container instance into a machine image and then use the googleapi client?
Just for FYI, what I found from the log-http logging(I know it is not the right answer)
==== request start ====
uri: https://compute.googleapis.com/batch/compute/v1
method: POST
== headers start ==
b'authorization': --- Token Redacted ---
b'content-length': b'479'
b'content-type': b'multipart/mixed; boundary="===============34234234234234=="'
b'user-agent': b'google-cloud-sdk gcloud/329.0.0 command/gcloud.compute.instances.create-with-container invocation-id/0dd6a37ac0624ac4b00e30a44da environment/devshell environment-version/None interactive/False from-script/False python/3.7.3 term/screen (Linux 5.4.89+)'
== headers end ==
"create-with-container" is passed through the header, I did not find the body for the same.
The code would benefit from refinement but here's a solution:
import os
from google.cloud import compute_v1
from google.cloud.compute_v1 import types
project=os.getenv("PROJECT")
number=os.getenv("NUMBER")
zone=os.getenv("ZONE")
machine_type=os.getenv("TYPE")
name=os.getenv("NAME")
# Compute Engine default
email=f"{number}-compute#developer.gserviceaccount.com"
# Define a non-trivial container with command and args
value="""
spec:
containers:
- name: foo
image: gcr.io/google-containers/busybox#sha256:d8d3bc2c183ed2f9f10e7258f84971202325ee6011ba137112e01e30f206de67
command:
- /bin/sh
args:
- -c
- |
while true
do
echo "Hello"
sleep 15
done
stdin: true
tty: true
restartPolicy: Always
"""
client = compute_v1.InstancesClient()
request = types.InsertInstanceRequest(
project=project,
zone=zone,
instance_resource=types.Instance(
name=name,
machine_type=f"zones/{zone}/machineTypes/{machine_type}",
disks=[
types.AttachedDisk(
device_name="foo",
boot=True,
auto_delete=True,
initialize_params=types.AttachedDiskInitializeParams(
disk_size_gb=10,
disk_type=f"zones/{zone}/diskTypes/pd-balanced",
source_image="projects/cos-cloud/global/images/cos-stable-101-17162-40-13",
),
),
],
network_interfaces=[
types.NetworkInterface(
access_configs=[
types.AccessConfig(
name="External NAT",
network_tier="STANDARD",
),
],
),
],
metadata=types.Metadata(
items=[{
"key":"gce-container-declaration",
"value": value,
}],
),
service_accounts=[
types.ServiceAccount(
email=email,
scopes=[
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring.write",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/trace.append"
],
),
],
),
)
response=client.insert(
request=request,
)
Is there such an option? My use case would be running a trigger for a production build (deploys to production). Ideally, that trigger doesn't need to listen to any change since it is invoked manually via chatbot.
I saw this video CI/CD for Hybrid and Multi-Cloud Customers (Cloud Next '18) announcing there's an API trigger support, I'm not sure if that's what I need.
I did same thing few days ago.
You can submit your builds using gcloud and rest api
gcloud:
gcloud builds submit --no-source --config=cloudbuild.yaml --async --format=json
Rest API:
Send you cloudbuild.yaml as JSON with Auth Token to this url https://cloudbuild.googleapis.com/v1/projects/standf-188123/builds?alt=json
example cloudbuild.yaml:
steps:
- name: 'gcr.io/cloud-builders/docker'
id: Docker Version
args: ["version"]
- name: 'alpine'
id: Hello Cloud Build
args: ["echo", "Hello Cloud Build"]
example rest_json_body:
{"steps": [{"args": ["version"], "id": "Docker Version", "name": "gcr.io/cloud-builders/docker"}, {"args": ["echo", "Hello Cloud Build"], "id": "Hello Cloud Build", "name": "alpine"}]}
This now seems to be possible via API:
https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.triggers/run
request.json:
{
"projectId": "*****",
"commitSha": "************"
}
curl request (with using a gcloud command):
PROJECT_ID="********" TRIGGER_ID="*******************"; curl -X POST -T request.json -H "Authorization: Bearer $(gcloud config config-helper \
--format='value(credential.access_token)')" \
https://cloudbuild.googleapis.com/v1/projects/"$PROJECT_ID"/triggers/"$TRIGGER_ID":run
You can use google client api to create build jobs with python:
import operator
from functools import reduce
from typing import Dict, List, Union
from google.oauth2 import service_account
from googleapiclient import discovery
class GcloudService():
def __init__(self, service_token_path, project_id: Union[str, None]):
self.project_id = project_id
self.service_token_path = service_token_path
self.credentials = service_account.Credentials.from_service_account_file(self.service_token_path)
class CloudBuildApiService(GcloudService):
def __init__(self, *args, **kwargs):
super(CloudBuildApiService, self).__init__(*args, **kwargs)
scoped_credentials = self.credentials.with_scopes(['https://www.googleapis.com/auth/cloud-platform'])
self.service = discovery.build('cloudbuild', 'v1', credentials=scoped_credentials, cache_discovery=False)
def get(self, build_id: str) -> Dict:
return self.service.projects().builds().get(projectId=self.project_id, id=build_id).execute()
def create(self, image_name: str, gcs_name: str, gcs_path: str, env: Dict = None):
args: List[str] = self._get_env(env) if env else []
opt_params: List[str] = [
'-t', f'gcr.io/{self.project_id}/{image_name}',
'-f', f'./{image_name}/Dockerfile',
f'./{image_name}'
]
build_cmd: List[str] = ['build'] + args + opt_params
body = {
"projectId": self.project_id,
"source": {
'storageSource': {
'bucket': gcs_name,
'object': gcs_path,
}
},
"steps": [
{
"name": "gcr.io/cloud-builders/docker",
"args": build_cmd,
},
],
"images": [
[
f'gcr.io/{self.project_id}/{image_name}'
]
],
}
return self.service.projects().builds().create(projectId=self.project_id, body=body).execute()
def _get_env(self, env: Dict) -> List[str]:
env: List[str] = [['--build-arg', f'{key}={value}'] for key, value in env.items()]
# Flatten array
return reduce(operator.iconcat, env, [])
Here is the documentation so that you can implement more functionality: https://cloud.google.com/cloud-build/docs/api
Hope this helps.
If you just want to create a function that you can invoke directly, you have two choices:
An HTTP trigger with a standard API endpoint
A pubsub trigger that you invoke by sending a message to a pubsub topic
The first is the more common approach, as you are effectively creating a web API that any client can call with an HTTP library of their choice.
You should be able to manually trigger a build using curl and a json payload.
For details see: https://cloud.google.com/cloud-build/docs/running-builds/start-build-manually#running_builds.
Given that, you could write a Python Cloud function to replicate the curl call via the requests module.
I was in search of the same thing (Fall 2022) and while I haven't tested yet I wanted to answer before I forget. It appears to be available now in gcloud beta builds triggers run TRIGGER
you can trigger a function via
gcloud functions call NAME --data 'THING'
inside your function you can do pretty much anything possibile within Googles Public API's
if you just want to directly trigger Google Cloud Builder from git then its probably advisable to use Release version tags - so your chatbot might add a release tag to your release branch in git at which point cloud-builder will start the build.
more info here https://cloud.google.com/cloud-build/docs/running-builds/automate-builds
I have tried the describe command listed here and I don't see the parameters. Is there another command that I should use to get this information, or some other API that would provide it?
TL;DR - You're missing the --full argument to the gcloud dataflow jobs describe command.
FLAGS
--full
Retrieve the full Job rather than the summary view
View full job info
If you're using gcloud to view the information about the GCP Dataflow job, this command will show the full info (which is actually quite a lot of info) about the job including any parameters that were passed to the job:
gcloud dataflow jobs describe JOB_ID --full
All the options are under the hierarchy environment.sdkPipelineOptions.options
View all options as JSON
To view all the options passed to the job (which prints actually more than just the command line arguments BTW) as a JSON, you can do:
$ gcloud dataflow jobs describe JOB_ID --full --format='json(environment.sdkPipelineOptions.options)'
{
"environment": {
"sdkPipelineOptions": {
"options": {
"apiRootUrl": "https://dataflow.googleapis.com/",
"appName": "WordCount",
"credentialFactoryClass": "com.google.cloud.dataflow.sdk.util.GcpCredentialFactory",
"dataflowEndpoint": "",
"enableCloudDebugger": false,
"enableProfilingAgent": false,
"firstArg": "foo",
"inputFile": "gs://dataflow-samples/shakespeare/kinglear.txt",
"jobName": "wordcount-tuxdude-12345678",
"numberOfWorkerHarnessThreads": 0,
"output": "gs://BUCKET_NAME/dataflow/output",
"pathValidatorClass": "com.google.cloud.dataflow.sdk.util.DataflowPathValidator",
"project": "PROJECT_NAME",
"runner": "com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner",
"secondArg": "bar",
"stableUniqueNames": "WARNING",
"stagerClass": "com.google.cloud.dataflow.sdk.util.GcsStager",
"stagingLocation": "gs://BUCKET_NAME/dataflow/staging/",
"streaming": false,
"tempLocation": "gs://BUCKET_NAME/dataflow/staging/"
}
}
}
}
View all options as a table
$ gcloud dataflow jobs describe JOB_ID --full --format='flattened(environment.sdkPipelineOptions.options)'
environment.sdkPipelineOptions.options.apiRootUrl: https://dataflow.googleapis.com/
environment.sdkPipelineOptions.options.appName: WordCount
environment.sdkPipelineOptions.options.credentialFactoryClass: com.google.cloud.dataflow.sdk.util.GcpCredentialFactory
environment.sdkPipelineOptions.options.dataflowEndpoint:
environment.sdkPipelineOptions.options.enableCloudDebugger: False
environment.sdkPipelineOptions.options.enableProfilingAgent: False
environment.sdkPipelineOptions.options.firstArg: foo
environment.sdkPipelineOptions.options.inputFile: gs://dataflow-samples/shakespeare/kinglear.txt
environment.sdkPipelineOptions.options.jobName: wordcount-tuxdude-12345678
environment.sdkPipelineOptions.options.numberOfWorkerHarnessThreads: 0
environment.sdkPipelineOptions.options.output: gs://BUCKET_NAME/dataflow/output
environment.sdkPipelineOptions.options.pathValidatorClass: com.google.cloud.dataflow.sdk.util.DataflowPathValidator
environment.sdkPipelineOptions.options.project: PROJECT_NAME
environment.sdkPipelineOptions.options.runner: com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner
environment.sdkPipelineOptions.options.secondArg: bar
environment.sdkPipelineOptions.options.stableUniqueNames: WARNING
environment.sdkPipelineOptions.options.stagerClass: com.google.cloud.dataflow.sdk.util.GcsStager
environment.sdkPipelineOptions.options.stagingLocation: gs://BUCKET_NAME/dataflow/staging/
environment.sdkPipelineOptions.options.streaming: False
environment.sdkPipelineOptions.options.tempLocation: gs://BUCKET_NAME/dataflow/staging/
Get the value of just a single option
To get the value of just a single option named --argName (whose value BTW is MY_ARG_VALUE), you can do:
$ gcloud dataflow jobs describe JOB_ID --full --format='value(environment.sdkPipelineOptions.options.argName)'
MY_ARG_VALUE
gcloud formatting
gcloud in general supports a wide range of formatting options in the output which is applicable to most gcloud commands which pull info from the server. You can read about them here.