For instance, I want to run the following gcloud CLI command,
gcloud run services delete [SERVICE]
But, from a triggered Google Cloud Function.
I've looked in a few places and have found a few similar things,
https://www.googlecloudcommunity.com/gc/Infrastructure-Compute-Storage/Automatic-Resource-Deletion/m-p/172865
https://github.com/PolideaInternal/cats-love-money
Create a Google function from a Google cloud function
But, I find them a bit tricky to follow and/or replicate.
The Google Cloud CLI is a Python program. That means a lot of dependencies and a requirement for a shell and OS environment. Cloud Functions does not provide either.
A better option for running the CLI is Cloud Run. This provides the additional benefit of being able to test the Cloud Run container locally. You will need to wrap the CLI with an HTTP server responding to HTTP requests which then execute the CLI.
However, most CLI commands can be easily duplicated with the Python SDKs and/or direct REST API calls which are supported by Cloud Functions. This will require a solid understanding of the services, APIs, and language.
Related
In Azure for example, I created a few bash scripts give me things like average daily CPU utilization over whatever time period I want for any/all VMs using their command line tool.
I can't seem to figure out how to do this in Google cloud except by manually using the console (automatically generated daily usage reports don't seem to give me any CPU info either), so far numerous searches have told me that using the monitoring function in the google cloud console is basically the only way I can do this, as the cli "gcloud" will only report quotas back which isn't really what I'm after here. I haven't bothered with the ops agent install yet, as my understanding is that this is just for adding additional metrics (to the console) and not functionality to the google cloud cli. Up to this point I've only ever managed Azure and some AWS, so maybe what I'm trying to do isn't even possible in Google cloud?
Monitoring (formerly Stackdriver) does seem to be neglected by the CLI (gcloud).
There is a gcloud monitoring "group" but even the gcloud alpha monitoring and gcloud beta monitoring commands are limited (and don't include e.g. metrics).
That said, gcloud implements Google's underlying (service) APIs and, for those (increasingly fewer) cases where the CLI does not yet implement an API and its methods, you can use APIs Explorer to find the underlying e.g. Monitoring service directly.
Metrics can be access through a query over the underlying time-series data, e.g. projects.timeseries.query. The interface provides a form that enables you to invoke service methods from the browser too.
You could then use e.g. curl to construct the queries you need for your bash scripts and other tools (e.g. jq) to post-process the data.
Alternatively, and if you want a more programmatic experience with good error-handling and control over the output formatting, you can use any of the language-specific SDKs (client libraries).
I'd be surprised if someone hasn't written a CLI tool to complement gcloud for monitoring as it's a reasonable need.
It may be worth submitting a feature request on Google's Issue Tracker. I'm unsure whether it would best be placed under Cloud CLI or Monitoring. Perhaps try Monitoring.
I'd like to call a Cloud Run app from inside a Cloud Function multiple times, given some logic. I've googled this quite a lot and don't find good solutions. Is this supported?
I've seen the Workflows Tutorials, but AFAIK they are meant to pass messages in series between different GPC services. My Cloud Function runs on a schedule every minute and it would only need to call the Cloud Run app a few times per day given some event. I've thought about having the entire app run in Cloud Run instead of the Cloud function. However, I think having it all in Cloud Run would be more expensive than running the Cloud function.
I went through your question, I have an alternative in my mind if you agree to the solution. You can use Cloud Scheduler to securely trigger a Cloud Run service asynchronously on a schedule.
You need to create a service account to associate with Cloud
Scheduler, and give that service account the permission to invoke
your Cloud Run service, i.e. Cloud Run invoker (You can use an
existing service account to represent Cloud Scheduler, or you can
create a new one for that matter)
Next, you have to create a Cloud Scheduler job that invokes your
service at specified times. Specify the frequency, or job interval,
at which the job is to run, using a configuration string. Specify the
fully qualified URL of your Cloud Run service, for example
https://myservice-abcdef-uc.a.run.app The job will send requests to
this URL.
Next, specify the HTTP method: the method must match what your
previously deployed Cloud Run service is expecting. When you deploy
the service using Cloud Scheduler, make sure you do not allow
unauthenticated invocations. Please go through this
documentation for details and try to implement the steps.
Back to your question, yes it's possible to call your Cloud Run service from inside Cloud Functions. Here, your Cloud Run service calls from another backend service i.e. Cloud Functions directly( synchronously) over HTTP, using its endpoint URL. For this use case, you should make sure that each service is only able to make requests to specific services.
Go through this documentation suggested by #John Hanley as it provides you with the steps you need to follow.
Recently I started working with GCP workflows, and functions. We are using serverless framework for the functions and we can run them in our on computers with the command serverless invoke local --function <function_name> so we don't have to spend cloud executions.
What I'm looking now is if there is a way to do the same thing with GCP workflows, to run them in our own computers instead of invoking them inside the cloud.
I already read the resources from google and from many different articles but I still not find the trick (if it actually exists)
Today, there is no emulator for Cloud Workflows. But if you can afford to deploy your cloud functions on GCP, Cloud workflows has a generous free tier: 5000 steps for free
I'm working on a Cloud Run docker application that handles a few long-running data integration processes.
I'm struggling to come up with a way to locally run/test my submissions to Cloud Tasks before actually deploying the container to Cloud Run.
Is there any way to do this?
A local emulator for Cloud Tasks is not available yet, in some cases you can substitute Cloud Tasks with Pub/Sub.
Also, consider to use non Google solutions such as Cloud-Tasks-In-Process-Emulator, gcloud-tasks-emulator 0.5.1 or Cloud tasks emulator.
As I can understand you want to test the cloud task locally! Yes it is possible by using ngrok. By using ngrok you can access your local application on public and for cloud task you need the public url for handling task.
I am trying to build an app where the user is able to upload a file to cloud storage. This would then trigger a model training process (and predicting later on). Initially I though I could do this with cloud functions/pubsub and cloudml, but it seems that cloud functions are not able to trigger gsutil commands which is needed for cloudml.
Is my only option to enable cloud-composer and attach GPUs to a kubernetes node and create a cloud function that triggers a dag to boot up a pod on the node with GPUs and mounting the bucket with the data? Seems a bit excessive but I can't think of another way currently.
You're correct. As for now, there's no possibility to execute gsutil command from a Google Cloud Function:
Cloud Functions can be written in Node.js, Python, Go, and Java, and are executed in language-specific runtimes.
I really like your second approach with triggering the DAG.
Another idea that comes to my mind is to interact with GCP Virtual Machines within Cloud Composer through the Python operator by using the Compute Engine Pyhton API. You can find more information in automating infrastructure and taking a deep technical dive into the core features of Cloud Composer here.
Another solution that you can think of is Kubeflow, which aims to make running ML workloads on Kubernetes. Kubeflow adds some resources to your cluster to assist with a variety of tasks, including training and serving models and running Jupyter Notebooks. Please, have a look on Codelabs tutorial.
I hope you find the above pieces of information useful.