I am reading this https://cloud.google.com/scheduler/docs/tut-pub-sub
They use the setup like below:
Cloud Scheduler -> PubSub -> Cloud Function-> external Service
and If I have a cron job for calling a service once a day, should I still need this pubsub in between?
I know there is an option for HTTP target type in Cloud Scheduler and I think the below setup without PubSub is good enough.
Cloud Scheduler -> Cloud Function-> external Service
Could you give some advice why I should/should not have the PubSub?
The example that you are looking at is Using Pub/Sub to trigger a Cloud Function so it'll include examples with Pub/Sub there. Instead you can deploy a HTTP Cloud function and use it's URL as the target URL as in below screenshot:
Here, Cloud Scheduler will trigger the function without Pub/Sub.
Related
I have a trigger on Eventarc that is supposed to run after each Cloud Scheduler invocation, which is google.cloud.scheduler.v1beta1.CloudScheduler.RunJob
However, it is not being triggered anyhow!
Other triggers, like force run, are working.
I want to trigger a Cloud Run after a Job execution. Is it possible or I am facing a bug?
If you are expecting your Cloud Run service to be executed at each scheduled invocation of Cloud Scheduler, it isn't possible to do so through Eventarc and Cloud Audit logs.
This is due to Cloud Scheduler not being in the list of services that write audit logs. Adding to that, the RunJob event you are filtering by will only get written if you manually execute a job (using the API), and not by your set CRON schedule.
A manual job run did trigger Eventarc when I tested this scenario, but I had to set my trigger as global.
If you would like to execute the Cloud Run service on a schedule, you can do that by having Cloud Scheduler send a request to the service URL directly. Another alternative is to instead of having Eventarc listen to Audit logs, have it listen to messages on a Pub/Sub topic, which will be sent by Cloud Scheduler. Let me know if this was helpful.
I'd like to find out what's the best architecture for a python app that gets triggered when a file is uploaded to Google Cloud Storage, does some processing and outputs a file to Google Drive?
I've tried using Cloud Functions but I'm getting a Function invocation was interrupted. Error: memory limit exceeded. in the logs.
I've also followed this tutorial Trigger Cloud Run with events from Eventarc so I know that one way is with EventArc and Cloud Audit logs.
2 questions:
What other methods are there since I require higher memory limits?
How do I get the bucket name and file name from cloud audit logs? through protoPayload.resourceName?
You can use PubSub. You can create a PubSub notification and create a push subscription to the service that you want.
Http cloud function
App Engine
Cloud Run
Any HTTP service running somewhere (VM, Kubernetes, on prem,...)
EventArc is mainly a wrapper of this process and can call only Cloud Run service (for now)
Is there a solution/service available on GCP in similar lines of Systems Manager?
My end goal is to run a shell script on GCP VM on specific events.
Like for AWS, via EventBridge I was able to trigger a Lambda Function and the function in turn triggered a SSM command for specific VM.
Is this possible on GCP?
There isn't a Systems Manager equivalent in GCP.
A Pub/Sub subscription from the VMs/compute units which triggers a lambda function (cloud function in GCP) is a suboptimal solution and different from what Systems Manager accomplishes..
I don't know what kind of events you have in mind that would trigger running a script but you can check out the tutorial how to run a function using pub/sub. It shows how to use scheduler based events but it's possible to use not-scheduled triggers;
Events are things that happen within your cloud environment that you might want to take action on. These might be changes to data in a database, files added to a storage system, or a new virtual machine instance being created. Currently, Cloud Functions supports events from the following providers:
HTTP
Cloud Storage
Cloud Pub/Sub
Cloud Firestore
Firebase (Realtime Database, Storage, Analytics, Auth)
Stackdriver Logging—forward log entries to a Pub/Sub topic by creating a sink. You can then trigger the function.
And here you can read on how to implement those triggers.
For example this documentation explains how to use storage based triggers in pub/sub.
If you provide more details of what exactly you want to achieve (what events have to trigger what) then I can point you to a more direct solution.
The approach depends on the exact use case you have in hand. One of the common architecture option could be using pub/sub with cloud functions. Based on messages published to Pub/Sub topics, cloud functions performing operations of our interest can be triggered/ invoked in the same cloud project as the function.
I have a database in Google Datastore. I don't know how to use cloud functions, but i want to trigger an event after a creation or an update.
Unfortunately the documentation is light on the subject : https://cloud.google.com/appengine/docs/standard/java/datastore/callbacks
I don't know how i could use #PostPut to trigger an event as soon as a line is created or updated.
Does anyone have a tutorial which a basic example ?
thank you
Dan MacGrath provided an answer to a similar request (callbacks are indeed discussed below. Such solution doesn't exist yet. As a workaround, taking into account the current available triggers:
HTTP—invoke functions directly via HTTP requests.
Cloud Storage
Cloud Pub/Sub
Firebase (DB, Storage, Analytics, Auth)
Stackdriver Logging—forward log entries to a Pub/Sub topic by creating a sink. You can then trigger the function.
I would suggest a couple of solutions:
Saving something in a specific bucket from Cloud Storage every time that a line is created or updated to trigger a linked Cloud Function. You can delete the bucket contents afterwards.
Create logs with the same name and then forward them to Pub/Sub, by creating a sink.
EDIT 1
Cloud Storage triggers for Cloud Functions: Official Google doc and tutorial with a sample code in node.js 6 in Github.
Cloud Pub/Sub triggers for Cloud Functions: Official Google doc and tutorial with a sample code in node.js 6 in Github (the same than before).
Cloud Datastore does not support real-time triggers on CRUD (Create, Read, Update, Delete) events.
However, you can migrate to Cloud Firestore which does support real-time triggers for those actions (by way of Cloud Pub/Sub which can be made to invoke a Cloud Function). Cloud Firestore is the successor to Cloud Datastore and may eventually supplant it at some point in future.
I am trying to trigger a google cloud function from stackdriver log, say if my VM goes down I need to perform some operation using function. Similar to how we do in AWS with cloudwatch events triggering lambda.
It's not possible to directly invoke a function from a log, but you can piece it together in two parts:
Export the logs you care about to Cloud Pub/Sub
https://cloud.google.com/logging/docs/export/
Subscribe a Cloud Function to the topic
https://cloud.google.com/functions/docs/calling/pubsub
and https://cloud.google.com/functions/docs/tutorials/pubsub
You'll want to configure StackDriver to call a webhook, which you then implement as a HTTP-triggered Cloud Function.