I use Cloud pub/sub and Cloud Functions.
Now, I want to publish a message to a topic which will trigger a background cloud function.
But I want to trigger my cloud function after a specific duration, like 30 seconds later.
How can I do this?
update:
Here is my architecture, is it correct?
Now, I want to publish a message to a topic which will trigger a
background cloud function. But I want to trigger my cloud function after a specific duration, like 30 seconds later.
If you setup PubSub to trigger Cloud Functions on publish events, Cloud Functions will be triggered almost immediately. There is no method to insert a delay.
You will need to implement your code as several major steps:
Setup PubSub Topic and Subscriptions. Do not trigger Cloud Functions on new messages. Messages will just sit waiting for delivery. Send messages to this topic.
Create a Cloud Function that processes PubSub subscriptions. Pull messages and process.
Use another service such as Cloud Tasks, Cloud Scheduler or App Engine Tasks to trigger your Cloud function after your desired delay.
You can use Cloud Tasks to schedule some work to happen on a delay.
Related
Currently, I create a service in Cloud Run to retrain ML models. The service will do the retrain process when there is an event from BigQuery called google.cloud.bigquery.v2.JobService.InsertJob. I use the EventArc in GCP to trigger the Retrain Service when that event happened. But, there is a problem. The trigger request to the service multiple times in one event. So, sometimes when the retraining process is done, the trigger requests the service again, and then the retraining process is active again. Is there something that I missed? Picture bellow is my EventArc setup.
As we can see in this picture that there are other requests while the first request is in process.
Eventarc is backed on PubSub. By default, and if you don't hack the default Eventarc configuration, the delivery timeout is set to 10s. (you can update manually the pubsub subscription created by eventarc. The Eventarc engineering team is aware of that not customizable parameter)
That's why, you should have a retry every 10s.
You have 2 solutions to that:
Either create an async process. I mean receive the PubSub message (tbe eventarc event), ack it immediately, and, in background, run your retrain
Or (not my preferred way), update the eventarc pubsub subscription and set the message retention duration to 5 seconds.
I'm writing a gcp cloud function and is there a feature to handle a batch of messages put on a pub/sub topic. I mean a single run of the cloud function can handle around 10-30 messages put on the queue. From the examples I have seen the cloud function gets invoked for each message. But in AWS I have seen the option where you can batch multiple messages into one Lambda.
With the traditional method Cloud functions + PubSub receiving messages via push, you won't be able to work with batches since every event will trigger the function.
You could perhaps create a different mechanism (trigger) for example a Cloud Scheduler to trigger the cloud function and pull all messages in the queue (pull mechanism): https://cloud.google.com/pubsub/docs/pull
I have a Google Cloud Storage Trigger set up on a Cloud Function with max instances of 5, to fire on the google.storage.object.finalize event of a Cloud Storage Bucket. The docs state that these events are "based on" the Cloud Pub/Sub.
Does anyone know:
Is there any way to see configuration of the topic or subscription in the console, or through the CLI?
Is there any way to get the queue depth (or equivalent?)
Is there any way to clear events?
No, No and No. When you plug Cloud Functions to Cloud Storage event, all the stuff are handle behind the scene by Google and you see nothing and you can't interact with anything.
However, you can change the notification mechanism. Instead of plugin directly your Cloud Functions on Cloud Storage Event, plug a PubSub on your Cloud Storage event.
From there, you have access to YOUR pubsub. Monitor the queue, purge it, create the subscription that you want,...
The recomended way to work with storage notifications is using Pubsub.
Legacy storage notifications still work, but with pubsub you can "peek" into the pubsub message queue and clear it if you need it.
Also, you can process pubsub events with cloud run - which is easier to develop and test (just web service), easier to deploy (just a container) and it can process several requests in parallel without having to pay more (great when you have a lot of requests together).
Where does pubsub storage notifications go?
You can see where gcloud notifications go with the gsutil command:
% gsutil notification list gs://__bucket_name__
projects/_/buckets/__bucket_name__/notificationConfigs/1
Cloud Pub/Sub topic: projects/__project_name__/topics/__topic_name__
Filters:
Event Types: OBJECT_FINALIZE
Is there any way to get the queue depth (or equivalent?)
In pubsub you can have many subsciptions to topics.
If there is no subsciption, messages get lost.
To send data to a cloud function or cloud run you setup a push subscription.
In my experience, you won't be able to see what happened because it faster that you can click: you'll find this empty 99.9999% of the time.
You can check the "queue" depht in the console (pubsub -> choose you topics -> choose the subscription).
If you need to troubleshoot this, set up a second subscription with a time to live low enough that it does not use a lot of space (you'll be billed for it).
Is there any way to clear events?
You can empty the messages from the pubsub subscription, but...
... if you're using a push notification agains a cloud function it will much faster than you can "click".
If you need it, it is on the web console (opent the pubsub subscription and click in the vertical "..." on the top right).
I have 2 cloud functions that run every 5 minutes, currently using two different Cloud Scheduler Jobs, is it possible to configure Cloud Scheduler to run them both at the same time using only 1 job instead of 2.
You have several options. The 2 easiest are:
With Cloud Scheduler publish a message in PubSub instead of calling a Cloud Function. Then add 2 push subscription to PubSub to call your Cloud Functions. The message in entry in the topic is duplicated in each subscription (here 2) and thus the functions are called in parallel. Note: The PubSub message format isn't exactly the same as your own specific for Cloud Functions (if you have data to POST to your function) and you need to rework on this entry point part
With Cloud Scheduler you can call Workflows and in your workflow you can call task in parallel. I wrote an article on that this week
In both cases, you can't do this out of the box and you need to use a intermediary component to perform the fan out of the only one scheduling event.
I am trying to trigger a google cloud function from stackdriver log, say if my VM goes down I need to perform some operation using function. Similar to how we do in AWS with cloudwatch events triggering lambda.
It's not possible to directly invoke a function from a log, but you can piece it together in two parts:
Export the logs you care about to Cloud Pub/Sub
https://cloud.google.com/logging/docs/export/
Subscribe a Cloud Function to the topic
https://cloud.google.com/functions/docs/calling/pubsub
and https://cloud.google.com/functions/docs/tutorials/pubsub
You'll want to configure StackDriver to call a webhook, which you then implement as a HTTP-triggered Cloud Function.