I have a cloud run service which will run upto 60 minutes.The pubsub is the trigger point for execution of cloud run service.
pubsub configuration for Retry policy is set to max (600s).
Now when a message is published from pubsub, cloud run starts executing, as the complete execution takes around 60 minutes to complete, but the pubsub message after 600s starts to retry again as it doesn't received any acknowledge from cloud run and again causing cloud run service executing again and again.
How to handle the pubsub retry here so that cloud run will not execute again and again because of retrying.
I was thinking to use Cloud Tasks, or Cloud Workflows as a proxy for your long running Cloud Run. Unfortunately both services have max timeout of 1800s (30minutes). By the way upcoming callback feature of Cloud Workflows will have 12h timeout. In the meantime I would create a proxy as Cloud Function triggered by PubSub message that will be immediately acknowledged, and the function will call your Cloud Run in async with the PubSub message and return right away.
With push subscriptions, such as what you'd use with a Cloud Run service, the maximum ack deadline for a message is indeed 600s. If using pull, one can call ModifyAckDeadline to extend the deadline for a message. In fact, the client libraries for Cloud Pub/Sub do this automatically for up to a configured amount of time (default is 60m).
There is not going to be a way to extend the deadline if using a push subscription. Therefore, your options are:
Switch to a pull subscription. You could potentially do this via Cloud Run, though it would not be the best fit. More likely, you want to spin up a job in an environment that can keep it running without any kind of trigger, e.g., GKE. If you switch to pull, you can extend the ack deadline, though note that duplicates are still possible, even if the ack deadline has not expired or the message has already been acknowledged. They should be rare, but you still have to account for it.
When you receive the message, persist it somewhere, either on disk or in a database, and then acknowledge the message once persisted. Once you are actually done processing the message an hour later, you remove it from this persistent storage. Of course, you could just persist the message instead of publishing it via Pub/Sub and rely on the persistence layer's notifications mechanisms to learn of the new message. For example, if you write to GCS, you could use Cloud Storage notifications via Pub/Sub. In this case, you probably want to have some periodic read from your storage to see if there are any messages that have not been processed for some period of time and if so, reprocess them. For example, if you write with the message the time at which processing started and if more than some amount of time has passed since then and the message is still present, you could start the processing over again.
Related
Currently, I create a service in Cloud Run to retrain ML models. The service will do the retrain process when there is an event from BigQuery called google.cloud.bigquery.v2.JobService.InsertJob. I use the EventArc in GCP to trigger the Retrain Service when that event happened. But, there is a problem. The trigger request to the service multiple times in one event. So, sometimes when the retraining process is done, the trigger requests the service again, and then the retraining process is active again. Is there something that I missed? Picture bellow is my EventArc setup.
As we can see in this picture that there are other requests while the first request is in process.
Eventarc is backed on PubSub. By default, and if you don't hack the default Eventarc configuration, the delivery timeout is set to 10s. (you can update manually the pubsub subscription created by eventarc. The Eventarc engineering team is aware of that not customizable parameter)
That's why, you should have a retry every 10s.
You have 2 solutions to that:
Either create an async process. I mean receive the PubSub message (tbe eventarc event), ack it immediately, and, in background, run your retrain
Or (not my preferred way), update the eventarc pubsub subscription and set the message retention duration to 5 seconds.
I have a Cloud Function that is being triggered from a Pub/Sub topic.
I want to rate limit my Cloud Function, so I set the max instances to 5. In my case, there will be a lot more produced messages than Cloud Functions (and I want to limit the number of running Cloud Functions).
I expected this process to behave like Kafka/queue - the topic messages will be accumulated, and the Cloud Function will slowly consume messages until the topic will be empty.
But it seems that all the messages that did not trigger cloud function (ack), simply sent a UNACK - and left behind. My subscription details:
The ack deadline max value is too low for me (it may take a few hours until the Cloud Function will get to messages due to the rate-limiting).
Anything I can change in the Pub/Sub to fit my needs? Or I'll need to add a queue? (Pub/Sub to send to a Task Queue, and Cloud Function consumes the Task Queue?).
BTW, The pub/sub data is actually GCS events.
If this was AWS, I would simply send S3 file-created events to SQS and have Lambdas on the other side of the queue to consume.
Any help would be appreciated.
The ideal solution is simply to change the retrying policy.
When using "Retry after exponential backoff delay", the Pub/Sub will keep retrying even after the maximum exponential delay (600 seconds).
This way, you can have a lot of messages in the Pub/Sub, and take care of them slowly with a few Cloud Functions - which fits our need of rate-limiting.
Basically, everything is the same but this configuration changed, and the result is:
Which is exactly what I was looking for :)
You cannot compare to kafka because your kafka consumer is pulling messages at its convenience, while Cloud Function(CF) creates a push subscription that is pushing messages to your CF.
So some alternatives:
Create a HTTP CF triggered by cloud scheduler that will pull messages from your PULL subscription. Max retention of unack messages are 7 days (hope it's enough)
Use Cloud for which you increase max concurrency (max concurrent request), with proper sizing for CPU and RAM. Of course your can control the max number of cloud run instances (different from max concurrency). And Use PUSH subscription pushing to cloud run. But here Also you will be limited by 10 minutes ack deadline.
We have a Cloud Functions with a Pubsub trigger and it will invoke an application on HTTP endpoint based on the message. When we need to update the backend application we want the Cloud Functions to pause and queue up the messages and then start again when the application is up.
Currently we are logging all the failed messages in Stackdriver and resubmitting them after the release. Is there any better way to do this?
Your system must be resilient to outage (managed by you when you perform an update, or unexpected). Therefore, I recommend you to handle both in the same way.
Set the retry parameter when you deploy your Cloud Functions (background functions, bind with PubSub topic). When your Cloud Functions can't process the message (the application isn't available), raise an error. The message will be retried later.
Like that, you have nothing to worry. When the message can be delivered, it is, and when it can't, there is an error and the message is retried.
I currently have a pub/sub push subscription that pushes to a http endpoint. This endpoint then triggers my cloud function. I am running into an issue where the same events that have already been sent to my cloud function are being resent by the pub/sub subscription. I increased my subscription's ack deadline to 3 minutes but after about a minute into my cloud functions execution, it will resend the same event that has already been processed. This leads to multiple invocations of my cloud function and further issues. I haven't seen any way to disable pub/sub retries but wondering if there are any suggestions as to a root cause of this or any work arounds?
Current set-up:
cloud function timeout limit: 120seconds
pub/sub subscription ack deadline: 180seconds
dead-lettering after 5 retries
You will need to consider idempotency and flag any recent retries to prevent them from firing again. This could be a timestamp stored in a database and filter based on time and any metadata you contain. Another important thing is to return a successful result.
Doug covers this concept in a video, while it doesn't reference pubsub, it is still just as valid: https://www.youtube.com/watch?v=Pwsy8XR7HNE
I started a GCE VM with a Docker image that runs a pub/sub subscriber, which handles the messages and start some big computational work (long running).
When we are ready to deploy new code, how do we ensure all the current running jobs are finished (make the deploy block on task finish). What's the best practice here?
I believe that you can look at Google Cloud Functions. Saying this, you can create a programming function that will respond to some specific events without the need to manage a server or runtime environment.
In particular, it is feasible subscribing specific Cloud function to Pub/Sub topic and every message published to this topic will trigger some custom code execution with message contents passed as input data generating google.pubsub.topic.publish event type.
Supposedly, you can compose some function, that would be subscribed to the same Pub/Sub topic as the message consumer from your example, triggering the desired deployment upon some condition match, checking the status of the long running job.