Pub Sub topic invokes a cloud function endpoint upon receiving a new message.
If any error happens inside the cloud function the function returns an error.
Whether the delivery will be retried by the PubSub in case of error?
The Cloud function deployed without the retry option. Want to have the retry control on the Pub Sub.
Tried a sample pub sub topic triggered cloud function which always returns the error on execution,
**
import (
"context"
"errors"
)
func PushBackOffTest(ctx context.Context, m PubSubMessage) error {
print(string(m.Data))
return errors.New("always returns error")
}
**
But the cloud function is not executed again.It ran only once.
ACK deadline 600 seconds. Max delivery attempts 6 . Configured from the G Cloud console.
If you want the event to be redelivered in the event of an error, then you need to enable retry in your Cloud Function by checking the "Retry on failure" box. Otherwise, Cloud Functions will acknowledge the message received from Pub/Sub regardless of the result of processing it. Checking this box is what tells Cloud Functions to use Cloud Pub/Sub's retry mechanism for unacknowledged messages.
Related
I am setting up a pub/sub trigger-based cloud function in GCP, the topic the cloud function listen to is us-pubsub1. When I deployed the cloud function and used the testing panel to send messages like:
{"index":123,
"video_name":'test.mp4'}
in cloud function to processing the message with key index and video_name, it has no issue. But when I sent a real message to us-pubsub1 to trigger the cloud function, it always failed for not able to find the 'index' in the message body. and when reading the pub/sub message in cloud function, it return me messages like:
{'#type': 'type.googleapis.com/google.pubsub.v1.PubsubMessage',
'attributes': None, 'data':
'eyJzZXNzaW9uX2lkIjogImUzYjM0MTJiLWQxNWUtNDM5My05YjEyLWI3ZGY1ZGE4MTQ0NCIsICJzZXNzaW9uX25hbWUiOiAiU0VTU0lPTl9DMjIjMwMTI1VDIwMTI1NCIsICJzaXRlX25hbWUiOiAiVkEgTUVESUNBTCBDRU5URVIgLSBQQUxPIEFMVE8gLSc3RlY3RvbXkiLCAiaHViX3NlcmlhbF9udW1iZXIiOiAiQzIyNC0wMDIwNSIsICJjYXN0X2FwcF92ZXJzaW9uIjogIjEyLjAuMzIuNyIsICJkdl9zeXN0ZW0iOiAiVUFUU0syMDA3IiwgImludGVybmFsX2tleSI6ICJtNjQzY2U1NS0zNjNiLQ=='}
I checked that the message arrive in us-pubsub1 correctly and just failed to process in cloud function.
Is there any thing I have missed for fetching the message body for real cloud function?
It's normal. You have a pubsub message enveloppe and your content is base64 encoded in the data field. Here the documentation details
FWIW, here the real content of your sample
{"session_id": "e3b3412b-d15e-4393-9b12-b7df5da81444", "session_name": "SESSION_C22#3#UC##SB"'6FUR#%dTD44TDU"Dstectomy", "hub_serial_number": "C224-00205", "cast_app_version": "12.0.32.7", "dv_system": "UATSK2007", "internal_key": "m643ce55-363b-
it is truncated, you might not share the whole content ;)
How can I bulk move messages from one topic to another in GCP Pub/Sub?
I am aware of the Dataflow templates that provide this, however unfortunately restrictions do not allow me to use Dataflow API.
Any suggestions on ad-hoc movement of messages between topics (besides one-by-one copy and pasting?)
Specifically, the use case is for moving messages in a deadletter topic back into the original topic for reprocessing.
You can't use snapshots, because snapshots can be applied only on subscriptions of the same topics (to avoid message ID overlapping).
The easiest way is to write a function that pull your subscription. Here, how I will do it:
Create a topic (named, for example, "transfer-topic") with a push subscription. Set the timeout to 10 minutes
Create a Cloud Functions HTTP triggered by PubSub push subscription (or a CLoud Run service). When you deploy it, set the timeout to 9 minutes for Cloud Function and to 10 minutes for Cloud Run. The content of the processing is the following
Read a chunk of messages (for examples 1000) from the deadletter pull subscription
Publish the messages (in bulk mode) into the initial topic
Acknowledge the messages of the dead letter subscription
Repeat this up to the pull subscription is empty
Return code 200.
The global process:
Publish a message in the transfer-topic
The message trigger the function/cloud run with a push HTTP
The process pull the messages and republish them into the initial topic
If the timeout is reached, the function crash and PubSub perform a retry of the HTTP request (according with an exponential backoff).
If all the message are processed, the HTTP 200 response code is returned and the process stopped (and the message into the transfer-topic subscription is acked)
this process allow you to process a very large amount of message without being worried about the timeout.
I suggest that you use a Python script for that.
You can use the PubSub CLI to read the messages and publish to another topic like below:
from google.cloud import pubsub
from google.cloud.pubsub import types
# Defining parameters
PROJECT = "<your_project_id>"
SUBSCRIPTION = "<your_current_subscription_name>"
NEW_TOPIC = "projects/<your_project_id>/topics/<your_new_topic_name>"
# Creating clients for publishing and subscribing. Adjust the max_messages for your purpose
subscriber = pubsub.SubscriberClient()
publisher = pubsub.PublisherClient(
batch_settings=types.BatchSettings(max_messages=500),
)
# Get your messages. Adjust the max_messages for your purpose
subscription_path = subscriber.subscription_path(PROJECT, SUBSCRIPTION)
response = subscriber.pull(subscription_path, max_messages=500)
# Publish your messages to the new topic
for msg in response.received_messages:
publisher.publish(NEW_TOPIC, msg.message.data)
# Ack the old subscription if necessary
ack_ids = [msg.ack_id for msg in response.received_messages]
subscriber.acknowledge(subscription_path, ack_ids)
Before running this code you will need to install the PubSub CLI in your Python environment. You can do that running pip install google-cloud-pubsub
An approach to execute your code is using Cloud Functions. If you decide to use it, pay attention in two points:
The maximum time that you function can take to run is 9 minutes. If this timeout get exceeded, your function will terminate without finishing the job.
In Cloud Functions you can just put google-cloud-pubsub in a new line of your requirements file instead of running a pip command.
I have a function with a simple test code like:
exports.helloPubSub = (event, context) => {
const message = event.data
? Buffer.from(event.data, 'base64').toString()
: 'Hello, World';
console.log(context);
throw new Error("Fail");
};
When I publish a message to Pub/sub the function fails and I expect that it will be called again with the same message after 600 seconds since the pub/sub subscription has Acknowledgement deadline set to 600 Seconds.
But it does not work as expected looks like it acks the message immediately despite the failure in a cloud function.
According to the doc: Cloud Functions acks the message internally upon successful function execution.
You must make sure that you have enabled retries on the function. From the "Retrying Background Functions" documentation (emphasis mine):
Cloud Functions guarantees at-least-once execution of a background
function for each event emitted by an event source. However, by
default, if a function invocation terminates with an error, the
function will not be invoked again, and the event will be dropped.
When you enable retries on a background function, Cloud Functions will
retry a failed function invocation until it completes successfully, or
the retry window (by default, 7 days) expires.
You can enable retries by providing the --retry option when creating the function via the gcloud command-line tool or checking the "Retry on failure" box when creating via the Cloud Console.
When a Cloud Function fails and retry is enabled, it nacks the message, which makes the message a candidate for redelivery. To delay the redelivery of nacked messages, you can set up a RetryPolicy on the subscription. To update the subscription to use a retry policy, find the name of the subscription created by Cloud Functions in the Cloud Console Pub/Sub section. Then, for example, you can issue this gcloud command to set the minimum retry delay to 1 second and the maximum delay to 60 seconds:
gcloud pubsub subscriptions update <subscription name> --min-retry-delay=1s --max-retry-delay=60s
I have got a topic, which on publish it pushes the event to a cloud run endpoint and I got a trigger on a storage bucket to publish for this topic. The container in the cloud run fails to process the event and it has been restarted over hundreds of times and I don't wanna waste money on this. How can I limit the retry on failure on a cloud run's container?
A possible answer to the puzzle might be the following notion.
If we read the documentation on PUSH subscriptions found here, we find the following:
... Pub/Sub retries delivery until the message expires after the
subscription's message retention period.
What this means is that if Pub/Sub pushes a message to Cloud Run and Cloud Run does not acknowledge the message by returning a 200 response code, then the message will be re-pushed for the "message retention period". By default, this is 7 days but according to the documentation, can be set to a minimum value of 10 minutes. What this seems to say to me is that we can stop a poison message after 10 minutes (minimum) of retries.
If a message is pushed and not acked, then it won't be pushed again immediately but instead be pushed as a function of a back-off algorithm described here.
If we look at the gcloud documentation we find reference to the concept of a maximum number of delivery attempts (--max-delivery-attempts). Associated with this is a topic called the dead letter topic (--dead-letter-topic). What this appears to define is that if an attempt to deliver a pub sub message more than the maximum number of times, the message will be removed from the queue of messages associated with the subscription and moved to the topic associated with the dead letter. If you define this for your environment, then your Cloud Run will only execute a finite number of times after which the poision messages will be moved elsewhere.
The official documentation does mention that Google Cloud Pub / Sub resends messages to subscribers until subscribers acknowledge the message receipt when using official Cloud Pub / Sub Node.js client.
But it does not explicitly mention this for background functions if they return a callback error. Refer https://cloud.google.com/functions/docs/writing/background.
If it helps - My background function does not use the official Cloud Pub / Sub Node.js client since I get all the required info from the event arguement itself.
From documentation: https://cloud.google.com/functions/docs/bestpractices/retries
Cloud Functions guarantees at-least-once execution of a background
function for each event emitted by an event source. However, by
default, if a function invocation terminates with an error, the
function will not be invoked again, and the event will be dropped.
When you enable retries on a background function, Cloud Functions will
retry a failed function invocation until it completes successfully, or
the retry window expires.
And described below, you can enable retry on errors:
In any of the above cases, the function stops executing by default and
the event is discarded. If you want to retry the function when an
error occurs, you can change the default retry policy by setting the
"retry on failure" property. This causes the event to be retried
repeatedly for up to multiple days until the function successfully
completes.