How can I acknowledge all messages that are currently in the backlog for Google PubSub Lite subscription. I tried using
gcloud pubsub lite-subscriptions ack-up-to SUBSCRIPTION \
--location LOCATION --partition PARTITION --offset OFFSET --project PROJECT
but I don't know how to set the offset accordingly.
To receive messages from a Lite subscription, request messages from the Lite subscription. The client library automatically connects to the partitions in the Lite topic attached to the Lite subscription. If more than one subscriber client is instantiated, messages will be distributed across all clients. The number of partitions in the topic determines the maximum number of subscriber clients that can simultaneously connect to a subscription. You can see more documentation and examples.
You can see these example to receive messages from Lite subscriptions:
gcloud pubsub lite-subscriptions subscribe SUBSCRIPTION_ID \
--location=LITE_LOCATION \
--auto-ack
But if you want to subscribe to a Pub/Sub Lite subscription and automatically acknowledge messages, you can see this example:
gcloud pubsub lite-subscriptions subscribe mysubscription --location=us-central1-a --auto-ack
To subscribe to specific partitions in a subscription, this example:
gcloud pubsub lite-subscriptions subscribe mysubscription --location=us-central1-a --partitions=0,1,2
You can see more documentation about these examples.
Related
In GCP console you have the possibility to purge messages in a subscription.
It looks like "gcloud pubsub subscriptions" do not support a "PURGE MESSAGES" command.
What are the best alternatives are there any other libraries or API?
With gcloud tool this functionality is provided by the seek method. It effectively resets a subscription's backlog to a point in time or to a given snapshot. So, to purge all messages from a subscription you'd pass the current time via the --time flag, for example:
gcloud pubsub subscriptions seek my_subscription__name --time=2022-02-10T11:00:00Z
Messages in the subscription that were published before this time are marked as acknowledged, and messages retained in the subscription that were published after this time are marked as unacknowledged.
The equivalent API endpoint for it is projects.subscriptions.seek.
I have an SNS Topic that had no subscriptions for a period of time. So any messages published to that topic during that period when there were no subscriptions were essentially lost. I am wondering if there are any logs or other records I could access to glean any info on things that could have been published during the time when there were no subscriptions.
I saw from an AWS forum post that
"From [the AWS] perspective, a publish is successful if we
successfully process the publish API call. We do not enumerate the
subscriptions list until later. If there are no subscriptions to a
topic, the message is eventually dropped."
I believe I could create a CloudWatch alarm to monitor the NumberOfMessagesPublished metric, per this article. From that article, I understand that Amazon SNS automatically sends the NumberOfMessagesPublished metric to CloudWatch, so if that metric shows nothing (count = 0) during the time period, I at least would have evidence of the lack of anything published to the topic. But if something was published, is there any evidence that remains--in logs or elsewhere--of the message that was published?
I want to deploy a Pubsub-triggered Cloud Function with message ordering:
https://cloud.google.com/pubsub/docs/ordering
gcloud functions deploy doesn't have an option to set an --enable-message-ordering option:
https://cloud.google.com/sdk/gcloud/reference/functions/deploy
Should I pre-create the subscription before deploying the function? If so, does Cloud Functions have a well-known format for how it matches to a subscription name? It seems maybe the format is: gcf-{function-name}-{region}-{topic-name}, but it also looks like the name format has changed over time, e.g. older deployed functions don't have the region name in the subscription. Is there a stable way to do this?
You must create message ordering pub/sub and Cloud function manually.
Fisrt, Create a pub/sub topic, and then create a subscription that subscribes pub/sub topic with --enable-message-ordering
Second, Create a Cloud function that will serve ordered pub/sub messages.
Last, back to the pub/sub subscription, Edit delivery type to push and specify your cloud function endpoint.
So final diagram is like below.
Publisher -> Pub/sub topic -> Pub/sub subscriber -> Cloud function
You tried to make a connection Pub/sub topic with Cloud function directly.
But for message ordering, Pub/sub needs topic -> subscriber connection.
So only pub/sub topic -> pub/sub subscriber -> Cloud function connection can delivers ordered messages to your function.
When declaring a Pub/Sub topic with a Cloud Function alike:
exports.pubsub = functions.pubsub.topic('some-topic').onPublish((message, context) => {}
The problem is that message ordering is only available for the subscription, but not for the topic.On deploy a push subscription is automatically being created at: cloudpubsub/subscription/list (the one which starts with gcf-*). This only appears to work when manually subscribing to a topic: Enabling message ordering. I haven't yet tried if it would pick up a subscription with the same name; if everything fails, one still could record the messages and then order by timestamp or a "sequence token": https://www.youtube.com/watch?v=nQ9_Xur2aM4.
I'm referring to Google provided dataflow Pub/Sub to Text Files on Cloud Storage.
The messages once read by dataflow don't get acknowledged. How do we ensure that messages once consumed by dataflow is acknowledged and is not available to any other subscriber?
To reproduce and test it, create 2 Jobs from the same template and you would see that both the job processing the same message.
Firstly, the messages are correctly acknowledge.
Then, to demonstrate this, and how your reproduction is wrong, I would like to focus on PubSub behavior.
One or several publishers publish messages in a topic
One or several subscription can be created on a topic
All the messages published in a topic are copied in each subscription
Subscription can have one or several subscribers.
Each subscriber receives a subset of the messages in the subscription.
Go back to your template. You specify only a topic, not a subscription. When your dataflow is running, go to the subscription, you will be able to see a new subscription created.
-> When you start a PubSub to TextFiles template a subscription is automatically created on the provided topic
Therefore, if you create 2 jobs, you will have 2 subscribtions, and thus, all the messages published in the topic are copied in each subscription. That's why you will have 2 times the same messages.
Now, keep your job up and go to the subscription. Here you can see the number of message in the queue and the unacked messages. You should see 0 in the unacked message graph.
I have my application which processes messages from a pubsub topic and if it fails the message is send to a separate dlq topic. I want to be able to set an alarm in monitoring that when during a day there were 30k messages sent to the dlq it notifies me and I can check why my service is not wokring.
I tried to set up some polices in gcp but I don't know and couldn't find anywhere in the docs how to setup a metric of daily processed messages on a topic.
Can anyone help me ?
You can create a new alert policy like this
PubSub subscription/unacked messages.
You can add a filter on your subscription name if you have several subscriptions in your project.
Add the notification channel that you want, an email in my case. After few minutes, you can see the first alert
And the email
EDIT
For the acked messages, you can do this
I never tried an aggregation over 1 day, but it should be OK.
Please check the following GCP community tutorials which outline how to create an alert-based event archiver with Stackdriver and Cloud Pub/Sub
https://cloud.google.com/community/tutorials/cloud-pubsub-drainer