I have created one actor that subscribes to one topic to get messages. This is the only actor that subscribes to this topic.
I wondered what will happen if for some reason the actor will unsubscribe and then subscribe again to the same topic (if actor restarts, for example).
From Akka types API documentation (https://doc.akka.io/docs/akka/2.6.19//typed/distributed-pub-sub.html):
When a topic actor has no subscribers for a topic it will deregister
itself from the receptionist meaning published messages for the topic
will not be sent to it.
What does it means? Does it mean that after restart - no one can send messages to this actor through this topic?
If the only actor subscribing to a topic unsubscribes, messages sent to that topic will no longer be delivered. When that topic has a subscriber again, messages sent after that point may be delivered (messages sent between the subscriptions will not be delivered).
So if you have an actor which is subscribing to a Distributed Pub Sub topic, it should resubscribe on every restart.
Related
I'm currently facing a problem when thinking about a event driven arch using SNS to decouple some applications.
Imagine a SNS Topic, and I have application A producing messages to it and application B will listen and consume messages from this topic.
This application B has a autoscaling group attached to it, so it can scales to more than one instance. How will SNS handle when application B scales? If now I got 2 instances of application B, SNS will send the message for all of them or it can realize that they are the same application and just send the message to one of them?
Think of SNS as a radio broadcast: Everyone who is listening will get your message. Meaning that every single of your subscribed servers will get notified.
SQS, on the other hand, is more like a todo list. Many subscribers can also listen to it, but every message is distributed to at least someone. Meaning that usually, only one server will get triggered.
If that suits you better, then you might consider using SQS instead of SNS.
I'm not sure what your desired outcome is here, so I'm splitting the answer into two parts:
a) You only want to process each message once:
A common pattern in this case is to subscribe an SQS queue to the SNS topic, and then have N application servers polling from this queue. That way, you can make sure that you process each message only once.
b) You want to process each message once on each server:
In this case, you can create one subscription for each server to the SNS topic. Each message published to the topic will be delivered once to each subscription.
I'm trying to learn about GCP Pub/Sub and I have a problem about the life of a message in Pub/Sub. In fact, I used this article as my reference. And in this article, they said:
Once at least one subscriber for each subscription has acknowledged the message, Pub/Sub deletes the message from storage.
So my first question is: for example I have a Subscription A which connects to Subscriber X et Subscriber Y. According to the docs, when the Subscriber X received the message and it sends an ACK to the Subscription A, the Pub/Sub will delete the message from storage without considering if the Subscriber Y received or not the message. In other words, Pub/Sub doesn't care if all subscribers have received messages or not, just one subscriber gets the message and Pub/Sub will delete the message from storage? Am I right, please?
Then, in the following part of the article, the article said:
Once all subscriptions on a topic have acknowledged a message, the message is asynchronously deleted from the publish message source and from storage.
And I feel a little bit confuse here. What I understood is that, for instance, I have a topic that has N subscriptions, each subscription has M subscriber, Pub/Sub just needs to known that for each subscription, at least one subscriber has acknowledged the message, it'll delete the message from storage. Am I right, please?
I also found that in the documentation, we have two concepts: Publishing Forwarder and Subscribing Forwarder. So may I ask some last questions:
What is the relationship between Subscription, Publishing Forwarder and Subscribing Forwarder? (for example, a Subscription consists only one Publishing Forwarder and one Subscribing Forwarder?)
The relationship between Publishing Forwarder and Subscribing Forwarder is one-to-one or one-to-many or many-to-one or many-to-many, please?
Can a Subscriber be associated with many Subscription or not, please?
Once a Subscriber consumes a message (here I say this message is not duplicated, it has no copy, it is unique), is it possible to this Subscriber re-consumes/re-reads exactly this message?
If I misunderstand something, please, point it out for me, I really appreciate that.
Thank you guys !!!
Quite a bit to unpack here. It is best not to think of a subscription as attaching to subscribers and also to understand that these two things are different. A subscription is a named entity that wants to receive all messages published to a topic. A subscriber is an actual client running to receive and process messages on behalf of a subscription. A topic can have many subscriptions. A subscription can have many subscribers. If there are multiple subscribers in a subscription, then, assuming there are no duplicate deliveries and subscriber ack all messages received, each message published to a topic will be delivered to one subscriber for the subscription. This is called load balancing: the processing of messages is spread out over many subscribers. If a topic has multiple subscriptions, each with one subscriber, then every subscriber will receive all messages. This is called fan out: each subscriber receives the complete set of messages published. Of course, it is possible to combine these two and have more than one subscriber for each subscription, in which case each message will be delivered to one subscriber for each subscription.
Forwarders are just the servers that are responsible for delivering messages. A publishing forwarder receives messages from publishers and a subscribing forwarder sends messages to subscribers. All of the relationships along the path of delivering a message, from publisher to publishing forwarder, publishing forwarder to subscribing forwarder, and subscribing forwarder to subscriber, can be many-to-many relationships.
A subscriber is associated with a single subscription. However, a job running could have multiple subscribers running within it, e.g., one could instantiate the subscriber client library several times on different subscriptions.
All of the above assumed an important caveat: assuming there are no duplicate deliveries. In general, Cloud Pub/Sub guarantees at least once delivery. That means that even a message that was properly acked by a subscriber could be redelivered--either to the same subscriber or a different subscriber--in which case the subscriber needs to ack the message on the subsequent delivery. Generally, duplicate rates should be very low, in the 0.1% range for a well-behaved subscriber that is acking messages before the ack deadline expires.
I'm wondering what would happen if there was an SNS topic having messages written to it, but for a period of time, there is no SQS queue. Let's say there was a container which normally was subscribed to the SNS topic to handle such messages, but it crashed and burned and spent 10 minutes getting resurrected; what would happen to any messages written to that topic, during which there is no queue? Do they disappear forever, or do they wait politely until some queue comes along, subscribes and picks up said messages?
They disappear forever.
SNS cannot know that some subscriber wants to subscribe but simply cannot right now. The topic either has subscribers or it does not. All current subscriber get the message, all future ones do not.
If you have subscriber but the delivery fails there is some SNS specific behaviour in regards to retries: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
If the subscriber fails to get the message, a retry mechanism in SNS kicks in as explained in the AWS docs:
When the delivery policy is exhausted, Amazon SNS stops retrying the delivery and discards the message—unless a dead-letter queue is attached to the subscription.
For SQS subscriber retry can be up to 100,015 times, over 23 days
If SQS Queue goes down then message won't disappear , Let's discuss this scenario:
Retry Policy :-** Let's say you set "Number of retries" as n and "Retry-backoff function" as Linear(you can select any other retry-backoff function) in SNS topic , then if SQS is not available then SNS will retry to send that message to subscriber(SQS) n number of times based on the "Retry-backoff function" .
But if you set Number of retries as 0 then your message will delete from SNS topic immediately if Subscriber(SQS) is not available
Basically I have a SNS topic with multiple SQS subscribers. By default its sending out the same message to all subscribers but what I need is one message is sent to exactly one subscriber.
One solution come to my mind but I want to know if I'm leaving out any possible solutions.
Use `Message Group Id`
A) Get all SQS subscribers for SNS topic
B) When sending out list of messages, we randomly choose a SQS subscriber
id, and use that as `Message Group Id`
C) When SNS message is received, it will go the appropriate SQS subscriber.
Is there any other solutions? for instance if I create a FIFO queue and attach subscribers, are they all getting the same messages?
Following is a message design pattern :
Step 1- Application sends message to SNS Topic
Step 2- SNS publishes message to subscribed SQS queue .
As per following definitions :
Broadcast : Message is published to all end points.
Multicast : Message is published to selected endpoints.
Above diagram can be interpreted as
Option 1 - Message is published to selected SQS queues which are subscribed to SNS topic , multicast pattern
OR
Option 2 - Message is published to all subscribed end points,broadcast pattern
how should this design pattern be interpreted ?
In the absence of clarification, this is likely to be broadcast.
The publisher can't select the queues that will receive the messages -- it will go to all of them, by default.
Historically, SNS fanout to SQS was always broadcast.
However, recent enhancements of SNS provide a capability for the subscriptions of each queue to the SNS topic to be "filtered" -- in which case, the publisher still can't directly select the queues that will receive the message (they're not explicitly addressable), but SNS makes decisions on where to deliver the messages based on the subscription filters... which might fit the multicast label, depending on the circumstances.
https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html