GCP pubsub docs mention load balancing for pull mode, it's not clear how to use it.
The Subsciption nor The Subscriber builder api, doesn't seem to have a method to turn this on.
Question: How to configure load balancing accross multiple pubsub subscribers?
Background:
We use multiple subscribers for the same topic, to achieve resilience.
(Multiple endpoints can be queried for data from the same data store).
The subscriptions persist the messages, but with out distribution, all subscriptions get all messages, leading to data duplication in our data store. Perhaps this background, will give ideas for another way to achieve resilience.
Things we have thought of ourselves:
Use multiple data stores...
Mark the messages, and do some sort of optimistic locking/versioning of rows in the data store.
Technologies:
GCP pubsub
Spring Boot / Data
JPA
Postgres DB.
If all subscribers are receiving all messages, then it is likely that you are using different subscriptions for each subscriber. Load balancing happens when you have different subscribers all pulling from the same subscription. From the subscriber guide description of load balancing: "Multiple subscribers can make pull calls to the same "shared" subscription. Each subscriber will receive a subset of the messages" (Emphasis mine). When you use different subscriptions, you get fanout, where all subscribers receive all messages.
Related
I am trying to send IoT commands using a push subscription. I have 2 reasons for this. Firstly, my devices are often on unstable connections so going through the pubsub let me have retries and I don't have to wait the QoS 1 timeout (I still need it because I log it for later use) at the time I send the message. The second reason is the push subscription can act as a load balancer. To my understanding, if multiple consumers listen to the same push subscription, each will receive a subset of the messages, effectively balancing my workload. Now my question is, this balancing is a behavior I observed on pull subscriptions, I want to know if:
Do push subscription act the same ?
Is it a reliable way to balance a workload ?
Am I garanteed that these commands will be executed at most once if there is, lets say, 15 instances listening to that subscription ?
Here's a diagram of what I'm trying to acheive:
Idea here is that I only interact with IoT Core when instances receive a subset of the devices to handle (when the push subscription triggers). Also to note that I don't need this perfect 1 instance for 1 device balancing. I just need the workload to be splitted in a semi equal manner.
EDIT: The question wasn't clear so I rewrote it.
I think you are a bit confused about the concepts behind Pub/Sub. In general, you publish messages to a topic for one or multiple subscribers. I prefer to compare Pub/Sub with a magazine that is being published by a big publishing company. People who like the magazine can get a copy of that magazine by means of a subscription. Then when a new edition of that magazine arrives, a copy is being sent to the magazine subscribers, having exactly the same content among all subscribers.
For Pub/Sub you can create multiple push subscriptions for a topic, up to the maximum of 10,000 subscriptions per topic (also per project). You can read more about those quotas in the documentation. Those push subscriptions can contain different endpoints, in your case, representing your IoT devices. Referring back to the publishing company example, those push endpoints can be seen as the addresses of the subscribers.
Here is an example IoT Core architecture, which focuses on the processing of data from your devices to a store. The other way around could also work. Sending a message (including device/registry ID) from your front-end to a Cloud Function wrapped in API gateway. This Cloud Function then publishes the message to a topic, which sends the message to a cloud Function that posts the message using the MQTT protocol. I worked out both flows for you that are loosely coupled so that if anything goes wrong with your device or processing, the data is not lost.
Device to storage:
Device
IoT Core
Pub/Sub
Cloud Function / Dataflow
Storage (BigQuery etc.)
Front-end to device:
Front-end (click a button)
API Gateway / Cloud Endpoints
Cloud Function (send command to pub/sub)
Pub/Sub
Cloud Function (send command to device with MQTT)
Device (execute the command)
I am trying to achieve a point in a system architecture on top of AWS infrastructure where a message is published from a data processor and sent to multiple subscribers (clients). This message would contain information that some - but not all - clients would want to receive.
Very similar question > Routing messages from Amazon SNS to SQS with filtering
So to do this message filtering I have turned to the message FilterPolicy functionality provided by SNS using one topic. Currently the system is reaching a point in time that clients have more granular and specific filtering rules so now I am reaching the filtering limits of the AWS SNS.
See more about the SNS filter policy here https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html (section “Filter policy constraints”)
One example of my limitation is the amount of filter values in a policy, on above link it states 150 values. Right now my subscribers would be interested in receiving messages with a specific attribute value. Although this one attribute could have several hundreds or thousands of different values.
I can not also group this attributes since they represent a non-sequential identity.
I seek some guidance over on a architectural solution that would allow me to keep using AWS SNS. I am limited to use some of the AWS infrastructure services, so no RabbitMQ for me.
In the Google Cloud Pub/Sub documentation about load balancing in pull delivery say:
Multiple subscribers can make pull calls to the same "shared"
subscription. Each subscriber will receive a subset of the
messages.
My concern is about the last phase. Can I decide the way to partition the topic? In others words, Can I decide the way the subsets are grouped?
For instance, in the Kinesis AWS service I can decide the partition key of the stream, in my case by user id, in consequence, a consumer recibe all the messages of a subset of users, or, from other point of view, all the messages of one user are consumed by the same consumer. The message stream of one user is not distributed between different consumers.
I want to do this kind of partition with the Google Pub/Sub service. Is that possible?
There is currently no way for the subscriber to specify a partition or set of keys for which they should receive messages in Google Cloud Pub/Sub, no. The only way to set up this partition would be to use separate topics.
I have an external data source as an ActiveMQ topic. I can only connect and consume messages. They come pretty rarely, about 1 message per 10-30 seconds.
I want to collect all the messages and put them into the database.
Also I'd like to have an active web page that can receive the new messages over WebSockets and draw a chart.
I have a prototype built with Python/Flask/MongoDB/SocketIO, BUT...
I would like to use Amazon AWS cloud infrastructure to avoid processing the data on servers.
I believe that AWS Lambda can accept the messages and store them into the database (DynamoDB?) and also send a notification (maybe using SQS) being transformed into WebSocket message. (Not everything is clear there yet, maybe simple ajax polling will be enough).
Here is a question: how it would be possible to consume the messages from external ActiveMQ topic and process it with AWS Lambda?
I was looking on Kinesis, but it looks it only supports the data being pushed to it, but not polling for the data by some protocol...
You can use Lambda as a cron-like facility and poll on a schedule. See Lambda Scheduled Events.
We need to sync data between different web servers. The idea is very basic: when one entity is created on one server, it should be sent to all the other servers. What's the right way to do it? We are currently evaluating 2 approaches: amazon's sqs and sns services and custom implementation with some key-value database (like memcached and memqueue). What are the common pitfalls of custom implementations? Any feedback will be highly appreciated.
SQS would work OK if you create a new queue for each server and write the data to each queue. The biggest downside is that you will need each server to poll for new messages.
SNS would work more efficiently because it allows you to broadcast a message to multiple locations. However, it's a one-shot try; if a machine can't receive its notification when SNS sends it SNS will not try again.
You don't specify how many messages you are sending or what your performance requirements are, but any SQS/SNS system will likely be much, much slower (mostly due to latencies between sending the message and the servers receiving it) then a local memcache/key-value server solution.
A mixed solution would be to use a persistant store (like SimpleDB) and use SNS to alert the servers that new data is available.