I'm developing an app that has chat between users. Does it make sense to use Pub/Sub for this purpose? I couldn't find in the documentation as a possible use case. Are there any good reasons why it's not a good fit?
Google Cloud Pub/Sub would not be a good fit for communicating with end user devices in a chat product. Cloud Pub/Sub is designed for torrents: a relatively small number of long-lived streams that require high throughput. In a chat app, you need trickles: a very large number of ephemeral streams that have a low throughput.
Cloud Pub/Sub quotas only allow for 10,000 topics project and 10,000 subscriptions per topic or per project. When setting up a new streaming pipeline, one generally creates the topic and its subscriptions once at the beginning and then publishes and subscribes from that point on. Therefore, Cloud Pub/Sub is a better fit for server-to-server communication or for large-scale streaming event ingestion, often in connection with Dataflow and/or BigQuery.
The permission model for Cloud Pub/Sub doesn't make it a good choice for a chat app, either. Permission to write to or read from a topic or subscription requires the user to be authenticated. Authenticating each individual user with permissions in your project is not really feasible, so you are left with the option of using common credentials for all of your users, which could be insecure.
That's not to say that Cloud Pub/Sub couldn't be part of a chat service, it just wouldn't be used to deliver messages to specific users. In fact, Cloud Pub/Sub can be used as the transport mechanism for messages sent to Hangouts Chat Bots. In this situation, though, the response to the user goes directly to the Hangouts Chat API, not back through Pub/Sub. Therefore, if you were implementing a chat service, you could potentially have your users send messages to a frontend server you design that then publishes those messages to Cloud Pub/Sub. Another server you build could be a subscriber to those messages and then deliver the appropriate messages to the appropriate individual users. In this situation, Cloud Pub/Sub is used for ingesting all of the messages from your frontend server (torrent) and delivery of messages to individual users falls to some other mechanism (trickles). The product in Google Cloud designed for this individual device delivery is Firebase Cloud Messaging,
Can Google Pub/Sub be used for chat messaging?
Yes. I would not use Pub/Sub for person-to-person chat messaging. Pub/Sub is designed for systems to de-couple themselves to support distributed systems.
Why not use Pub/Sub:
Permissions. You will need to create service account credentials and distribute this Json file to each subscriber. This is not secure.
Each chat person will need a subscription to a Pub/Sub Topic and then poll that subscription. This means one subscription per chat person. This is neither a good design nor inexpensive.
There are much better solutions for chat.
Pub/Sub for chat? No. As it is rightly mentioned above, Pub/sub has 2 major use cases, for data ingestion and for decoupling the system.
I would recommend you search a bit about web sockets if are still interested making a chat app.
Related
I am new to Google Clout Iot core. I just tried the iot core, registered the device and published some data on a topic.
Now i am wondering is only Google functions are used to get/subscribe data from it? or can i also used the data published from a device directly into my applications e.g. node.js or python? like it is usually subscribed through a normal MQTT topic? I dont need a serverless architecture i want to use the data from google iot into my server and store/use it from there..
thanks a lot!
When a message arrives over MQTT or HTTP from a device over IoT Core, that message is published on a GCP Pub/Sub topic using the GCP Pub/Sub infrastructure. What this means is that you can develop a consuming application however you like as long as that application consumes the message from Pub/Sub.
The diagram shown here from the documentation is a good reference.
GCP exposes the API for Pub/Sub to be a subscriber. There are language bindings for a variety of languages.
Cloud Functions is typically seen as a serverless consumer of Pub/Sub messages as it scales to zero and can scale up if there is bursty load. However, there is nothing to prevent you from writing Compute Engine hosted applications or Container hosted applications which can act as subscribers.
References:
Cloud Pub/Sub documentation
Cloud Pub/Sub Client Libraries
I think your question is more on the architectural side of things. You can use IoT Core with something like Pub/Sub and write the data to Cloud Storage in order to later consume it with an application.
I would recommend you to check some of the diagrams from this page if you are interested in knowing which products you could use to best fit your needs.
I hope you find this useful!
I am working on an assignment where our customers will sync their crm data to our systems. The sync will be ongoing process. Any best practices or suggestions on google pub/sub for sharing one of our existing (or a new one) topic that our customer will publish too and we consume? Idea here is to keep sync asynchronous.
#Alex-hong is true if your customer is on GCP. If not, you have 2 solutions:
You can generate a service account and send it the JSON key file. Define the right role on the service account (as described by Alex) and let your customer use the Service Account and to publish to the PubSub topic. This solution implies an important development by your customer (Use external JSON key, use new libraries to push messages,...)
You can deploy a Cloud Function/Cloud Run endpoint and let your customer to simply perform an HTTP request. Of course, you can secure the call with Basic Authentication or APIkey (or something like that, that you check in your Cloud Function). The function only check the security and publish to PubSub. It's often simpler and more standard for your customer.
On the last point, it could be possible to set up an ESP in place of the function, but I never tested for publishing directly to PubSub.
You can give the other party the roles/pubsub.publisher role for your specific project if you own the topic. Alternatively, if they own the topic, they can grant you the role roles/pubsub.subscriber which will let you subscribe to that topic.
For more information, see Access Control
I'm new to the GCP and currently have a microservice architecture using GKE and gRPC. The microservices are publishing events to Google Cloud Pub/Sub. My Web-UI is
using Google Cloud Endpoints to send requests to the Microservices.
I want to have a lot of live/push updates on the website (such as live updating user statistics etc) and now wonder how this is done best. Is it a bad practice to let the Web-UI subscribe to a topic in Google Cloud Pub/Sub? Are there other technologies in the GCP that may be better for this case?
Cloud Pub/Sub is intended for `torrents` use cases, where you have a lot of data communicated between relatively few publishers/subscribers.
Firebase Cloud Messaging might better fit the particular `trickles` use case you describe, where you are sending smaller updates to many, possibly transient, subscribers.
I have industrial internet of things (IIoT) data coming from machines out in the wilderness. Currently the data are going from the machines to an Amazon Web Services (AWS) Simple Queueing Service (SQS) queue, for consumption by a third-party company.
I want to do two things now:
Put the data into the SQS queues for the third-party company, as before, and
Store the data in an Amazon Relational Database Service (RDS) database for analysis purposes
To accomplish this, should I send the data to an Amazon Simple Notification Service (SNS) topic first, and then push the data from there to both my database, and an SQS queue for the third-party company?
Is there a better way of accomplishing my objective, or am I on the right track?
I ask partly because it seems that Amazon is steering me toward a noSQL database called DynamoDB, but I think I want a structured relational database for my IIoT data.
Amazon’s IoT “message broker”, which communicates with my IoT devices using MQTT, is an SNS topic, which makes a lot of sense.
SNS message topics are different than SQS queues. Queues can only be read by one user, and then the message(s) get deleted from the queue—it’s just temporary storage. So we couldn’t share an SQS queue with the third-party company.
SNS topics, by contrast, send push notifications (and the messages) to multiple users instantly, and all “subscribed” users get the message(s). SNS is exactly like subscribing someone’s Twitter or Instagram feed, where all “subscribers” get notified of new messages from “publishers” to the message “topic”.
Long story short, we can use AWS IoT “rules” to send our devices' MQTT-secured “messages” to as many “subscribers” or AWS services as we want. So we’ll send one copy to the SQS queues, and another copy to an AWS database of some sort.
I want to use google cloud pub sub to implement Facebook kind of notifications in blog application which has following architecture
I have a both web and android device as a client for rest services exposed.
Those rest services has been secured using OAuth2 (spring security).
When user create a blog then if someone comment on that post, the concern owner can get the notifications. Only authorized user can create a post or comment and the concern person can see the notifications
I don't want user to authorize again with Google so how I can incorporate the api's in my architecture using pull notification
You typically wouldn't use Google Cloud Pub/Sub in the client in the type of setup you describe. Cloud Pub/Sub isn't designed to scale to a number of subscriptions that are on the order of number of phones. Instead, you'd want to use Firebase Cloud Messaging. Firebase has a model that handles registration of devices to receive messages that is more in line with what you are thinking of, e.g., Set Up a Firebase Cloud Messaging Client App on Android and
Implementing a Firebase Cloud Messaging Client on Chrome
What you may use Cloud Pub/Sub for is communication between different services. For example, perhaps the service that handles blog posts publishes messages on a Cloud Pub/Sub topic that your notification service subscribes to. The notification service would process the message and send out Firebase Cloud Messaging messages to end users as appropriate.