I'm new to the GCP and currently have a microservice architecture using GKE and gRPC. The microservices are publishing events to Google Cloud Pub/Sub. My Web-UI is
using Google Cloud Endpoints to send requests to the Microservices.
I want to have a lot of live/push updates on the website (such as live updating user statistics etc) and now wonder how this is done best. Is it a bad practice to let the Web-UI subscribe to a topic in Google Cloud Pub/Sub? Are there other technologies in the GCP that may be better for this case?
Cloud Pub/Sub is intended for `torrents` use cases, where you have a lot of data communicated between relatively few publishers/subscribers.
Firebase Cloud Messaging might better fit the particular `trickles` use case you describe, where you are sending smaller updates to many, possibly transient, subscribers.
Related
Amazon EventBridge enables developers to connect 3rd party event-driven applications with Amazon services. Amazon AppFlow offers event-driven integration with 3rd party apps as well.
What is the difference between the two services for the event-driven scenarios and when to use one over the other?
The AppFlow FAQ contains this info:
Amazon EventBridge enables developers to build event driven
applications that interact with SaaS applications and AWS services.
SaaS applications that have integrated with EventBridge emit events to
the customer’s event bus, which can then be routed to targets such as
Amazon EC2 instances or Lambda functions for processing. AppFlow
supports bi-directional transfer of data between SaaS applications and
AWS services that may be initiated by humans using a UI, a schedule,
or events - all with a point and click interface.
AppFlow is great for anyone who wants to connect together their applications without writing any code. EventBridge works with Step Functions and Lambda (and various other AWS services), and so is well suited for developers who need the extra flexibility that those services provide. Right now EventBridge is only one way, whereas AppFlow allows you to send data back to SaaS apps. The list of supported partners for each service is also different.
For some of AppFlow's partners, data is received via API polling (from the FAQ: "AppFlow is a fully managed API integration service that replaces custom connectors"), which is slightly different from EventBridge which receives data in an event-driven manner, where an event is sent via an HTTP call from the partner as soon as a change occurs. The pricing of the two services also differs: EventBridge charges $1/million events, and AppFlow charges $0.001 per flow (with an additional charge per GB of data processed).
I am new to Google Clout Iot core. I just tried the iot core, registered the device and published some data on a topic.
Now i am wondering is only Google functions are used to get/subscribe data from it? or can i also used the data published from a device directly into my applications e.g. node.js or python? like it is usually subscribed through a normal MQTT topic? I dont need a serverless architecture i want to use the data from google iot into my server and store/use it from there..
thanks a lot!
When a message arrives over MQTT or HTTP from a device over IoT Core, that message is published on a GCP Pub/Sub topic using the GCP Pub/Sub infrastructure. What this means is that you can develop a consuming application however you like as long as that application consumes the message from Pub/Sub.
The diagram shown here from the documentation is a good reference.
GCP exposes the API for Pub/Sub to be a subscriber. There are language bindings for a variety of languages.
Cloud Functions is typically seen as a serverless consumer of Pub/Sub messages as it scales to zero and can scale up if there is bursty load. However, there is nothing to prevent you from writing Compute Engine hosted applications or Container hosted applications which can act as subscribers.
References:
Cloud Pub/Sub documentation
Cloud Pub/Sub Client Libraries
I think your question is more on the architectural side of things. You can use IoT Core with something like Pub/Sub and write the data to Cloud Storage in order to later consume it with an application.
I would recommend you to check some of the diagrams from this page if you are interested in knowing which products you could use to best fit your needs.
I hope you find this useful!
I'm developing an app that has chat between users. Does it make sense to use Pub/Sub for this purpose? I couldn't find in the documentation as a possible use case. Are there any good reasons why it's not a good fit?
Google Cloud Pub/Sub would not be a good fit for communicating with end user devices in a chat product. Cloud Pub/Sub is designed for torrents: a relatively small number of long-lived streams that require high throughput. In a chat app, you need trickles: a very large number of ephemeral streams that have a low throughput.
Cloud Pub/Sub quotas only allow for 10,000 topics project and 10,000 subscriptions per topic or per project. When setting up a new streaming pipeline, one generally creates the topic and its subscriptions once at the beginning and then publishes and subscribes from that point on. Therefore, Cloud Pub/Sub is a better fit for server-to-server communication or for large-scale streaming event ingestion, often in connection with Dataflow and/or BigQuery.
The permission model for Cloud Pub/Sub doesn't make it a good choice for a chat app, either. Permission to write to or read from a topic or subscription requires the user to be authenticated. Authenticating each individual user with permissions in your project is not really feasible, so you are left with the option of using common credentials for all of your users, which could be insecure.
That's not to say that Cloud Pub/Sub couldn't be part of a chat service, it just wouldn't be used to deliver messages to specific users. In fact, Cloud Pub/Sub can be used as the transport mechanism for messages sent to Hangouts Chat Bots. In this situation, though, the response to the user goes directly to the Hangouts Chat API, not back through Pub/Sub. Therefore, if you were implementing a chat service, you could potentially have your users send messages to a frontend server you design that then publishes those messages to Cloud Pub/Sub. Another server you build could be a subscriber to those messages and then deliver the appropriate messages to the appropriate individual users. In this situation, Cloud Pub/Sub is used for ingesting all of the messages from your frontend server (torrent) and delivery of messages to individual users falls to some other mechanism (trickles). The product in Google Cloud designed for this individual device delivery is Firebase Cloud Messaging,
Can Google Pub/Sub be used for chat messaging?
Yes. I would not use Pub/Sub for person-to-person chat messaging. Pub/Sub is designed for systems to de-couple themselves to support distributed systems.
Why not use Pub/Sub:
Permissions. You will need to create service account credentials and distribute this Json file to each subscriber. This is not secure.
Each chat person will need a subscription to a Pub/Sub Topic and then poll that subscription. This means one subscription per chat person. This is neither a good design nor inexpensive.
There are much better solutions for chat.
Pub/Sub for chat? No. As it is rightly mentioned above, Pub/sub has 2 major use cases, for data ingestion and for decoupling the system.
I would recommend you search a bit about web sockets if are still interested making a chat app.
I'm looking for something like AWS's device management functionality (https://aws.amazon.com/iot-device-management/) that exists on Google's Firebase service.
I found this: https://cloud.google.com/community/tutorials/cloud-iot-firestore-config
But I'm not sure if I can achieve the same functionality as with AWS IoT Device Management (I'm still new to Firebase).
Is there a Firebase equivalent to AWS IoT Device Management, specifically regarding onboarding and organization?
There is no equivalent in Firebase.
As has been mentioned, no, there's nothing directly in Firebase.
Have you looked at Cloud IoT Core? It has a lot of the features you're probably looking for in a device manager. From there, there are data pipeline tools (like Google Cloud Functions, or Cloud Dataflow) which can ferry the telemetry data from your devices into Firebase (or, potentially a better option, Firestore).
I did a session at the Google Cloud Next conference this year (2018) which talked about the end-to-end IoT application, and we published the code we built on stage which includes IoT device management, and used Cloud Functions to push the data into Firestore if you're interested. Link can be found here. The Cloud Functions that moved the data into Firestore are in the gcf folder. The device folder has the Python code we used on device to send telemetry up to IoT Core.
Is Pub/Sub significantly faster way of communicating between, say, Kubernetes Engine (GKE) api server and a Cloud Function (GCF)?
Is it possible to use Pub/Sub to have such communication between GKE from one Google Cloud Project and GCF from another Google Cloud Project?
What is the way to communicate with Cloud Functions from another Google Cloud Project with low latency?
I think a global answer will clarify your questions. For this particular case, there are two ways to trigger a Google Cloud Function (GCF). You can directly make an HTTP request or you can subscribe the GCF to a topic by using Pub/Sub [https://cloud.google.com/functions/docs/calling ].
If your requests are occasional, an HTTP request will be faster because you don't need an intermediary. If that's not the case, then the Pub/Sub subscription queues the messages and ensures the delivery by retrying them until it receives confirmation.
To communicate between Google Kubernetes Engine (GKE) from one Google Cloud Project and Google Cloud Function (GCF) to another Google Cloud Project you can use either option. Trigger the GCF by the HTTP request directly or do it by publishing the message. When publishing, specify the project where you are sending it and the desirable topic in that project.
Also you need to give the proper permission to the service account to access from one project to the other:
For Pub/Sub https://cloud.google.com/pubsub/docs/authentication
For HTTP request
https://cloud.google.com/solutions/authentication-in-http-cloud-functions.
Google Cloud Function HTTP triggers documentation here: https://cloud.google.com/functions/docs/calling/http
Pub/Sub documentation here:
https://cloud.google.com/pubsub/docs/reference/libraries (you can
access to GitHub by the links in the code and see functions examples
for each language)