I'm trying to use Google Cloud Platform to make a little IoT project.
I've created a registry and a device in the "IoT Core" section and connected the registry to a default topic.
I've also specified three subfolders for that topic: "events", "config" and "status".
Now, I would like to connect a "Cloud Function" for the incoming messages but I can't find how to configure a single subfolder to monitor neither how to publish messages on them in the "Cloud Pub/Sub" section.
All the documentation talks about single topics so.... Am I missing some base concept on how it works?
Need to back up a step. What are you trying to do with the various subfolders? This may be a misunderstanding with how communication works with the device to and from the Cloud.
So, there's 3 MQTT topics, those are the events/config/state (not status). Those DON'T map to Pub/Sub topics at all. They're each handled separately in IoT Core.
Events is device->Cloud, and gets put into the Pub/Sub topic you specify when creating the registry. Setting subfolders is all about splitting telemetry from the same device to multiple places for handling. So, e.g. you have temp data you want to go in one pub/sub topic, and pressure data you want into another. The other way to handle this would be to attach the function to the main pub/sub topic, parse the telemetry, and then re-issue a Pub/Sub message to different places depending on the payload itself.
Config is Cloud->device, and is initiated by calls to the IoT Core Admin SDK. If your device subscribes to the /config/ MQTT topic, then they'll get a callback on the MQTT client's on_message handler (exact code depends on library being used of course) when some external entity sends a config message to IoT Core for your device.
State is device->Cloud but is specially handled, and doesn't go to any Pub/Sub topic. The results can be retrieved by the IoT Core Admin SDK. It's a way for the device to report its state and then external processes/applications to get that status without having to call back down to the device itself. This is particularly useful when you have devices which don't remain connected, for example, but you still want to be able to do things based on the last known state.
So the only one that you can use Cloud Functions with is the /events/ topic. That's done by deploying a Cloud Function and choosing the Pub/Sub publish event hook as the firing mechanism for the function, and specifying the IoT Core's registry Pub/Sub topic as the source of the events. Then anytime your device publishes telemetry to the /events/ MQTT topic, it'll get published to that Pub/Sub topic (confusing I know because we call them both topics) and the Cloud Function will fire.
Hopefully this clarifies what's going on? Or did I totally miss the question? :)
I believe the correct way to set this up would be to set up one pubsub topic for each subfolder. From the cloud iot core documentation:
Devices can publish data to separate Pub/Sub topics by specifying a subfolder in the MQTT topic. The subfolder is the subtopic after {device-id}/events. For example, if the device publishes to the MQTT topic /devices/{device-id}/events/alerts, the subfolder is the string alerts. This subfolder must be configured in the device registry resource's eventNotificationConfigs.subfolderMatches field with a matching Pub/Sub topic in the eventNotificationConfigs.pubsubTopicName field. When data is sent to a subfolder, it is published to the subfolder's matching Pub/Sub topic.
https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events_to_separate_pubsub_topics
Related
Can someone verify for me what I believe I've discovered about Google IOT?
iotClient.sendCommandToDevice(...) goes straight to the MQTT broker, and does not seem to involve itself with PubSub. PubSub Subscriptions seem to only work with telemetry from the device and subtopics thereof. We would like to have some process (perhaps a firebase function triggered on the topic) see all commands going to all devices, and have not been able to do this.
Connecting to the MQTT broker directly requires a public/private keypair associated with a single device ID, and one apparently cannot subscribe to messages to other devices (for example, one cannot subscribe to "/devices/+/commands/#" or just "#".
We could certainly create a function that duplicated the data, sending it to the device and to a PubSub topic, but we'd much rather have something that more reliably connected to all commands going to devices without requiring that they go through our function.
I am trying to send IoT commands using a push subscription. I have 2 reasons for this. Firstly, my devices are often on unstable connections so going through the pubsub let me have retries and I don't have to wait the QoS 1 timeout (I still need it because I log it for later use) at the time I send the message. The second reason is the push subscription can act as a load balancer. To my understanding, if multiple consumers listen to the same push subscription, each will receive a subset of the messages, effectively balancing my workload. Now my question is, this balancing is a behavior I observed on pull subscriptions, I want to know if:
Do push subscription act the same ?
Is it a reliable way to balance a workload ?
Am I garanteed that these commands will be executed at most once if there is, lets say, 15 instances listening to that subscription ?
Here's a diagram of what I'm trying to acheive:
Idea here is that I only interact with IoT Core when instances receive a subset of the devices to handle (when the push subscription triggers). Also to note that I don't need this perfect 1 instance for 1 device balancing. I just need the workload to be splitted in a semi equal manner.
EDIT: The question wasn't clear so I rewrote it.
I think you are a bit confused about the concepts behind Pub/Sub. In general, you publish messages to a topic for one or multiple subscribers. I prefer to compare Pub/Sub with a magazine that is being published by a big publishing company. People who like the magazine can get a copy of that magazine by means of a subscription. Then when a new edition of that magazine arrives, a copy is being sent to the magazine subscribers, having exactly the same content among all subscribers.
For Pub/Sub you can create multiple push subscriptions for a topic, up to the maximum of 10,000 subscriptions per topic (also per project). You can read more about those quotas in the documentation. Those push subscriptions can contain different endpoints, in your case, representing your IoT devices. Referring back to the publishing company example, those push endpoints can be seen as the addresses of the subscribers.
Here is an example IoT Core architecture, which focuses on the processing of data from your devices to a store. The other way around could also work. Sending a message (including device/registry ID) from your front-end to a Cloud Function wrapped in API gateway. This Cloud Function then publishes the message to a topic, which sends the message to a cloud Function that posts the message using the MQTT protocol. I worked out both flows for you that are loosely coupled so that if anything goes wrong with your device or processing, the data is not lost.
Device to storage:
Device
IoT Core
Pub/Sub
Cloud Function / Dataflow
Storage (BigQuery etc.)
Front-end to device:
Front-end (click a button)
API Gateway / Cloud Endpoints
Cloud Function (send command to pub/sub)
Pub/Sub
Cloud Function (send command to device with MQTT)
Device (execute the command)
I am currently working on an IoT device using AWS IoT core. I am new to working with IoT device. What is the standard/best way for determining whether the device is online and connected to the internet?
Thanks you!
Since you have been using AWS IoT Core, I would recommend that you stay in fully managed services provided by AWS IoT suite. No need to reinvent the wheel such as provisioning a separate database for a basic requirement of pretty much every IoT-enabled solution.
What I understand is that you want to monitor your IoT device fleets for state changes or failures in operation, and to trigger actions when such events occur. To address this challenge, I'd suggest using AWS IoT Events. It accepts inputs from many different IoT telemetry data sources including smart sensors, edge devices, management applications, and other AWS IoT services. You can easily push any telemetry data input to AWS IoT Events by using a standard API interface.
In specific to device heartbeat, please take a look at this sample detector model. A detector model simply represents your equipment or process. On the console, you can find some other pre-made detector model templates which you can customize based on your use-case.
One way to know if a device is online is to check for a heartbeat.
A device heartbeat is a small mqtt message to a topic that the device sends every 5 minutes.
In IoT Core, you would configure a rule that would update a Dynamodb table with a timestamp each time a message is sent to the heartbeat topic.
By checking this timestamp in Dynamodb, you can confirm if your device is currently online.
You can follow this Developer Guide to get connect disconnect events. it works on MQTT topics so we can use rules to trigger Lambda or other services.
I am using google IoT core and pubsub services for my IoT devices. I am publishing data using pubsub to the database. but I think its quite expensive to store every data into the database. I have some data like if the device is on or off and a configuration file which has some parameter which I need to process my IoT payload. Now I am not able to understand if configuration and state topic in IoT is expensive or not? and how long the data is stored in the config topic and is it feasible that whenever the parameter is changed in the config file it publish that data into config topic? and what if I publish my state of a device that if it is online or not every 3 seconds or more into the state topic?
You are mixing different things. There is Cloud IoT, where you have a device registry, with metadata, configuration and states. You also have PubSub topic in which you can publish message about IoT payload that can contain configuration data (I assume that is that you means in this sentence: "it publish that data into config topic").
In definitive it's simple.
All the management operations on Cloud IoT are free (device registration, configuration, metadata,...). There is no limitation and no duration limit. The only one which exists in the quotas for rate limit and configuration size.
The inbound and outbound traffic from and to the IoT devices is billed as described here
If you use PubSub for pushing your messages, Cloud Functions (or Cloud Run, or other compute option), a database (Cloud SQL or Datastore/Firestore), all these services are billed as usual, there is no relation with Cloud IoT service & billing. The constraints of each services are applied as a regular usage. For example, a PubSub message live up to 7 days (by default) in a subscription and until it hasn't acknowledged.
EDIT
Ok, got it, I took time to understood what you wanted to achieve.
The state is designed for getting the internal representation of the devices, but the current limitation doesn't allow you to update it automatically when you received message.
You have 2 solutions:
Either you can update your devices and send an update message only when its state changes (it's for this kind of use case that the feature is designed!)
Or, let the device published the messages every 3 seconds, but in the event PubSub topic. Get the events in a function which get the state list, get the first one (the most recent) and compare the value with the PubSub message. If different, update the state. This workflow also work with external database like Datastore or Firestore.
Is it possible to have the Video Intelligence API respond on a pub/sub topic? It makes no sense to have a cloud function (or something else) waiting for minutes until the video processing operation completes.
The documentation says you can specify an output_uri for the bucket location and have a topic on that specific bucket, but that may add extra delay.
Also the longrunning package mentions something generic on managing operations on Pub/Sub.
Thanks,
You can specify outputUri, as you already mentioned, as described in documentation. Now, Google Cloud Storage provides option to send notifications about object changes/inserts/deletions to PubSub topic. It's easy to setup, then your downstream app is just plug-and-play on PubSub topic with all info required.