One of the core features of AWS IOT is to provide bi-directional communication through the device gateway. Knowing that the implementation is private, can anyone conceptually explain how the gateway is able to push messages to a device on my local network? Does the service maintain a persistent connection to the device?
To add context, in my experience prior to AWS IOT, if I wanted to communicate with my embedded app (using HTTP) I would need to set up port-forwarding, so it's a mystery to me why device gateway could work without this step.
AWS IOT is implemented with MQTT, an MQTT connection is permanent TCP connection from the device to the gateway (broker), meaning that when there is a need to send a message to the device this connection is used.
Because the connection is initiated from the device to the gateway there is no need to set up any port forwarding (apart from that which is automatically handled by a home router doing NAT)
AWS IoT, uses the Pub/Sub pattern to provide communication between devices and message broker.
Publish–subscribe is a messaging pattern where senders of messages,
called publishers, do not program the messages to be sent directly to
specific receivers, called subscribers, but instead categorize
published messages into classes without knowledge of which
subscribers, if any, there may be. Similarly, subscribers express
interest in one or more classes and only receive messages that are of
interest, without knowledge of which publishers, if any, there are.
More about this pattern here: https://msdn.microsoft.com/en-us/library/ff649664.aspx
Related
I need to connect my mqtt-spy client to mqtt.googleapis.com with rsa_public.pem and root.ca
What configuration parameters should be used?.
I attempted many possible combinations of options none worked.
That endpoint is an MQTT bridge used by Cloud IoT. It is extensively documented and I encourage you to review the docs:
https://cloud.google.com/iot/docs/how-tos/mqtt-bridge
It's possible Google applies constraints to the bridge to focus its use to client devices interacting with Cloud IoT so it may not present with 100% fidelity as an MQTT broker but, you should be able to configure mqtt-spy to connect to it as long as you're able to present as a device with certs.
I am working on a project wherein a mobile app (Flutter app) should create a unique identity (IOT thing) of my IOT device machine (on AWS) as soon as the user logins onto the app and also it should establish a connection (and subsequent communications) with the isolated IOT device after its configuration (Somewhat similar to Google Home configuration process).
The process starts with user logging onto the app (using AWS Amplify/Congnito to automatically add the user account in the cloud). Subsequently, the app needs to configure the IOT device (ESP32) and send Wifi credentials of home network by connecting to ESP's WiFi AP. However, since both the app and IOT device are isolated from each other (except during the configuration process), I don't have much idea on how exactly will mobile app connect (creating a unique IOT thing automatically) and communicate with the isolated IOT device over AWS IOT cloud.
I have worked with some of the AWS services like AWS Lambda, DynamoDB, AWS IOT for quite a while, hence I am aware of the basics. One of the possible solutions for the above maybe Fleet Provisioning by a Trusted User feature provided by AWS IOT (I did read it's documentation, but still confused). We can also use ESP32's Bluetooth feature for the app to scan nearby devices with which it needs to connect with, but I am not sure how will the connection establish through AWS cloud.
Since I am a beginner to AWS cloud services, it'd be great if someone can provide a possible solution for the above in detail. Also if possible, please provide a solution which would work incase of a large-scale deployment of the above project.
The parts you may be missing are missing are 1) MQTT messaging, which is the IoT message broker service provided by IoT Core, and 2) the need for an application program interface to handle creating and activating devices.
MQTT is a lightweight and widely adopted messaging protocol that is designed for constrained devices.
Your devices (things) will need to communicate back to AWS via MQTT messages, which are captured and processed by the IoT Rules service.
These rules typically trigger Lambda functions, which implement the process logic you need for your application.
See https://mqtt.org/ and https://docs.aws.amazon.com/iot/latest/developerguide/what-is-aws-iot.html
2} you will want to implement a simple API to handle device activation. The AWS API Gateway service makes it easy to implement APIs. For example, you can implement an endpoint for POST /device/ to create a device. That endpoint can trigger a Lambda handles device provisionsing.
My Agents running on various environments/devices are going to drop periodic messages from public network.
These messages will be processed by my AWS Lambda.
The systems are asynchronous.
I am thinking of using SQS to feed the Lambda.
Just that, SQS endpoint will be open to internet.
How can I validate the messages posted on AWS SQS.
Most of the devices/agents pushing messages will be on customer VPN. So, establishing a private-vpn-link is a possible solution.
You shoudn't make an SQS queue public so that anyone without AWS credentials could use it. Its not a good practice.
A better option is to uses API gateway in front of your SQS queue:
Creating an AWS Service Proxy for Amazon SQS
This way you can make your API gateway endpoint public, control its throughput, limits, throttling, access using API keys, and more.
The API gateway would be integrated with your SQS queue which would allow you to trigger your lambda function.
With the use of API keys or lambda authorizes you will be able to control access of your devices/agents to the API gateway, and subsequently, to the SQS.
The Amazon SQS endpoint is on the Internet not in a VPC. There is no benefit to using a VPN connection, unless you wish to create a policy that only allows messages to be sent to the queue if they come via the VPN connection.
Your use-case might be better-suited for AWS IoT rather than Amazon SQS:
It is designed to handle large quantities of data some from various devices, with in-built rules for handling the messages.
If you're looking to minimise your messages traversing the public internet space the you have a few options that are available to you.
Firstly, as you mention you have devices that can connect via a VPN, if this is possible you can create a VPC endpoint for SQS using a custom domain name.
In your application by referring to the custom domain name or a private IP address from one of the ENIs it might be possible to actually interact through the VPC privately.
You could enhance your security checking by adding a proxy layer in front to validate the input to perform any checks against the content. Your application would pass to this validation layer, and then add to the queue if it passes.
One thing worth pointing out is even with VPN the traffic is traversing the internet, however it is encrypted in transit. The best line of security is combining HTTPS alongside private credentials (preferably as an IAM role) to reduce the chance of someone being able to add improper data to your queue.
Say I have 1000 IoT devices. Each device sends telemetry data to the Azure cloud each second. There is mobile app which displays that telemetry data. Mobile user can see data from specific IoT device only, based on his login. It's not an issue to receive all those d2c messages in the cloud, for example with Event Hubs (initially i wanted to use IoT Hub, but it's pretty expensive with huge amount of messages). It's also clear how to get general event stream (for example, by listening to events endpoint in IoT hub). However, this stream gives all messages, from all devices. Receiving all messages in mobile app and filtering them there does not look like right choice. How can I subscribe to events from specific IoT device only? Can I do this with Event Hubs and/or IoT Hub only, or do I need to utilize other Azure services? If yes, what will be the most cheap solution.
Update:
Original question was just about Azure. But is it possible to achieve what I need with AWS without requiring adding multiple parts to the system. Is Message Broker for AWS IoT what I need here (as I understood, this is built in functionality of AWS IoT Core)?
I'm trying to create a collaborative web application where multiple users can work together on various (shared) projects. So far I have a JavaScript client and one local jWebSocket server.
To remain scalable upon deployment, I thought of two options:
Option 1
I can use AWS IoT instead of multiple jWebSocket servers. Publishing changes of a project is easy, I would just need to publish to e.g. /project/{project-id}. But how would the traditional request-response mechanism work?
The Problem: EC2 instances handling requests would be reachable by publishing to distinct topics (e.g. /server/1). But when the JS client connects to AWS IoT, it does not know of any EC2 instance to send requests to. How could I assign each client to an instance/topic?
Option 2
Run jWebSocket servers on multiple EC2 instances behind an AWS Application Load Balancer. The balancer would simply assign each client to a server and the traditional request-response flow would not be a problem. But what about pushing changes?
The Problem: Because each server has its own set of connected clients, it can not push changes to clients connected to another server.
Remarks
Mixing jWebSocket to send requests to and AWS IoT to receive events from seems like a sloppy solution.
I assume I can programmatically adapt the IoT policies per cognito identity to allow/deny the subscription to specific projects.
Using AWS Lambda and relinquishing servers altogether is not an option due to the high latency introduced by Lambda (if you've made different experiences, please share).
Related posts
IoT request response protocol
Thanks for any thought you could give me on this issue.
I've got it. The first suggestion in this question pointed me into the right direction. The solution allows all clients to maintain a direct WebSocket connection to the server they originally connected to, without subscribing to specific topics.
It works as follows:
When a client connects to a server, the server subscribes to the client's channel
If a server needs to send a message to a client that is not connected, it publishes that message to the client's channel
(you guessed it) The server that is subscribed to the channel can then process the message on the first server's behalf
"Pusher" in the diagram describes this SaaS, but can of course be replaced by any other messaging service.