I am using AWS IoT Service.
When a device sends a registration message to MQTT broker, I have a rule to store it in a SQS queue.
A Lambda function is triggered, when the message is added to the Queue. The Thing is created for the device and it's certificate is registered.
While carrying out the load testing, I observed that, after some time, the incoming messages are not received on the AWS MQTT broker and are not processed.
I have written some test clients which run on EC2 instances to simulate the MQTT clients.
If I restart the test clients after some time, again I can see the messages coming to AWS IoT.
I am not sure, if this is the issue of MQTT broker or if it is the issue with the clients running on EC2 instances.
I can think of possible issues because of limits on AWS IoT ,
https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot
I want to know what are the possible AWS IoT matrices, I need to monitor for this or which IoT specific alarms I need to configure?
Is it a possible issue on EC2 side? ( maybe network out bytes per second, etc.)
There is another load testing scenario, in which I am not doing registration of devices, but just capturing the connect or disconnect events. In this case, I am not observing similar issues.
As you know, there is some limits about AWS IoT.
API Transactions per Second
CreateCertificateFromCsr 15
CreateDynamicThingGroup 5
CreateJob 10
CreatePolicy 10
CreatePolicyVersion 10
CreateRoleAlias 10
CreateThing 15
Generally, AWS API throws Exception when it run over limts.
How about catch Exception?
If you want to check EC2 network issue, use some command ( netstat, tcpdump, ... )
Related
In AWS IOT we can make device subscribe to a topic. When a message is received on a topic, the device can be programmed to execute some code.
AWS IOT Jobs seems similar in that the device listens on the job and executes certain code when job is received.
How are AWS IOT Jobs different to Topic subscription?
The primary purpose of jobs is to notify devices of a software or
firmware update.
AWS IOT Job Doc
AWS IOT Events activities (like subscribing to a topic) would be the generic implementation for doing stuff when a device gets a message. IOT jobs are more of a managed workflow for doing a specific activity- like notifying devices of a firmware update and using CodeSigning.
Just want to add an important point to what #Bobshark wrote.
Yes, Amazon engineers implemented a set of endpoints to manage a whole job lifecycle on a single device and the process of gradually rolling out jobs over a fleet of devices.
However, IoT jobs are not tied down to using MQTT as the transport protocol. As the AWS docs [1] mention:
Devices can communicate with the AWS IoT Jobs service through these methods:
MQTT
HTTP Signature Version 4
HTTP TLS
My personal advice: Use jobs if you would have to implement your own update procedure (such as progress reporting, gradual rollouts, etc.) otherwise.
[1] https://docs.aws.amazon.com/iot/latest/developerguide/jobs-devices.html
I've got rather rare requirement to deliver SNS topic message to all micro service instances.
Basically it's kind of notification that related data had changed
and all micro service instances should reload their internals from data source.
We are using TerraForm to create our infrastructure, with Kong api gateway.
Micro Service instances could be created 'on the fly' as system load is increased,
so subscriptions to topic could not be created in TerraForm stage.
Micro Service is standard SpringBoot app.
My first approach is:
micro service is exposing http endpoint that can be subscribed to SNS topic
micro service on start will subscribe itself (above endpoint) to required SNS topic, unsubscribe on service shutdown.
My problem is to determine individual micro service instances urls, that can be used in subscription process.
Alternative approach would be to use SQS, create SQS queue per micro srv instance (subscribe it to sns).
Maybe I'm doing it wrong on conceptual level ?
Maybe different architecture approach is required ?
It might be easier for the microservices to check an object in Amazon S3 to "pull" the configuration updates (or at least call HeadObject to check if the configuration has changed) rather than trying to "push" the configuration update to all servers.
Or, use AWS Systems Manager Parameter Store and have the servers cache the credentials for a period (eg 5 minutes) so they aren't always checking the configuration.
Kinda old right now but here is my solution:
create SNS, subscribe with SQS, publish the SQS to redis pub/sub, subscribe to pub/sub
now all your instances will get the event.
Jobs and Messages are both just transactions of text between AWS IoT service and devices.
Why should I use jobs than messages or the other way around?
They are transaction but they have their differences
Messages - The AWS IoT message broker is a publish/subscribe broker service that enables the sending and receiving of messages to
and from AWS IoT. The act of sending the message is referred to as
publishing. The act of registering to receive messages for a topic
filter is referred to as subscribing.
Example - When communicating with AWS IoT, a client sends a message addressed to a topic like Sensor/temp/room1. The message broker, in turn, sends the message to all clients that have registered to receive messages for that topic.
Jobs - AWS IoT jobs can be used to define a set of remote operations that are sent to and executed on one or more devices
connected to AWS IoT.
Example - you can define a job that instructs a set of devices to download and install application or firmware updates, reboot, rotate certificates, or perform remote troubleshooting operations.
To use Jobs or Messages is up to your requirements. If you want to update a set of devices Jobs seems to do the work, or its just one device message will do.
I used Aws IoT Device management to register my laptop as a device and run a node js script on my laptop.
My laptop will publish the message to a topic and it also can subscribe to a topic.
But what I want is, aws cloud send msg to my deivce( which is my laptop), periodically, if my laptop doesn't give the responses back to the cloud, cloud will know that my device is dead.
But how to send msg to my device and wait for the response? I read the tutorial about the jobs and shadow in aws IoT device management, none of them are sending something to device, they are just existing on the cloud and device will go and grab them.
How to make aws IoT device management automatically send msg or publish the msg to the device? All I know is go to the "Test" page and publish to the topic the manually.
Okay, so you're trying to use AWS IoT to create a monitoring system to detect if your device (laptop) is alive or not.
Sending data from the cloud -> laptop is not needed. You can do this by only sending data from your laptop -> cloud.
Configure the Node.JS script on your laptop to periodically send messages to a topic in AWS IoT. Configure an AWS IoT Rule on that topic that will publish a metric to AWS CloudWatch. This metric will represent your laptop sending a heartbeat to the cloud.
You can additionally configure an alarm in AWS CloudWatch to watch this metric and perform an action if the alarm threshold is ever breached. For your use case you can probably set the alarm to breach when it's missing data points because that'll mean your laptop has stopped sending messages.
I'd like to use AWS IoT to manage a grid of devices. Data by device must be sent to a queue service (RabbitMQ) hosted on an EC2 instance that is the starting point for a real time control application. I read how to make a rule to write data to other Service: Here
However there isn't an example for EC2. Using the AWS IoT service, how can I connect to a service on EC2?
Edit:
I have a real time application developed with storm that consume data from RabbitMQ and puts the result of computation in another RabbitMQ queue. RabbitMQ and storm are on EC2. I have devices producing data and connected to IoT. Data produced by devices must be redirected to the queue on EC2 that is the starting point of my application.
I'm sorry if I was not clear.
The AWS IoT supports pushing the data directly to other AWS services. As you have probably figured out by now publishing to third party APIs isn't directly supported.
From the choices AWS offers Lambda, SQS, SNS and Kinesis would probably work best for you.
With Lambda you could directly forward the incoming message using the one of Rabbit MQs APIs.
With SQS you would put it into an AWS queue first and than poll this queue transfering it to RabbitMQ.
Kinesis would allow more sophisticated processing, but is probably too complex.
I suggest you program a Lamba with the programming language of your choice using one of the numerous RabbitMQ APIs.