Jobs and Messages are both just transactions of text between AWS IoT service and devices.
Why should I use jobs than messages or the other way around?
They are transaction but they have their differences
Messages - The AWS IoT message broker is a publish/subscribe broker service that enables the sending and receiving of messages to
and from AWS IoT. The act of sending the message is referred to as
publishing. The act of registering to receive messages for a topic
filter is referred to as subscribing.
Example - When communicating with AWS IoT, a client sends a message addressed to a topic like Sensor/temp/room1. The message broker, in turn, sends the message to all clients that have registered to receive messages for that topic.
Jobs - AWS IoT jobs can be used to define a set of remote operations that are sent to and executed on one or more devices
connected to AWS IoT.
Example - you can define a job that instructs a set of devices to download and install application or firmware updates, reboot, rotate certificates, or perform remote troubleshooting operations.
To use Jobs or Messages is up to your requirements. If you want to update a set of devices Jobs seems to do the work, or its just one device message will do.
Related
I have two services, one is the producer (Service A) and one is a consumer (Service B). So Service A will produce a message which will be published to Amazon SQS service and then it will be delivered to Service B as it has subscribed to the queue. So, this works fine until I have a single instance of Service B.
But when I start another instance of Service B, such that now there are 2 instances of Service B, both of which are subscribing to the same queue, as it is the same service, I observe that the messages from SQS are now being delivered in round-robin fashion. Such that at a given time, only one instance of Service B receives the message that is published by Service A. I want that when a message is published to this queue, it should be received by all the instances of Service B.
How can we do this? I have developed these services as Springboot applications, along with Spring cloud dependencies.
Please see the diagram below for reference.
If you are interested in building functionality like this, use SNS, not SQS. We have a Spring BOOT example that shows how to build a web app that lets users sign up for email subscriptions and then when a message is published, all subscribed emails get the message.
The purpose of this example is to get you up and running building a Spring BOOT app using the Amazon Simple Notification Service. That is, you can build this app with Spring BOOT and the official AWS Java V2 API:
Creating a Publish/Subscription Spring Boot Application
While your message may appear to be read in a round robbin fashion, they are not actually consumed in a round robin. SQS works by making all messages available to any consumer (that has the appropriate IAM permissions) and hides the message as soon as one consumer fetches the message for a pre-configured amount of time that you can configure, effectively "locking" that message. The fact that all of your consumer seem to be operating in a round robin way is most likely coincidental.
As others have mentioned you could use SNS instead of SQS to fanout messages to multiple consumers at once, but that's not as simple a setup as it may sound. If your service B is load balanced, the HTTP endpoint subscriber will point to the Load Balancer's DNS name, and thus only one instance will get the message. Assuming your instances have a public IP, you could modify your app so that it self-registers as an HTTP subscriber to the topic when the application wakes up. The downsides here are that you're not only bypassing your Load Balancer, you're also losing the durability guarantees that come with SQS since an SNS topic will try to send the message X times, but will simply drop the message after that.
An alternative solution would be to change the message hiding timeout setting on the SQS queue to 0, that way the message is never locked and every consumer will be able to read it. That will also mean you'll need to modify your application to a) not process messages twice, as the same message will likely be read more than once by the time it has finished processing and b) handle failure gracefully when one of the instance deletes the message from the queue and other instances try to delete that message from the queue after that.
Alternatively, you might want to use some sort of service mesh, or service discovery mechanism so that instances can communicate between each other in a peer-to-peer fashion so that one instance can pull the message from the SQS queue and propagate it to the other instances of the service.
You could also use a distributed store like Redis or DynamoDB to persist the messages and their current status so that every instance can read them, but only one instance will ever insert a new row.
Ultimately there's a few solutions out there for this, but without understanding the use-case properly it's hard to make a hard recommendation.
Implement message fanout using Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS). There is a hands-on Getting Started example of this.
Here's how it works: in the fanout model, service A publishes a message to an SNS topic. Each instance of service B has an associated SQS queue which is subscribed to that SNS topic. The published message is delivered to each subscribed queue and hence to each instance of service B.
In AWS IOT we can make device subscribe to a topic. When a message is received on a topic, the device can be programmed to execute some code.
AWS IOT Jobs seems similar in that the device listens on the job and executes certain code when job is received.
How are AWS IOT Jobs different to Topic subscription?
The primary purpose of jobs is to notify devices of a software or
firmware update.
AWS IOT Job Doc
AWS IOT Events activities (like subscribing to a topic) would be the generic implementation for doing stuff when a device gets a message. IOT jobs are more of a managed workflow for doing a specific activity- like notifying devices of a firmware update and using CodeSigning.
Just want to add an important point to what #Bobshark wrote.
Yes, Amazon engineers implemented a set of endpoints to manage a whole job lifecycle on a single device and the process of gradually rolling out jobs over a fleet of devices.
However, IoT jobs are not tied down to using MQTT as the transport protocol. As the AWS docs [1] mention:
Devices can communicate with the AWS IoT Jobs service through these methods:
MQTT
HTTP Signature Version 4
HTTP TLS
My personal advice: Use jobs if you would have to implement your own update procedure (such as progress reporting, gradual rollouts, etc.) otherwise.
[1] https://docs.aws.amazon.com/iot/latest/developerguide/jobs-devices.html
I am using AWS IoT Service.
When a device sends a registration message to MQTT broker, I have a rule to store it in a SQS queue.
A Lambda function is triggered, when the message is added to the Queue. The Thing is created for the device and it's certificate is registered.
While carrying out the load testing, I observed that, after some time, the incoming messages are not received on the AWS MQTT broker and are not processed.
I have written some test clients which run on EC2 instances to simulate the MQTT clients.
If I restart the test clients after some time, again I can see the messages coming to AWS IoT.
I am not sure, if this is the issue of MQTT broker or if it is the issue with the clients running on EC2 instances.
I can think of possible issues because of limits on AWS IoT ,
https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot
I want to know what are the possible AWS IoT matrices, I need to monitor for this or which IoT specific alarms I need to configure?
Is it a possible issue on EC2 side? ( maybe network out bytes per second, etc.)
There is another load testing scenario, in which I am not doing registration of devices, but just capturing the connect or disconnect events. In this case, I am not observing similar issues.
As you know, there is some limits about AWS IoT.
API Transactions per Second
CreateCertificateFromCsr 15
CreateDynamicThingGroup 5
CreateJob 10
CreatePolicy 10
CreatePolicyVersion 10
CreateRoleAlias 10
CreateThing 15
Generally, AWS API throws Exception when it run over limts.
How about catch Exception?
If you want to check EC2 network issue, use some command ( netstat, tcpdump, ... )
I have made two ec2 instances listening to one particular aws sqs queue. Now both the instances are connected via load balancer. The problem is , as both the instances are listening on the same queue, isn't the message going to be received by both the instances at the same time. If it is received on both the instances at the same time, how to prevent duplication of the transactions?
Your architecture does not describe a normal usage pattern for Amazon SQS.
Amazon Simple Queue Service (SQS) is a queueing service where you can create a queue and send messages to a queue. Amazon SQS retains the message for up to 14 days.
Applications can then request to receive a message from the queue. This makes the message invisible while the application is processing the message. Once the application finishes processing the message, to tells SQS to delete the message from the queue. If the application fails while processing a message, Amazon SQS will make the message visible again after a period of time so that it can be processed by another application server.
Things to note:
SQS does not send messages. Rather, applications request to receive a message from the queue. Think of the apps as "retrieving" a message rather than SQS transmitting a message.
The instances retrieving the messages should not be sitting behind a Load Balancer. There is nothing sent to a Load Balancer in this process. Instead, the instances themselves connect to SQS and request a message.
If you wish to send a message to multiple systems at the same time, you can use Amazon Simple Notification Service (SNS).
I used Aws IoT Device management to register my laptop as a device and run a node js script on my laptop.
My laptop will publish the message to a topic and it also can subscribe to a topic.
But what I want is, aws cloud send msg to my deivce( which is my laptop), periodically, if my laptop doesn't give the responses back to the cloud, cloud will know that my device is dead.
But how to send msg to my device and wait for the response? I read the tutorial about the jobs and shadow in aws IoT device management, none of them are sending something to device, they are just existing on the cloud and device will go and grab them.
How to make aws IoT device management automatically send msg or publish the msg to the device? All I know is go to the "Test" page and publish to the topic the manually.
Okay, so you're trying to use AWS IoT to create a monitoring system to detect if your device (laptop) is alive or not.
Sending data from the cloud -> laptop is not needed. You can do this by only sending data from your laptop -> cloud.
Configure the Node.JS script on your laptop to periodically send messages to a topic in AWS IoT. Configure an AWS IoT Rule on that topic that will publish a metric to AWS CloudWatch. This metric will represent your laptop sending a heartbeat to the cloud.
You can additionally configure an alarm in AWS CloudWatch to watch this metric and perform an action if the alarm threshold is ever breached. For your use case you can probably set the alarm to breach when it's missing data points because that'll mean your laptop has stopped sending messages.