I have two result topics where messages arrive from different sources "almost nearly at the same time".
Topic: sensor1/result --receiving payload--> { "output_from_sensor1": {"result":"OK"} }
Topic: sensor2/result --receiving payload--> { "output_from_sensor2": {"result":"OK"} }
I would like to create an AWS IoT Rule which scans these two topics "simultaneously in one query" and take an action.
I am not sure if AWS IoT SQL support "scanning multiple topics" in one query. No such references found in AWS docs.
During the way, I have tried these IoT queries (from my knowledge of SQL syntax) but no luck so far :(
SELECT output_from_sensor1.result AS final_output.result FROM ‘sensor1/result’ WHERE (SELECT output_from_sensor2.result FROM ‘sensor2/result’)=‘OK’
(SELECT output_from_sensor1.result FROM 'sensor1/result') UNION (SELECT output_from_sensor2.result FROM 'sensor2/result')
Thanks much!
AWS IoT rules are triggered by a single MQTT message and the rule actions only process the message that triggered the rule. So while the + and # Wildcards can be used to select from multiple topics, each invocation of the rule only handles one message.
Your assumption that it is possible to 'scan multiple topics' in one query implies that multiple messages are involved (to each topic).
Depending on the problem you are trying to solve, it may make sense to buffer the messages in a queue (e.g. SQS). The processing can then check if multiple messages appear in a given time window to perform a single action on both messages.
I am not sure if AWS IoT SQL support "scanning multiple topics" in one query. No such references found in AWS docs.
I haven't found a definitive statement in the documentation that rules this out. But the wording is consistent with a rule being triggered by one message.
e.g. From the rules tutorial
The rule is triggered when an MQTT message that matches the topic filter is received on a topic.
The FROM Clause subscribes the rule to a topic or topic filter using the MQTT + and # wildcards.
There are operators like AND and OR but these are not used in the FROM clause. The operators documentation states:
The following operators can be used in SELECT and WHERE clauses.
Related
I'm using the Serverless Framework and I need to subscribe my Lambda function to an AWS Iot Jobs topic, more specifically the "start-next/#" topic. I already have another Lambda function subscribed to another topic of the AWS Iot tools, using this piece of code in the serverless.yml file:
events:
- iot:
sql: "SELECT topic() AS topic, * FROM '$aws/things/+/shadow/update'"
This works just fine, the Lambda function is triggered and I'm able to process what I need, but when I try to use this piece of code:
events:
- iot:
sql: "SELECT topic() AS topic, * FROM '$aws/things/+/jobs/start-next/#'"
it simply will not work no matter what I try to do.
Could someone give me any sugestions of what I can do to trigger my Lambda function when new messages are published to the '$aws/things/+/jobs/start-next' topic?
Using the wildcard # at the end of the query you make your rule subscribes to any topic beginning with
'$aws/things/thingName/jobs/start-next/[...]'
'$aws/things/thingName/jobs/start-next/[...]/[...]'
'...'
If you are publishing a message to a topic like this by any chance (with nothing after /start-next)
$aws/things/thingName/jobs/start-next
Then it won't match your rule. In such scenario, you should query without the '#' wildcard at the end. Something like this
'sql: "SELECT topic() AS topic, * FROM '$aws/things/+/jobs/start-next'"'
You can find more details about such wildcards and its coverage right here
So... Reading the MQTT Client Help on the IOT page, I've actually found this part:
Reserved topics
Topics beginning with $ are considered reserved and are not supported for publishing and
subscribing except when working with the Thing Shadows service. To learn more,
see Thing Shadows.
which would explain why the shadows topic triggers the Lambda as I mentioned in the question, but not my other Lambda based on the jobs topics.
I guess I'll have to find another way to make this work, but thanks to who tried to help :)
I want to publish a notification using SNS and I want subscribers to be able to filter on multiple message attribute(s). One of such message attribute is going to be a String.Array. For example, the notification can have two attributes fruit_found and all_fruits_found.
"fruit_found": ["Apple"],"all_fruits_found":["Mango","Apple","Banana"]
There can be use cases where a subscriber might need to know if both Mango and Apple were found and only then consume the notification else drop it. Is it possible to do so in SNS?
So I had to talk to the SNS customer support team and found out that they don't have AND operation within a String.array message attributes.
A workaround that I found was to replicate the same message attributes for the number of filters you want to provide. For the message in the question, it should have structure like:
"fruit_found": ["Apple"],
"all_fruits_found_filter_1":["Mango","Apple","Banana"],
"all_fruits_found_filter_2":["Mango","Apple","Banana"]
The filter policy defined for when both Mango and Apple are found would be:
"all_fruits_found_filter_1": ["Mango"] //and
"all_fruits_found_filter_2": ["Apple"]
However, there is a limitation of at max 10 message attributes per SNS message. So if you are within that boundary the above solution works fine. Else you would have to refer to the answer from Ali.
You cannot achieve this using SNS alone you might require a lambda function to receive SNS message and separate it based on the string and publish again to a topic.
You might need to create three SNS topic:
For lambda processing
For Mango subscribers
For Apple subscribers
Simple question, but I suspect it doesn't have a simple or easy answer. Still, worth asking.
We're creating an implementation for push notifications using AWS with our Web Server running on EC2, sending messages to a queue on SQS, which is dealt with using Lambda, which is sent finally to SNS to be delivered to the iOS/Android apps.
The question I have is this: is there a way to query SNS endpoints based on the custom user data that you can provide on creation? The only way I see to do this so far is to list all the endpoints in a given platform application, and then search through that list for the user data I'm looking for... however, a more direct approach would be far better.
Why I want to do this is simple: if I could attach a User Identifier to these Device Endpoints, and query based on that, I could avoid completely having to save the ARN to our DynamoDB database. It would save a lot of implementation time and complexity.
Let me know what you guys think, even if what you think is that this idea is impractical and stupid, or if searching through all of them is the best way to go about this!
Cheers!
There isn't the ability to have a "where" clause in ListTopics. I see two possibilities:
Create a new SNS topic per user that has some identifiable id in it. So, for example, the ARN would be something like "arn:aws:sns:us-east-1:123456789:know-prefix-user-id". The obvious downside is that you have the potential for a boat load of SNS topics.
Use a service designed for this type of usage like PubNub. Disclaimer - I don't work for PubNub or own stock but have successfully used it in multiple projects. You'll be able to target one or many users this way.
According the the [AWS documentation][1] if you try and create a new Platform Endpoint with the same User Data you should get a response with an exception including the ARN associated with the existing PlatformEndpoint.
It's definitely not ideal, but it would be a round about way of querying the User Data Endpoint attributes via exception.
//Query CustomUserData by exception
CreatePlatformEndpointRequest cpeReq = new CreatePlatformEndpointRequest().withPlatformApplicationArn(applicationArn).withToken("dummyToken").withCustomUserData("username");
CreatePlatformEndpointResult cpeRes = client.createPlatformEndpoint(cpeReq);
You should get an exception with the ARN if an endpoint with the same withCustomUserData exists.
Then you just use that ARN and away you go.
When receiving events in Lambda from SNS the outer structure of the event will look somewhat like:
{ "Records": [...] }
In all the tutorials I have seen there has only ever been 1 record in the records field.
Is it safe to make the assumption that the "Records"-array will only ever contain 1 item?
Each SNS notification will contain not more than one message.
Refer Reliability section in SNS FAQ : https://aws.amazon.com/sns/faqs/
Having said that, each lambda function trigger will have just a single record
Just ran into the same thing, and for posterity, I think it is worth adding that in the official AWS tutorial, they rely on there being exactly one record:
https://docs.aws.amazon.com/lambda/latest/dg/with-sns-create-package.html
I have a system in which I get a lot of messages. Each message has a unique ID, but it can also receives updates during its lifetime. As the time between the message sending and handling can be very long (weeks), they are stored in S3. For each message only the last version is needed. My problem is that occasionally two messages of the same id arrive together, but they have two versions (older and newer).
Is there a way for S3 to have a conditional PutObject request where I can declare "put this object unless I have a newer version in S3"?
I need an atomic operation here
That's not the use-case for S3, which is eventually-consistent. Some ideas:
You could try to partition your messages - all messages that start with A-L go to one box, M-Z go to another box. Then each box locally checks that there are no duplicates.
Your best bet is probably some kind of database. Depending on your use case, you could use a regular SQL database, or maybe a simple RAM-only database like Redis. Write to multiple Redis DBs at once to avoid SPOF.
There is SWF which can make a unique processing queue for each item, but that would probably mean more HTTP requests than just checking in S3.
David's idea about turning on versioning is interesting. You could have a daemon that periodically trims off the old versions. When reading, you would have to do "read repair" where you search the versions looking for the newest object.
Couldn't this be solved by using tags, and using a Condition on that when using PutObject? See "Example 3: Allow a user to add object tags that include a specific tag key and value" here: https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html#tagging-and-policies