I'm trying to set an AWS IOT rule to send data to DynamoDB without the help of a lambda.
My rule query statement is : SELECT *, topic() AS topic, timestamp() AS timestamp FROM '+/#'
My data is fine in AWS IOT as I'm successfully retrieving it with a lambda. However, even by following the developer guide to create the rule, in order to get the information passed on to Dynamo, by setting the 2 form fields with ${topic} and ${timestamp} as it should work, I get nothing in DynamoDB and I can find the following exception in Cloudwatch :
MESSAGE:Dynamo Insert record failed. The error received was NoSuchElementException. Message arrived on: myTopic/data, Action: dynamo, Table: myTable, HashKeyField: topic, HashKeyValue: , RangeKeyField: Some(timestamp), RangeKeyValue:
HashKeyValue and RangeKeyValue seem to be empty. Why ?
I also posted the question on the AWS forum : https://forums.aws.amazon.com/thread.jspa?threadID=267987
Suppose your devide sends this payload:
mess={"reported":
{"light": "blue","Temperature": int(temp_data)),
"timestamp": str(pd.to_datetime(time.time()))}}
args.message=mess
You should query as:
SELECT message.reported.* FROM '#'
Then, set up DynamoDB hash key value as ${MessageID()}
You will get:
MessageID || Data
1527010174562 { "light" : { "S" : "blue" }, "Temperature" : { "N" : "41" }, "timestamp" : { "S" : "1970-01-01 00:00:01.527010174" }}
Then you can easily extract values using Lambda and send to S3 via Data Pipeline or to Firehose to create a data stream.
Related
Here is a modified version of an Event type I am receiving in my handler for a lambda function with a DynamoDB someTableName table trigger that I logged using cargo lambda.
Event {
records: [
EventRecord {
change: StreamRecord {
approximate_creation_date_time: ___
keys: {"id": String("___")},
new_image: {
....
"valid": Boolean(true),
},
...
},
...
event_name: "INSERT",
event_source: Some("aws:dynamodb"),
table_name: None
}
]
}
Goal: Correctly filter with event_name=INSERT && valid=false
I have tried a number of options, for example;
{"eventName": ["INSERT"]}
While the filter is added correctly, it does not trigger the lambda on item inserted.
Q1) What am I doing incorrectly here?
Q2) Why is table_name returning None? The lambda function is created with a specific table name as trigger. The returned fields are returning an option (Some(_)) so I'm asssuming it returns None if the table name is specified on lambda creation, but seems odd to me?
Q3) From AWS Management Console > Lambda > ... > Trigger Detail, I see the following (which is slightly different from my code mentioned above), where does "key" come from and what does it represent in the original Event?
Filters must follow the documented syntax for filtering in the Event Source Mapping between Lambda and DynamoDB Streams.
If you are entering the filter in the Lambda console:
{ "eventName": ["INSERT"], "dynamodb": { "NewImage": {"valid": { "BOOL" : [false]}} } }
The attribute name is actually eventName, so your filter should look like this:
{"eventName": ["INSERT"]}
Hello I currently have an event notification set up with my s3 bucket. This notification is sent to a SNS topic, then a SQS Queue, and finally a lambda.
My ultimate goal is for my lambda to read the event notification json and parse out the bucket and key.
The problem is that I see the bucket name in the json but not the key when I print out the 'event' object using python. How/where should I go debug to figure out what is going on? I do remember seeing the key in the json from previous implementations
the json looks like:
{
'Records': [{
'messageId': '15d42178-c59c-4f3a-8efa-cce8a20acd5b',
'receiptHandle': 'AQEBnPN7q4+jLFQfExOytZYH69w4kvI4ohjJGFqUqOAvCjRHMfbFFvgEeLVjonZ5q4GAYyzLzDSRQmZv3+YTvE3VYqKmU+Nt0rgX824LoMMkKKMuSWBT6c1a0X5dXRJRzFaOKjpniONRg5Gdm1V9I/7mW0x+Zfi0PXr5cQZXVA1NNUdJ4tIkwtpuC+Rh/dbGFQmAo6fQDuCnpzRW1NKGGda440t3ivtUQMvrniwY8ILKVoX9pnS1rAVgVPGBUo8mXyH9ec9p/Er9O9N5Kxc3xQE44MhHUygD1iJbRROBHG9m0Mj6qbKx4uI7S4KQVWRK8hHkxYFUtP4NzhzcGP1LfY91+zG4mNweGzQkfDbvn0LG9+6guxv9dW+uGz1c3f9My7272s+ABfksvfbNRgPSgwJecg==',
'body': '{\n "Type" : "Notification",\n "MessageId" : "78cadcbb-f349-5bae-b39b-85504866b186",\n "TopicArn" : "<topic arn>",\n "Subject" : "Amazon S3 Notification",\n "Message" : "{\\"Service\\":\\"Amazon S3\\",\\"Event\\":\\"s3:TestEvent\\",\\"Time\\":\\"2021-10-21T19:01:03.083Z\\",\\"Bucket\\":\\"<s3 bucket>\\",\\"RequestId\\":\\"TCJP8AZ6S75XXXPN\\",\\"HostId\\":\\"VYNq+Jh5Hkg+Vykp2RcIy9lSca7uJyhzLPfE8tcgnt3Je9kH0I+H3zvzvJkd6IvfZKZm2jYqu4Q=\\"}",\n "Timestamp" : "2021-10-21T19:01:03.285Z",\n "SignatureVersion" : "1",\n "Signature" : "EE9xsZx8hezxh8Yhyj8DLc+VSGYowl641kHgqr8tWq2msNwOBv4KEZoTtHJ/hdnfNYLEBsR7imsfv5ZrX7nKRKL2kR8xax57tcih7GRbifIuFyrs9wAhtcuclf2NJQG4eY9OrOHHxPN3fSvNI9xduPeBrxB2TAfbTcWq4AeN0C4KriV18J2dU28ecMJGtmqK0JM+2KLEuwQe/dyYiEnEnWu5EfGweDhYCRvmB1aUPRcW4s3yOHIckklmHhBLkbmufl1me/hdO7GEGa1ju8wJDF33hmmCCSE6M7ITl9niWICBtvWlFz1Md5OiswyriRyN4LZjmvEjzRZtNwy/qMkDYA==",\n "SigningCertURL" : "<cert url>",\n "UnsubscribeURL" : "<unsubscribe url>"\n}',
'attributes': {
'ApproximateReceiveCount': '34',
'SentTimestamp': <timestamp>,
'SenderId': <senderid>,
'ApproximateFirstReceiveTimestamp': <timestamp>
},
'messageAttributes': {},
'md5OfBody': <md5>,
'eventSource': 'aws:sqs',
'eventSourceARN': <queue-arn>,
'awsRegion': 'us-east-1'
}]
}
In your Amazon SNS subscription, activate Amazon SNS raw message delivery.
This will pass-through the S3 Event in a cleaner form, with the body containing a string-version of the JSON from S3. You'll need to use JSON.parse() to convert it to an object.
I'm new to AWS AppSync and I have adopted the single table pattern in DynamoDB. Now I am trying to create an item based on a particular field value in the existing item in the same table. For example, I have a table called transaction which holds 2 types of records.
Request
Response
As you can see the above table, I can insert (PutItem) multiple responses for a particular request. Before I insert a new response, I need to validate whether the request (RequestID) is already exists. Is there any way to do via conditional expression in the resolver? Below is my current request resolver code which is not working as expected.
#set( $Id = $util.autoId() )
{
"version" : "2017-02-28",
"operation" : "PutItem",
"key" : {
"PK": $util.dynamodb.toDynamoDBJson("USER#$ctx.args.input.UserId"),
"SK": $util.dynamodb.toDynamoDBJson("RESPONSE#$Id"),
},
"attributeValues" : $util.dynamodb.toMapValuesJson($ctx.args.input),
"condition": {
"expression": "SK = :SK",
"expressionValues" : {
":SK" : {
"S" : "REQUEST#${ctx.args.input.RequestId}"
}
}
}
}
You could do this in one request mapping template by using DynamoDB transactions, see (https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-dynamodb-transact.html).
In one of the "transactWriteItems", you update an arbitrary value (or perhaps, number of responses) in the request item with a condition that checks if the request item exists with that requestId in the SK. If the conditions succeeds, the request item is updated.
Also make sure you have the response item in your "transactWriteItems" so that the response item is also written if the request condition passes.
You won't be able to do a conditional PutItem based on another entry no. In this case you'd want to use pipeline resolvers. In your first function you'd fetch the request item and in the second you can then do your PutItem condition - the previous GetItem result will be available as $ctx.prev.result.
Is there a way to filter messages based on Regex or substring in AWS SNS?
AWS Documentation for filtering messages mentions three types of filtering for strings:
Exact matching (whitelisting)
Anything-but matching (blacklisting)
Prefix matching
I want to filter out messages based on substrings in the messages, for example
I have a S3 event that sends a message to SNS when a new object is added to S3, the contents of the message are as below:
{
"Records": [
{
"s3": {
"bucket": {
"name": "images-bucket"
},
"object": {
"key": "some-key/more-key/filteringText/additionaldata.png"
}
}
}
]
}
I want to keep the messages if only filteringText is present in key field.
Note: The entire message is sent as text by S3 notification service, so Records is not a json object but string.
From what I've seen in the documentation, you can't do regex matches or substrings, but you can match prefixes and create your own attributes in the MessageAttributes field.
To do this, I send the S3 event to a simple Lambda that adds MessageAttributes and then sends to SNS.
In effect, S3 -> Lambda -> SNS -> other consumers (with filtering).
The Lambda can do something like this (where you'll have to programmatically decide when to add the attribute):
let messageAttributes = {
myfilterkey: {DataType: "String", StringValue:"filteringText"}
};
let params = {
Message: JSON.stringify(payload),
MessageAttributes: messageAttributes,
MessageStructure: 'json',
TargetArn: SNS_ARN
};
await sns.publish(params).promise();
Then in SNS you can filter:
{"myfilterkey": ["filtertext"]}
It seems a little convoluted to put the Lambda in there, but I like the idea of being able to plug and unplug consumers from SNS on the fly and use filtering to determine who gets what.
My weather-station is publishing its status via MQTT to AWS IoT.
The message is published to topic
$aws/things/my-weather-station-001/shadow/update and looks like this:
{
"state": {
"reported": {
"temperature" : 22,
"humidity" : 70,
....
"wind" : 234,
"air" : 345
}
}
After message is received I have create a rule to store it in AWS DynamoDB the rules select statement is:
SELECT state.reported.* FROM $aws/things/+/shadow/update/accepted
And when this works well, whilst I am sending messages containing state.reported field.
However sometimes to the topic $aws/things/weather-station-0001/shadow/update are sent "control" messages telling device to switch on an LED or some other part. These messages would be usually sent by an app or a controlling server and look like this notice that instead of reported field it hasdesired
{
"state": {
"desired": {
"led1" : "on",
"locked" : true
}
}
So when these messages are arriving, they ARE STILL processed by the rule and arrive to the DynamoDb table with {} empty payload.
Is there any way to force the Rule to ignore messages not containing state.reported element?
You can add a where clause to your SQL statement. Try
SELECT state.reported.* FROM $aws/things/+/shadow/update/accepted WHERE state.reported <> ''