Here is a modified version of an Event type I am receiving in my handler for a lambda function with a DynamoDB someTableName table trigger that I logged using cargo lambda.
Event {
records: [
EventRecord {
change: StreamRecord {
approximate_creation_date_time: ___
keys: {"id": String("___")},
new_image: {
....
"valid": Boolean(true),
},
...
},
...
event_name: "INSERT",
event_source: Some("aws:dynamodb"),
table_name: None
}
]
}
Goal: Correctly filter with event_name=INSERT && valid=false
I have tried a number of options, for example;
{"eventName": ["INSERT"]}
While the filter is added correctly, it does not trigger the lambda on item inserted.
Q1) What am I doing incorrectly here?
Q2) Why is table_name returning None? The lambda function is created with a specific table name as trigger. The returned fields are returning an option (Some(_)) so I'm asssuming it returns None if the table name is specified on lambda creation, but seems odd to me?
Q3) From AWS Management Console > Lambda > ... > Trigger Detail, I see the following (which is slightly different from my code mentioned above), where does "key" come from and what does it represent in the original Event?
Filters must follow the documented syntax for filtering in the Event Source Mapping between Lambda and DynamoDB Streams.
If you are entering the filter in the Lambda console:
{ "eventName": ["INSERT"], "dynamodb": { "NewImage": {"valid": { "BOOL" : [false]}} } }
The attribute name is actually eventName, so your filter should look like this:
{"eventName": ["INSERT"]}
Related
I'm trying to update a method in a file in AWS lambda function. Previously, the method accepted URLs such as
https://yh74mgrokc.execute-api.us-west-1.amazonaws.com/dev/ga/answer/qn_type?tag=alexa
I updated the function to accept URL like the ones below, as well as new tags like to and from.
https://yh74mgrokc.execute-api.us-west-1.amazonaws.com/dev/ga/answer/qn_type?tag=alexa&from=2020-05-29&to=2021-05-29
So, I modified one of the methods as follows:
exports.getAnswerQnTypeSchema = [
query('tag')
.exists()
.withMessage('tag is required'),
query().custom((query) => {
const allowedKeys = [
"id",
"name",
"tag",
"from",
"to"
];
for (const key of Object.keys(query)) {
if (!allowedKeys.includes(key)) {
throw new Error(`Unknown property: ${key} please resolve`,query);
}
}
return true;
})
];
to allow for 'from' and 'to' tags. Then I redeployed the function and uploaded the updated lambda function. But still the function works as the previous version. when I send the request
https://yh74mgrokc.execute-api.us-west-1.amazonaws.com/dev/ga/answer/qn_type?tag=alexa&from=2020-05-29&to=2021-05-29
It does not support the from and to tags, and the error message is as follows:
I'm not sure why this is the case. I'm new to AWS services, and any assistance with this matter would be greatly appreciated.
I'm trying to modify a Dynamodb table each time a Lambda function is executed.
Specifically, I create a simple lambda function that returns a list of S3 bucket names and this function run each minute thanks to a Cloudwatch's rule.
However, as I said before, my goal is to also update a Dynamodb each time the same function is executed. Specifically I want to add each time a new Item with the same attribute (so let's say the function is executed 1000 times, I want 1K items/rows).
However I don't know how to do it. Any suggestions? Here's the code:
import json
import boto3
s3 = boto3.resource('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Table')
def lambda_handler(event, context):
bucket_list = []
for b in s3.buckets.all():
print(b.name)
bucket_list.append(b.name)
response = "done"
table.put_item(
Item = {
"Update": response
}
)
return {
"statusCode": 200,
"body": bucket_list
}
Thank you in advance
Your problem is that PutItem does overwrite exiting items, if they are the same. So every time you try to insert Update=done, it just overwrites the same item.
The very first sentence of the documentation states:
Creates a new item, or replaces an old item with a new item.
So what you need to do is to put something in your item that is unique, so that a new item is created instead of the old one being overwritten.
You could create a UUID or something like that, but I think it would be beneficial to use the time of execution. This way you could see when your last execution was etc.
from datetime import datetime
[...]
table.put_item(
Item = {
"Update": response,
"ProcessingTime": datetime.now().isoformat()
}
)
Adding to what Jens stated, which is 100% correct.
You could use data from the event. The event will look something like this:
{
"id": "cdc73f9d-aea9-11e3-9d5a-835b769c0d9c",
"detail-type": "Scheduled Event",
"source": "aws.events",
"account": "123456789012",
"time": "1970-01-01T00:00:00Z",
"region": "us-west-2",
"resources": [
"arn:aws:events:us-west-2:123456789012:rule/ExampleRule"
],
"detail": {}
}
The id value will be 100% unique, and the time value will be the time it was triggered.
We have two types of SNS messages coming in:
1. has MessageAttributes empty like this:
"MessageAttributes": {}
2. has MessageAttributes coming in like this:
"MessageAttributes": {
"Generator": {
"Type": "String",
"Value": "some-service"
}
}
I would like to use a filter subscription policy that ignores the second type but passes the first type to the subscriber.
So I tried this for the policy:
{
"Generator": [
{
"exists": false
}
]
}
I thought this would mean it will only pass along messages that do NOT contain the Generator key in MessageAttributes
However I am seeing now that no messages are getting passed along.
The AWS Subscription Filter docs seem to support this as a solution, but they only show the opposite way of checking that a key does exist, so I'm not sure if they support checking a key doesn't exist: https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html#attribute-key-matching
Is this possible?
The answer from #David Adams is out of date. See the Attribute key matching docs.
Use "exists": false to return incoming messages that don't include the specified attribute.
It is now possible to exclude any messages that have a particular key by using the policy:
{
"key": [
{
"exists": false
}
]
}
Late response but may be helpful to someone.
Filtering out by lack of existance is not possible. See the bottom of https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html#attribute-key-matching
Note: You cannot use the exists operator to match messages in which an attribute does not exist. Filtering will NOT match any messages if you set [{"exists": false}].
You could pass a string 'null' or similar to the generator attribute if it is non existant maybe?
Is there a way to filter messages based on Regex or substring in AWS SNS?
AWS Documentation for filtering messages mentions three types of filtering for strings:
Exact matching (whitelisting)
Anything-but matching (blacklisting)
Prefix matching
I want to filter out messages based on substrings in the messages, for example
I have a S3 event that sends a message to SNS when a new object is added to S3, the contents of the message are as below:
{
"Records": [
{
"s3": {
"bucket": {
"name": "images-bucket"
},
"object": {
"key": "some-key/more-key/filteringText/additionaldata.png"
}
}
}
]
}
I want to keep the messages if only filteringText is present in key field.
Note: The entire message is sent as text by S3 notification service, so Records is not a json object but string.
From what I've seen in the documentation, you can't do regex matches or substrings, but you can match prefixes and create your own attributes in the MessageAttributes field.
To do this, I send the S3 event to a simple Lambda that adds MessageAttributes and then sends to SNS.
In effect, S3 -> Lambda -> SNS -> other consumers (with filtering).
The Lambda can do something like this (where you'll have to programmatically decide when to add the attribute):
let messageAttributes = {
myfilterkey: {DataType: "String", StringValue:"filteringText"}
};
let params = {
Message: JSON.stringify(payload),
MessageAttributes: messageAttributes,
MessageStructure: 'json',
TargetArn: SNS_ARN
};
await sns.publish(params).promise();
Then in SNS you can filter:
{"myfilterkey": ["filtertext"]}
It seems a little convoluted to put the Lambda in there, but I like the idea of being able to plug and unplug consumers from SNS on the fly and use filtering to determine who gets what.
I'm trying to set an AWS IOT rule to send data to DynamoDB without the help of a lambda.
My rule query statement is : SELECT *, topic() AS topic, timestamp() AS timestamp FROM '+/#'
My data is fine in AWS IOT as I'm successfully retrieving it with a lambda. However, even by following the developer guide to create the rule, in order to get the information passed on to Dynamo, by setting the 2 form fields with ${topic} and ${timestamp} as it should work, I get nothing in DynamoDB and I can find the following exception in Cloudwatch :
MESSAGE:Dynamo Insert record failed. The error received was NoSuchElementException. Message arrived on: myTopic/data, Action: dynamo, Table: myTable, HashKeyField: topic, HashKeyValue: , RangeKeyField: Some(timestamp), RangeKeyValue:
HashKeyValue and RangeKeyValue seem to be empty. Why ?
I also posted the question on the AWS forum : https://forums.aws.amazon.com/thread.jspa?threadID=267987
Suppose your devide sends this payload:
mess={"reported":
{"light": "blue","Temperature": int(temp_data)),
"timestamp": str(pd.to_datetime(time.time()))}}
args.message=mess
You should query as:
SELECT message.reported.* FROM '#'
Then, set up DynamoDB hash key value as ${MessageID()}
You will get:
MessageID || Data
1527010174562 { "light" : { "S" : "blue" }, "Temperature" : { "N" : "41" }, "timestamp" : { "S" : "1970-01-01 00:00:01.527010174" }}
Then you can easily extract values using Lambda and send to S3 via Data Pipeline or to Firehose to create a data stream.