Trigger DynamoDB Stream for Large Data Set - amazon-web-services

Question

I have a lambda which gets triggered by DynamoDB Streams.
I would like to trigger that lambda for each item in the table (more than 100.000 items in one table - one item is around 500bytes).
How could I achieve that?
I have created a lambda which queries the DynamoDB table, gets each item and sends a message on the EventBridge containing the ID, so that another Lambda (Triggered by that ID) could update the item with a field 'createdAt' where I would just pass the EPOCH time.
After 300 items queried, I receive a timeout because memory of the lambda is exceed (256mb). This is unfortunately not a good solution.
My code looks something like this
updateByMId: async (mId, fn) => {
const paginator = paginateQuery(paginatorConfig, {
TableName: tableName,
KeyConditionExpression: '#pk = :pk',
ExpressionAttributeNames: { '#pk': 'mId' },
ExpressionAttributeValues: marshall({ ':pk': mId }, { removeUndefinedValues: true }),
})
const promises: Promise<void>[] = []
for await (const page of paginator) {
const records = (page.Items || []).map(item => unmarshall(item) as Record)
for (const record of records) {
promises.push(fn(record))
}
}
await Promise.all(promises)
},
the function which is being passed to this method is:
putEvent: async (message) => {
const output = await client.send(new eventbridge.PutEventsCommand({
Entries: [{
EventBusName: eventBusName,
Source: 'foo',
DetailType: 'bar',
Detail: JSON.stringify(message),
}],
}))
if (output.FailedEntryCount !== undefined && output.FailedEntryCount > 0) {
throw new Error(`Error putting event on bus! ${JSON.stringify(output.Entries)}`)
}
logger.info(`Successfully put event on bus`, { message })
},

Answer 1

Here's one way to do it that's reasonably tolerant of failures:
scan the table
write the item keys to an SQS queue
configure a Lambda function on that SQS queue to process the queue in batches of, say, 10 messages
have the Lambda function write a date attribute to the item associated with each key, as needed
Personally, I would not use a Lambda function to perform that initial scan.