Can I export data from s3 to dynamoDB using lambda? - amazon-web-services

Question

I want to build a lambda process that automatically sends data to dynamodb when s3 comes in. But as dynamodb is not set as destination like the picture below, what should I do? (The permission is admin.)

Answer 1

The best way would be to setup notifications on your s3 bucket to trigger for new object. The notifications would launch your lambda function, which would then update dynamodb.
If you already have objects in your bucket, you could use S3 batch operations to process all of them with your lambda function.

Answer 2

You should understand how to work with lambda event source (event trigger). Here the event source is the S3 where once there is a stored object, the S3 will trigger an event to the lambda function. To get this work, you have to add the permission on lambda for S3 event. Check it out :
Using Lambda Function with Amazon S3
Now, every objects put into S3 will trigger an event to Lambda. Something like tell the lambda for new coming S3 object. You can check the event object from lambda code like this sample code :
exports.handler = async (event) => {
event.Records.forEach((record, i)=>{
if (record.eventName == 'ObjectCreated:Put')
console.log(record);
})
return;
}
Test the file upload to your s3 bucket and go to cloudwatch to check your lambda log.
Next if you want to store the file content into dynamodb, you will add policies for lambda role and write some lines on the lambda function, for the sample policy to add permission s3:GetObject and dynamodb:PutItem for lambda role :
{
"Sid": "Stmt1583413548180",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "your_s3Bucket_ARN"
},
{
"Sid": "Stmt1583413573162",
"Action": [
"dynamodb:PutItem"
],
"Effect": "Allow",
"Resource": "your_dynamodbTable_ARN"
}
And this is the sample of lambda code :
let s3 = new AWS.S3();
let dynamoDB = new AWS.DynamoDB.DocumentClient();
let TextDecoder = require("util").TextDecoder;
exports.handler = async (event) => {
let records = [];
event.Records.forEach((record, i)=>{
if (record.eventName == 'ObjectCreated:Put')
records.push(fileContent());
if (i == event.Records.length-1)
if (records.length > 0)
return Promise.all(records).then(()=>{
console.log("All events completed")
return "All events completed";
}).catch((e)=>{
console.log("The tasks error: ",e)
throw "The tasks error";
})
else
return "All events completed";
})
}
/* Get the file content and put new dynamodb item */
function fileContent(obj) {
let params = {
Bucket: obj.bucket.name,
Key: obj.object.key
}
return s3.getObject(params).promise().then((content)=>{
console.log("GetObject succeeded");
content = new TextDecoder("utf-8").decode(content.Body);
let Item = {
Key: obj.object.key,
dataContent: content
}
return dynamoDB.put({
TableName: 'table_name',
Item:Item
}).promise();
})
}
Well, let me resume the steps :
Add permission for s3 event on your lambda function
Add the IAM policy to lambda role for actions s3:GetObject and dynamodb:PutItem
Update your lambda function code to export the s3 file to dynamodb item