I want to get real-time data from a stock API (IEX API) and load it to DynamoDB. Most of the tutorials I've watched so far (like this around 15:10) show how to create Lambda functions and integrate with API gateway. However, they are still manually entering the data to load to the table. How do I use the API Gateway and Lambda to get data from the IEX API and load it to DynamoDB rather than writing the data myself?
Following some tutorials, my Lambda function for loading data to the table is:
const AWS = require('aws-sdk');
exports.handler = async (event, context) => {
const documentClient = new AWS.DynamoDB.DocumentClient();
let responseBody = "";
let statusCode = 0;
const { id, stocksymbol} = JSON.parse(event.body);
const params = {
TableName: "StockData",
Item: {
id: id,
stocksymbol: stocksymbol
}
};
try {
const data = await documentClient.put(params).promise();
responseBody = JSON.stringify(data);
statusCode = 201;
} catch(err) {
responseBody = `Unable to put item: ${err}`;
statusCode = 403;
}
const response = {
statusCode: statusCode,
headers: {
"Content-Type": "application/json"
},
body: responseBody
};
return response
};
I would be getting more data from the API (opening price, closing price etc.) but this is what I have for now.
Assuming you need to utilize pull mechanism (you need to get data from API yourself), you can use AWS EventBridge rule to trigger your lambda periodically with interval of your preference - https://docs.aws.amazon.com/eventbridge/latest/userguide/create-eventbridge-scheduled-rule.html. In lambda you download the API data and store them in DynamoDb.
If you can use push mechanism (you can get data pushed to you, e.g. https://iexcloud.io/docs/api/?gclid=CjwKCAiA_9r_BRBZEiwAHZ_v17o9kJuPyF5Do_E3_mwC0uHEh2yXqqOdtVgqvc34yEk5RR8W8028HRoC0HUQAvD_BwE#webhooks), you can set the your API gateway resource URL as a target path, while having your lambda function as the handler for the API Gateway resource URL, storing the pushed data to the DynamoDb.
Related
My project is having public website and Content Management System(CMS).
I am using Lambda and API Gateway for the api.
The CMS currently has an api GET request to get ALL the data from the table below.
“Banner” Table
attribute:
-id: string(primary key/partition key)
-title: string
-isActive: boolean
...
-----------------------------------------------
Id isActive title
1 true title1
2 false title2
3 true title3
4 true title4
----------------------------------------------
This is My lambda function:
getBanner.js
'use strict'
const AWS = require('aws-sdk');
exports.handler = async function (event, context, callback) {
const documentClient = new AWS.DynamoDB.DocumentClient();
let responseBody = "";
let statusCode = 0;
const params = {
TableName : "Banner",
};
try{
const data = await documentClient.scan(params).promise();
responseBody = JSON.stringify(data.Items);
statusCode = 200
}catch(err){
statusCode = 403
}
const response = {
statusCode: statusCode,
body: responseBody
}
return response
}
I need an api to get all banners with isActive = true.
There are 2 approaches I can think of
1.Modify the existing lambda function
I can add something like below:
…
if(event.queryStringParameters.isActive === true){
// add filter or query to get all result
}
...
But everyone is able to get all the data(including the results with isActive = false) if they do not use the queryStringParameters, which is what I want to avoid since the data with isActive = false should not be seen by the public.
2.Create new lambda function
This is probably the best way to protect the data.
But I have a lot of API encountering the same situation(having “inActive” attribute), it means I need to create a lot of public api.
Which method should I use?
You could create a public API Gateway that points to the same Lambda functions, and in the API Gateway configuration have it always pass the isActive=true parameter so that users of the public API wouldn't be able to override that.
I want the frontend to decide the currentPage, pageSize(number of data in a page) and the sorting(+ mean ascending/- mean descending) in event.queryStringParameters.
Dynamodb attribute
-id(string)(partition key)
-createdAt(date)
-url(string)
-createdBy(string)
-title(string)
But I am troubled to create this function, I can only set the pageSize, and I don't know how to deal with currentPage and sorting in my function.
Below is my current lambda function.
getBanner.js
'use strict'
const AWS = require('aws-sdk');
exports.handler = async function (event, context, callback) {
const documentClient = new AWS.DynamoDB.DocumentClient();
let responseBody = "";
let statusCode = 0;
const queryParams = JSON.stringify(event.queryStringParameters);
/* queryParams will look like this
{
"currentPage": "0",
"pageSize": "30",
"sorting": "-status"
}
*/
const params = {
TableName : "Banner",
Limit:queryParams.pageSize
};
try{
const data = await documentClient.scan(params).promise();
responseBody = JSON.stringify(data.Items);
statusCode = 200
}catch(err){
responseBody = `Unable to get Alerts: ${err}`;
statusCode = 403
}
const response = {
statusCode: statusCode,
headers:{
"Content-Type": "application/json",
'Access-Control-Allow-Origin': '*', // Required for CORS support to work
},
body: responseBody
}
return response
}
Pagination doesn't work that way in DynamoDB (link to docs). This is another example of where knowledge of SQL database practices can hinder your understanding of DynamoDB.
DynamoDB has a limit of 1MB responses for scan and query operations. Anything larger than that will be paginated. This means that DynamoDB will paginate results on the size of the response, not the number of items in the response. This limitation ensures that your responses stay small and manageable, while your queries remain performant.
Fetching 1MB of data at a time is different than fetching N rows at a time, as is common with pagination in SQL databases. Take some time to read up on how DDB handles pagination with the LastEvaluatedKey and ExclusiveStartKey before implementing a front-end solution.
I want to save every request to my S3 object in DynamoDB.
This is my pipeline.
I'm deploying the function to Lambda#Edge with viewer-response as the trigger and just to make sure that the function runs I'm also inserting a custom header with the image served. The header is coming as expected hence the function runs.
When I test this function via the Lambda Console, it is inserting the entry in DynamoDB. The problem is that while it's returning the custom header, it's not making a DB entry when I go to the CloudFront CDN URL.
For example when I goto https://d********e.cloudfront.net/test-image.png it should -
Serve the Respective Image (Working Properly)
Have the custom header (Working Properly)
Store entry in DynamoDB (Not working)
Here's the code for reference. Please note that the function is storing an entry in DB while running a test event in Lambda Console and I want this function to run every time the CDN serves an image.
const AWS = require('aws-sdk');
const dynamoDb = new AWS.DynamoDB.DocumentClient();
const shortid = require('shortid');
module.exports.image = async (event, context, callback) => {
const response = event.Records[0].cf.response;
const headers = response.headers;
headers['mycustomheader'] = [{ key: 'My-Custom-Header', value: new Date().getTime().toString() }];
await storeClickEvent(event);
callback(null, response);
};
const storeClickEvent = async (data) => {
const params = {
TableName: 'my-dummy-table-name',
Item: {
event: data,
id: shortid.generate(),
createdAt: new Date().getTime(),
}
};
try {
await dynamoDb.put(params).promise();
} catch (err) {
console.error('Error occurred =>', err);
}
}
I have a simple AWS Lambda function which makes an S3.getObject() call as follows:
const AWS = require('aws-sdk');
AWS.config.logger = console;
const s3 = new AWS.S3();
exports.handler = async (event) => {
return await getObject({
Bucket: "<MY-BUCKET>",
Key: "<MY-KEY>"
}).then( (res) => {
console.log('Retrieved object from S3');
console.log(res);
return res.Body.toString('ascii');
})
};
async function getObject(params){
return await s3.getObject(params).promise();
}
I've enabled logging SDK calls as per this document.
How do I get response headers for the s3.getObject() SDK call that was made? I am basically trying to retrieve the S3 request ID and extended request ID.
The in-built logger added via the "AWS.config.logger = console;" line does not seem to log response headers. How else do I get response headers for AWS JavaScript SDK calls?
P.S: Bonus points if you can let me know whether or not I need two await keywords in the code above.
Listen to httpHeaders event.
var requestObject = s3.getObject(params);
requestObject.on('httpHeaders', (statusCode, headers, response, statusMessage) => {
// your code here.
});
requestObject.promise()
.then(response => { ... })
I am learning the AWS services for a use case. After going through the docs I came came up with the a simple flow. I want to ingest data into the Kinesis streams by using the Streams API and the KPL. I use the example putRecord method to ingest data to the streams. I am ingesting the this JSON to the stream -
{"userid":1234,"username":"jDoe","firstname":"John","lastname":"Doe"}
Once the data is ingested i get the following response in putRecordResult -
Put Result :{ShardId: shardId-000000000000,SequenceNumber: 49563097246355834103398973318638512162631666140828401666}
Put Result :{ShardId: shardId-000000000000,SequenceNumber: 49563097246355834103398973318645765717549353915876638722}
Put Result :{ShardId: shardId-000000000000,SequenceNumber: 49563097246355834103398973318649392495008197803400757250}
Now I write a Lambda function to get these data and push into a DynamoDB table. Here is my Lambda function -
console.log('Loading function');
var AWS = require('aws-sdk');
var tableName = "sampleTable";
var doc = require('dynamodb-doc');
var db = new doc.DynamoDB();
exports.handler = (event, context, callback) => {
//console.log('Received event:', JSON.stringify(event, null, 2));
event.Records.forEach((record) => {
// Kinesis data is base64 encoded so decode here
const payload = new Buffer(record.kinesis.data, 'base64').toString('ascii');
console.log('Decoded payload:', payload);
var userid = event.userid;
var username = event.username;
var firstname = event.firstname;
console.log(userid + "," + username +","+ firstname);
var item = {
"userid" : userid,
"username" : username,
"firstname" : firstname
};
var params = {
TableName : tableName,
Item : item
};
console.log(params);
db.putItem(params, function(err, data){
if(err) console.log(err);
else console.log(data);
});
});
callback(null, `Successfully processed ${event.Records.length} records.`);
};
Somehow I am not able to see the console.logs in the lambda functions execution. I see in the streams page there have been putRecord to the stream and get as well but somehow i can see nothing in the Lambdafunction page nor in the DynamoDB table.
I have an IAM policy for the Java code for the ingestion of the data into Kinesis, another for the Lambda function that is lambda-kinesis-execution-role and a policy for the DynamoDB to ingest data into the tables.
Is there any tutorial that shows how it is done in the right way? I am getting a feeling that I am missing many points in this process for example how to link all those IAM policies and make them in sync so that when the data is put into the stream it is processed by Lambda and ends up in Dynamo?
Any pointers and help is deeply appreciated.
If you're code above is a direct copy of the code you're using, you're referencing event.userid but you should be using payload.userid. You've decoded the Kinesis record into the payload variable.
You Can use Lambda function
1.Create IAM role for both Kinesis and Dynamodb
2.Now Create a Lambda function from blue print of dynamodb-process-stream
3.Select the execution role which we created from IAM
4.Click Create Function
Now Go to Edit code section and write the following code
const AWS =require('aws-sdk');
const docClient =new AWS.DynamoDB.DocumentClient({region : 'us-east-1'});
exports.handler = (event, context, callback) => {
event.Records.forEach((record) => {
var params={
Item :{
ROWTIME:Date.now().toString(),//Dynamodb column name
DATA:new Buffer(record.kinesis.data, base64').toString('ascii')//Dynamodb column name
},
TableName:'mytable'//Dynamodb Table Name
};
docClient.put(params,function(err,data){
if(err){
callback(err,null);
}
else{
callback(null,data);
}
});
});
};