I'm developing some business administration software in JS and I find myself in need of ACID transactions with DynamoDb. Lucky of mine, AWS just released the transactGet and transactWrite APIs that covers just the right use case!
I'm using the AWS.DynamoDb.DocumentClient object to make my other calls and it seems like the transact operations are not exposed for me to use.
I hacked inside the aws-sdk code and found the interface exports like such ("aws-sdk": "^2.368.0", document_client.d.js, line 2076):
export interface TransactWriteItem {
/**
* A request to perform a check item operation.
*/
ConditionCheck?: ConditionCheck;
/**
* A request to perform a PutItem operation.
*/
Put?: Put;
/**
* A request to perform a DeleteItem operation.
*/
Delete?: Delete;
/**
* A request to perform an UpdateItem operation.
*/
Update?: Update;
}
However, whenever I try to call the client with this methods, I get back a Type error, namely:
TypeError: dynamoDb[action] is not a function
Locally, I can just hack the sdk and expose everything, but it is not accepted on my deployment environment.
How should I proceed?
Thank you very much!
Edit:
If it is worth anything, there is the code I'm using to do the calls:
dynamo-lib.js:
import AWS from "aws-sdk";
export function call(action, params) {
const dynamoDb = new AWS.DynamoDB.DocumentClient();
return dynamoDb[action](params).promise();
}
Lambda code:
import * as dynamoDbLib from '../libs/dynamodb-lib';
import { success, failure } from '../libs/response-lib';
export default async function main(params, callback) {
try {
const result = await dynamoDbLib.call("transactWrite", params);
callback(null, success(result));
} catch (e) {
console.log(e);
callback(null, failure({"status": "Internal server error"}));
}
}
Edit 2:
Seems like the document client really does not export the method.
AWS.DynamoDB.DocumentClient = AWS.util.inherit({
/**
* #api private
*/
operations: {
batchGetItem: 'batchGet',
batchWriteItem: 'batchWrite',
putItem: 'put',
getItem: 'get',
deleteItem: 'delete',
updateItem: 'update',
scan: 'scan',
query: 'query'
},
...
As mentioned in accepted answer, the workaround is to use the transactWriteItems method straight from the DynamoDB object.
Thanks for the help! =D
Cheers!
UPDATE:
The issue has been resolved
Github Thread
AWS SDK is not currently supporting transactions in dynamodb document client i have raised the issue on github a current workaround is just not to use document client
let aws=require("aws-sdk");
aws.config.update(
{
region:"us-east-1",
endpoint: "dynamodb.us-east-1.amazonaws.com"
}
);
var dynamodb = new aws.DynamoDB();
dynamodb.transactWriteItems();
Also make sure you are using SDK v2.365.0 plus
Update the aws-sdk package (>2.365)
Use
var dynamodb = new aws.DynamoDB();
instead of
var dynamodb = new aws.DynamoDB.DocumentClient();
Related
I was working on something that would modify my promotional sms messages. I read that it is possible via CampaignHook in Pinpoint. But from the documentation, I couldn't gather how this will actually work. I have followed it until adding permission and linking the pinpoint app id and with it. I have followed this link: https://github.com/Ryanjlowe/lambda-powered-pinpoint-templates
For some reason, I am not able to follow what I need to do on the Lambda (boto3) function side to try and make this work. Is there an example code (python) or well-documented example for this? It would help me a lot. Thanks!
The Pinpoint developer guide describes how to setup a CampaignHook under the chapter "Customizing segments with AWS Lambda".
https://docs.aws.amazon.com/pinpoint/latest/developerguide/segments-dynamic.html
"To assign a Lambda function to a campaign, you define the campaign's CampaignHook settings by using the Campaign resource in the Amazon Pinpoint API. These settings include the Lambda function name. They also include the CampaignHook mode, which specifies whether Amazon Pinpoint receives a return value from the function."
The docs show an example Lambda function:
'use strict';
exports.handler = (event, context, callback) => {
for (var key in event.Endpoints) {
if (event.Endpoints.hasOwnProperty(key)) {
var endpoint = event.Endpoints[key];
var attr = endpoint.Attributes;
if (!attr) {
attr = {};
endpoint.Attributes = attr;
}
attr["CreditScore"] = [ Math.floor(Math.random() * 200) + 650];
}
}
console.log("Received event:", JSON.stringify(event, null, 2));
callback(null, event.Endpoints);
};
"In this example, the handler iterates through each endpoint in the event.Endpoints object, and it adds a new attribute, CreditScore, to the endpoint. The value of the CreditScore attribute is simply a random number."
Let's say I have an "Banner" Table.
There are 2 possible use cases for this table.
1.get all banner data from table
My lambda function might like below:
'use strict'
const AWS = require('aws-sdk');
exports.handler = async function (event, context, callback) {
const documentClient = new AWS.DynamoDB.DocumentClient();
let responseBody = "";
let statusCode = 0;
const params = {
TableName : "Banner",
};
try{
const data = await documentClient.scan(params).promise();
responseBody = JSON.stringify(data.Items);
statusCode = 200
}catch(err){
responseBody = `Unabel to get products: ${err}`;
statusCode = 403
}
const response = {
statusCode: statusCode,
headers:{
"Content-Type": "application/json",
'Access-Control-Allow-Origin': '*', // Required for CORS support to work
},
body: responseBody
}
return response
}
2.Query by user partition key/GSI
I may need to query based on banner id or banner title to get the corresponding table.
At first, I was thinking combine this two user case in one single lambda function.
until I opened below post.
aws - how to set lambda function to make dynamic query to dynamodb
One of the comments provide a way for me to do the dynamic query for these 2 user case, but he/she also mention that:
you are giving anyone invoke the request the ability to put any query in the request, that might put you vulnerable to some type of SQL Injection attacks.
This makes me thinking whether I should separate these 2 user cases in two lambda function?
What is the general practise for these kinds of things?
Generally speaking, also if the "SQL Injection" can be blocked it's good to separate this into 2 functions, Lambda handler should be single responsibility. If you want to reuse the code you can create some common DAL that you can create with the common code.
I think this comes down to personal preference, but I'd recommend splitting the functionality into two lambdas.
It sounds like you have two access patterns:
Get all banners
Get banners by user
I would probably implement these with two separate lambdas. If I were exposing this functionality via an API, I'd probably create two endpoints:
GET /banners. (fetches all banners)
GET /users/[user_id]/banners. (fetches all banners for a given user)
Each of these endpoints would route to their own lambda that services the specific request. If you serve the request with a single lambda, you'll have to introduce logic within your lambdas to determine which type of request you're fulfilling. I can't imagine what you'd gain by using only one lambda.
Keep your lambda code focused on a single responsibility, it'll make it easier to develop, test, and debug.
I am trying to create an aws lambda function that will read rows from multiple Google Sheets documents using the Google Sheet API and will merge them afterwards and write in another spreadsheet. To do so I did all the necessary steps according to several tutorials:
Create credentials for the AWS user to have the key pair.
Create a Google Service Account, download the credentials.json file.
Share each necessary spreadsheet with the Google Service Account client_email.
When executing the program locally it works perfectly, it successfully logins using the credentials.json file and reads & writes all necessary documents.
However when uploading it to AWS Lambda using the serverless framework and google-spreadsheet, the program fails silently in the authentication step. I've tried changing the permissions as recommended in this question but it still fail. The file is read properly and I can print it to the console.
This is the simplified code:
async function getData(spreadsheet, psychologistName) {
await spreadsheet.useServiceAccountAuth(clientSecret);
// It never gets to this point, it fails silently
await spreadsheet.loadInfo();
... etc ...
}
async function main() {
const promises = Object.entries(psychologistSheetIDs).map(async (psychologistSheetIdPair) => {
const [psychologistName, googleSheetId] = psychologistSheetIdPair;
const sheet = new GoogleSpreadsheet(googleSheetId);
psychologistScheduleData = await getData(sheet, psychologistName);
return psychologistScheduleData;
});
//When all sheets are available, merge their data and write back in joint view.
Promise.all(promises).then(async (psychologistSchedules) => {
... merge the data ...
});
}
module.exports.main = async (event, context, callback) => {
const result = await main();
return {
statusCode: 200,
body: JSON.stringify(
result,
null,
2
),
};
I solved it,
While locally having a Promise.all(promises).then(result =>...) eventually returned the value and executed what was inside the then(), aws lambda returned before the promises were resolved.
This solved it:
const res = await Promise.all(promises);
mergeData(res);
I have a lambda function which does a series of actions. I have a react application which triggers the lambda function.
Is there a way I can send a partial response from the lambda function after each action is complete.
const testFunction = (event, context, callback) => {
let partialResponse1 = await action1(event);
// send partial response to client
let partialResponse2 = await action2(partialResponse1);
// send partial response to client
let partialResponse3 = await action3(partialResponse2);
// send partial response to client
let response = await action4(partialResponse3);
// send final response
}
Is this possible in lambda functions? If so, how we can do this. Any ref docs or sample code would be do a great help.
Thanks.
Note: This is fairly a simple case of showing a loader with % on the client-side. I don't want to overcomplicate things SQS or step functions.
I am still looking for an answer for this.
From what I understand you're using API Gateway + Lambda and are looking to show the progress of the Lambda via UI.
Since each step must finish before the next step begin I see no reason not to call the lambda 4 times, or split the lambda to 4 separate lambdas.
E.g.:
// Not real syntax!
try {
res1 = await ajax.post(/process, {stage: 1, data: ... });
out(stage 1 complete);
res2 = await ajax.post(/process, {stage: 2, data: res1});
out(stage 2 complete);
res3 = await ajax.post(/process, {stage: 3, data: res2});
out(stage 3 complete);
res4 = await ajax.post(/process, {stage: 4, data: res3});
out(stage 4 complete);
out(process finished);
catch(err) {
out(stage {$err.stage-number} failed to complete);
}
If you still want all 4 calls to be executed during the same lambda execution you may do the following (this especially true if the process is expected to be very long) (and because it's usually not good practice to execute "long hanging" http transaction).
You may implement it by saving the "progress" in a database, and when the process is complete save the results to the database as well.
All you need to do is query the status every X seconds.
// Not real syntax
Gateway-API --> lambda1 - startProcess(): returns ID {
uuid = randomUUID();
write to dynamoDB { status: starting }.
send sqs-message-to-start-process(data, uuid);
return response { uuid: uuid };
}
SQS --> lambda2 - execute(): returns void {
try {
let partialResponse1 = await action1(event);
write to dynamoDB { status: action 1 complete }.
// send partial response to client
let partialResponse2 = await action2(partialResponse1);
write to dynamoDB { status: action 2 complete }.
// send partial response to client
let partialResponse3 = await action3(partialResponse2);
write to dynamoDB { status: action 3 complete }.
// send partial response to client
let response = await action4(partialResponse3);
write to dynamoDB { status: action 4 complete, response: response }.
} catch(err) {
write to dynamoDB { status: failed, error: err }.
}
}
Gateway-API --> lambda3 -> getStatus(uuid): returns status {
return status from dynamoDB (uuid);
}
Your UI Code:
res = ajax.get(/startProcess);
uuid = res.uuid;
in interval every X (e.g. 3) seconds:
status = ajax.get(/getStatus?uuid=uuid);
show(status);
if (status.error) {
handle(status.error) and break;
}
if (status.response) {
handle(status.response) and break;
}
}
Just remember that lambda's cannot exceed 15 minutes execution. Therefore, you need to be 100% certain that whatever the process does, it never exceeds this hard limit.
What you are looking for is to have response expose as a stream where you can write to the stream and flush it
Unfortunately its not there in Node.js
How to stream AWS Lambda response in node?
https://docs.aws.amazon.com/lambda/latest/dg/programming-model.html
But you can still do the streaming if you use Java
https://docs.aws.amazon.com/lambda/latest/dg/java-handler-io-type-stream.html
package example;
import java.io.InputStream;
import java.io.OutputStream;
import com.amazonaws.services.lambda.runtime.RequestStreamHandler;
import com.amazonaws.services.lambda.runtime.Context;
public class Hello implements RequestStreamHandler{
public void handler(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
int letter;
while((letter = inputStream.read()) != -1)
{
outputStream.write(Character.toUpperCase(letter));
}
}
}
Aman,
You can push the partial outputs into SQS and read the SQS messages to process those message. This is a simple and scalable architecture. AWS provides SQS SDKs in different languages, for example, JavaScript, Java, Python, etc.
Reading and writing into SQS is very easy using SDK and that too can be implemented in serverside or in your UI layer (with proper IAM).
I found AWS step function may be what you need:
AWS Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly.
Check this link for more detail:
In our example, you are a developer who has been asked to create a serverless application to automate handling of support tickets in a call center. While you could have one Lambda function call the other, you worry that managing all of those connections will become challenging as the call center application becomes more sophisticated. Plus, any change in the flow of the application will require changes in multiple places, and you could end up writing the same code over and over again.
I'm facing some timeout problems with a "middleware" (from now on file-service) service developed with NestJS and AWS-S3.
The file-service has two main purposes:
Act as an object storage abstraction layer, to allow the backend to upload files to different storage services completely transparent to the user.
Receive signed tokens as a url query parameter with file/object information, verify access to resource and stream it.
Upload works without problem.
Download small files has no problems too.
But when I try to download large files (> 50MB), after a few seconds, the connection beaks down because of a timeout and as you can figure out the download fails.
I've been spending some days looking for a solutions and reading docs.
Here some of them:
About KeepAlive
Use an instance of S3 each time
But nothing works.
Here the code:
Storage definition class
export class S3Storage implements StorageInterface {
config: any;
private s3;
constructor() {}
async initialize(config: S3ConfigInterface): Promise<void> {
this.config = config;
this.s3 = new AWS.S3();
// initialize S3 Configuration
AWS.config.update({
accessKeyId: config.accessKeyId,
secretAccessKey: config.secretAccessKey,
region: config.region
});
}
async downloadFile(target: FileDto): Promise<Readable> {
const params = {
Bucket: this.config.Bucket,
Key: target.sourcePath
};
return this.s3.getObject(params).createReadStream();
}
}
Download method
private async downloadOne(target: FileDto, request, response) {
const storage = await this.provider.getStorage(target.datasource);
response.setHeader('Content-Type', mime.lookup(target.filename) || 'application/octet-stream');
response.setHeader('Content-Disposition', `filename="${path.basename(target.filename)}";`);
const stream = await storage.downloadFile(target);
stream.pipe(response);
// await download and exit
await new Promise((resolve, reject) => {
stream.on('end', () => {
resolve(`${target.filename} has been downloaded`);
});
stream.on('error', () => {
reject(`${target.filename} could not be downloaded`);
});
});
}
If any one has faced the same issue (or similar) or any one has any idea (useful or not), I will appreciate any help or advice.
Thank you in advance.
I had the same issue, and here is how I solved it on my side: instead of processing the file by directly getting the stream from S3, I decided to download the content to a temp file (Amazon backend server for my API) and process already the stream from that temp file. Afterwards, I removed the temp file in order not to fill the hard drive.