DynamoDB JavaScript PutItemCommand is neither failing nor working - amazon-web-services

Please note: although this question mentions AWS SAM, it is 100% a DynamoDB JavaScript SDK question at heart and can be answered by anyone with experience writing JavaScript Lambdas (or any client-side apps) against DynamoDB using the AWS DynamoDB client/SDK.
So I used AWS SAM to provision a new DynamoDB table with the following attributes:
FeedbackDynamoDB:
Type: AWS::DynamoDB::Table
Properties:
TableName: commentary
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 5
WriteCapacityUnits: 5
StreamSpecification:
StreamViewType: NEW_IMAGE
This configuration successfully creates a DynamoDB table called commentary. However, when I view this table in the DynamoDB web console, I noticed a few things:
it has a partition key of id (type S)
it has no sort key
it has no (0) indexes
it has a read/write capacity mode of "5"
I'm not sure if this raises any red flags with anyone but I figured I would include those details, in case I've configured anything incorrectly.
Now then, I have a JavaScript (TypeScript) Lambda that instantiates a DynamoDB client (using the JavaScript SDK) and attempts to add a record/item to this table:
// this code is in a file named app.ts:
import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import { User, allUsers } from './users';
import { Commentary } from './commentary';
import { PutItemCommand } from "#aws-sdk/client-dynamodb";
import { DynamoDBClient } from "#aws-sdk/client-dynamodb";
export const lambdaHandler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
try {
const ddbClient = new DynamoDBClient({ region: "us-east-1" });
let status: number = 200;
let responseBody: string = "\"message\": \"hello world\"";
const { id, content, createdAt, providerId, receiverId } = JSON.parse(event.body);
const commentary = new Commentary(id, content, createdAt, providerId, receiverId);
console.log("deserialized this into commentary");
console.log("and the deserialized commentary has content of: " + commentary.getContent());
await provideCommentary(ddbClient, commentary);
responseBody = "\"message\": \"received commentary -- check dynamoDb!\"";
return {
statusCode: status,
body: responseBody
};
} catch (err) {
console.log(err);
return {
statusCode: 500,
body: JSON.stringify({
message: err.stack,
}),
};
}
};
const provideCommentary = async (ddbClient: DynamoDBClient, commentary: Commentary) => {
const params = {
TableName: "commentary",
Item: {
id: {
S: commentary.getId()
},
content: {
S: commentary.getContent()
},
createdAt: {
S: commentary.getCreatedAt()
},
providerId: {
N: commentary.getProviderId()
},
receiverId: {
N: commentary.getReceiverId()
}
}
};
console.log("about to try to insert commentary into dynamo...");
try {
console.log("wait for it...")
const rc = await ddbClient.send(new PutItemCommand(params));
console.log("DDB response:", rc);
} catch (err) {
console.log("hmmm something awry. something....in the mist");
console.log("Error", err.stack);
throw err;
}
};
Where commentary.ts is:
class Commentary {
private id: string;
private content: string;
private createdAt: Date;
private providerId: number;
private receiverId: number;
constructor(id: string, content: string, createdAt: Date, providerId: number, receiverId: number) {
this.id = id;
this.content = content;
this.createdAt = createdAt;
this.providerId = providerId;
this.receiverId = receiverId;
}
public getId(): string {
return this.id;
}
public getContent(): string {
return this.content;
}
public getCreatedAt(): Date {
return this.createdAt;
}
public getProviderId(): number {
return this.providerId;
}
public getReceiverId(): number {
return this.receiverId;
}
}
export { Commentary };
When I update the Lambda with this handler code, and hit the Lambda with the following curl (the Lambda is invoked by an API Gateway URL that I can hit via curl/http):
curl -i --request POST 'https://<my-api-gateway>.execute-api.us-east-1.amazonaws.com/Stage/feedback' \
--header 'Content-Type: application/json' -d '{"id":"123","content":"test feedback","createdAt":"2022-12-02T08:45:26.261-05:00","providerId":457,"receiverId":789}'
I get the following HTTP 500 response:
{"message":"SerializationException: NUMBER_VALUE cannot be converted to String\n
Am I passing it a bad request body (in the curl) or do I need to tweak something in app.ts and/or commentary.ts?

Interestingly the DynamoDB API expects numerical fields of items as strings. For example:
"N": "123.45"
The doc says;
Numbers are sent across the network to DynamoDB as strings, to maximize compatibility across languages and libraries. However, DynamoDB treats them as number type attributes for mathematical operations.
Have you tried sending your input with the numerical parameters as strings as shown below? (See providerId and receiverId)
{
"id":"123",
"content":"test feedback",
"createdAt":"2022-12-02T08:45:26.261-05:00",
"providerId":"457",
"receiverId":"789"
}
You can convert these IDs into string when you're populating your input Item:
providerId: {
N: String(commentary.getProviderId())
},
receiverId: {
N: String(commentary.getReceiverId())
}
You could also use .toString() but then you'd get errors if the field is not set (null or undefined).

Try using a promise to see the outcome:
client.send(command).then(
(data) => {
// process data.
},
(error) => {
// error handling.
}
);
Everything seems alright with your table setup, I believe it's Lambda async issue with the JS sdk. I'm guessing Lambda is not waiting on your code and exiting early. Can you include your full lambda code.

Related

Function scan in DynamoDB doesn't bring some of the results

I got a function in AWS Lambda that lists every patient in a table from DynamoDB. I realized that some items from the table were not on the list. This is my function to list:
module.exports.listPatients = async (event) => {
try {
const queryString = {
limit: 5,
...event.queryStringParameters,
};
const { limit, next, name } = queryString;
const localParams = {
...patientsParams,
Limit: limit,
FilterExpression: "contains(full_name, :full_name)",
ExpressionAttributeValues: { ":full_name": name },
};
if (next) {
localParams.ExclusiveStartKey = {
id: next,
};
}
const data = await dynamoDb.scan(localParams).promise();
const nextToken = data.LastEvaluatedKey ? data.LastEvaluatedKey.id : "";
const result = {
items: data.Items,
next_token: nextToken,
};
return {
statusCode: 200,
body: JSON.stringify(result),
};
} catch (error) {
console.log("Error: ", error);
return {
statusCode: error.statusCode ? error.statusCode : 500,
body: JSON.stringify({
error: error.name ? error.name : "Exception",
message: error.message ? error.message : "Unknown error",
}),
};
}
};
Am I missing something?
I tried with and without a limit, removed the filters, and yet nothing.
I tested one of the ids with get() to test with the server can find one of those who are missing, and it worked.
I am using Serverless to deploy the code, and when I try offline, it's working.
Stackoverflow recommended this post when writing my question, but I am using DynamoDB.DocumentClient without specifying the full attribute type in the filter expression:
How to scan in DynamoDB without primary sort key with Nodejs
Looks like you are paginating using scan(). Using query() with some Global Secondary Indexes and ScanIndexForward would give you a much better performance. scan() doesn't scale well when your data grows.

Calling AWS AppSync graphql API from Lambda

I am trying to update the value of a table using the AWS-app sync graphql API,
I am able to create data and add it in a table using graphql mutation in lambda
but when I am trying to update the data its not working.
I am calling this lambda service from an API Gateway.
I am referring this article to code
https://cloudonaut.io/calling-appsync-graphql-from-lambda/
I would like to mentioned git no error in cloud watch log
Here is the schema for my graphql
type Mutation {
createLib_content(input: CreateLib_contentInput!): lib_content
#aws_iam
updateLib_content(input: UpdateLib_contentInput!): lib_content
#aws_iam
deleteLib_content(input: DeleteLib_contentInput!): lib_content
}
input CreateLib_contentInput {
content: String
userId: String
}
input UpdateLib_contentInput {
content: String
id: ID!
}
Create Mutation
graphqlData = await clientDetails.mutate({
mutation: gql(`
mutation CreateLibContent($input: CreateLib_contentInput!) {
createLib_content(input: $input) {
id
content
}
}`),
variables: {
input: {
content : {},
userId : identitiesDetails.userId
}
},
});
Update Mutation
const mutation = gql(`
mutation UpdateLibContent($input: UpdateLib_contentInput!) {
updateLib_content(input: $input) {
userId
content
}
}`);
await clientDetails.mutate({
mutation,
variables: {
input: {
id : "2947c37e-6f76-40d8-8c10-4cd6190d3597",
content : JSON.stringify(event)
}
}
}).promise;
Thanks to #cppgnlearner your guess were right.
I just removed the .promise from my update code
And it started working.
can't believe such a small thing took my whole day.

Limit the size dynamoDB primary Key using typescript

I am building an application with ionic3 and AWS lanbda, where i use dynamodb for my database, the primary key generated as the "Id" ( in my case "groupId" ) of the table is too long afc9c380-db7d-11e7-a2c5-3bce7c435d3e I would like to know if there is a way to limit the no of digits to 6 characters? Thank you
My typescript code - "groupId is auto Genrated"
addGroup(form) {
this.submitted = true;
if (form && this.formData.name) {
let group = {
name: this.formData.name,
description: this.formData.description,
profileImageURI: this.profileImageURI,
userId: this.globals.getUserId(),
imageUrl: this.formData.imageUrl
};
this.globals.displayLoader("Adding...");
this.client.getClient().groupsCreateByUserId(this.globals.getUserId(), group).subscribe(
(data) => {
this.globals.dismissLoader();
this.globals.displayToast(`Group successfully added.`);
this.navCtrl.pop();
},
(err) => {
this.globals.dismissLoader();
this.globals.displayAlert('Error encountered',
`An error occurred when trying to add the group. Please check the console logs for more information.`);
console.error(err);
}
);
}
this.globals.dismissLoader();
}

AWS DynamoDB returns validation error when called from AWS Lambda

I'm using AWS Lambda and try to write something to AWS DynamoDB. I use the following code:
var tableName = "locations";
var item = {
deviceId: {
S: event.deviceId
},
timestamps: {
S: event.timestamp
}
}
var params = {
TableName: tableName,
Item: item
};
dynamo.putItem(params, function(err, data) {
if (err) {
context.fail(new Error('Error ' + err));
} else {
context.success(null);
}
});
And I get the following error:
returns Error ValidationException: One or more parameter values were invalid: Type mismatch for key deviceId expected: S actual: M
This happened because the aws sdk for Nodejs had changed!
If you are using:
var doc = require('dynamodb-doc');
var dynamo = new doc.DynamoDB();
Then the parameters to the putItem call (and most other calls) have changed and instead needs to be:
var tableName = "locations";
var item = {
deviceId: event.deviceId,
timestamp: event.timestamp,
latitude: Number(event.latitude),
longitude: Number(event.longitude)
}
var params = {
TableName: tableName,
Item: item
};
Read all about the new sdk here: https://github.com/awslabs/dynamodb-document-js-sdk

How Do I Make a Faster Riak MapReduce Query?

How can we make our MapReduce Queries Faster?
We have built an application using a five node Riak DB cluster.
Our data model is composed of three buckets: matches, leagues, and teams.
Matches contains links to leagues and teams:
Model
var match = {
id: matchId,
leagueId: meta.leagueId,
homeTeamId: meta.homeTeamId,
awayTeamId: meta.awayTeamId,
startTime: m.match.startTime,
firstHalfStartTime: m.match.firstHalfStartTime,
secondHalfStartTime: m.match.secondHalfStartTime,
score: {
goals: {
a: 1*safeGet(m.match, 'score.goals.a'),
b: 1*safeGet(m.match, 'score.goals.b')
},
corners: {
a: 1*safeGet(m.match, 'score.corners.a'),
b: 1*safeGet(m.match, 'score.corners.b')
}
}
};
var options = {
index: {
leagueId: match.leagueId,
teamId: [match.homeTeamId, match.awayTeamId],
startTime: match.startTime || match.firstHalfStartTime || match.secondHalfStartTime
},
links: [
{ bucket: 'leagues', key: match.leagueId, tag: 'league' },
{ bucket: 'teams', key: match.homeTeamId, tag: 'home' },
{ bucket: 'teams', key: match.awayTeamId, tag: 'away' }
]
};
match.model = 'match';
modelCache.save('matches', match.id, match, options, callback);
Queries
We write a query that returns results from several buckets, one way is to query each bucket separately. The other way is to use links to combine results from a single query.
Two versions of the query we tried both take over a second, no matter how small our bucket size.
The first version uses two map phases, which we modeled after this post (Practical Map-Reduce: Forwarding and Collecting).
#!/bin/bash
curl -X POST \
-H "content-type: application/json" \
-d #- \
http://localhost:8091/mapred \
<<EOF
{
"inputs":{
"bucket":"matches",
"index":"startTime_bin",
"start":"2012-10-22T23:00:00",
"end":"2012-10-24T23:35:00"
},
"query": [
{"map":{"language": "javascript", "source":"
function(value, keydata, arg){
var match = Riak.mapValuesJson(value)[0];
var links = value.values[0].metadata.Links;
var result = links.map(function(l) {
return [l[0], l[1], match];
});
return result;
}
"}
},
{"map":{"language": "javascript", "source": "
function(value, keydata, arg) {
var doc = Riak.mapValuesJson(value)[0];
return [doc, keydata];
}
"}
},
{"reduce":{
"language": "javascript",
"source":"
function(values) {
var merged = {};
values.forEach(function(v) {
if(!merged[v.id]) {
merged[v.id] = v;
}
});
var results = [];
for(key in merged) {
results.push(merged[key]);
}
return results;
}
"
}
}
]
}
EOF
In the second version we do four separate Map-Reduce queries to get the objects from the three buckets:
async.series([
//First get all matches
function(callback) {
db.mapreduce
.add(inputs)
.map(function (val, key, arg) {
var data = Riak.mapValuesJson(val)[0];
if(arg.leagueId && arg.leagueId != data.leagueId) {
return [];
}
var d = new Date();
var date = data.startTime || data.firstHalfStartTime || data.secondHalfStartTime;
d.setFullYear(date.substring(0, 4));
d.setMonth(date.substring(5, 7) - 1);
d.setDate(date.substring(8, 10));
d.setHours(date.substring(11, 13));
d.setMinutes(date.substring(14, 16));
d.setSeconds(date.substring(17, 19));
d.setMilliseconds(0);
startTimestamp = d.getTime();
var short = {
id: data.id,
l: data.leagueId,
h: data.homeTeamId,
a: data.awayTeamId,
t: startTimestamp,
s: data.score,
c: startTimestamp
};
return [short];
}, {leagueId: query.leagueId, page: query.page}).reduce(function (val, key) {
return val;
}).run(function (err, matches) {
matches.forEach(function(match) {
result.match[match.id] = match; //Should maybe filter this
leagueIds.push(match.l);
teamIds.push(match.h);
teamIds.push(match.a);
});
callback();
});
},
//Then get all leagues, teams and lines in parallel
function(callback) {
async.parallel([
//Leagues
function(callback) {
db.getMany('leagues', leagueIds, function(err, leagues) {
if (err) { callback(err); return; }
leagues.forEach(function(league) {
visibleLeagueIds[league.id] = true;
result.league[league.id] = {
r: league.regionId,
n: league.name,
s: league.name
};
});
callback();
});
},
//Teams
function(callback) {
db.getMany('teams', teamIds, function(err, teams) {
if (err) { callback(err); return; }
teams.forEach(function(team) {
result.team[team.id] = {
n: team.name,
h: team.name,
s: team.stats
};
});
callback();
});
}
], callback);
}
], function(err) {
if (err) { callback(err); return; }
_.each(regionModel.getAll(), function(region) {
result.region[region.id] = {
id: region.id,
c: 'https://d1goqbu19rcwi8.cloudfront.net/icons/silk-flags/' + region.icon + '.png',
n: region.name
};
});
var response = {
success: true,
result: {
modelRecords: result,
paging: {
page: query.page,
pageSize: 50,
total: result.match.length
},
time: moment().diff(a)/1000.00,
visibleLeagueIds: visibleLeagueIds
}
};
callback(null, JSON.stringify(response, null, '\t'));
});
How do we make these queries faster?
Additional info:
We are using riak-js and node.js to run our queries.
One way to make it at least a bit faster would be to deploy the JavaScript mapreduce functions to the server instead of passing them through as part of the job. (see description of js_source_dir parameter here). This is usually recommended if you have a JavaScript functions that you run repeatedly.
As there is some overhead associated with running JavaScript mapreduce functions compared to native ones implemented in Erlang, using non-JavaScript functions where possible may also help.
The two map phase functions in your first query appear to be designed to work around the limitation that a normal linking phase (which I believe is more efficient) does not pass on the record being processed (the matches record). The first function includes all the links and passes on the match data as additional data in JSON form, while the second passes on the data of the match as well as the linked record in JSON form.
I have written a simple Erlang function that includes all links as well as the ID of the record passed in. This could be used together with the native Erlang function riak_kv_mapreduce:map_object_value to replace the two map phase functions in your first example, removing some of the JavaScript usage. As in the existing solution, I would expect you to receive a number of duplicates as several matches may link to the same league/team.
-module(riak_mapreduce_example).
-export([map_link/3]).
%% #spec map_link(riak_object:riak_object(), term(), term()) ->
%% [{{Bucket :: binary(), Key :: binary()}, Props :: term()}]
%% #doc map phase function for adding linked records to result set
map_link({error, notfound}, _, _) ->
[];
map_link(RiakObject, Props, _) ->
Bucket = riak_object:bucket(RiakObject),
Key = riak_object:key(RiakObject),
Meta = riak_object:get_metadata(RiakObject),
Current = [{{Bucket, Key}, Props}],
Links = case dict:find(<<"Links">>, Meta) of
{ok, List} ->
[{{B, K}, Props} || {{B, K}, _Tag} <- List];
error ->
[]
end,
lists:append([Current, Links]).
The results of these can either be sent back to the client for aggregation or passed into a reduce phase function as in the example you provided.
The example function would need to be compiled and installed on all nodes, and may require a restart.
Another way to improve performance (that very well may not be an option for you) would perhaps be alter the data model in order to avoid having to use mapreduce queries for performance critical queries altogether.