I got a function in AWS Lambda that lists every patient in a table from DynamoDB. I realized that some items from the table were not on the list. This is my function to list:
module.exports.listPatients = async (event) => {
try {
const queryString = {
limit: 5,
...event.queryStringParameters,
};
const { limit, next, name } = queryString;
const localParams = {
...patientsParams,
Limit: limit,
FilterExpression: "contains(full_name, :full_name)",
ExpressionAttributeValues: { ":full_name": name },
};
if (next) {
localParams.ExclusiveStartKey = {
id: next,
};
}
const data = await dynamoDb.scan(localParams).promise();
const nextToken = data.LastEvaluatedKey ? data.LastEvaluatedKey.id : "";
const result = {
items: data.Items,
next_token: nextToken,
};
return {
statusCode: 200,
body: JSON.stringify(result),
};
} catch (error) {
console.log("Error: ", error);
return {
statusCode: error.statusCode ? error.statusCode : 500,
body: JSON.stringify({
error: error.name ? error.name : "Exception",
message: error.message ? error.message : "Unknown error",
}),
};
}
};
Am I missing something?
I tried with and without a limit, removed the filters, and yet nothing.
I tested one of the ids with get() to test with the server can find one of those who are missing, and it worked.
I am using Serverless to deploy the code, and when I try offline, it's working.
Stackoverflow recommended this post when writing my question, but I am using DynamoDB.DocumentClient without specifying the full attribute type in the filter expression:
How to scan in DynamoDB without primary sort key with Nodejs
Looks like you are paginating using scan(). Using query() with some Global Secondary Indexes and ScanIndexForward would give you a much better performance. scan() doesn't scale well when your data grows.
I am trying to update the value of a table using the AWS-app sync graphql API,
I am able to create data and add it in a table using graphql mutation in lambda
but when I am trying to update the data its not working.
I am calling this lambda service from an API Gateway.
I am referring this article to code
https://cloudonaut.io/calling-appsync-graphql-from-lambda/
I would like to mentioned git no error in cloud watch log
Here is the schema for my graphql
type Mutation {
createLib_content(input: CreateLib_contentInput!): lib_content
#aws_iam
updateLib_content(input: UpdateLib_contentInput!): lib_content
#aws_iam
deleteLib_content(input: DeleteLib_contentInput!): lib_content
}
input CreateLib_contentInput {
content: String
userId: String
}
input UpdateLib_contentInput {
content: String
id: ID!
}
Create Mutation
graphqlData = await clientDetails.mutate({
mutation: gql(`
mutation CreateLibContent($input: CreateLib_contentInput!) {
createLib_content(input: $input) {
id
content
}
}`),
variables: {
input: {
content : {},
userId : identitiesDetails.userId
}
},
});
Update Mutation
const mutation = gql(`
mutation UpdateLibContent($input: UpdateLib_contentInput!) {
updateLib_content(input: $input) {
userId
content
}
}`);
await clientDetails.mutate({
mutation,
variables: {
input: {
id : "2947c37e-6f76-40d8-8c10-4cd6190d3597",
content : JSON.stringify(event)
}
}
}).promise;
Thanks to #cppgnlearner your guess were right.
I just removed the .promise from my update code
And it started working.
can't believe such a small thing took my whole day.
I am building an application with ionic3 and AWS lanbda, where i use dynamodb for my database, the primary key generated as the "Id" ( in my case "groupId" ) of the table is too long afc9c380-db7d-11e7-a2c5-3bce7c435d3e I would like to know if there is a way to limit the no of digits to 6 characters? Thank you
My typescript code - "groupId is auto Genrated"
addGroup(form) {
this.submitted = true;
if (form && this.formData.name) {
let group = {
name: this.formData.name,
description: this.formData.description,
profileImageURI: this.profileImageURI,
userId: this.globals.getUserId(),
imageUrl: this.formData.imageUrl
};
this.globals.displayLoader("Adding...");
this.client.getClient().groupsCreateByUserId(this.globals.getUserId(), group).subscribe(
(data) => {
this.globals.dismissLoader();
this.globals.displayToast(`Group successfully added.`);
this.navCtrl.pop();
},
(err) => {
this.globals.dismissLoader();
this.globals.displayAlert('Error encountered',
`An error occurred when trying to add the group. Please check the console logs for more information.`);
console.error(err);
}
);
}
this.globals.dismissLoader();
}
I'm using AWS Lambda and try to write something to AWS DynamoDB. I use the following code:
var tableName = "locations";
var item = {
deviceId: {
S: event.deviceId
},
timestamps: {
S: event.timestamp
}
}
var params = {
TableName: tableName,
Item: item
};
dynamo.putItem(params, function(err, data) {
if (err) {
context.fail(new Error('Error ' + err));
} else {
context.success(null);
}
});
And I get the following error:
returns Error ValidationException: One or more parameter values were invalid: Type mismatch for key deviceId expected: S actual: M
This happened because the aws sdk for Nodejs had changed!
If you are using:
var doc = require('dynamodb-doc');
var dynamo = new doc.DynamoDB();
Then the parameters to the putItem call (and most other calls) have changed and instead needs to be:
var tableName = "locations";
var item = {
deviceId: event.deviceId,
timestamp: event.timestamp,
latitude: Number(event.latitude),
longitude: Number(event.longitude)
}
var params = {
TableName: tableName,
Item: item
};
Read all about the new sdk here: https://github.com/awslabs/dynamodb-document-js-sdk
How can we make our MapReduce Queries Faster?
We have built an application using a five node Riak DB cluster.
Our data model is composed of three buckets: matches, leagues, and teams.
Matches contains links to leagues and teams:
Model
var match = {
id: matchId,
leagueId: meta.leagueId,
homeTeamId: meta.homeTeamId,
awayTeamId: meta.awayTeamId,
startTime: m.match.startTime,
firstHalfStartTime: m.match.firstHalfStartTime,
secondHalfStartTime: m.match.secondHalfStartTime,
score: {
goals: {
a: 1*safeGet(m.match, 'score.goals.a'),
b: 1*safeGet(m.match, 'score.goals.b')
},
corners: {
a: 1*safeGet(m.match, 'score.corners.a'),
b: 1*safeGet(m.match, 'score.corners.b')
}
}
};
var options = {
index: {
leagueId: match.leagueId,
teamId: [match.homeTeamId, match.awayTeamId],
startTime: match.startTime || match.firstHalfStartTime || match.secondHalfStartTime
},
links: [
{ bucket: 'leagues', key: match.leagueId, tag: 'league' },
{ bucket: 'teams', key: match.homeTeamId, tag: 'home' },
{ bucket: 'teams', key: match.awayTeamId, tag: 'away' }
]
};
match.model = 'match';
modelCache.save('matches', match.id, match, options, callback);
Queries
We write a query that returns results from several buckets, one way is to query each bucket separately. The other way is to use links to combine results from a single query.
Two versions of the query we tried both take over a second, no matter how small our bucket size.
The first version uses two map phases, which we modeled after this post (Practical Map-Reduce: Forwarding and Collecting).
#!/bin/bash
curl -X POST \
-H "content-type: application/json" \
-d #- \
http://localhost:8091/mapred \
<<EOF
{
"inputs":{
"bucket":"matches",
"index":"startTime_bin",
"start":"2012-10-22T23:00:00",
"end":"2012-10-24T23:35:00"
},
"query": [
{"map":{"language": "javascript", "source":"
function(value, keydata, arg){
var match = Riak.mapValuesJson(value)[0];
var links = value.values[0].metadata.Links;
var result = links.map(function(l) {
return [l[0], l[1], match];
});
return result;
}
"}
},
{"map":{"language": "javascript", "source": "
function(value, keydata, arg) {
var doc = Riak.mapValuesJson(value)[0];
return [doc, keydata];
}
"}
},
{"reduce":{
"language": "javascript",
"source":"
function(values) {
var merged = {};
values.forEach(function(v) {
if(!merged[v.id]) {
merged[v.id] = v;
}
});
var results = [];
for(key in merged) {
results.push(merged[key]);
}
return results;
}
"
}
}
]
}
EOF
In the second version we do four separate Map-Reduce queries to get the objects from the three buckets:
async.series([
//First get all matches
function(callback) {
db.mapreduce
.add(inputs)
.map(function (val, key, arg) {
var data = Riak.mapValuesJson(val)[0];
if(arg.leagueId && arg.leagueId != data.leagueId) {
return [];
}
var d = new Date();
var date = data.startTime || data.firstHalfStartTime || data.secondHalfStartTime;
d.setFullYear(date.substring(0, 4));
d.setMonth(date.substring(5, 7) - 1);
d.setDate(date.substring(8, 10));
d.setHours(date.substring(11, 13));
d.setMinutes(date.substring(14, 16));
d.setSeconds(date.substring(17, 19));
d.setMilliseconds(0);
startTimestamp = d.getTime();
var short = {
id: data.id,
l: data.leagueId,
h: data.homeTeamId,
a: data.awayTeamId,
t: startTimestamp,
s: data.score,
c: startTimestamp
};
return [short];
}, {leagueId: query.leagueId, page: query.page}).reduce(function (val, key) {
return val;
}).run(function (err, matches) {
matches.forEach(function(match) {
result.match[match.id] = match; //Should maybe filter this
leagueIds.push(match.l);
teamIds.push(match.h);
teamIds.push(match.a);
});
callback();
});
},
//Then get all leagues, teams and lines in parallel
function(callback) {
async.parallel([
//Leagues
function(callback) {
db.getMany('leagues', leagueIds, function(err, leagues) {
if (err) { callback(err); return; }
leagues.forEach(function(league) {
visibleLeagueIds[league.id] = true;
result.league[league.id] = {
r: league.regionId,
n: league.name,
s: league.name
};
});
callback();
});
},
//Teams
function(callback) {
db.getMany('teams', teamIds, function(err, teams) {
if (err) { callback(err); return; }
teams.forEach(function(team) {
result.team[team.id] = {
n: team.name,
h: team.name,
s: team.stats
};
});
callback();
});
}
], callback);
}
], function(err) {
if (err) { callback(err); return; }
_.each(regionModel.getAll(), function(region) {
result.region[region.id] = {
id: region.id,
c: 'https://d1goqbu19rcwi8.cloudfront.net/icons/silk-flags/' + region.icon + '.png',
n: region.name
};
});
var response = {
success: true,
result: {
modelRecords: result,
paging: {
page: query.page,
pageSize: 50,
total: result.match.length
},
time: moment().diff(a)/1000.00,
visibleLeagueIds: visibleLeagueIds
}
};
callback(null, JSON.stringify(response, null, '\t'));
});
How do we make these queries faster?
Additional info:
We are using riak-js and node.js to run our queries.
One way to make it at least a bit faster would be to deploy the JavaScript mapreduce functions to the server instead of passing them through as part of the job. (see description of js_source_dir parameter here). This is usually recommended if you have a JavaScript functions that you run repeatedly.
As there is some overhead associated with running JavaScript mapreduce functions compared to native ones implemented in Erlang, using non-JavaScript functions where possible may also help.
The two map phase functions in your first query appear to be designed to work around the limitation that a normal linking phase (which I believe is more efficient) does not pass on the record being processed (the matches record). The first function includes all the links and passes on the match data as additional data in JSON form, while the second passes on the data of the match as well as the linked record in JSON form.
I have written a simple Erlang function that includes all links as well as the ID of the record passed in. This could be used together with the native Erlang function riak_kv_mapreduce:map_object_value to replace the two map phase functions in your first example, removing some of the JavaScript usage. As in the existing solution, I would expect you to receive a number of duplicates as several matches may link to the same league/team.
-module(riak_mapreduce_example).
-export([map_link/3]).
%% #spec map_link(riak_object:riak_object(), term(), term()) ->
%% [{{Bucket :: binary(), Key :: binary()}, Props :: term()}]
%% #doc map phase function for adding linked records to result set
map_link({error, notfound}, _, _) ->
[];
map_link(RiakObject, Props, _) ->
Bucket = riak_object:bucket(RiakObject),
Key = riak_object:key(RiakObject),
Meta = riak_object:get_metadata(RiakObject),
Current = [{{Bucket, Key}, Props}],
Links = case dict:find(<<"Links">>, Meta) of
{ok, List} ->
[{{B, K}, Props} || {{B, K}, _Tag} <- List];
error ->
[]
end,
lists:append([Current, Links]).
The results of these can either be sent back to the client for aggregation or passed into a reduce phase function as in the example you provided.
The example function would need to be compiled and installed on all nodes, and may require a restart.
Another way to improve performance (that very well may not be an option for you) would perhaps be alter the data model in order to avoid having to use mapreduce queries for performance critical queries altogether.