Best way to query DynamoDB - amazon-web-services

I'm trying to write a GraphQL server that queries an AWS DynamoDB store.
For the purpose of this question, the GraphQL part is irrelevant except for the fact that the arguments come in the form:
{
key1: value1,
key2: value2,
key3: value3
}
These key/value pairs are used to query against the database. Strict equality only, nothing fancy. All arguments are optional.
Here's what I came up with:
import { DynamoDB } from 'aws-sdk';
function constructParams(tableName, fields = {}) {
const keys = Object.keys(fields);
if (keys.length === 0) {
return {
TableName: tableName,
};
}
const filters = keys.map(key => `#${key} = :${key}`);
const attributeNames = keys.reduce((memo, key) => Object.assign(memo, {
[`#${key}`]: key,
}), {});
const attributeValues = keys.reduce((memo, key) => Object.assign(memo, {
[`:${key}`]: fields[key],
}), {});
return {
TableName: tableName,
FilterExpression: filters.join(' AND '),
ExpressionAttributeNames: attributeNames,
ExpressionAttributeValues: attributeValues,
};
}
function query(tableName, fields = {}) {
const docClient = new DynamoDB.DocumentClient({ region: 'ap-southeast-2' });
const params = constructParams(tableName, fields);
return new Promise((resolve, reject) => {
docClient.scan(params, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data.Items);
}
});
});
}
export default query;
This works fine. But goodness it's cumbersome. My constructParams function feels like so much unnecessary boilerplate. Such a function would be completely unnecessary with MongoDB.
Put it this way, why does it feel like I'm writing SQL for a NoSQL database?

What you've written is indeed the shortest code possible for what you are trying to achieve, because that's how DynamoDB expects filtering information for a Scan.
You can also consider moving filtering from DynamoDB-side to client-side, if it helps decrease code complexity. This won't change the consumed read capacity units, so your read cost will remain the same.

Sadly, what I'm trying to achieve just isn't possible with DynamoDB.
The scan method superficially offers this functionality. But it really just returns the entire table set and then filters afterward. Once your data set reaches a certain size, scan ceases to work.
The query method is what you really should be using. But it won't permit queries on more than two keys. This is simply appalling. It is profoundly inadequate for a so called database.
In other words, the DynamoDB API isn't just syntactically primitive. It's also functionally primitive.

Related

how to find the whole array using the array's element in dynamodb?

In DB there is an attribute (name "user_ids") in form of an array that contains user-id [a, b, c, d...]. I want to search that whole array using a single user-id.
Unfortunately, in this case, you have to scan the whole table. DDB is not optimized for this type of operation.
var params = {
TableName: 'my-table-name',
FilterExpression: "#users = :id",
ExpressionAttributeNames: {
"#users": "users"
},
ExpressionAttributeValues: {
":id": ["KwV-yfctBcwCHIw="] // user-id
}
};
dynamo.scan(params, (err, data) => {
if (err) console.error({ err });
console.log(data); // output -> { 'room-id': 'group-2', link: 'asdf', users: ["KwV-yfctBcwCHIw=", "Kqc-wfctacwCsww=", "lqw-yfftBcwqwIw="] },
})
yes you can do this using filterexpression, it's depends how you are storing the data .
either it's "Document Types" (List/Map) or it's Sets,just give a try to filterexpression
you could refer initial aws documentation or there refer ton of example available online.
please refer this link... have some sample code.

aws dynamodb - how to query by date for sort key

I want to query the table and get data which is after certain date.
Hash Key: id
Sort Key: timestamp
error
ValidationException: Query condition missed key schema element: id
index.js
var aws = require('aws-sdk');
const dynamodb = new aws.DynamoDB();
exports.handler = async (event, context, callback) => {
const documentClient = new aws.DynamoDB.DocumentClient();
const params = {
TableName : 'dynamodb-log-testing',
KeyConditionExpression: '#myTimestamp >= :myDate',
ExpressionAttributeNames: {
"#myTimestamp": "timestamp"
},
ExpressionAttributeValues: {
':myDate': '2017-11-17'
}
};
try{
const data = await documentClient.query(params).promise();
console.log(data)
}catch(err){
console.log(err)
}
};
All queries on a DynamoDB table must include the partition key (aka HASH). You can use scan, but it's not recommended for most use cases. You can create a GSI where the partition on the index is a fixed value, and have the same timestamp value for the sort key. That will allow you to query the index in the way you are describing (except that you'll include the fixed value in the query as well). Your query would look like this:
var aws = require('aws-sdk');
const dynamodb = new aws.DynamoDB();
exports.handler = async (event, context, callback) => {
const documentClient = new aws.DynamoDB.DocumentClient();
const params = {
TableName : 'dynamodb-log-testing',
IndexName: 'myGSI1',
KeyConditionExpression: 'gsi1pk = :fixedValue and #myTimestamp >= :myDate',
ExpressionAttributeNames: {
"#myTimestamp": "timestamp"
},
ExpressionAttributeValues: {
':myDate': '2017-11-17',
':fixedValue': 'some fixed value'
}
};
try{
const data = await documentClient.query(params).promise();
console.log(data)
}catch(err){
console.log(err)
}
};
Keep in mind that this model has a strong potential for hot partitions on the GSI if your data is large. For that reason you may want to rethink the access pattern a bit. If you can do something like maybe include the date part of the timestamp in the partition and just include the time in the sort that would help. That does mean that you can't query across days in a single query.
Best Practices for Designing and Architecting with DynamoDB
has a lot of good information on best practices for DynamoDB.

Is it possible to build dynamic queries for Amplify Datastore?

I am looking to create a query-builder for my Amplify Datastore.
The function should process an an array of conditions, that need to be applied to the query and return the according Predicate.
This is easily done, if there is only one filter, but I would like to be able to process any amount of filters.
My goal is to be able to write the queries like so:
Datastore.query(Post, *queryBuilder(filters)*)
Where I can pass an array of filters with a filter looking like this:
filter = {
connector: 'or' |
property: rating
predicate: 'gt'
value: 4
}
and the query builder returns the Predicate in the below mentioned format.
I have tried to chain and return multiple functions in the query builder, but I was not able to figure out a pattern for how to create the correct predicate function.
For reference, this is how queries are built according to the docs: https://docs.amplify.aws/lib/datastore/data-access/q/platform/js#predicates
const posts = await DataStore.query(Post, c => c.rating("gt", 4));
and for multiple conditions:
const posts = await DataStore.query(Post, c =>
c.rating("gt", 4).status("eq", PostStatus.PUBLISHED)
);
Let's say we have the model:
type Post #model{
id: ID!
category: String
city: String
content: String
}
And we want to query & filter by city and category by a dynamic amount of variables. Then we can make a function as such on our script:
const fetchData = async props => {
/*
More configurable wrapper for Datastore.query calls
#param props: {model: Model, criteria: [{fieldId, predicate, value}]}.
*/
try {
let criteria;
if (props.criteria && typeof props.criteria === 'object') {
criteria = c => {
props.criteria.forEach(item => {
const predicate = item.predicate || 'eq';
c[item.fieldId](predicate, item.value);
});
return c;
};
} else {
criteria = props.criteria;
}
return await DataStore.query(props.model, criteria);
} catch (e) {
throw new Error(e);
}
}
So now if we want to execute this we can pass the parameters:
// where Post = models.Post
const myResult = fetchData({model: Post, criteria: [
{ fieldId: 'category',
predicate: 'eq',
value: 'news'
},
{
fieldId: 'city',
predicate: 'eq',
value: 'SomeCityName'
}]
})
Unfortunately I do not know of a way to also query linked relationships as you would using a direct graphQL api query while using DataStore and this method I presented only uses implicit AND between criteria.
I don't know if this has changed since you asked the question but, based on the documents, it looks like multiple conditions have an implicit and, but you can explicitly chain them with or/and/not:
const posts = await DataStore.query(Post, c => c.or(
c => c.rating("gt", 4).status("eq", PostStatus.PUBLISHED)
));

how can I get order '2' < '10' with filter in loopback?

My data like ['2', '13', '13A', '14-1'], How can i get the correct order with filter? Thanks everyone.
IIUC, you are storing numbers (2, 10, etc.) as strings ('2', '10', etc.) in your database.
LoopBack relies on the database to perform ordering (sorting).
Here are few things to try:
Modify your model definition to store the property as number. LoopBack is smart and will coerce string values provided by the user (REST API clients) to numbers before they are stored in the database. This would be my preferred solution, because it does not require any complex code in your application and preserves performance.
Depending on the database you are using, it may be possible to configure it to treat string values as numbers for sorting. This is not LoopBack specific, I can't really help you with that.
As a last resort, you can sort the records in-memory, LoopBack is already doing that for location-based queries when the database does not support them. The idea is to tell the database to return all records matching the filter criteria and then apply order, limit, skip and other options inside your Node.js process. Please note this comes with a severe performance hit and will work only for reasonably-sized data.
As for the 3rd option: implementation wise, you need to override find method in your model class.
// common/models/my-model.js
module.exports = function(MyModel) {
MyModel.on('modelRemoted', () => {
MyModel._findRaw = MyModel.find;
MyModel.find = findWithCustomSort;
});
}
function findWithCustomSort(filter, options, cb) {
if (!cb) {
if (typeof options === 'function') {
cb = options;
options = undefined;
} else if (!options && typeof filter === 'function') {
cb = filter;
filter = undefined;
}
}
const dbFilter = {
where: filter.where,
include: filter.include,
fields: filter.fields,
};
if (cb) {
this._findRaw(dbFilter, options, (err, found) => {
if (err) return cb(err);
else cb(null, sortResults(filter, found))
});
} else {
return this._findRaw(dbFilter, options)
.then(found => sortResults(filter, found));
}
}
function sortResults(filter, data) {
// implement your sorting rules, don't forget about "limit", "skip", etc.
}
UPDATE
Is there a way to use sql for query in custom method?
Yes, you can execute any SQL by using MyModel.dataSource.connector.execute function, see Executing native SQL. There is one catch though - this method is callback based, you cannot use Promise API or async/await.
const idValue = 1;
MyModel.dataSource.connector.execute(
'SELECT * FROM MyModel WHERE id=?',
[idValue]
(err, results) => {
if (err) console.error('query failed', err);
else console.log('found data', results);
});

Loopback: MySQL functions in order filter

I want to use MySQL functions in order filter with Node API.
The function that I need right now is FIELD.
Thanks everyone!
Well Loopback ORM as of this moment doesn't have support for it(and it probably never will) but you can always use the MySQL driver directly:
var ids = [4,5,6,7]
YourModel.dataSource.connector.query('SELECT * FROM something ORDER BY FIELD(id, ?)', ids, (err, results) => {
//...
});
But if you want to be database agnostic, you can do it in Javascript:
var ids = [4,5,6,7]
SomethingModel.find().then(rows => {
rows.sort(function (a, b) {
return ids.indexOf(a) - ids.indexOf(b);
});
//...
});
Or a little bit faster for larger arrays(by indexing):
var idList = [4,5,6,7];
var idMap = {};
idList.forEach(function (id, index) {
idMap[id] = index;
});
SomethingModel.find().then(rows => {
rows.sort(function (a, b) {
return idMap[a] - idMap[b];
});
//...
});
If you do this in many places across your code, you can also live a little dangerously and monkey-patch connector.buildOrderBy to do this natively.