AWS Lambda and Dynamo db: How to filter the result of scan by multiple parameters? - amazon-web-services

I am new to AWS and dynamo. I working on my project with React.js front-end and AWS (Gateway API, Lambda, Dynamo) backend.
This is my app location:
https://www.alphaux.com
After I click "Get Hint", I receive server response. If I click on a keywords - these keywords will be added to the list of GET params for the request like:
topic=blah&keywords=blah1,blah2,blah3
Here are the details of my problem:
In my Lambda:
..
const docClient = new AWS.DynamoDB.DocumentClient({region: 'us-west-2'});
exports.handler = async (event) => { ..
from GET I am receiving the following params:
const topicName = event.queryParams.topic="React";
const keywords = event.queryParams.topic.keywords="blah1,blah2,blah3";
..some code here converts keywords string into array:
const keywordsArray = [blah1,blah2,blah3];
In dynamodb they exist in the following way:
[ { "S" : "javascript" }, { "S" : "programming" }, { "S" : "React" } ]
My Primary partition key is: id (Number)
I have tried different ways and approaches, used scan and query - nothing works. I am stuck..
I have tried the following approach:
const listToObjectMappings = () => {
let x = {};
/* keywords hardcoded for now: */
const keywords = ["javascript", "React"];
keywords.map(item => x[':' + item] = item)
return x
}
let mappings = listToObjectMappings()
let joined = Object.keys(mappings).join();
var params2 = {
TableName : "my-little-table",
FilterExpression: 'topic = :topic and #keywords IN (' + joined + ')',
ExpressionAttributeNames: {
'#keywords' : 'keywords'
},
ExpressionAttributeValues:{
":topic" : "React"
}
};
var params = {
TableName: "my-little-table",
FilterExpression: "#topic = :topic",
ExpressionAttributeNames: {
"#topic": "topic"
},
ExpressionAttributeValues: {
":topic": "React"
}
};
let result;
try {
/* scan DB */
result = await docClient.scan(params).promise();
}
catch(ex) {
result = ex;
}
When I use "scan", the result is always empty [] if I am trying to use "keywords".
It works only with bare minimum like:
const params = {
TableName : currentTable,
FilterExpression:'topic = :topic',
ExpressionAttributeValues:{
":topic" : requestedTopic
}
};
..which gives me all records based on requestedTopic.
If I am using "query" it complains that my key name ("id") is too short and has to be at least 3 chars long.
I am stuck and gracefully asking for your help!
Thank you!

If you're trying to match the topic and any one of the keywords then use something like this:
const params = {
TableName: 'mytable',
FilterExpression: '#tp = :tp AND (contains(#kw, :kw1) OR contains(#kw, :kw2))',
ExpressionAttributeNames: {
'#tp': 'topic',
'#kw': 'keywords',
},
ExpressionAttributeValues: {
':tp': 'React',
':kw1': 'react',
':kw2': 'react-router',
},
};
If you're trying to match the topic and all of the keywords then use something like this:
const params = {
TableName: 'mytable',
FilterExpression: '#tp = :tp AND contains(#kw, :kw1) AND contains(#kw, :kw2)',
ExpressionAttributeNames: {
'#tp': 'topic',
'#kw': 'keywords',
},
ExpressionAttributeValues: {
':tp': 'React',
':kw1': 'react',
':kw2': 'react-router',
},
};

Related

get row by partition key on aws dynamodb

I'm trying to return a a row with id (partition key)
var db = new AWS.DynamoDB.DocumentClient();
const id = "123";
var queryParams = {
TableName: "students",
IndexName: "id",
KeyConditionExpression: "id = :id",
ExpressionAttributeValues: {
":id": id,
},
};
const query = db.query(queryParams).promise();
but is throwing this error:
ValidationException: 1 validation error detected: Value 'id' at 'indexName' failed to satisfy constraint: Member must have length greater than or equal to 3
isn't partition key enough to get a record? I'm confused partition key is not the same as index name?
I also tried with
var db = new AWS.DynamoDB.DocumentClient();
var queryParams = {
TableName: "students",
Key: {
id: "123",
},
};
const query = db.get(queryParams).promise();
and it doesn't return anything
You don't actually have an index named id. You don't need to provide an index when querying on the partition key. Simply remove this line:
IndexName: "id",
Also, note that both table and index names must be 3-255 characters in length (inclusive), which explains the error message that you see.
The following code (both query and get) works fine for me:
const AWS = require('aws-sdk');
AWS.config.update({ region: 'us-east-1' });
const db = new AWS.DynamoDB.DocumentClient();
const ID = "123";
const TABLE_NAME = "students";
const getParams = {
TableName: TABLE_NAME,
Key: {
id: ID,
},
};
const queryParams = {
TableName: TABLE_NAME,
KeyConditionExpression: "id = :id",
ExpressionAttributeValues: {
":id": ID,
},
};
(async () => {
const rc1 = await db.get(getParams).promise();
console.log('get rc:', rc1);
const rc2 = await db.query(queryParams).promise();
console.log('query rc:', rc2);
})();

Having problem in understanding to update data in Dynamodb. I can't understand what is happening in this updateTodo

I have a lambda function which should update the field on dynamodb using AppSync. But i am having difficulty to understand the code. As I cant understand what is the purpose of the for loop and creating the variable "attributes" and the purpose of all those params and the prefix variable. Where as id the the primary key of my dynamo table
const AWS = require('aws-sdk');
const docClient = new AWS.DynamoDB.DocumentClient();
type Params = {
TableName: string | undefined,
Key: string | {},
ExpressionAttributeValues: any,
ExpressionAttributeNames: any,
UpdateExpression: string,
ReturnValues: string
}
async function updateTodo(todo: any) {
let params: Params = {
TableName: process.env.TODOS_TABLE,
Key: {
id: todo.id
},
ExpressionAttributeValues: {},
ExpressionAttributeNames: {},
UpdateExpression: "",
ReturnValues: "UPDATED_NEW"
};
let prefix = "set ";
let attributes = Object.keys(todo);
for (let i = 0; i < attributes.length; i++) {
let attribute = attributes[i];
if (attribute !== "id") {
params["UpdateExpression"] += prefix + "#" + attribute + " = :" + attribute;
params["ExpressionAttributeValues"][":" + attribute] = todo[attribute];
params["ExpressionAttributeNames"]["#" + attribute] = attribute;
prefix = ", ";
}
}
try {
await docClient.update(params).promise()
return todo
} catch (err) {
console.log('DynamoDB error: ', err)
return null
}
}
export default updateTodo;
This method is building the required attributes for the updateItem method.
Here's an example of what the parameters being passed to the upateItem method should look like:
const params = {
TableName: "YOUR_TABLE_NAME",
Key: {
"id": "1"
},
UpdateExpression: "set #attribute1 = :a1, #attribute2 = :a2",
ExpressionAttributeNames: {
"#attribute1": "attribute1_name"
"#attribute1": "attribute2_name"
},
ExpressionAttributeValues: {
":a1": "attribute 1 value",
":a2": "attribute 2 value"
}
};
The updateTodo method is building this parameter hash dynamically in several steps:
This block of code is creating a map called params. It's specifying the table name by looking up the TODOS_TABLE environment variable. The Key refers to the partition key, which in this example is named id.
let params: Params = {
TableName: process.env.TODOS_TABLE,
Key: {
id: todo.id
},
ExpressionAttributeValues: {},
ExpressionAttributeNames: {},
UpdateExpression: "",
ReturnValues: "UPDATED_NEW"
};
Notice how the ExpressionAttributeValues,ExpressionAttributeNames and UpdateExpression keys all have empty values.
The next step is to set the params[ExpressionAttributeValues] and params[ExpressionAttributeNames] as required by the updateItem API.
let prefix = "set ";
let attributes = Object.keys(todo);
for (let i = 0; i < attributes.length; i++) {
let attribute = attributes[i];
if (attribute !== "id") {
params["UpdateExpression"] += prefix + "#" + attribute + " = :" + attribute;
params["ExpressionAttributeValues"][":" + attribute] = todo[attribute];
params["ExpressionAttributeNames"]["#" + attribute] = attribute;
prefix = ", ";
}
}
The end result is a params map that would look something like this:
const params = {
TableName: "YOUR_TABLE_NAME",
Key: {
"id": "1"
},
UpdateExpression: "set #attribute1 = :a1, #attribute2 = :a2",
ExpressionAttributeNames: {
"#attribute1": "attribute1_name"
"#attribute1": "attribute2_name"
},
ExpressionAttributeValues: {
":a1": "attribute 1 value",
":a2": "attribute 2 value"
}
};

More than 2 conditions in FilterExpression

The scan works fine if I provide two or less filter conditions, like this:
FilterExpression: '#locationID = :areaID and #price >= :minPrice'
But if I add a third condition, I get no results:
FilterExpression: '#locationID = :areaID and #price >= :minPrice and #offeringType = :offeringType'
I tried using only #offeringType, the condition is working properly by itself.
Can I have more than two conditions?
Full code:
let params = {
TableName: 'properties',
ExpressionAttributeNames: {
'#locationID': 'locationID',
'#price': 'price',
'#offeringType': 'offeringType',
},
ExpressionAttributeValues: {
':areaID': areaID,
':minPrice': parseInt(minPrice),
':offeringType': offeringType
},
FilterExpression: '#locationID = :areaID and #price >= :minPrice and #offeringType = :offeringType'
}
documentClient.scan(params, (err, data) => {
if (err) {
console.log(err)
reject()
} else {
resolve(data)
}
})
Thanks
The expression was acting weird because #price wasn't a Number in the database. Chaining expressions with AND works fine.

AWS RDS Data API executeStatement not return column names

I'm playing with the New Data API for Amazon Aurora Serverless
Is it possible to get the table column names in the response?
If for example I run the following query in a user table with the columns id, first_name, last_name, email, phone:
const sqlStatement = `
SELECT *
FROM user
WHERE id = :id
`;
const params = {
secretArn: <mySecretArn>,
resourceArn: <myResourceArn>,
database: <myDatabase>,
sql: sqlStatement,
parameters: [
{
name: "id",
value: {
"stringValue": 1
}
}
]
};
let res = await this.RDS.executeStatement(params)
console.log(res);
I'm getting a response like this one, So I need to guess which column corresponds with each value:
{
"numberOfRecordsUpdated": 0,
"records": [
[
{
"longValue": 1
},
{
"stringValue": "Nicolas"
},
{
"stringValue": "Perez"
},
{
"stringValue": "example#example.com"
},
{
"isNull": true
}
]
]
}
I would like to have a response like this one:
{
id: 1,
first_name: "Nicolas",
last_name: "Perez",
email: "example#example.com",
phone: null
}
update1
I have found an npm module that wrap Aurora Serverless Data API and simplify the development
We decided to take the current approach because we were trying to cut down on the response size and including column information with each record was redundant.
You can explicitly choose to include column metadata in the result. See the parameter: "includeResultMetadata".
https://docs.aws.amazon.com/rdsdataservice/latest/APIReference/API_ExecuteStatement.html#API_ExecuteStatement_RequestSyntax
Agree with the consensus here that there should be an out of the box way to do this from the data service API. Because there is not, here's a JavaScript function that will parse the response.
const parseDataServiceResponse = res => {
let columns = res.columnMetadata.map(c => c.name);
let data = res.records.map(r => {
let obj = {};
r.map((v, i) => {
obj[columns[i]] = Object.values(v)[0]
});
return obj
})
return data
}
I understand the pain but it looks like this is reasonable based on the fact that select statement can join multiple tables and duplicated column names may exist.
Similar to the answer above from #C.Slack but I used a combination of map and reduce to parse response from Aurora Postgres.
// declarative column names in array
const columns = ['a.id', 'u.id', 'u.username', 'g.id', 'g.name'];
// execute sql statement
const params = {
database: AWS_PROVIDER_STAGE,
resourceArn: AWS_DATABASE_CLUSTER,
secretArn: AWS_SECRET_STORE_ARN,
// includeResultMetadata: true,
sql: `
SELECT ${columns.join()} FROM accounts a
FULL OUTER JOIN users u ON u.id = a.user_id
FULL OUTER JOIN groups g ON g.id = a.group_id
WHERE u.username=:username;
`,
parameters: [
{
name: 'username',
value: {
stringValue: 'rick.cha',
},
},
],
};
const rds = new AWS.RDSDataService();
const response = await rds.executeStatement(params).promise();
// parse response into json array
const data = response.records.map((record) => {
return record.reduce((prev, val, index) => {
return { ...prev, [columns[index]]: Object.values(val)[0] };
}, {});
});
Hope this code snippet helps someone.
And here is the response
[
{
'a.id': '8bfc547c-3c42-4203-aa2a-d0ee35996e60',
'u.id': '01129aaf-736a-4e86-93a9-0ab3e08b3d11',
'u.username': 'rick.cha',
'g.id': 'ff6ebd78-a1cf-452c-91e0-ed5d0aaaa624',
'g.name': 'valentree',
},
{
'a.id': '983f2919-1b52-4544-9f58-c3de61925647',
'u.id': '01129aaf-736a-4e86-93a9-0ab3e08b3d11',
'u.username': 'rick.cha',
'g.id': '2f1858b4-1468-447f-ba94-330de76de5d1',
'g.name': 'ensightful',
},
]
Similar to the other answers, but if you are using Python/Boto3:
def parse_data_service_response(res):
columns = [column['name'] for column in res['columnMetadata']]
parsed_records = []
for record in res['records']:
parsed_record = {}
for i, cell in enumerate(record):
key = columns[i]
value = list(cell.values())[0]
parsed_record[key] = value
parsed_records.append(parsed_record)
return parsed_records
I've added to the great answer already provided by C. Slack to deal with AWS handling empty nullable character fields by giving the response { "isNull": true } in the JSON.
Here's my function to handle this by returning an empty string value - this is what I would expect anyway.
const parseRDSdata = (input) => {
let columns = input.columnMetadata.map(c => { return { name: c.name, typeName: c.typeName}; });
let parsedData = input.records.map(row => {
let response = {};
row.map((v, i) => {
//test the typeName in the column metadata, and also the keyName in the values - we need to cater for a return value of { "isNull": true } - pflangan
if ((columns[i].typeName == 'VARCHAR' || columns[i].typeName == 'CHAR') && Object.keys(v)[0] == 'isNull' && Object.values(v)[0] == true)
response[columns[i].name] = '';
else
response[columns[i].name] = Object.values(v)[0];
}
);
return response;
}
);
return parsedData;
}

DynamoDB Update - ExpressionAttributeNames can only be specified when using expressions

I need another set of eyes on this. For the life of me I see no issues with this parameter set, used for Dynamo DocumentClient, update method - (here: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#update-property ).
{
TableName: "mygame-dev",
Key: { pk: "09d017aa-cbf7-42ce-be6a-a94ecb58f9a7", sk: "GAME" },
ExpressionAttributeNames: { "#GAMELASTUPDATED": "gameLastUpdated", "#GAMETITLE": "gameTitle" },
ExpressionAttributeValues: { ":gamelastupdated": 1556376010704, ":gametitle": "test title 1" },
UpdateExpression: "SET #GAMELASTUPDATED = :gamelastupdated, #GAMETITLE = :gametitle",
ReturnValues: "ALL_NEW"
};
Error:
ValidationException: ExpressionAttributeNames can only be specified
when using expressions
Any thoughts?
Disregard, this was a copy and paste issue... I was using "query" not "update"
Was:
const updateGameResult = await ddbCall("query", params);
Should have been:
const updateGameResult = await ddbCall("update", params);