i have the objects each having timestamp field , consist datetime like
data1 --> "2014-06-26T11:06:19.315404+00:00"
data2 --> "2014-06-26T08:04:29.755019+00:00"
data3 --> "2014-06-26T08:03:51.008563+00:00"
using underscore.js i;m trying to group the data by timestamp field like
grp_data = _.groupBy(result, "timestamp")
Expected result:
{"2014-06-26": [data1, data2, data3] }
what is happening is :
{"2014-06-26": [data1],
{"2014-06-26": [data2],
{"2014-06-26": [data3],
}
the reason of above behavior is since the each timestamp of data1, data2, data3 are different because of T11:06:19.315404+00:00" which is appending in timestamp.
that is why data is not grouped expectedly.
Is there any way around this ?
Pass a function to the groupBy that extracts the date part:
var result = [
{ id: 'data1', timestamp: "2014-06-26T11:06:19.315404+00:00" },
{ id: 'data2', timestamp: "2014-06-26T11:06:19.315404+00:00" },
{ id: 'data3', timestamp: "2014-06-26T11:06:19.315404+00:00" }
];
var extractDate = function(datum){
return datum.timestamp.substring(0,10);
}
var grp_data = _.groupBy(result, extractDate);
Related
I am using JS SDK with DynamoDB to fetch data.
I am able to fetch data from my table using simple query with partition key and sort key.
My sort key sk has records -
Year#Batch#Rate
If I pass var sk = "2006#CSE#90"; it returns all of records matching this,
Requirement - How can I get all products with year 2006 , Batch CSE AND Rate =>90
readItem_pbro(){
console.log("inside pbro");
var table2 = "pbro";
var pk = "1";
var sk = "2006#CSE#90";
var params2 = {
TableName: table2,
Key:{
"pk": pk,
"sk": sk
}
};
Edit 1 :: Created a different column for score/rate as score. It is numeric.
Now my query in JS is -
but I am getting error - ValidationException: The provided key element does not match the schema
readItem_score_filter(){
console.log("inside pbro");
var table2 = "pbro";
var pk = "1"; // string
var sk = "2006#CSE"; // string
var score = 90; //number
var params2 = {
TableName: table2,
Key:{
"pk": pk,
"sk": sk,
FilterExpression:'score >=:score',
}
};
what is wrong in my FilterExpression.
Edit 2 :: Added Key condition Expression but issue still remains the same
Error: ValidationException: The provided key element does not match the schema
Here is my complete function now:
readItem_score_filter(){
console.log("inside pbro");
var table2 = "pbro";
var pk = "1"; //string
var sk = "2006#CSE"; // string
var score = 90; //number
var params2 = {
TableName: table2,
Key:{
"pk": pk,
"sk": sk,
"score": score,
KeyConditionExpression: 'pk = :pk AND sk=:sk',
FilterExpression: "score >=:score",
}
};
this.user.docClient.get(params2, function(err, data) {
if (err) {
console.log(err);
} else {
console.log(data);
}
});
}
Screenshot of table attached incase you need to see::
If "2006#CSE#90" this is the value of sort key column then you cant do anything at Dynamodb level..
comparings like this can be done through regular expressions but DynamoDB doesn't support Regular Expressions.
you simply need to get results and then seperate these values and compare ..
Updated :- use different column for score.
And use Filter Expression to get records having rate more than 90.
I dont know python , but still am trying here
var params2 = {
TableName: "pbro",
KeyConditionExpression: "pk = :pk AND sk =:sk",
FilterExpression: "score >= :score"
};
I'm playing with the New Data API for Amazon Aurora Serverless
Is it possible to get the table column names in the response?
If for example I run the following query in a user table with the columns id, first_name, last_name, email, phone:
const sqlStatement = `
SELECT *
FROM user
WHERE id = :id
`;
const params = {
secretArn: <mySecretArn>,
resourceArn: <myResourceArn>,
database: <myDatabase>,
sql: sqlStatement,
parameters: [
{
name: "id",
value: {
"stringValue": 1
}
}
]
};
let res = await this.RDS.executeStatement(params)
console.log(res);
I'm getting a response like this one, So I need to guess which column corresponds with each value:
{
"numberOfRecordsUpdated": 0,
"records": [
[
{
"longValue": 1
},
{
"stringValue": "Nicolas"
},
{
"stringValue": "Perez"
},
{
"stringValue": "example#example.com"
},
{
"isNull": true
}
]
]
}
I would like to have a response like this one:
{
id: 1,
first_name: "Nicolas",
last_name: "Perez",
email: "example#example.com",
phone: null
}
update1
I have found an npm module that wrap Aurora Serverless Data API and simplify the development
We decided to take the current approach because we were trying to cut down on the response size and including column information with each record was redundant.
You can explicitly choose to include column metadata in the result. See the parameter: "includeResultMetadata".
https://docs.aws.amazon.com/rdsdataservice/latest/APIReference/API_ExecuteStatement.html#API_ExecuteStatement_RequestSyntax
Agree with the consensus here that there should be an out of the box way to do this from the data service API. Because there is not, here's a JavaScript function that will parse the response.
const parseDataServiceResponse = res => {
let columns = res.columnMetadata.map(c => c.name);
let data = res.records.map(r => {
let obj = {};
r.map((v, i) => {
obj[columns[i]] = Object.values(v)[0]
});
return obj
})
return data
}
I understand the pain but it looks like this is reasonable based on the fact that select statement can join multiple tables and duplicated column names may exist.
Similar to the answer above from #C.Slack but I used a combination of map and reduce to parse response from Aurora Postgres.
// declarative column names in array
const columns = ['a.id', 'u.id', 'u.username', 'g.id', 'g.name'];
// execute sql statement
const params = {
database: AWS_PROVIDER_STAGE,
resourceArn: AWS_DATABASE_CLUSTER,
secretArn: AWS_SECRET_STORE_ARN,
// includeResultMetadata: true,
sql: `
SELECT ${columns.join()} FROM accounts a
FULL OUTER JOIN users u ON u.id = a.user_id
FULL OUTER JOIN groups g ON g.id = a.group_id
WHERE u.username=:username;
`,
parameters: [
{
name: 'username',
value: {
stringValue: 'rick.cha',
},
},
],
};
const rds = new AWS.RDSDataService();
const response = await rds.executeStatement(params).promise();
// parse response into json array
const data = response.records.map((record) => {
return record.reduce((prev, val, index) => {
return { ...prev, [columns[index]]: Object.values(val)[0] };
}, {});
});
Hope this code snippet helps someone.
And here is the response
[
{
'a.id': '8bfc547c-3c42-4203-aa2a-d0ee35996e60',
'u.id': '01129aaf-736a-4e86-93a9-0ab3e08b3d11',
'u.username': 'rick.cha',
'g.id': 'ff6ebd78-a1cf-452c-91e0-ed5d0aaaa624',
'g.name': 'valentree',
},
{
'a.id': '983f2919-1b52-4544-9f58-c3de61925647',
'u.id': '01129aaf-736a-4e86-93a9-0ab3e08b3d11',
'u.username': 'rick.cha',
'g.id': '2f1858b4-1468-447f-ba94-330de76de5d1',
'g.name': 'ensightful',
},
]
Similar to the other answers, but if you are using Python/Boto3:
def parse_data_service_response(res):
columns = [column['name'] for column in res['columnMetadata']]
parsed_records = []
for record in res['records']:
parsed_record = {}
for i, cell in enumerate(record):
key = columns[i]
value = list(cell.values())[0]
parsed_record[key] = value
parsed_records.append(parsed_record)
return parsed_records
I've added to the great answer already provided by C. Slack to deal with AWS handling empty nullable character fields by giving the response { "isNull": true } in the JSON.
Here's my function to handle this by returning an empty string value - this is what I would expect anyway.
const parseRDSdata = (input) => {
let columns = input.columnMetadata.map(c => { return { name: c.name, typeName: c.typeName}; });
let parsedData = input.records.map(row => {
let response = {};
row.map((v, i) => {
//test the typeName in the column metadata, and also the keyName in the values - we need to cater for a return value of { "isNull": true } - pflangan
if ((columns[i].typeName == 'VARCHAR' || columns[i].typeName == 'CHAR') && Object.keys(v)[0] == 'isNull' && Object.values(v)[0] == true)
response[columns[i].name] = '';
else
response[columns[i].name] = Object.values(v)[0];
}
);
return response;
}
);
return parsedData;
}
{ _id: 1, results: [ "tokyo", "japan" ] }
{ _id: 2, results: [ "sydney", "australia" ] }
db.scores.find(
{ results: { $elemMatch: { $regex: *some regex* } } }
)
How do you convert this simple elemMatch example using spring mongodb data Query Criteria?
If the array contains object I can do it this way
Criteria criteria =
Criteria.where("results").
elemMatch(
Criteria.where("field").is("tokyo")
);
But in my question, I dont have the "field"
Update:
I thought the Veeram's answer was going to work after trying it out
Criteria criteria =
Criteria.where("results").
elemMatch(
new Criteria().is("tokyo")
);
It does not return anything. Am I missing something?
When i inspect the query object, it states the following:
Query: { "setOfKeys" : { "$elemMatch" : { }}}, Fields: null, Sort: null
On the other hand, If i modify the criteria using Criteria.where("field") like above,
Query: { "setOfKeys" : { "$elemMatch" : { "field" : "tokyo"}}}, Fields: null, Sort: null
I'm getting something but that's not how my data was structured, results is an array of strings not objects.
I actually need to use regex, for simplicity , the above example is using .is
You can try below query.
Criteria criteria = Criteria.where("results").elemMatch(new Criteria().gte(80).lt(85));
Try this
Criteria criteria = Criteria.where("results").regex(".*tokyo.*","i");
In one of my mongodb collection, I have a date string that has a mm/dd/yyyy format. Now, I want to query the 'mm' string.
Example, 05/20/2016 and 04/05/2015.
I want to get the first 2 characters of the string and query '05'. With that, the result I will get should only be 05/20/2016.
How can I achieve this?
Thanks!
For a regex solution, the following will suffice
var search = "05",
rgx = new RegExp("^"+search); // equivalent to var rgx = /^05/;
db.collection.find({ "leave_start": rgx });
Testing
var leave_start = "05/06/2016",
test = leave_start.match(/^05/);
console.log(test); // ["05", index: 0, input: "05/06/2016"]
console.log(test[0]); // "05"
or
var search = "05",
rgx = new RegExp("^"+search),
leave_start = "05/12/2016";
var test = leave_start.match(rgx);
console.log(test); // ["05", index: 0, input: "05/06/2016"]
console.log(test[0]); // "05"
Another alternative is to use the aggregation framework and take advantage of the $substr operator to extract the first 2 characters of a field and then the $match operator will filter documents based on the new substring field above:
db.collection.aggregate([
{
"$project": {
"leaves_start": 1,
"monthSubstring": { "$substr": : [ "$leaves_start", 0, 2 ] }
}
},
{ "$match": { "monthSubstring": "05" } }
])
I am so sorry, but after one day researching and trying all different combinations and npm packages, I am still not sure how to deal with the following task.
Setup:
MongoDB 2.6
Node.JS with Mongoose 4
I have a schema like so:
var trackingSchema = mongoose.Schema({
tracking_number: String,
zip_code: String,
courier: String,
user_id: Number,
created: { type: Date, default: Date.now },
international_shipment: { type: Boolean, default: false },
delivery_info: {
recipient: String,
street: String,
city: String
}
});
Now user gives me a search string, a rather an array of strings, which will be substrings of what I want to search:
var search = ['15323', 'julian', 'administ'];
Now I want to find those documents, where any of the fields tracking_number, zip_code, or these fields in delivery_info contain my search elements.
How should I do that? I get that there are indexes, but I probably need a compound index, or maybe a text index? And for search, I then can use RegEx, or the $text $search syntax?
The problem is that I have several strings to look for (my search), and several fields to look in. And due to one of those aspects, every approach failed for me at some point.
Your use case is a good fit for text search.
Define a text index on your schema over the searchable fields:
trackingSchema.index({
tracking_number: 'text',
zip_code: 'text',
'delivery_info.recipient': 'text',
'delivery_info.street': 'text',
'delivery_info.city': 'text'
}, {name: 'search'});
Join your search terms into a single string and execute the search using the $text query operator:
var search = ['15232', 'julian'];
Test.find({$text: {$search: search.join(' ')}}, function(err, docs) {...});
Even though this passes all your search values as a single string, this still performs a logical OR search of the values.
Why just dont try
var trackingSchema = mongoose.Schema({
tracking_number: String,
zip_code: String,
courier: String,
user_id: Number,
created: { type: Date, default: Date.now },
international_shipment: { type: Boolean, default: false },
delivery_info: {
recipient: String,
street: String,
city: String
}
});
var Tracking = mongoose.model('Tracking', trackingSchema );
var search = [ "word1", "word2", ...]
var results = []
for(var i=0; i<search.length; i++){
Tracking.find({$or : [
{ tracking_number : search[i]},
{zip_code: search[i]},
{courier: search[i]},
{delivery_info.recipient: search[i]},
{delivery_info.street: search[i]},
{delivery_info.city: search[i]}]
}).map(function(tracking){
//it will push every unique result to variable results
if(results.indexOf(tracking)<0) results.push(tracking);
});
Okay, I came up with this.
My schema now has an extra field search with an array of all my searchable fields:
var trackingSchema = mongoose.Schema({
...
search: [String]
});
With a pre-save hook, I populate this field:
trackingSchema.pre('save', function(next) {
this.search = [ this.tracking_number ];
var searchIfAvailable = [
this.zip_code,
this.delivery_info.recipient,
this.delivery_info.street,
this.delivery_info.city
];
for (var i = 0; i < searchIfAvailable.length; i++) {
if (!validator.isNull(searchIfAvailable[i])) {
this.search.push(searchIfAvailable[i].toLowerCase());
}
}
next();
});
In the hope of improving performance, I also index that field (also the user_id as I limit search results by that):
trackingSchema.index({ search: 1 });
trackingSchema.index({ user_id: 1 });
Now, when searching I first list all substrings I want to look for in an array:
var andArray = [];
var searchTerms = searchRequest.split(" ");
searchTerms.forEach(function(searchTerm) {
andArray.push({
search: { $regex: searchTerm, $options: 'i'
}
});
});
I use this array in my find() and chain it with an $and:
Tracking.
find({ $and: andArray }).
where('user_id').equals(userId).
limit(pageSize).
skip(pageSize * page).
exec(function(err, docs) {
// hooray!
});
This works.