Match with substring in mongodb aggregation - regex

In my mongodb collection, I have a time_stamp="2013-06-30 23:58:37 928".
I need to use "$match" with only the date, like time_stamp="2013-06-30". So how can I get the substring like that ?
Previously I've tried with $substr, but it shows an error "errmsg" : "exception: invalid operator: $substr"

I think you are trying to make query using aggregation framework since you tried $match & $substr operators. I have created a simple example to show how you can use $substr to achive result you wanted on aggregation framework.
I have inserted following data into the MongoDB.
{ "_id" : ObjectId("528b343881d4fe2cfe0b1b25"), "time_stamp" : "2013-06-30 23:58:37 928" }
{ "_id" : ObjectId("528b343b81d4fe2cfe0b1b26"), "time_stamp" : "2013-06-30 23:58:37 928" }
{ "_id" : ObjectId("528b344c81d4fe2cfe0b1b27"), "time_stamp" : "2013-06-30 12:58:37 928" }
{ "_id" : ObjectId("528b344f81d4fe2cfe0b1b28"), "time_stamp" : "2013-06-30 12:58:23 928" }
{ "_id" : ObjectId("528b345381d4fe2cfe0b1b29"), "time_stamp" : "2013-06-31 12:58:23 928" }
{ "_id" : ObjectId("528b345981d4fe2cfe0b1b2a"), "time_stamp" : "2013-07-31 12:58:23 933" }
I wrote following code to group by date by using $substr operator.
db.myObject.aggregate(
{$project : {new_time_stamp : {$substr : ["$time_stamp",0, 10]}}},
{$group:{_id:"$new_time_stamp", "count": {$sum:1}}}
);

$substr operation failed because you there is no such operation for find. You can only use it in aggregation framework. You mentioned that you have a timestamp. In such a case you need to query by time interval.
{time_stamp: {
$gte: ISODate("2013-06-30T00:00:00Z"),
$lt: ISODate("2013-06-31T00:00:00Z")
}}
I hope that this month has 31 days. If it is a string ( which is really bad because dates should be stored as dates) you have to use a regex.

Related

Spring data mongo query with regex within an array

I have a collection with structure somewhat like this :
{
"organization" : "Org1",
"active" : true,
"fields" : [
{
"key" : "key1",
"value" : "table"
},
{
"key" : "key2",
"value" : "Harrison"
}
]
}
I need to find all documents with organization : "Org1", active : true, and regex match the 'value' in fields.
In mongo shell, it works perfectly. I tried the query:
db.collection.find({"organization" : "Org1", "active" : true, "fields" : {$elemMatch : {"key" : "key2","value" : {$regex : /iso/i}}}}).pretty()
But when I tried to convert it to a Java code with Spring, it gives wrong results.
1. This one will give documents even if it didn't match the pattern:
#Query("{'organization' : ?0, 'active' : true, 'fields' : {$elemMatch : {'key' : ?1, 'value' : {$regex : ?2}}}}")
List<ObjectCollection> findFieldDataByRegexMatch(String org, String key, String pattern);
This one doesn't give any documents even though it should.
MongoTemplate MONGO_TEMPLATE = null;
try {
MONGO_TEMPLATE = multipleMongoConfig.secondaryMongoTemplate();
} catch (Exception e) {
e.printStackTrace();
}
List<Criteria> criteriaListAnd = new ArrayList<Criteria>();
Criteria criteria = new Criteria();
String pattern = "/iso/i";
criteriaListAnd.add(Criteria.where("organization").is("Org1"));
criteriaListAnd.add(Criteria.where("active").is(true));
criteriaListAnd.add(Criteria.where("fields").elemMatch(Criteria.where("key").is(key).and("value").regex(pattern)));
criteria.andOperator(criteriaListAnd.toArray(new Criteria[criteriaListAnd.size()]));
Query query = new Query();
query.addCriteria(criteria);
List<ObjectCollection> objects = MONGO_TEMPLATE.find(query, ObjectCollection.class);
What am I missing here and how should I form my query?
You are making a very small mistake, in the pattern you are passing / which is the mistake, it took me half an hour to identify it, finally, I got it after enabling the debug log of spring boot.
For the first query, it should be called as below:
springDataRepository.findFieldDataByRegexMatch("Org1", "key2", "iso")
And the query should be modified in the Repository as to hanlde the case sensetivity:
#Query("{'organization' : ?0, 'active' : true, 'fields' : {$elemMatch : {'key' : ?1, 'value' : {$regex : ?2, $options: 'i'}}}}")
List<Springdata> findFieldDataByRegexMatch(String org, String key, String pattern);
The same issue in your second query also, just change String pattern = "/iso/i"; to String pattern = "iso" or String pattern = "iso.*" ;
Both will start working, For details please check the my GitHub repo https://github.com/krishnaiitd/learningJava/blob/master/spring-boot-sample-data-mongodb/src/main/java/sample/data/mongo/main/Application.java#L60
I hope this will resolve your problem.

Ordering a term aggregation with a multi-bucket sub-aggregation

Given a term aggregation (label), I would like to sort the bucket by a string field (energy).
The problem is that we cannot use a multi-bucket value in the order clause.
For a given label, I'm sure that there is only one energy. What I would like to do is to use the first (and only) result of my energy sub aggregation.
I'm using the AWS elasticsearch service which is in a 1.5 version, and scripts are disabled, so I did not find a way to sort the bucket by another term :(
Any idea ?
{
"aggs" : {
"label" : {
"terms" : { "field" : "label" },
"order" : { "energy[0]" : "desc" } // cannot do this
},
"aggs" : {
"energy" : {
"terms" : {
"field" : "energy",
"size" : 1
}
}
}
}
}

Can I do a MongoDB "starts with" query on an indexed subdocument field?

I'm trying to find documents where a field starts with a value.
Table scans are disabled using notablescan.
This works:
db.articles.find({"url" : { $regex : /^http/ }})
This doesn't:
db.articles.find({"source.homeUrl" : { $regex : /^http/ }})
I get the error:
error: { "$err" : "table scans not allowed:moreover.articles", "code" : 10111 }
There are indexes on both url and source.homeUrl:
{
"v" : 1,
"key" : {
"url" : 1
},
"ns" : "mydb.articles",
"name" : "url_1"
}
{
"v" : 1,
"key" : {
"source.homeUrl" : 1
},
"ns" : "mydb.articles",
"name" : "source.homeUrl_1",
"background" : true
}
Are there any limitations with regex queries on subdocument indexes?
When you disable table scans, it means that any query where a table scan "wins" in the query optimizer will fail to run. You haven't posted an explain but it's reasonable to assume that's what is happening here based on the error. Try hinting the index explicitly:
db.articles.find({"source.homeUrl" : { $regex : /^http/ }}).hint({"source.homeUrl" : 1})
That should eliminate the table scan as a possible choice and allow the query to return successfully.

Some magic in find

In Mongo FIND( regexp ), the number of found records is different from Meteor.
mongodb side:
> db.products.find({name:/clp/ig}).count()
4
crome js console:
> products.find({name:/clp/ig}).count()
2
Data are very simple:
{ "_id" : ObjectId("514d39f087696bb4cc3b549d"), "code" : "P615", "name" : "PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 68" }
{ "_id" : ObjectId("514d39f087696bb4cc3b549e"), "code" : "P616", "name" : "PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 100" }
{ "_id" : ObjectId("514d39f087696bb4cc3b549f"), "code" : "P617", "name" : "PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 150" }
{ "_id" : ObjectId("514d39f087696bb4cc3b54a0"), "code" : "P618", "name" : "PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 220" }
Collection
var products = new Meteor.Collection('products');
Meteor.publish( 'products', function(){
products.find();
});
Could be a bug? And where?
ps: checked on different computers. same magic.
The data may not yet be on the client when you make the call. Do you use it anywhere? The data is only fetched when it's used, so when you do the find on the client, you get result with what's already there, and the other items just start their way through the net.
To test if this is the case:
Do the same find on the server (in Meteor.startup for example) and log the results.
On the client, do Products.find({}).fetch();. Wait some time, then do your query.
All data is too old(may be year).
products.find({name:/clp/ig}).fetch();
answer is:
P615 PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 68
P617 PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 150
other
products.find({name:/clp 220/ig}).fetch();
answer is
P618 PENNASOL WERKZEUGMASCHINENGETRIEBEOEL CLP 220
It's real magic.

combining regex and embedded objects in mongodb queries

I am trying to combine regex and embedded object queries and failing miserably. I am either hitting a limitation of mongodb or just getting something slightly wrong maybe someone out ther has encountered this. The documentation certainly does'nt cover this case.
data being queried:
{
"_id" : ObjectId("4f94fe633004c1ef4d892314"),
"productname" : "lightbulb",
"availability" : [
{
"country" : "USA",
"storeCode" : "abc-1234"
},
{
"country" : "USA",
"storeCode" : "xzy-6784"
},
{
"country" : "USA",
"storeCode" : "abc-3454"
},
{
"country" : "CANADA",
"storeCode" : "abc-6845"
}
]
}
assume the collection contains only one record
This query returns 1:
db.testCol.find({"availability":{"country" : "USA","storeCode":"xzy-6784"}}).count();
This query returns 1:
db.testCol.find({"availability.storeCode":/.*/}).count();
But, this query returns 0:
db.testCol.find({"availability":{"country" : "USA","storeCode":/.*/}}).count();
Does anyone understand why? Is this a bug?
thanks
You are referencing the embedded storecode incorrectly - you are referencing it as an embedded object when in fact what you have is an array of objects. Compare these results:
db.testCol.find({"availability.0.storeCode":/x/});
db.testCol.find({"availability.0.storeCode":/a/});
Using your sample doc above, the first one will not return, because the first storeCode does not have an x in it ("abc-1234"), the second will return the document. That's fine for the case where you are looking at a single element of the array and pass in the position. In order to search all of the objcts in the array, you want $elemMatch
As an example, I added this second example doc:
{
"_id" : ObjectId("4f94fe633004c1ef4d892315"),
"productname" : "hammer",
"availability" : [
{
"country" : "USA",
"storeCode" : "abc-1234"
},
]
}
Now, have a look at the results of these queries:
PRIMARY> db.testCol.find({"availability" : {$elemMatch : {"storeCode":/a/}}}).count();
2
PRIMARY> db.testCol.find({"availability" : {$elemMatch : {"storeCode":/x/}}}).count();
1