Let say my mongodb data store some data with int
num: 850531015931
num: 860338053336
num: 859923992712
Now i would like to query aggregate with regex
those num start with 85, how do i do this with aggregate function?
First you have to convert integer in to string using $substr operator than perform $match funtion using $regex operator
db.collection.aggregate([{$project :{numstring : {$substr :["$number",0,12]}}},{$match :{"numstring":{$regex : /^8.5/}}}])
$substr can be used in a $project stage to convert the num to a string value. You can then have a $match stage with the regex.
Project Stage:
{
$project:{
numAsString: { $substr : ["$num", 0, -1 ] }
}
}
Match Stage:
{
$match : { numAsString: { $regex: /^85.*/ } }
}
Although this can be done by regexes, I would like to suggest an alternate method. The problem with regex is that it wont allow you to index elements. Hence as your collection size increases, your queries will become slower and slower.
You can just go for the basics and do the following checks
{$or: [
{$and: [ {$gte: ['$num', 85] }, {$lte: ['$num', 85] } ]},
{$and: [ {$gte: ['$num', 850] }, {$lte: ['$num', 859] } ]},
{$and: [ {$gte: ['$num', 8500] }, {$lte: ['$num', 8599] } ]},
{$and: [ {$gte: ['$num', 85000] }, {$lte: ['$num', 85999] } ]},
{$and: [ {$gte: ['$num', 850000] }, {$lte: ['$num', 859999] } ]},
{$and: [ {$gte: ['$num', 8500000] }, {$lte: ['$num', 8599999] } ]},
{$and: [ {$gte: ['$num', 85000000] }, {$lte: ['$num', 85999999] } ]},
{$and: [ {$gte: ['$num', 850000000] }, {$lte: ['$num', 859999999] } ]},
{$and: [ {$gte: ['$num', 8500000000] }, {$lte: ['$num', 8599999999] } ]},
{$and: [ {$gte: ['$num', 85000000000] }, {$lte: ['$num', 85999999999] } ]},
]}
Keep doing this till you reach the max possible value in num. Sorry for the ugly code, but it should run faster.
There's no point in using a regular expression in your specific case. As the others already pointed out the conversion to a string can be done using $substr but the all it takes for the filtering is a standard equals comparison:
db.collection.aggregate([{
$addFields: {
"firstTwoDigits": { // create a new field called "firstTwoDigits"
$substr: ["$num", 0, 2] // extract the first two digits only
}
}
}, {
$match: {
"firstTwoDigits": "85" // simple filter
}
}])
Related
Can anyone please guide me steps to create multiple triggers types one with conditional and other with scheduled trigger type in single workflow
So far I have used create_trigger function . But above requirement not sure how to address.
Can any one help here please.
I have tried with below syntax didn't work
response = client.create_trigger(
Name='two_triggers',
WorkflowName='wf_With_two_tirggers',
Type='SCHEDULED',
Schedule='cron(0 12 * * ? *)',
Actions=[
{
'JobName': 'abc_dev',
'Arguments': {
'string': 'string'
},
'Timeout': 123,
'SecurityConfiguration': 'string',
'NotificationProperty': {
'NotifyDelayAfter': 123
},
'Trigger': 'string'
},
],
Type='CONDITIONAL',
Predicate={
'Logical': 'ANY',
'Conditions': [
{
'LogicalOperator': 'EQUALS',
'JobName': 'def_dev',
'State': 'SUCCEEDED'
},
]
},
Actions=[
{
'JobName': 'ghi_dev',
'Arguments': {
'string': 'string'
},
'Timeout': 123,
'SecurityConfiguration': 'string',
'NotificationProperty': {
'NotifyDelayAfter': 123
},
'CrawlerName': 'string'
},
],
Description='string',
StartOnCreation=True,
Tags={
'string': 'string'
}
)
Below is the design workflow struggling to write code for. Tried with above code for below design using boto3 didn't work
Yes I figured out on an answer. Below is the code for design given in question
import boto3
import os
import logging
glue = boto3.client(service_name="glue", region_name='us-east-1')
response = glue.create_workflow(
Name="dual_trigger_wf")
response1 = glue.create_trigger(
Name="trigger_one_to_many",
WorkflowName="dual_trigger_wf",
Type="SCHEDULED",
Schedule="cron(0 8 * * ? *)",
Actions=[
{
"JobName": "abc",
"Arguments": {"string": "string"},
"Timeout": 123,
"SecurityConfiguration": "string",
"NotificationProperty": {"NotifyDelayAfter": 123},
},
{
"JobName": "def",
"Arguments": {"string": "string"},
"Timeout": 123,
"SecurityConfiguration": "string",
"NotificationProperty": {"NotifyDelayAfter": 123},
},
],
Description="string",
StartOnCreation=False,
)
response2 = glue.create_trigger(
Name="trigger_many_to_one",
WorkflowName="dual_trigger_wf",
Type="CONDITIONAL",
Predicate={
"Logical": "AND",
"Conditions": [
{
"LogicalOperator": "EQUALS",
"JobName": "abc",
"State": "SUCCEEDED",
},
{
"LogicalOperator": "EQUALS",
"JobName": "def",
"State": "SUCCEEDED",
},
],
},
Actions=[
{
"JobName": "ghi",
"Arguments": {"string": "string"},
"Timeout": 123,
"SecurityConfiguration": "string",
"NotificationProperty": {"NotifyDelayAfter": 123},
}
],
Description="string",
StartOnCreation=False,
)
My ListInputSecurityGroup task returns this json:
{
"output": [
{
"Arn": "arn:aws:medialive:eu-north-1:xxx:inputSecurityGroup:1977625",
"Id": "1977625",
"Inputs": [],
"State": "IDLE",
"Tags": {},
"WhitelistRules": [
{
"Cidr": "5.5.5.5/32"
}
]
},
{
"Arn": "arn:aws:medialive:eu-north-1:xxx:inputSecurityGroup:5411101",
"Id": "5411101",
"Inputs": [],
"State": "IDLE",
"Tags": {
"use": "some_other_use"
},
"WhitelistRules": [
{
"Cidr": "1.1.1.1/0"
}
]
},
{
"Arn": "arn:aws:medialive:eu-north-1:xxx:inputSecurityGroup:825926",
"Id": "825926",
"Inputs": [
"4011716"
],
"State": "IN_USE",
"Tags": {
"use": "for_rtmp_pipeline"
},
"WhitelistRules": [
{
"Cidr": "0.0.0.0/0"
}
]
}
]
}
I want to use OutputPath to extract the InputSecurityGroup with the tag {use:for_rtmp_pipeline}. According to this JSONPath tester this expression works $.output[?(#.Tags.use == for_rtmp_pipeline)] and it returns the 3rd element in this array. But when used in the StepFunction itself, or in the Data Flow Simulator, it doesn't return anything. Is this a limitation of the JSONPath engine in AWS, or is there a different syntaxis? How can I extract the one element I want?
Note that in the tester the searched string should be in quotes, while in AWS there's no need for quotes.
Not able to get desired output. Getting “Unrecognized expression ‘$regex’” error
[
{
'$lookup': {
'from': 'profiles',
'let': {
'userId': '$userId',
},
'pipeline': [
{
'$match': {
$expr: {
$and: [
{ '$eq': ['$uniqueId', '$$mcontactId'] },
{
$or: [{ 'birthDate': { '$regex': '$$today' } },
{ 'spouseBirthdate': { '$regex': '$$today' } },
{ 'weddingAnniversary': { '$regex': '$$today' } },
],
},
],
},
},
},
{
'$project': {
'userId': 1,
'uniqueId': 1,
'mobileNumber': 1,
'whatsApp': 1,
'emailId': 1,
'lastName': 1,
'firstName': 1,
'address': 1,
'signature': 1,
},
},
],
'as': 'profile',
},
},
]
$regex is a query operator, you are trying to use it within an $expr which uses the "aggregation" language as oppose to the "query" language normally used within a $match stage.
Apart from that you have some other issue's in your pipeline, for example you only define $userId as a variable for the $lookup stage but in it you're trying to use $$today and $$mcontactId which are not defined anywhere.
Regardless once you sort out those issue's you have two options:
if the regex match is not related to the input variables just use $regex outside the $expr, like so:
{
'$match': {
$and: [
{
$expr: {
'$eq': [
'$uniqueId',
'$$userId',
],
},
},
{
$or: [
{
'birthDate': {
'$regex': '03-05',
},
},
],
},
],
},
},
Mongo Playground
if the regex does not to use an input variable from the $lookup then you need to use an aggregation operator, like $regexMatch to do the match within the $expr
My doc looks as follows
doc = {
name: 'abc',
age:20
}
and my query looks like
{ $expr: {$and:[{ $gt:[ "$age", 10 ] },
{ $regex:["$name",'ab']}
]
}
} }
But it's not working and I get an error
Unrecognized expression '$regex'
How can I make it work?
My original query looks like this
db.orders.aggregate([{
$match: {}},
{$lookup: {
from: "orders",
let: {
"customer_details": "$customerDetails"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{ $or: [
{
$eq: ["$customerDetails.parentMobile","$$customer_details.parentMobile"]
},
{$eq: ["$customerDetails.studentMobile","$$customer_details.parentMobile"]
},
{$eq: ["$customerDetails.studentMobile","$$customer_details.parentMobile"]
},
{$eq: ["$customerDetails.studentMobile","$$customer_details.studentMobile"]
}
]
},
{$eq: ["$customerDetails.zipCode","$$customer_details.zipCode"]},
{$eq: ["$customerDetails.address","$$customer_details.address"]}
]
}
}
}],
as: "oldOrder"
}
}])
I want to use regex for matching address.
Any help will be greatly appreciated. Thanks in advance.
If your mongoDB version is 4.2, then you can use $regexMatch
try this
db.collection.find({
$expr: {
$and: [
{
$gt: [
"$age",
10
]
},
{
$regexMatch: {
input: "$name",
regex: "ab"
}
}
]
}
})
check this Mongo Playground
$regex is a query operator you cannot use inside $expr because it only supports aggregation pipeline operators.
{
"$expr": { "$gt": ["$age", 10] } ,
"name": { "$regex": "ab" }
}
If you have mongodb 4.2, you can use $regexMatch
{ "$expr": {
"$and": [
{ "$gt": ["$age", 10] },
{
"$regexMatch": {
"input": "$name",
"regex": "ab", //Your text search here
"options": "i",
}
}
]
}}
Mongodb allows regex expression of pattern /pattern/ without using $regex expression.
http://docs.mongodb.org/manual/reference/operator/query/in/
How can i do it using morphia ?
If i give Field criteria with field operator as in and value of type "java.util.regex.Pattern" then the equivalent query generated in
$in:[$regex: 'given pattern'] which wont return expected results at all.
Expectation: $in :[ /pattern1 here/,/pattern2 here/]
Actual using 'Pattern' object : $in : [$regex:/pattern1 here/,$regex:/pattern 2 here/]
I'm not entirely sure what to make of your code examples, but here's a working Morphia code snippet:
Pattern regexp = Pattern.compile("^" + email + "$", Pattern.CASE_INSENSITIVE);
mongoDatastore.find(EmployeeEntity.class).filter("email", regexp).get();
Note that this is really slow. It can't use an index and will always require a full collection scan, so avoid it at all cost!
Update: I've added a specific code example. The $in is not required to search inside an array. Simply use /^I/ as you would in string:
> db.profile.find()
{
"_id": ObjectId("54f3ac3fa63f282f56de64bd"),
"tags": [
"India",
"Australia",
"Indonesia"
]
}
{
"_id": ObjectId("54f3ac4da63f282f56de64be"),
"tags": [
"Island",
"Antigua"
]
}
{
"_id": ObjectId("54f3ac5ca63f282f56de64bf"),
"tags": [
"Spain",
"Mexico"
]
}
{
"_id": ObjectId("54f3ac6da63f282f56de64c0"),
"tags": [
"Israel"
]
}
{
"_id": ObjectId("54f3ad17a63f282f56de64c1"),
"tags": [
"Germany",
"Indonesia"
]
}
{
"_id": ObjectId("54f3ad56a63f282f56de64c2"),
"tags": [
"ireland"
]
}
> db.profile.find({ tags: /^I/ })
{
"_id": ObjectId("54f3ac3fa63f282f56de64bd"),
"tags": [
"India",
"Australia",
"Indonesia"
]
}
{
"_id": ObjectId("54f3ac4da63f282f56de64be"),
"tags": [
"Island",
"Antigua"
]
}
{
"_id": ObjectId("54f3ac6da63f282f56de64c0"),
"tags": [
"Israel"
]
}
{
"_id": ObjectId("54f3ad17a63f282f56de64c1"),
"tags": [
"Germany",
"Indonesia"
]
}
Note: The position in the array makes no difference, but the search is case sensitive. Use /^I/i if this is not desired or Pattern.CASE_INSENSITIVE in Java.
Single RegEx Filter
use .filter(), .criteria(), or .field()
query.filter("email", Pattern.compile("reg.*exp"));
// or
query.criteria("email").contains("reg.*exp");
// or
query.field("email").contains("reg.*exp");
Morphia converts this into:
find({"email": { $regex: "reg.*exp" } })
Multiple RegEx Filters
query.or(
query.criteria("email").contains("reg.*exp"),
query.criteria("email").contains("reg.*exp.*2"),
query.criteria("email").contains("reg.*exp.*3")
);
Morphia converts this into:
find({"$or" : [
{"email": {"$regex": "reg.*exp"}},
{"email": {"$regex": "reg.*exp.*2"}},
{"email": {"$regex": "reg.*exp.*3"}}
]
})
Unfortunately,
You cannot use $regex operator expressions inside an $in.
MongoDB Manual 3.4
Otherwise, we could do:
Pattern[] patterns = new Pattern[] {
Pattern.compile("reg.*exp"),
Pattern.compile("reg.*exp.*2"),
Pattern.compile("reg.*exp.*3"),
};
query.field().in(patterns);
hopefully, one day morphia will support that :)