I have a scenario where inside the should clause in elastic search query I need to have a must clause.
eg I need to filter data in such a way that if data should come only for orders that have dispatch Area ids as 10 only and carrier Ids as 1,2,3 only but should also pull data for all orders having driver id as 1,2,3.
In my current scenario its pulling data for all carrier Ids as 1,2,3 and dispatch area ids 10 and also dispatch areas which are not 10 .i.e if carrier id 1 had dispatch area-id as 9 that data is also coming.
how can I add a must clause in should query.
{
"from": 0,
"size": 10000,
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
[
{
"terms": {
"dispatchAreaId": [
10
]
}
},
{
"terms": {
"carrierId": [
1,2,3
]
}
},
{
"terms": {
"driverIds": [
1,2,3
]
}
}
]
]
}
}
]
}
}
}
Try Below Query
{
"from": 0,
"size": 10000,
"query": {
"bool": {
"should": [
{
"terms": {
"dispatchAreaId": [
10
]
}
},
{
"bool": {
"must": [
{
"terms": {
"dispatchAreaId": [
10
]
}
},
{
"terms": {
"carrierId": [
1,
2,
3
]
}
}
]
}
}
],
"minimum_should_match": 1
}
}
}
Hope this helps!!
Related
I am making a search query in Elastic Search and I want to treat the fields the same when they match. For example if I search for field field1 and it matches, then the _score is increase by 10(for example), same for the field2.
I was tried function_score but it's not working. It throws an error.
"caused_by": {
"type": "class_cast_exception",
"reason": "class
org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData
cannot be cast to class
org.elasticsearch.index.fielddata.IndexNumericFieldData
(org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData
and org.elasticsearch.index.fielddata.IndexNumericFieldData are in unnamed
module of loader 'app')"
}
The query:
{
"track_total_hits": true,
"size": 50,
"query": {
"function_score": {
"query": {
"bool": {
"must": [
{
"term": {
"field1": {
"value": "Value 1"
}
}
},
{
"term": {
"field2": {
"value": "value 2"
}
}
}
]
}
},
"functions": [
{
"field_value_factor": {
"field": "field1",
"factor": 10,
"missing": 0
}
},
{
"field_value_factor": {
"field": "field2",
"factor": 10,
"missing": 0
}
}
],
"boost_mode": "multiply"
}
}
}
You can use function score with filter function to boost.
assuming that your mapping looks like the one below
{
"mappings": {
"properties": {
"field_1": {
"type": "keyword"
},
"field_2": {
"type": "keyword"
}
}
}
}
with documents
{"index":{}}
{"field_1": "foo", "field_2": "bar"}
{"index":{}}
{"field_1": "foo", "field_2": "foo"}
{"index":{}}
{"field_1": "bar", "field_2": "bar"}
you can use weight parameter to boost the documents matched for each query.
{
"query": {
"function_score": {
"query": {
"match_all": {}
},
"functions": [
{
"filter": {
"term": {
"field_1": "foo"
}
},
"weight": 10
},
{
"filter": {
"term": {
"field_2": "foo"
}
},
"weight": 20
}
],
"score_mode": "multiply"
}
}
}
You can refer below solution if you want to provide manual weight for different field in query. This will always replace highest weight field on top of your query response -
Elasticsearch query different fields with different weight
I have a ES document like below :
{
"_id" : "test#domain.com",
"age" : 12,
"hobbiles" : ["Singing", "Dancing"]
},
{
"_id" : "test1#domain.com",
"age" : 7,
"hobbiles" : ["Coding", "Chess"]
}
I am storing email as id, age and hobbiles, hobbies is nested type, age is long I want to query with id, age and hobbiles, something like below :
Select * FROM tbl where _id IN ('val1', 'val2') AND age > 5 AND hobbiles should match with Chess or Dancing
How can I do in Elastic Search ? I am using OpenSearch 1.3 (latest) : AWS
I will suspect that field hobbiles is keyword, then the query suggested:
PUT test
{
"mappings": {
"properties": {
"age": {
"type": "long"
},
"hobbiles": {
"type": "keyword"
}
}
}
}
POST test/_doc/test#domain.com
{
"age": 12,
"hobbiles": [
"Singing",
"Dancing"
]
}
POST test/_doc/test1#domain.com
{
"age": 7,
"hobbiles": [
"Coding",
"Chess"
]
}
GET test/_search
{
"query": {
"bool": {
"filter": [
{
"terms": {
"_id": [
"test1#domain.com",
"test#domain.com"
]
}
}
],
"must": [
{
"range": {
"age": {
"gt": 5
}
}
},
{
"terms": {
"hobbiles": [
"Coding",
"Chess"
]
}
}
]
}
}
}
I have a problem while using Opensearch which based on Elasticsearch when I use range as filter query I get all the data that apply to the filter query despite it doesn't match the search query with a score of 0.0 down below sample of the query I use
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"match": {
"FIELD": "TEXT"
}
},
{
"match": {
"FIELD": "TEXT"
}
}
]
}
},
{
"bool": {
"must": [
{
"match": {
"FIELD": "TEXT"
}
},
{
"match": {
"FIELD": "TEXT"
}
}
]
}
}
],
"filter": [
{
"range": {
"FIELD": {
"gt": 1,
"lt": 500
}
}
}
]
}
}
}
what I want is to get all the data that only match the search Query and it matches the filter at the same time and I don't know what I'm doing wrong
any help.
Thanks in advance
My sample JSON file for postman runner:
[ { "name": "runner", "hitler_id": "4006abc", "year": "2017", "boolean": "false", "expected": 717962 } ]
Pre request script:
var member = data.name; var booking = data.boolean; var fyyear = data.year; var sid = data.hitler_id;
console.log(data.name); console.log(data.boolean); console.log(data.year); console.log(data.hitler_id);
Body with parameters:
{ "size": 0, "query": { "bool": { "filter": [ { "terms": { "name": [ "{{name}}" ] } }, { "terms": { "salesman_id": [ "{{sid}}" ] } }, { "terms": { "fyyear": [ "{{fyyear}}" ] } }, { "terms": { "boolean": [ "{{boolean}}" ] } } ] } }, "aggs": { "year": { "terms": { "field": "year" }, "aggs": { "value": { "sum": { "field": "value" } } } } } }
For only string variables are accepted - name and boolean fields are working and the value is populated
for the other two, the variable values are not passed.
The variables are not used in your request body that way.
Either you have to store them in environment oder global variables via
pm.globals.set("variable_key", variable_value)
pm.environment.set("variable_key", "variable_value");
or just skip the pre-request script if you just want to use your data and reference the fields directly in your body:
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"terms": {
"name": [
"{{name}}"
]
}
},
{
"terms": {
"salesman_id": [
"{{hitler_id}}"
]
}
},
{
"terms": {
"fyyear": [
{{year}}
]
}
},
{
"terms": {
"boolean": [
{{boolean}}
]
}
}
]
}
},
"aggs": {
"year": {
"terms": {
"field": "year"
},
"aggs": {
"value": {
"sum": {
"field": "value"
}
}
}
}
}
}
However take care you're storing the values in your data file. You stored the bool and the year as strings". But they should be represented as you already did for the "expected" var.
I'm using elasticsearch on AWS to store logs from Cloudfront. I have created a simple query that will give me all entries from the past 24h, sorted from new to old:
{
"from": 0,
"size": 1000,
"query": {
"bool": {
"must": [
{ "match": { "site_name": "some-site" } }
],
"filter": [
{
"range": {
"timestamp": {
"lt": "now",
"gte": "now-1d"
}
}
}
]
}
},
"sort": [
{ "timestamp": { "order": "desc" } }
]
}
Now, there a are certain sources (based on the user agent) for which I would like to exclude results. So my question boils down to this:
How can I filter out entries from the results when a certain field contains a certain string? Or:
query.filter.where('cs_user_agent').does.not.contain('Some string')
(This is not real code, obviously.)
I have tried to make sense of the Elasticsearch documentation, but I couldn't find a good example of how to achieve this.
I hope this makes sense. Thanks in advance!
Okay, I figured it out. What I've done is use a Bool Query in combination with a wildcard:
{
"from": 0,
"size": 1000,
"query": {
"bool": {
"must": [
{ "match": { "site_name": "some-site" } }
],
"filter": [
{
"range": {
"timestamp": {
"lt": "now",
"gte": "now-1d"
}
}
}
],
"must_not": [
{ "wildcard": { "cs_user_agent": "some string*" } }
]
}
},
"sort": [
{ "timestamp": { "order": "desc" } }
]
}
This basically matches any user agent string containing "some string", and then filters it out (because of the "must_not").
I hope this helps others who run into this problem.
nod.js client version:
const { from, size, value, tagsIdExclude } = req.body;
const { body } = await elasticWrapper.client.search({
index: ElasticIndexs.Tags,
body: {
from: from,
size: size,
query: {
bool: {
must: {
wildcard: {
name: {
value: `*${value}*`,
boost: 1.0,
rewrite: 'constant_score',
},
},
},
filter: {
bool: {
must_not: [
{
terms: {
id: tagsIdExclude ? tagsIdExclude : [],
},
},
],
},
},
},
},
},
});