My sample JSON file for postman runner:
[ { "name": "runner", "hitler_id": "4006abc", "year": "2017", "boolean": "false", "expected": 717962 } ]
Pre request script:
var member = data.name; var booking = data.boolean; var fyyear = data.year; var sid = data.hitler_id;
console.log(data.name); console.log(data.boolean); console.log(data.year); console.log(data.hitler_id);
Body with parameters:
{ "size": 0, "query": { "bool": { "filter": [ { "terms": { "name": [ "{{name}}" ] } }, { "terms": { "salesman_id": [ "{{sid}}" ] } }, { "terms": { "fyyear": [ "{{fyyear}}" ] } }, { "terms": { "boolean": [ "{{boolean}}" ] } } ] } }, "aggs": { "year": { "terms": { "field": "year" }, "aggs": { "value": { "sum": { "field": "value" } } } } } }
For only string variables are accepted - name and boolean fields are working and the value is populated
for the other two, the variable values are not passed.
The variables are not used in your request body that way.
Either you have to store them in environment oder global variables via
pm.globals.set("variable_key", variable_value)
pm.environment.set("variable_key", "variable_value");
or just skip the pre-request script if you just want to use your data and reference the fields directly in your body:
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"terms": {
"name": [
"{{name}}"
]
}
},
{
"terms": {
"salesman_id": [
"{{hitler_id}}"
]
}
},
{
"terms": {
"fyyear": [
{{year}}
]
}
},
{
"terms": {
"boolean": [
{{boolean}}
]
}
}
]
}
},
"aggs": {
"year": {
"terms": {
"field": "year"
},
"aggs": {
"value": {
"sum": {
"field": "value"
}
}
}
}
}
}
However take care you're storing the values in your data file. You stored the bool and the year as strings". But they should be represented as you already did for the "expected" var.
Related
Hi I am trying to get a query based on the following documents.
document 1
{
"date" : "2021-05-19",
"items" : [
{
"keyA" : "ValueA"
}
]
}
document 2
{
"date" : "2021-05-19",
"items" : [
{
"keyB" : "ValueB"
}
]
}
Output I expect
{
"date" : "2021-05-19",
"items" : [
{
"keyA" : "ValueA"
},
{
"keyB" : "ValueB"
}
]
}
I don't want to update this structure in ES. Only when I query, I want the result to be in this format.
Is it possible? or should I handle it after receiving the result?
You can use a combination of terms and top_hits aggregation
{
"size": 0,
"aggs": {
"dateagg": {
"terms": {
"field": "date"
},
"aggs": {
"top_keys": {
"top_hits": {
"_source": {
"includes": [
"items.*"
]
}
}
}
}
}
}
}
Search Result will be
"aggregations": {
"dateagg": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": 1621382400000,
"key_as_string": "2021-05-19T00:00:00.000Z", // note this
"doc_count": 2,
"top_keys": {
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "67598919",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"items": [
{
"keyA": "ValueA" // note this
}
]
}
},
{
"_index": "67598919",
"_type": "_doc",
"_id": "2",
"_score": 1.0,
"_source": {
"items": [
{
"keyB": "ValueB" // note this
}
]
}
}
]
}
}
}
]
}
}
I'm extending Istio to provide Thrift features. The Istio component I'm working on right now is Pilot (Envoy config service). I've extended it with basic Thrift routing so that it can provide an Envoy configuration to route a listener to the correct cluster. My development environment looks something like:
Now I'm trying to add rate limiting. I'm following the docs. I've patched my config with a RouteAction that doesn't get applied, because there is no rate limit filter. This works fine, and passes traffic as before:
...
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"stat_prefix": "10.97.28.169_9090",
"transport": "HEADER",
"protocol": "BINARY",
"route_config": {
"name": "outbound|9090||backend.default.svc.cluster.local",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "outbound|9090||backend.default.svc.cluster.local",
"rate_limits": [
{
"actions": [
{
"request_headers": {
"header_name": ":method-name",
"descriptor_key": "method-name"
}
}
]
}
]
}
}
]
}
}
}
]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
},
"last_updated": "2019-10-30T16:26:39.203Z"
},
...
I've built a function to create an envoy.filters.thrift.rate_limit filter that makes Envoy to call the rate limit service I've set up (I've tried both GoogleGrpc and EnvoyGrpc):
func buildThriftRatelimit(ratelimitServiceUri, domain string) *thrift_ratelimit.RateLimit {
var thriftRateLimit *thrift_ratelimit.RateLimit
timeout := 2000 * time.Millisecond
thriftRateLimit = &thrift_ratelimit.RateLimit{
Domain: domain,
Timeout: &timeout,
FailureModeDeny: false,
RateLimitService: &ratelimit.RateLimitServiceConfig{
GrpcService: &core.GrpcService{
TargetSpecifier: &core.GrpcService_GoogleGrpc_{
GoogleGrpc: &core.GrpcService_GoogleGrpc{
StatPrefix: ratelimitServiceUri,
TargetUri: ratelimitServiceUri,
},
},
},
},
}
thriftRateLimit.Validate()
if err := thriftRateLimit.Validate(); err != nil {
panic(err)
}
return thriftRateLimit
}
Which produces:
...
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"stat_prefix": "10.97.28.169_9090",
"transport": "HEADER",
"protocol": "BINARY",
"route_config": {
"name": "outbound|9090||backend.default.svc.cluster.local",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "outbound|9090||backend.default.svc.cluster.local",
"rate_limits": [
{
"actions": [
{
"request_headers": {
"header_name": ":method-name",
"descriptor_key": "method-name"
}
}
]
}
]
}
}
]
},
"thrift_filters": [
{
"name": "envoy.filters.thrift.rate_limit",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.thrift.rate_limit.v2alpha1.RateLimit",
"domain": "backend.default.svc.cluster.local",
"timeout": "2s",
"rate_limit_service": {
"grpc_service": {
"google_grpc": {
"target_uri": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80",
"stat_prefix": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80"
}
}
}
}
}
]
}
}
]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
},
"last_updated": "2019-10-30T16:26:39.203Z"
},
...
When the rate limit filter is applied, connections to the backend die and no error is returned to the client or displayed in Envoy's logs.
If Thrift filters are provided, you need to add the Router filter as the last filter in the chain like so:
...
{
"filters": [
{
"name": "envoy.filters.network.thrift_proxy",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.network.thrift_proxy.v2alpha1.ThriftProxy",
"stat_prefix": "10.97.28.169_9090",
"transport": "HEADER",
"protocol": "BINARY",
"route_config": {
"name": "outbound|9090||backend.default.svc.cluster.local",
"routes": [
{
"match": {
"method_name": ""
},
"route": {
"cluster": "outbound|9090||backend.default.svc.cluster.local",
"rate_limits": [
{
"actions": [
{
"request_headers": {
"header_name": ":method-name",
"descriptor_key": "method-name"
}
}
]
}
]
}
}
]
},
"thrift_filters": [
{
"name": "envoy.filters.thrift.rate_limit",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.filter.thrift.rate_limit.v2alpha1.RateLimit",
"domain": "backend.default.svc.cluster.local",
"timeout": "2s",
"rate_limit_service": {
"grpc_service": {
"google_grpc": {
"target_uri": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80",
"stat_prefix": "istio-lyft-ratelimit.istio-system.svc.cluster.local:80"
}
}
}
}
},
{
"name": "envoy.filters.thrift.router"
}
]
}
}
]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
},
"last_updated": "2019-10-30T16:26:39.203Z"
},
...
I am looking for appropriate elasticsearch query for,
SELECT col1,col2 FROM myTable WHERE col1="value1" AND col2 = "value2"
eg:
This is my mapping,
{
"mapping": {
"doc": {
"properties": {
"book": {
"properties": {
"name": {
"type": "text"
},
"price": {
"type": "integer"
},
"booktype": {
"properties": {
"booktype": {
"type": "text"
}
}
}
}
}
}
}
}
}
I am trying to write a query which will give me price and name which has booktype=Fiction
Try this:
GET myTable/_search
{
"size": 1000,
"_source": [
"price",
"name"
],
"query": {
"bool": {
"must": [
{
"match": {
"booktype.booktype": "Fiction"
}
}
]
}
}
}
Note: you might need to adapt "size" to fit your needs.
I followed this link :
https://developers.google.com/assistant/sdk/guides/service/python/extend/custom-actions
And I successfully create three actions files. and it works individually. but when I test all together like deploying three action packages:
./gactions update --action_package MqttAct.json --action_package action.json --action_package MesureTemp.json --project rpi3-0001-ga-******
I found that only two packages will work. it's like we can't have more than two action packages?
I'm surprised you can even deploy two packages at the same time. The idea is that you have one action package, which can contain a variety of actions merged together.
Here is a merged action package, based on the documentation:
{
"manifest": {
"displayName": "Blinky light",
"invocationName": "Blinky light",
"category": "PRODUCTIVITY"
},
"actions": [
{
"name": "com.example.actions.BlinkLight",
"availability": {
"deviceClasses": [
{
"assistantSdkDevice": {}
}
]
},
"intent": {
"name": "com.example.intents.BlinkLight",
"parameters": [
{
"name": "number",
"type": "SchemaOrg_Number"
},
{
"name": "speed",
"type": "Speed"
}
],
"trigger": {
"queryPatterns": [
"blink ($Speed:speed)? $SchemaOrg_Number:number times",
"blink $SchemaOrg_Number:number times ($Speed:speed)?"
]
}
},
"fulfillment": {
"staticFulfillment": {
"templatedResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "Blinking $number times"
}
},
{
"deviceExecution": {
"command": "com.example.commands.BlinkLight",
"params": {
"speed": "$speed",
"number": "$number"
}
}
}
]
}
}
}
},
{
"name": "com.example.actions.BlonkLight",
"availability": {
"deviceClasses": [
{
"assistantSdkDevice": {}
}
]
},
"intent": {
"name": "com.example.intents.BlonkLight",
"parameters": [
{
"name": "number",
"type": "SchemaOrg_Number"
},
{
"name": "speed",
"type": "Speed"
}
],
"trigger": {
"queryPatterns": [
"blonk ($Speed:speed)? $SchemaOrg_Number:number times",
"blonk $SchemaOrg_Number:number times ($Speed:speed)?"
]
}
},
"fulfillment": {
"staticFulfillment": {
"templatedResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "Blonking $number times"
}
},
{
"deviceExecution": {
"command": "com.example.commands.BlonkLight",
"params": {
"speed": "$speed",
"number": "$number"
}
}
}
]
}
}
}
}
],
"types": [
{
"name": "$Speed",
"entities": [
{
"key": "slowly",
"synonyms": [
"slowly",
"slow"
]
},
{
"key": "normally",
"synonyms": [
"normally",
"regular"
]
},
{
"key": "quickly",
"synonyms": [
"quickly",
"fast",
"quick"
]
}
]
}
]
}
I'm using elasticsearch on AWS to store logs from Cloudfront. I have created a simple query that will give me all entries from the past 24h, sorted from new to old:
{
"from": 0,
"size": 1000,
"query": {
"bool": {
"must": [
{ "match": { "site_name": "some-site" } }
],
"filter": [
{
"range": {
"timestamp": {
"lt": "now",
"gte": "now-1d"
}
}
}
]
}
},
"sort": [
{ "timestamp": { "order": "desc" } }
]
}
Now, there a are certain sources (based on the user agent) for which I would like to exclude results. So my question boils down to this:
How can I filter out entries from the results when a certain field contains a certain string? Or:
query.filter.where('cs_user_agent').does.not.contain('Some string')
(This is not real code, obviously.)
I have tried to make sense of the Elasticsearch documentation, but I couldn't find a good example of how to achieve this.
I hope this makes sense. Thanks in advance!
Okay, I figured it out. What I've done is use a Bool Query in combination with a wildcard:
{
"from": 0,
"size": 1000,
"query": {
"bool": {
"must": [
{ "match": { "site_name": "some-site" } }
],
"filter": [
{
"range": {
"timestamp": {
"lt": "now",
"gte": "now-1d"
}
}
}
],
"must_not": [
{ "wildcard": { "cs_user_agent": "some string*" } }
]
}
},
"sort": [
{ "timestamp": { "order": "desc" } }
]
}
This basically matches any user agent string containing "some string", and then filters it out (because of the "must_not").
I hope this helps others who run into this problem.
nod.js client version:
const { from, size, value, tagsIdExclude } = req.body;
const { body } = await elasticWrapper.client.search({
index: ElasticIndexs.Tags,
body: {
from: from,
size: size,
query: {
bool: {
must: {
wildcard: {
name: {
value: `*${value}*`,
boost: 1.0,
rewrite: 'constant_score',
},
},
},
filter: {
bool: {
must_not: [
{
terms: {
id: tagsIdExclude ? tagsIdExclude : [],
},
},
],
},
},
},
},
},
});