I have the following data
{
"companyID" : "amz",
"companyType" : "ret",
"employeeID" : "ty-5a62fd78e8d20ad"
},
{
"companyID" : "amz",
"companyType" : "ret",
"employeeID" : "ay-5a62fd78e8d20ad"
},
{
"companyID" : "mic",
"companyType" : "cse",
"employeeID" : "by-5a62fd78e8d20ad"
},
{
"companyID" : "ggl",
"companyType" : "cse",
"employeeID" : "ply-5a62fd78e8d20ad"
},
{
"companyID" : "ggl",
"companyType" : "cse",
"employeeID" : "wfly-5a62ad"
}
I want the following result. basically combination of values like this mic-cse,ggl-cse,amz-ret .
"agg_by_company_type" : {
"buckets" : [
{
"key" : "ret",
"doc_count" : 1
},
{
"key" : "cse",
"doc_count" : 2
}
]
How do I do it?
I have tried the following aggregations:
"agg_by_companyID_topHits": {
"terms": {
"field": "companyID.keyword",
"size": 100000,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": true,
"order": {
"_key": "asc"
}
},
"aggs": {
"agg_by_companyType" : {
"top_hits": {
"size": 1,
"_source": {
"includes": ["companyType"]
}
}
}
}
}
But this just gives me first groupBy of company id now on top of that data I want count of company type.
this is the response I get
"agg_by_companyID_topHits" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "amz",
"doc_count" : 2,
"doc_count_error_upper_bound" : 0,
"agg_by_companytype" : {
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "my-index",
"_type" : "_doc",
"_id" : "uytuygjuhg",
"_score" : 0.0,
"_source" : {
"companyType" : "ret"
}
}
]
}
}
},
{
"key" : "mic",
"doc_count" : 1,
"doc_count_error_upper_bound" : 0,
"agg_by_companytype" : {
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "my-index",
"_type" : "_doc",
"_id" : "uytuygjuhg",
"_score" : 0.0,
"_source" : {
"companyType" : "cse"
}
}
]
}
}
},
{
"key" : "ggl",
"doc_count" : 2,
"doc_count_error_upper_bound" : 0,
"agg_by_companytype" : {
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "my-index",
"_type" : "_doc",
"_id" : "uytuygjuhg",
"_score" : 0.0,
"_source" : {
"companyType" : "ret"
}
}
]
}
}
},
]
}
If it were spark, it would be simple to partition by companyID, group it and then group by companyType and count to get the desired result but not sure how to do it in ES.
Important Note: I am working with Opensearch.
Possible solution for this in elastic search multi-terms-aggregation
is not available in versions before v7.12.
So wondering how it was done before this feature in ES.
We came across this issue because AWS migrated from ES to Opensearch.
use multi_terms agg doc here
GET /products/_search
{
"aggs": {
"genres_and_products": {
"multi_terms": {
"terms": [{
"field": "companyID"
}, {
"field": "companyType"
}]
}
}
}
}
can you use script in terms agg ,like this:
GET b1/_search
{
"aggs": {
"genres": {
"terms": {
"script": {
"source": "doc['companyID'].value+doc['companyType'].value",
"lang": "painless"
}
}
}
}
}
I am currently running elasticsearch cluster version 6.3.1 on AWS and here is template file which I need to upload but can't
```
{
"template" : "logstash-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "enabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "enabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true, "ignore_above" : 256}
}
}
}
}, {
"float_fields" : {
"match" : "*",
"match_mapping_type" : "float",
"mapping" : { "type" : "float", "doc_values" : true }
}
}, {
"double_fields" : {
"match" : "*",
"match_mapping_type" : "double",
"mapping" : { "type" : "double", "doc_values" : true }
}
}, {
"byte_fields" : {
"match" : "*",
"match_mapping_type" : "byte",
"mapping" : { "type" : "byte", "doc_values" : true }
}
}, {
"short_fields" : {
"match" : "*",
"match_mapping_type" : "short",
"mapping" : { "type" : "short", "doc_values" : true }
}
}, {
"integer_fields" : {
"match" : "*",
"match_mapping_type" : "integer",
"mapping" : { "type" : "integer", "doc_values" : true }
}
}, {
"long_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "long", "doc_values" : true }
}
}, {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}, {
"geo_point_fields" : {
"match" : "*",
"match_mapping_type" : "geo_point",
"mapping" : { "type" : "geo_point", "doc_values" : true }
}
} ],
"properties" : {
"#timestamp": { "type": "date", "doc_values" : true },
"#version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"ip": { "type": "ip", "doc_values" : true },
"location" : { "type" : "geo_point", "doc_values" : true },
"latitude" : { "type" : "float", "doc_values" : true },
"longitude" : { "type" : "float", "doc_values" : true }
}
}
}
}
}
}'
I tried loading the template via Dev Tools in Kibana and got the following error
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [_default_]: No field type matched on [float], possible values are [object, string, long, double, boolean, date, binary]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [_default_]: No field type matched on [float], possible values are [object, string, long, double, boolean, date, binary]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field type matched on [float], possible values are [object, string, long, double, boolean, date, binary]"
}
},
"status": 400
}
Can somebody please help with what I need to do to have this working on version 6 elasticsearch. I am completely new to elasticsearch and am just looking to setup logging from cloudtrail -> s3 -> AWS elasticsearch -> kibana.
In order to work on 6.3, the correct mapping for the logstash index would need to be (taken from here):
{
"template" : "logstash-*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"#timestamp": { "type": "date"},
"#version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}
I'm attempting my first ember-based project, as well as integrating emberfire into it to get Firebase capabilities. I'm building a tree menu builder, for another project.
The json for the firebase database is as follows:
{
"default" : {
"_type" : "arrayObject",
"name" : "unnamed",
"_name" : "default",
"menus" : {
"Top" : {
"menu" : {
"2" : {
"text" : "0-2",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 0,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : 0,
"a" : {
"_type" : "object"
},
"r" : 0
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : 208,
"a" : {
"_type" : "object"
},
"r" : 208
},
"row" : 2,
"id" : 2
},
"5" : {
"text" : "0-5",
"actions" : [ {
"action" : 3
} ],
"column" : 0,
"color" : {
"b" : 0,
"g" : 0,
"a" : 255,
"r" : 0
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : 208,
"g" : 208,
"a" : 255,
"r" : 208
},
"row" : 5,
"id" : 5
},
"12" : {
"text" : "1-3",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 3,
"id" : 12
},
"15" : {
"text" : "1-6",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 6,
"id" : 15
},
"8" : {
"text" : "0-8",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 0,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 8,
"id" : 8
},
"_type" : "objectArray",
"7" : {
"text" : "0-7",
"actions" : [ {
"action" : 3
} ],
"column" : 0,
"color" : {
"b" : 0,
"g" : 0,
"a" : 255,
"r" : 0
},
"_type" : "object",
"visibility" : "visible",
"page" : 0,
"bgColor" : {
"b" : 208,
"g" : 208,
"a" : 255,
"r" : 208
},
"row" : 7,
"id" : 7
},
"17" : {
"text" : "1-8",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 8,
"id" : 17
},
"1" : {
"text" : "0-1",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 0,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : 0,
"a" : {
"_type" : "object"
},
"r" : 0
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : 208,
"a" : {
"_type" : "object"
},
"r" : 208
},
"row" : 1,
"id" : 1
},
"4" : {
"text" : "0-4",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3
}
},
"column" : 0,
"color" : {
"b" : 0,
"g" : 0,
"a" : 255,
"r" : 0
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : 208,
"g" : 208,
"a" : 255,
"r" : 208
},
"row" : 4,
"id" : 4
},
"11" : {
"text" : "1-2",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 2,
"id" : 11
},
"14" : {
"text" : "1-5",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 5,
"id" : 14
},
"6" : {
"text" : "0-6",
"actions" : [ {
"action" : 3
} ],
"column" : 0,
"color" : {
"b" : 0,
"g" : 0,
"a" : 255,
"r" : 0
},
"_type" : "object",
"visibility" : "visible",
"page" : 0,
"bgColor" : {
"b" : 208,
"g" : 208,
"a" : 255,
"r" : 208
},
"row" : 6,
"id" : 6
},
"0" : {
"text" : "0-0",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 0,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : 0
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : 208
},
"row" : 0,
"id" : "0"
},
"9" : {
"text" : "1-0",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 0,
"id" : 9
},
"16" : {
"text" : "1-7",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 7,
"id" : 16
},
"3" : {
"text" : "0-3",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 0,
"color" : {
"b" : 0,
"_type" : "objectArray",
"g" : 0,
"a" : {
"_type" : "object"
},
"r" : 0
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : 208,
"_type" : "objectArray",
"g" : 208,
"a" : {
"_type" : "object"
},
"r" : 208
},
"row" : 3,
"id" : 3
},
"10" : {
"text" : "1-1",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 1,
"id" : 10
},
"13" : {
"text" : "1-4",
"actions" : {
"_type" : "objectArray",
"0" : {
"action" : 3,
"_type" : "object"
}
},
"column" : 1,
"color" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"_type" : "object",
"visibility" : "hidden",
"page" : 0,
"bgColor" : {
"b" : {
"_type" : "object"
},
"_type" : "objectArray",
"g" : {
"_type" : "object"
},
"a" : {
"_type" : "object"
},
"r" : {
"_type" : "object"
}
},
"row" : 4,
"id" : 13
}
},
"_type" : "object"
},
"_type" : "objectArray"
},
"id" : "default"
},
"_type" : "objectArray",
"app2" : {
"_type" : "arrayObject",
"name" : "app2",
"_name" : "app2",
"menus" : {
"Top" : [ {
"text" : "0-0"
} ],
"_type" : "objectArray"
}
}
}
Anyhow, I've succeeded using a fork of emberFire that provides an emberFire object out of my initial array.(https://github.com/ember-meteor/emberFire/blob/refactor/emberfire-latest.js)
That works.
I have an embedded array that I turn into another emberFire ObjectArray, and
my code:https://github.com/jondthompson/menutest.
I used a similar behavior to the original ObjectArray creation to create a second one for the 'ssMenus' array nested within the 'ssApp', but it appears to not be populating with the proper data from Firebase, nor is the router even firing with an empty array.
The proper behavior would be when you click on an 'app', the name of the app appears at the bottom (works!), with the names of the menus appearing next to it (doesn't).
Anyhow, Help, please!
I figured it out.
I was writing the child firebase reference manually. I figured out how to do it via the original firebase object.
I wasn't encapsulating the child reference in a firebase object, but rather I was trying to inject it somewhere it didn't belong.
I was trying to create a second route, when I didn't need it.
I may have missed something else too. I consider myself a beginner programmer, so this is stretching my abilities.
My original AppController..
App.SsAppController = Ember.ObjectController.extend({
needs: ["selected_ssApp", 'ssMenus'],
selected: function(){
return this.get('controllers.selected_ssApp.model') === this.get('model');
}.property('controllers.selected_ssApp.model', 'model'),
actions: {
select: function(){
var model = this.get('model');
var menuList = this.get('controllers.ssMenus')
Ember.debug("MenuList: "+ menuList);
this.set('controllers.selected_ssApp.model', model);
menuList.set('ref', new Firebase("https://menutest.firebaseio.com/jt-test/"+model.content._name+"/menus"));
}
}
});
My new and improved (read: working) MenuController..
App.SsAppController = Ember.ObjectController.extend({
needs: ["selected_ssApp", 'ssMenus'],
menus: function(){
var model = this.get('model'),
ref = model.get('ref');
return EmberFire.ObjectArray.create({ ref: ref.child('menus')});
},
selected: function(){
return this.get('controllers.selected_ssApp.model') === this.get('model');
}.property('controllers.selected_ssApp.model', 'model'),
actions: {
select: function(){
var model = this.get('model'),
menusObj = model.get('menus');
this.set('controllers.selected_ssApp.model', model);
this.set('controllers.ssMenus.content', this.menus());
}
}
});