Elastic Search Index template for NGINX custom log - templates

Question

I have the following log from NGINX:
111.111.111.111, 11.11.11.11 - 11.11.11.11 [06/May/2016:08:26:10 +0000] "POST /some-service/GetSomething HTTP/1.1" 499 0 "-" "Jakarta Commons-HttpClient/3.1" "7979798797979799" 59.370 - "{\x0A\x22correlationId\x22 : \x22TestCorr1\x22\x0A}"
Logstash will be like this:
input { stdin {} }
output { stdout { codec => "rubydebug" } }
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG} %{QS:partner_id} %{NUMBER:req_time} %{GREEDYDATA:extra_fields}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
mutate {
gsub => ["extra_fields", "\"","",
"extra_fields", "\\x0A","",
"extra_fields", "\\x22",'\"',
"extra_fields", "(\\)",""
]
}
json {
source => "extra_fields"
target => "extra_fields_json"
}
mutate {
add_field => {
"correlationId" => "%{[extra_fields_json][correlationId]}"
}
}
}
The problem is req_time is string, so I need to convert to float using the following template:
{
"template" : "filebeat*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"properties" : {
"#timestamp": { "type": "date" },
"partner_id": { "type": "string", "index": "not_analyzed" },
"#version": { "type": "string", "index": "not_analyzed" },
"req_time" : { "type" : "float", "index" : "not_analyzed" },
"res_time" : { "type" : "string", "index" : "not_analyzed" },
"purchaseTime" : { "type" : "date", "index" : "not_analyzed" },
"received_at" : { "type" : "date", "index" : "not_analyzed" },
"itemPrice" : { "type" : "double", "index" : "not_analyzed" },
"total" : { "type" : "integer", "index" : "not_analyzed" },
"bytes" : { "type" : "double", "index" : "not_analyzed" }
}
}
}
}
Verified using:
curl -XGET 'http://localhost:9200/filebeat-2016.06.30/_mapping/field/req_time'
I am getting:
{"filebeat-2016.06.30":{"mappings":{"nginxlog":{"req_time": {"full_name":"req_time","mapping":{"req_time":{"type":"string"}}}}}}}
so my template definitely does not work. Anyone can help?

Answer 1

At the end, I just removed the template, and let ES guest the field type. It did work.