Extracting a substring after a match position using grok in logstash - regex

Question

Objective : I have a log file from where I want to extract the amount details after the string Amount::: in the below given log file.
What I have Done so far: Since it is a Custom Parsing, I have created a custom pattern using RegEx and I am trying to Implement it using logstash.
here is my log file -
28-04-2017 14:45:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 3000.00
28-04-2017 12:45:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 31000.00
28-04-2017 14:15:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 10000.00
28-04-2017 11:45:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 9000.00
28-04-2017 08:15:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 7000.00
I have used Regex to find the string Amount:::
Note : I want to extract the sub string which is coming after the string Amount:::
here are my Custom Patterns I have used in Grok:
(but it doesn't yield good results)
CUSTOM_AMOUNT (?<= - Amount::: ).*
CUSTOM_AMOUNT (?<=Amount::: )%{BASE16FLOAT}
here is my logstacsh.conf-
input {
file {
path => "D:\elk\data\amnt_parse.txt"
type => "customgrok"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter{
if[type]== "customgrok" {
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "D:\elk\logstash-5.2.1\vendor\bundle\jruby\1.9\gems\logstash-patterns-core-4.0.2\patterns\custom"
match => { "message" => "%{CUSTOM_AMOUNT:amount" }
add_field => { "subType" => "Amount" }
}
}
}
mutate {
gsub => ['message', "\t", " "]
} } }
output {
stdout {
codec => "rubydebug"
}
elasticsearch {
index => "amnt_parsing_change"
hosts =>"localhost"
}
}
Our intension is to Visualize and to perform aggregation operations based on the extracted substring using Kibana and Elasticsearch.
but it stores the log file into the variable "message". as you can see here, match => { "message" => "%{CUSTOM_AMOUNT:amount" }.
here is how the line is stored inside "message", when I tried to view it in Kibana -
"message": "28-04-2017 11:45:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 9000.00\r",
"message": "28-04-2017 12:45:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 31000.00\r",
"message": "28-04-2017 11:45:50 INFO abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 9000.00\r",
Logstash file is loading the Data(log file) and Index is also getting created but Custom Pattern isn't giving expected result.
what are possibilities to extract the sub string which I have mentioned above ? or do we have any alternatives?

Answer 1

Here is what you have to do :
filter {
grok {
match => {
"message" => "%{DATESTAMP:Date} %{WORD:LogSeverity}\s+%{WORD:LogInfo} \(%{NOTSPACE:JavaClass}\) \- Amount::: %{NUMBER:Amount}"
}
}
mutate
{
gsub =>
[
"Data"," ","-"
]
#If you dont want those fields
remove_field => ["Date","LogSeverity","LogInfo","JavaClass"]
}
}
I recommend you to read the documentations :
Grok Documentation
Grok Patterns
You can use the following debugger :
GrokDebbuger