Regex in logstash mutate gsub to replace a character in a string - regex

Question

I am trying to get double quotes (placed at random places) from a string replaced with something else.
This is the logline-
msg="AUT30544: User chose to proceed on the sign-in notification page "Sign-In Notification Message""
Actually this was part of KV parsing in logstash's filter section. If you notice there is a quoted string inside of a string that itself is in double-quotes.
However, Below string gets correctly parsed in KV-
msg="AUT23278: User Limit realm restrictions successfully passed for /google_auth "
Now I created a regex to remove the double-quotes in problematic string-
https://regex101.com/r/o00oot/1/
Applied it in logstash but nothing changed.
Below is my config file-
input {
tcp {
port => 1301
}
}
filter {
if "type=vpn" in [message] {
dissect {
mapping => { "message" => "%{reserved} id=firewall %{message1}" }
}
#mutate { gsub => ["message1",':'," "] }
#mutate { gsub => ["message1",'"',''] }
mutate {gsub => ["msg","(.*)\"(.*)\"(\")", "\1 '\2 '\3"] }
kv { source => "message1" value_split => "=" whitespace => "strict" } #field_split => " " remove_char_value => '"' }
geoip { source => "src" }
# \/ end of if vpn type log
}
else { drop {} }
}
A similar logline that I could capture using tcpdump is-
<134>Oct 2 11:24:45 1xx.xx.43.101 1 2021-10-02T11:24:45+05:30 canopus.domain1.com2 PulseSecure: - - - id=firewall time="2021-10-02 11:24:45" pri=6 fw=172.20.43.101 vpn=ive user=user1 realm="google_auth" roles="" proto=auth src=2xx.176.114.94 dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="AUT30544: User chose to proceed on the sign-in notification page "Sign-In Notification Message""
The stdout of same kind of message on stdout. I can see the double-quotes being escaped but still they create problem in parsing.
{
"type" => "vpn",
"user" => "user1",
"fw" => "1xx.xx.43.101",
"host" => "1xx.xx.4.63",
"realm" => "google_auth",
"src" => "1xx.66.50.112",
"port" => 33003,
"#version" => "1",
"message" => "<13>Oct 2 11:54:39 1xx.xx.43.101 396 <134>1 2021-10-02T11:54:39+05:30 canopus.domain1.com2 PulseSecure: - - - id=firewall time=\"2021-10-02 11:54:39\" pri=6 fw=1xx.xx.43.101 vpn=ive user=user1 realm=\"google_auth\" roles=\"\" proto=auth src=1xx.66.50.112 dst= dstname= type=vpn op= arg=\"\" result= sent= rcvd= agent=\"\" duration= msg=\"AUT30544: User chose to proceed on the sign-in notification page \"Sign-In Notification Message\"\"",
"geoip" => {
"location" => {
"lon" => 77.5937,
"lat" => 12.9719
},
If someone knows a KV plugin's native solution to this problem, I dont need to go through hassles of regex in gsub.

Answer 1

I'm not sure if you can use kv on whole message as you have, try to split it up so you get key/value part of the message in separate field and then use kv on it. That being said, I would suggest you to skip using gsub here completely because there is option called trim_value for kv filter.
With that option your configuration would look something like this. Disclaimer, this is not tested, maybe you'll have to play with regex inside of trim_value, but this is easier way to handle it.
input {
tcp {
port => 1301
}
}
filter {
if "type=vpn" in [message] {
dissect {
mapping => { "message" => "%{reserved} id=firewall %{message1}" }
}
kv {
source => "message1"
value_split => "="
whitespace => "strict"
trim_value => "\\\""
}
geoip {
source => "src"
}
}
else {
drop { }
}
}