fluentd multiline parser in parser filter - regex

Question

I'm trying to parse multiline logs from my applications in fluentd on kubernetes.
I currently have the following filter dropped-in my fluentd container:
<filter kubernetes.**>
#type parser
key_name log
emit_invalid_record_to_error false # do not fail on non-matching log messages
reserve_data true # keep the log key (needed for non-matching records)
<parse>
#type multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3})\s+(?<level>\S+)(?:\s+\[[^\]]*\])?\s+(?<pid>\d+)\s+---\s+\[\s*(?<thread>[^\]]+)\]\s+(?<class>\S+)\s+:\s+(?<message>.*)/
time_format %Y-%m-%d %H:%M:%S.%L
types pid:integer
</parse>
</filter>
This filter should parse spring boot style logs (which is not that important, as it is not working for my other filters as well).
Single line logs are parsed fine! All capture groups are detected and time format and pid type is also saved as an integer. But in case of a multi line log statement, the next line is just left as it is and saved as its own entry.
I got the idea for this parser from the fluentd documentation: https://docs.fluentd.org/parser/multiline
The documentation says currently in_tail plugin works with multiline but other input plugins do not work with it.
The container I'm using uses the in_tail plugin to get the logs. But I'm using the parser inside a filter. Not sure if this might be the problem? In the documentation the parser filter plugin (https://docs.fluentd.org/filter/parser) just links to the Parser Plugin Overview (https://docs.fluentd.org/parser) without mentioning anything about single parsers not working.
Would be great if someone could point me into the right direction!
Thanks in advance!

Answer 1

I came recently to exactly the same issue and still couldn't find obvious solution so I had to figure it out myself. It is exactly as it is in doc - this parser you mentioned works only as Parser section in Input plugin ('in_tail' only). It doesn't work in filter plugin unfortunately.
But for me this plugin helped:
https://github.com/fluent-plugins-nursery/fluent-plugin-concat
You just have to add one filter section above your main one where you do this concat, e.g. my example looks exactly like this (indicator of real new log is timestamp, if there is no timestamp it is always stacktrace of errors where the problem appears):
<filter XYZ.**>
#type concat
key log
multiline_start_regexp /\d{4}-\d{1,2}-\d{1,2}/
</filter>
<filter>
# here the original filter
</filter>