So, I'm a little bit new at the elk-stack, and I’m having an issue with further experiment with the tools. I'm using a linux machine.
First of all, here's my config-file :
input {
file {
type => "openerp"
path => "/home/jvc/Documents/log/openerp-cron.log.2014-11-20.txt"
start_position => "beginning"
codec => multiline{
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
filter{
if [type]=="openerp"{
date{
match => ["timestamp","yyyy-MM-dd HH:mm:ss,SSS"]
}
grok{
patterns_dir => "./patterns"
match => { "message" => "%{ODOOLOG}" }
}
}
}
output{
file{
path => "/home/jvc/Bureau/testretour.txt"
}
}
I have some patterns too :
REQUESTTIMESTAMP %{MONTHDAY}/%{MONTH}/%{YEAR} %{TIME}
REQUEST %{IPORHOST:client} %{USER:ident} %{USER:auth} [%{REQUESTTIMESTAMP:request_timestamp}] "%{WORD:request_type} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} -
ODOOMISC %{GREEDYDATA}
ODOOLOG %{TIMESTAMP_ISO8601:timestamp} %{POSINT:pid} %{LOGLEVEL:level} (?:%{USERNAME:user}|\?) %{PROG:module}: (?:%{REQUEST}|%{ODOOMISC:misc})
Some examples of the logs :
2014-11-21 08:00:16,715 17798 DEBUG noe openerp.addons.base.ir.ir_cron: cron.object.execute('noe', 1, '*', u'crossovered.budget.lines', u'computeblank')
2014-11-21 08:00:17,172 17798 WARNING noe openerp.osv.orm.browse_record.noe_utils.synchro_date: Field 'conform' does not exist in object 'browse_record(noe_utils.synchro_date, 13)'
2014-11-21 08:00:17,172 17798 ERROR noe openerp.sql_db: Programming error: can't adapt type 'browse_record', in query SELECT id
FROM crossovered_budget_lines
WHERE is_blank='t'
AND general_budget_id in %s
AND date_from <= %s AND date_to >= %s
2014-11-21 08:00:17,173 17798 ERROR noe openerp.addons.base.ir.ir_cron: Call of self.pool.get('crossovered.budget.lines').computeblank(cr, uid, *()) failed in Job 10
I'm having trouble with this config. For some reason that I can't find, this produces nothing.
What I have tried - done :
-First of all, I tested my grok, and multiline pattern in some grok debugger I have find on the web. All of them matches my logs.
-Before using the codec for multiline, i used the multiline filter. This one worked, but seems to be deprecated. So it's not a solution.
-I know that logstash keep in mind what he had read or not with the "sincedb" files : I delete these before every test, but you know what happens.
-I tried to run logstash with the -verbose, but nothing wrong is displayed.
-I don't really know if I must write the ".txt" at the end of my paths. But anyway, none of them works.
Have I missed something ? Thank you in advance for helping hands.
So, with more test i succeeded. I copied the content of one of my logs file and pasted it in another file : It works.
But, there is now another question : if deleting the "sincedb" file doesn't work, how can i "empty" the cache of logstash ?
Related
This is a log line
2015-10-05 12:04:19.199 INFO 4808 --- [metrics-logger-reporter-2-thread-1] com.example.metrics : type=TIMER, name=demo.ws.rest.controllers.ItemController.getAllItems, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
I tried to learn grok and this is what i have so far
"message" => "%{TIMESTAMP_ISO8601:time}%{SPACE}%{WORD}%{SPACE}%{NUMBER}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD}%{NOTSPACE}%{WORD:metrictype}%{NOTSPACE}%{SPACE}%{WORD:vardspirms}%{DATA:pirms}%{JAVAFILE:javafilename}%{NOTSPACE:peec}%{SPACE}%{WORD}%{NOTSPACE}%{NUMBER:count}%{GREEDYDATA:debuginfo}"
And it looks so long, inefficient and bad practise. I would like to know, how can I add ignore inside grok. so i can ignore everything between INFO and type. Sorry for my english, Im not a native speaker.
I found a pretty handy solution.
kv {
source => "debuginfo" # new field generated by grok before
field_split => ", " # split fields by semicolon
}
Seems to split everything in debuginfo pretty good.
Let's say you have 2 very different types of logs such as FORTINET and NetASQ logs and you want:
grok FORTINET using a regex, ang grok NETASQ using an other regex.
I know that with "type"in the input file and "condition" in the filter we can resolve this problem.
So I used this confing file to do it :
input {
file {
type => "FORTINET"
path => "/fortinet/*.log"
sincedb_path=>"/logstash-autre_version/var/.sincedb"
start_position => 'beginning'
}
file {
type => "NETASQ"
path => "/home/netasq/*.log"
}
}
filter {
if [type] == "FORTINET" {
grok {
patterns_dir => "/logstash-autre_version/patterns"
match => [
"message" , "%{FORTINET}"
]
tag_on_failure => [ "failure_grok_exemple" ]
break_on_match => false
}
}
if [type] == "NETASQ" {
# .......
}
}
output {
elasticsearch {
cluster => "logstash"
}
}
And i'm getting this error :
Got error to send bulk of actions: no method 'type' for arguments(org.jruby.RubyArray) on Java::OrgElasticsearchActionIndex::IndexRequest {:level=>:error}
But if don't use "type" and i grok only FORTINET logs it wroks.
What should i do ?
I'm not sure about this but maybe it helps:
I have the same error and I think that it is caused by the use of these if statements:
if [type] == "FORTINET"
your type field is compared to "FORTINET" but this is maybe not possible because "FORTINET" is a string and type isn't. Some times by setting a type to an input, if there is already a type, the type isn't replaced, but the new type is added to a list with the old type. You should have a look to your data in kibana (or wherever) and try to find something like this:
\"type\":[\"FORTINET\",\"some-other-type\"]
maybe also without all those \" .
If you find something like this try not to set the type of your input explicitly and compare the type in your if-statement to the some-other-type you have found.
Hope this works (I'm working with more complex inputs/forwarders and for me it doesn't, but it is worth a try)
My logs are formatted like this:
2014-06-19 02:26:05,556 INFO ok
2014-06-19 02:27:05,556 ERROR
message:space exception
at line 85
solution:increase space
remove files
There are 2 types of events:
-log on one line like the first
-log on multiple line like the second
I am able to process the one line event, but I am not able to process the second type, where I would like to stock the message in one variable and the solution in another.
This is my config:
input {
file {
path => ["logs/*"]
start_position => "beginning"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
filter {
#parsing of one line event
grok {
patterns_dir => "./patterns"
match=>["message","%{TIMESTAMP_ISO8601:timestamp} %{WORD:level} ok"]
}
#the parsing fail, so we assumed we are in multiline events, now I process them and I am stuck when I am getting to the new line.
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "./patterns"
match=>["message","%{TIMESTAMP_ISO8601:timestamp} %{WORD:level}\r\n"]
}
}
}
So this is what I have done, and I would like to have in my console output the following:
{
"#timestamp" => "2014-06-19 00:00:00,000"
"path" => "logs/test.log"
"level"=>"INFO"
},
{
"#timestamp" => "2014-06-19 00:00:00,000"
"path" => "logs/test.log"
"level"=>"ERROR"
"message" => "space exception at line 85"
"solution"=>"increase space remove files"
}
Concretely, I would like to get all the expression between two words ("message" and "solution" for the message variable, "solution" and the end of event for the solution variable), and that no matter if the expression is on one or multiple lines.
Thanks in advance
As for multiline grok, it's best to use special flag for pattern string:
grok {
match => ["message", "(?m)%{SYSLOG5424LINE}"]
}
It looks like you have two issues:
You need to correctly combine your multilines:
filter
{
multiline
{
pattern => "^ "
what => "previous"
}
}
This will combine any line that begins with a space into the previous line. You may end up having to use a "next" instead of a "previous".
Replace Newlines
I don't believe that grok matches across newlines.
I got around this by doing the following in your filter section. This should go before the grok section:
mutate
{
gsub => ["message", "\n", "LINE_BREAK"]
}
This allowed me to grok multilines as one big line rather than matching only till the "\n".
I have a line in my log files that literally just have a semi colon in them. I am assuming it is attached to the previous line. Logstash is constantly printing them, and I want to drop these when ever there is a line that begins with a ;.
This is what logstash prints:
"message" => ";/r"
"#version" => "1"
"#timestamp" => 2014-06-24T15:39:00.655Z,"
"type" => "BCM_Core",
"host => XXXXXXXXXXX",
"Path => XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"tags" => [
[0] "_grokparsefailureZ"
],
"BCM_UTC_TIME" =>"2014-06-24%{time}Z"
I've attempted to use multiline to append to previous line so logstash would stop printing:
multiline{
type => "BCM_Core"
pattern => "\;"
negate => true
what => "previous"
}
but logstash is still printing them out. How can I make logstash drop it?
Just use a drop filter to drop any line that starts with ;:
filter {
if ([message] =~ "^;") {
drop {}
}
}
Although based on your output, it really ;/r not ;\r, so you might need to adjust if your output is not just an example.
You can also just drop anything that fails to grok:
if "_grokparsefailure" in [tags] { drop {} }
Is it possible at all? My use case is setting wildcard domain's docroot. I have a variable
$docroot = "/var/www"
and, inside apache::vhost {} I'm trying to set
virtual_docroot' => "{$docroot}/%-2+",
That however throws an error saying:
Error: Could not parse for environment production: Syntax error at ' => "$docroot/%-2+",
docroot => $docroot,
serveraliases => ['; expected '}' at /tmp/vagrant-puppet-1/manifests/init.pp:22 on node localhost
My guess is that the whole expression is treated as a regexp, and $ treated as a part of the whole expression. Hence my doubts whether that's possible at all.
The answer turns out to be an orphaned apostrophe, misplaced after virtual_docroot, and copied from puppetlabs-apache documentation, where the faulty code is:
apache::vhost { 'subdomain.loc':
vhost_name => '*',
port => '80',
virtual_docroot' => '/var/www/%-2+',
docroot => '/var/www',
serveraliases => ['*.loc',],
}
Removing the apostrophe fixed the problem. Answer to my question is to set that particular configuration line as:
virtual_docroot => "$docroot/%-2+",
Notice the omitted {} characters.