Grok parse error when using custom pattern definitions - regex

I'm trying to use a grok filter in logstash version 1.5.0 to parse several fields of data from a log file.
I'm able to parse a simple WORD field with no issues, but when I try to define a custom pattern and add that in as well, the grok parse fails.
I've tried using a couple of grok debuggers which have been recommended elsewhere to find an issue:
http://grokconstructor.appspot.com/do/match
and
http://grokdebug.herokuapp.com/
both say that my regex should be fine, and return the fields that I want, but when I add it to my logstash.conf, grok fails to parse the log line and simply passes through the raw data to elasticsearch.
My sample line is as follows:
APPERR [2015/06/10 11:28:56.602] C1P1405 S39 (VPTestSlave002_001)| 8000B Connect to CGDialler DB (VPTest - START)| {39/A612-89A0-A598/60B9-1917-B094/9E98F46E} Failed to get DB connection: SQLConnect failed. 08001 (17) [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied.
My logstash.conf grok config looks like this:
grok
{
patterns_dir => ["D:\rt\Logstash-1.5.0\bin\patterns"]
match => {"message" => "%{WORD:LogLevel} \[%{KERNELTIMESTAMP:TimeStamp}\]"}
}
and the contents of my custom pattern file are:
KERNELTIMESTAMP %{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
I am expecting this to return the following set of data:
{
"LogLevel": [
[
"APPERR"
]
],
"TimeStamp": [
[
"2015/06/10 11:28:56.602"
]
],
"YEAR": [
[
"2015"
]
],
"MONTHNUM": [
[
"06"
]
],
"MONTHDAY": [
[
"10"
]
],
"HOUR": [
[
"11",
null
]
],
"MINUTE": [
[
"28",
null
]
],
"SECOND": [
[
"56.602"
]
],
"ISO8601_TIMEZONE": [
[
null
]
]
}
Can anyone tell me where my issue is?

Related

Logstash grok filter apache pattern

This is a sample Apache Tomcat log:
portal.portal.some.thing.int:8443 13.233.220.113 - - [09/Sep/2019:00:08:02 +0200] "GET /en/search-results?p_p_id=portal201_WAR_portal201_INSTANCE_q8EzsBteHybf&p_p_lifecycle=1&p_p_state=normal&queryText=Poll&facet.collection=AΜLex%2CAMsom%2CAMss%2WebPage%2SummariesOfSomething&startRow=1&resultsPerPage=10&SEARCH_TYPE=SIMPLE HTTP/1.1" 230 334734 6261 - - 35S64857F6860FDFC0F60B5B47A97E18
10.235.350.103 94.62.15.157, 10.435.230.101,10.134.046.2
I would like to capture the following variables
09/Sep/2019:00:08:02 +0200
/en/search-results?p_p_id=portal2....
35S64857F6860FDFC0F60B5B47A97E18
Can you help me with that? I want to index only those and drop the others, is it possible? Thank you
Use this grok pattern:
%{GREEDYDATA:field1} %{IP:ip1} - - \[%{GREEDYDATA:date}] \"%{WORD:method} %{GREEDYDATA:request}" %{WORD:numbers} %{WORD:numbers} %{WORD:numbers} - - %{WORD:last_parameter}
input:
portal.portal.some.thing.int:8443 13.233.220.113 - - [09/Sep/2019:00:08:02 +0200] "GET /en/search-results?p_p_id=portal201_WAR_portal201_INSTANCE_q8EzsBteHybf&p_p_lifecycle=1&p_p_state=normal&queryText=Poll&facet.collection=AΜLex%2CAMsom%2CAMss%2WebPage%2SummariesOfSomething&startRow=1&resultsPerPage=10&SEARCH_TYPE=SIMPLE HTTP/1.1" 230 334734 6261 - - 35S64857F6860FDFC0F60B5B47A97E18
10.235.350.103 94.62.15.157, 10.435.230.101,10.134.046.2
output:
{
"field1": [
[
"portal.portal.some.thing.int:8443"
]
],
"ip1": [
[
"13.233.220.113"
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
"13.233.220.113"
]
],
"date": [
[
"09/Sep/2019:00:08:02 +0200"
]
],
"method": [
[
"GET"
]
],
"request": [
[
"/en/search-results?p_p_id=portal201_WAR_portal201_INSTANCE_q8EzsBteHybf&p_p_lifecycle=1&p_p_state=normal&queryText=Poll&facet.collection=AΜLex%2CAMsom%2CAMss%2WebPage%2SummariesOfSomething&startRow=1&resultsPerPage=10&SEARCH_TYPE=SIMPLE HTTP/1.1"
]
],
"numbers": [
[
"230",
"334734",
"6261"
]
],
"last_parameter": [
[
"35S64857F6860FDFC0F60B5B47A97E18"
]
]
}
fields you want are:
date
request
last_parameter
You can remove other fields using remove field in mutate filter.

How do you extract a time stamp using logstash and grok?

I'm trying to extract a timestamp using TIME from grok in logstash, but the extraction is unsucessful.
I'm using a grok pattern, but it is not matching or returning anything.
2019-07-30 14:12:23 - main - INFO - metro crawler completed runtime:00:00:02
%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}-%{SPACE}%{GREEDYDATA:crawler}%{SPACE}-%{SPACE}%{LOGLEVEL:level}%{TIME:time}
I'm getting no matches
You may use
%{TIMESTAMP_ISO8601:timestamp}%{SPACE}-%{SPACE}%{DATA:crawler}%{SPACE}-%{SPACE}%{LOGLEVEL:level}%{DATA}%{TIME:time}
See the debug output:
{
"timestamp": [
[
"2019-07-30 14:12:23"
]
],
"crawler": [
[
"__main__"
]
],
"level": [
[
"INFO"
]
],
"time": [
[
"00:00:02"
]
]
}

Nesting Angular component files into one in Visual Studio

Which setting in Visual Studio 2017 allows for bundling Angular component files into one file?
In the image below the files in the red square should be visually presented as one that then can be expanded.
Names of the files are:
navmenu.component.css
navmenu.component.html
navmenu.component.ts
They only differ by extension. The html file should be the main one with ts and css being presented as sub files.
It was working fine until I updated Visual Studio 2017 to version 15.3.3 (before it was 15.1).
I finally found a solution to that issue on that thread. It requires editing the toolSettings.json file.
It requires editing toolSettings.json, located in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Web Tools\ProjectSystem
The following entries need to be added
".ts": [
".html"
],
under extensionToExtension.
And ".html" in the "css" array.
The whole file looks like that:
{
"dependentFileProviders": {
"addedExtension": {},
"pathSegment": {
"*": [
".js",
".css",
".html",
".htm",
".less",
".scss",
".coffee",
".iced",
".config",
".cs",
".vb",
".json"
]
},
"extensionToExtension": {
".js": [
".coffee",
".iced",
".ts",
".tsx",
".jsx"
],
".ts": [
".html"
],
".css": [
".less",
".scss",
".sass",
".styl",
".html"
],
".html": [
".md",
".mdown",
".markdown",
".mdwn",
],
".map": [
".js",
".css"
],
".svgz": [
".svg"
],
".designer.cs": [
".resx"
],
".cs.d.ts": [
".cs"
]
},
"fileToFile": {
".bowerrc": [
"bower.json"
],
".npmrc": [
"package.json"
],
"npm-shrinkwrap.json": [
"package.json"
],
"yarn.lock": [
"package.json"
],
".yarnclean": [
"package.json"
],
".yarnignore": [
"package.json"
],
".yarn-integrity": [
"package.json"
],
".yarnrc": [
"package.json"
]
},
"filePartToExtension": {
"-vsdoc.js": [
".js"
]
},
"allExtensions": {
"*": [
".tt"
]
}
}
}
The accepted answer got me on the right track, but I have a couple of improvements to contribute:
First, rather than editing the Visual Studio default rules, I would recommend creating your own nesting rules. You can either create one just for your personal use, or create a .filenesting.json file in your Project or Solution folder so your teammates get the same experience by default.
Second, since in Angular it's the typescript that defines the component and sometimes references HTML, CSS, etc., for me it makes more sense to reverse the relationship.
Here's the .filenesting.json file I ended up using for our web app project, in case it's useful as a minimalist starting point for other Angular and Visual Studio users.
{
"help": "https://go.microsoft.com/fwlink/?linkid=866610",
"root": true,
"dependentFileProviders": {
"add": {
"addedExtension": {},
"pathSegment": {
"add": {
".spec.ts": [
".ts"
]
}
},
"extensionToExtension": {
"add": {
".html": [
".ts"
],
".scss": [
".ts"
]
}
}
}
}
}

Can't do regex matching in Swift

I am working on a Swift project and I need to use this regex to check email is valid or not but when the app start the checking the app crash and give me this error:
NSInternalInconsistencyException', reason: 'Can't do regex matching, reason: Can't open pattern U_REGEX_MISSING_CLOSE_BRACKET
This is my REGEX:
^(([^<>()[\\]\\.,;:\\s#\\\"]+(\\.[^<>()[\\]\\.,;:\\s#\\\"]+)*)|(\\\".+\\\"))#((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|(([a-zA-Z\\-0-9]+[\\.]*)+[a-zA-Z]{2,}))$
Check unescaped brackets in your regex pattern:
let pattern
= "^(([^<>()[\\]\\.,;:\\s#\\\"]+(\\.[^<>()[\\]\\.,;:\\s#\\\"]+)*)|(\\\".+\\\"))"
// [ [ ] [ [ ]
+ "#((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|(([a-zA-Z\\-0-9]+[\\.]*)+[a-zA-Z]{2,}))$"
// [ ] [ ] [ ] [ ] [ ] [ ] [ ]
You have some mismatching brackets [ ] in the first half of your pattern.
In some dialects of regex, you have no need to escape [ between [ and ], but in some other dialects, you need it.
Try adding some escapes to your regex:
let pattern
= "^(([^<>()\\[\\]\\.,;:\\s#\\\"]+(\\.[^<>()\\[\\]\\.,;:\\s#\\\"]+)*)|(\\\".+\\\"))"
// [ ^^ ] [ ^^ ]
+ "#((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\])|(([a-zA-Z\\-0-9]+[\\.]*)+[a-zA-Z]{2,}))$"
// [ ] [ ] [ ] [ ] [ ] [ ] [ ]

Chaining grok filter patterns for logstash

I am trying to configure logstash to manage my various log sources, one of which is Mongrel2. The format used by Mongrel2 is tnetstring, where a log message will take the form
86:9:localhost,12:192.168.33.1,5:57089#10:1411396297#3:GET,1:/,8:HTTP/1.1,3:200#6:145978#]
I want to write my own grok patterns to extract certain fields from the above format. I received help on this question trying to extract the host. So if in grok-patterns I define
M2HOST ^(?:[^:]*\:){2}(?<hostname>[^,]*)
and then in the logstash conf specify
filter {
grok {
match => [ "message", "%{M2HOST}" ]
}
}
it works as expected. The problem I now have is I want to specify multiple patterns e.g. M2HOST, M2ADDR etc. I tried defining additional ones in the same grok-patterns file
M2HOST ^(?:[^:]*\:){2}(?<hostname>[^,]*)
M2ADDR ^(?:[^:]*\:){3}(?<address>[^,]*)
and changing the logstash conf
filter {
grok {
match => [ "message", "%{M2HOST} %{M2ADDR}" ]
}
}
but now I just get the error _grokparsefailure.
with your sample input from other question and with some guessing about the values names the full match would be:
(?:[^:]*:){2}(?<hostname>[^,]*)[^:]*:(?<address>[^,]*)[^:]*:(?<pid>[^#]*)[^:]*:(?<time>[^#]*)[^:]*:(?<method>[^,]*)[^:]*:(?<query>[^,]*)[^:]*:(?<protocol>[^,]*)[^:]*:(?<code>[^#]*)[^:]*:(?<bytes>[^#]*).*
Producing:
{
"hostname": [
[
"localhost"
]
],
"address": [
[
"192.168.33.1"
]
],
"pid": [
[
"57089"
]
],
"time": [
[
"1411396297"
]
],
"method": [
[
"GET"
]
],
"query": [
[
"/"
]
],
"protocol": [
[
"HTTP/1.1"
]
],
"code": [
[
"200"
]
],
"bytes": [
[
"145978"
]
]
}