This is the an example log:
2022-01-07 11:05:01,185 [http-nio-8080-exec-526] WARN de.web.Main| = - Error while execute Request
javax.servlet.ServletException: com.cg.blart.web.HttpInvocationException: Invalid tenant session
at com.cg.blume.web.DispatchingServlet.doGet(DispatchingServlet.java:169)
at de.jinx.lee.web.AutoDBUpgradeDispatchingServlet.lambda$doGet$0(AutoDBUpgradeDispatchingServlet.java:200)
at de.jinx.lee.web.AutoDBUpgradeDispatchingServlet.addMDCKey(AutoDBUpgradeDispatchingServlet.java:297)
Caused by: com.cg.blume.web.HttpInvocationException: Invalid tenant session
at com.cg.blume.web.procedure.HttpSessionManager.get(HttpSessionManager.java:190)
at de.jinx.lee.web.session.leeHttpSessionManager.get(leeHttpSessionManager.java:76)
... 41 more
This ist the pattern so far:
^.*?Exception: +%{DATA:exception}$
The output i get is close but no cigar: "com.cg.blart.web.HttpInvocationException: Invalid tenant session". For the live of me i dont get it to match the whole line. Can you show me the way? The desired output is:
javax.servlet.ServletException: com.cg.blart.web.HttpInvocationException: Invalid tenant session
Here is a link for a test tool: https://grokdebug.herokuapp.com/
Solution was this pattern:
(?<DATA:mps_exception>.*?(Exception:).*)
It is explained here: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_custom_patterns
Related
I'm new to fluentd.
I have a log that I want to push to AWS with fluentd but I can't figure out what the regexp should be.
All the log lines, except the multilines, start with a UUID.
Here's a sample log:
6b0815f2-8ff1-4181-a4e6-058148288281 2020-11-03 13:00:05.976366 [DEBUG] switch_core_state_machine.c:611 (some_other_data) State Change CS_REPORTING -> CS_DESTROY
And, I'm trying to get UUID, DateTime, and Message.
With this regex:
/^(?<UUID>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) (?<time>.*) (?<message>[^ ]*)/gm
I'm getting the last word CS_DESTROY.
I tried fluentular and still got:
text:
f6a6e1ae-e52e-4aba-a8a5-4e3cc7f40914 2020-11-03 14:32:34.975779 [CRIT] mod_dptools.c:1866 audio3: https://mydomain.s3-eu-west-1.amazonaws.com/media/576d06e5-04fc-11eb-a52c-020fd8c14d18/5f9ddf2d5df0f698094395.mpg
regexp:
^(?<UUID>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) (?<time>.*) (?<message>[^ ]*)$
and got:
time 2020/11/03 14:32:34 +0000
UUID f6a6e1ae-e52e-4aba-a8a5-4e3cc7f40914
message https://mydomain.s3-eu-west-1.amazonaws.com/media/576d06e5-04fc-11eb-a52c-020fd8c14d18/5f9ddf2d5df0f698094395.mpg
It's missing what's between the datetime and "https".
Try:
^(?<UUID>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) (?<time>[^\[]*) (?<message>\[.*)$
Live at rubular: https://rubular.com/r/JQQXs5VTkr2IxM
Here's the output for both logs:
Match 1
UUID 6b0815f2-8ff1-4181-a4e6-058148288281
time 2020-11-03 13:00:05.976366
message [DEBUG] switch_core_state_machine.c:611 (some_other_data) State Change CS_REPORTING -> CS_DESTROY
Match 2
UUID f6a6e1ae-e52e-4aba-a8a5-4e3cc7f40914
time 2020-11-03 14:32:34.975779
message [CRIT] mod_dptools.c:1866 audio3: https://mydomain.s3-eu-west-1.amazonaws.com/media/576d06e5-04fc-11eb-a52c-020fd8c14d18/5f9ddf2d5df0f698094395.mpg
I am trying to send a large chunk emails and got error with message "Domain contains dot-dot".
Does anyone know why I am getting this error?
#0 /var/www/vhosts/domainame.com/vendor/aws/aws-sdk- php/src/Aws/Common/Exception/NamespaceExceptionFactory.php(76): Aws\Common\Exception\NamespaceExceptionFactory->createException('Aws\\Ses\\Excepti...', Object(Guzzle\Http\Message\EntityEnclosingRequest), Object(Guzzle\Http\Message\Response), Array)
#1 /var/www/vhosts/domain.com/vendor/aws/aws-sdk-php/src/Aws/Common/Exception/ExceptionListener.php(55): Aws\Common\Exception\NamespaceExceptionFactory->fromResponse(Object(Guzzle\Http\Message\EntityEnclosingRequest), Object(Guzzle\Http\Message\Response))
#2 [internal function]: Aws\Common\Exception\ExceptionListener->onRequestError(Object(Guzzle\Common\Event), 'request.error', Object(Symfony\Compone in /var/www/vhosts/domainname.com/vendor/guzzle/guzzle/src/Guzzle/Service/Exception/CommandTransferException.php on line 25
PHP Fatal error: Uncaught exception 'Guzzle\Service\Exception\CommandTransferException' with message 'Errors during multi transfer
(Aws\Ses\Exception\SesException) /var/www/vhosts/domainname.com/vendor/aws/aws-sdk-php/src/Aws/Common/Exception/NamespaceExceptionFactory.php line 91
Domain contains dot-dot
Domain name of some mail address contains more then one dot sign (.) literally.
eg. Alan#stackoverflow..com
Reference:
1.
https://java.net/projects/javamail/sources/mercurial/diff/mail/src/oldtest/java/javax/mail/internet/addrlist?rev1=61&rev2=62
https://java.net/projects/javamail/sources/mercurial/content/mail/src/main/java/javax/mail/internet/InternetAddress.java?rev=557
I'm creating log parser to parse the log message from different source like rsyslog, logback extension, nxlog etc.
I have to extract exception message fields. But I stuck while generating regex for below test string.
Test String:
2014-10-16 01:32:22,780 ERROR main Sample.main - java.lang.NullPointerException: Sample Log4j Exception
at Sample.errorLevel3(Sample.java:35)
at Sample.errorLevel2(Sample.java:31)
at Sample.errorLevel1(Sample.java:27)
at Sample.main(Sample.java:16)
Note: \n and \t are received as literal #012#011 in the string escaped by rsyslog
Expected Match:
2014-10-16 01:32:22,780
java.lang.NullPointerException
Sample Log4j Exception
----------
Sample.errorLevel3
Sample.java
35
----------
Sample.errorLevel2
Sample.java
31
----------
Sample.errorLevel1
Sample.java
27
Example:
info: 2014-10-28T22:39:46.593Z - info: an error occurred while trying
to handle command: PlaceMarketOrderCommand, xkkdAAGRIl. Error:
Insufficient Cash #userId=5 #orderId=Y5545
pattern:
> %{LOGLEVEL:stream_level}: %{TIMESTAMP_ISO8601:timestamp} -
> %{LOGLEVEL:log_level}: %{MESSAGE:message}
> (#userId=%{USER_ID:user_id})? (#orderId=%{ORDER_ID:order_id})?
extra patterns used:
USER_ID (\d+|None)
ORDER_ID .*
ORDER_ID_HASH \s*(#orderId=%{ORDER_ID:order_id})?
USER_ID_HASH \s*(#userId=%{USER_ID:user_id})?
MESSAGE (.*?)
Works fine:
removing the optional last orderId also works
info: 2014-10-28T22:39:46.593Z - info: an error occurred while trying
to handle command: PlaceMarketOrderCommand, xkkdAAGRIl. Error:
Insufficient Cash #userId=5
but if I keep the orderId and remove the userId then I get a "no match"
info: 2014-10-28T22:39:46.593Z - info: an error occurred while trying
to handle command: PlaceMarketOrderCommand, xkkdAAGRIl. Error:
Insufficient Cash #orderId=Y5545
Also the user_id group is ending with a ? as an optional group..
working with the grok debugger in heroku:
Is this a bug? (logstash 1.4.2) missing something with the regex? (more probable.. but what?)
I looked at the regex lib grok is using and looks this syntax supposed to work. It does work for the last group (orderId) but not for the one before..
Thanks for the help!
You are forcing a space to be before your optional last... you need to do ?:
%{LOGLEVEL:stream_level}: %{TIMESTAMP_ISO8601:timestamp} -> %{LOGLEVEL:log_level}: %{MESSAGE:message} ?(#userId=%{USER_ID:user_id})? ?(#orderId=%{ORDER_ID:order_id})?
I have tried to write a regex for fail2ban for my exim mail server but I can't seem to get any matches. even on the rejected logfile.
Here is a line from my exim_mainlog:
2014-09-18 16:34:30 dovecot_login authenticator failed for xx-xx-78-xx.dedicated.abac.net (User) [xx.xx.78.xx]:64298: 535 Incorrect authentication data (set_id=sexy)
2014-09-18 16:50:17 dovecot_login authenticator failed for (User) [xx.xx.xx.231]:9859: 535 Incorrect authentication data (set_id=evans)
2014-09-18 16:52:30 dovecot_login authenticator failed for (User) [xx.xx.16.128]:60350: 535 Incorrect authentication data (set_id=orange)
2014-09-18 17:10:19 dovecot_login authenticator failed for XXXX.onlinehome-server.com (User) [xx.xx.96.171]:52799: 535 Incorrect authentication data (set_id=matrix)
the same entries are in my exim_rejectlog.
Here is the filter I tried on both the exim_mainlog and exim_reject log
failregex = \[<HOST>\]: 535 Incorrect authentication data
But I don't get any hits even from the ones that are in the rejectlog
Lines: 257 lines, 0 ignored, 0 matched, 257 missed
Im not very good at regex and would appreciate any help to keep these spammers away
Your server is configured to also log the incoming port, not just the ip address. Change your regex to accommodate the extra characters:
failregex = \[<HOST>\]:\d+: 535 Incorrect authentication data
Official released version with exim parsing (it now handles the port if it's present). https://github.com/fail2ban/fail2ban/blob/master/config/filter.d/exim.conf