fail2ban need help to create regex rules - regex

I try to protect my serveur from xmlrpc.php ddos.
I use fail2ban, but the regex I found dont seems to be ok. Can you have a look:
This is the log:
Aug 2 17:33:11 myserver pound: my.web.site 188.209.49.38 - -
[02/Aug/2015:17:33:11 +0200] "POST /xmlrpc.php HTTP/1.0" 404 410 ""
"Mozilla/5.0 (compatible; Googlebot/2.1;
http://www.google.com/bot.html)"
Aug 2 16:27:49 myserver pound:
(7fec610c5700) e503 no back-end "POST /xmlrpc.php HTTP/1.0" from
185.62.188.25
filter.d/xmlrpc.conf
[Definition]
failregex = ^<HOST> .*POST .*xmlrpc\.php.*
ignoreregex =
jail.local
[xmlrpc]
enabled = true
filter = xmlrpc
action = iptables[name=xmlrpc, port=http, protocol=tcp]
logpath = /var/log/pound.log
bantime = 43600
maxretry = 2
And the test
fail2ban-regex /var/log/pound.log /etc/fail2ban/filter.d/xmlrpc.conf
/usr/share/fail2ban/server/filter.py:442: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
Running tests
=============
Use regex file : /etc/fail2ban/filter.d/xmlrpc.conf
Use log file : /var/log/pound.log
Results
=======
Failregex
|- Regular expressions:
| [1] ^<HOST> .*POST .*xmlrpc\.php.*
|
`- Number of matches:
[1] 0 match(es)
Ignoreregex
|- Regular expressions:
|
`- Number of matches:
Summary
=======
Sorry, no match
Look at the above section 'Running tests' which could contain important
information.
root#myserver:/etc/fail2ban#
Any idea?
Thks

I edited the type format, so I have now this kind of log
Aug 3 06:25:51 ns111111 pound: 141.101.96.94 POST /xmlrpc.php HTTP/1.1 - HTTP/1.1 200 OK
So I tried this, and it's ok :
fail2ban-regex 'Aug 3 06:25:51 ns111111 pound: 141.101.96.94 POST /xmlrpc.php HTTP/1.1 - HTTP/1.1 200 OK' 'ns111111 pound: <HOST> .*POST .*xmlrpc\.php.*'

Related

Rate Limit Bot Traffic Using Fail2ban

We have configured Fail2ban in Suse-Linux-Enterprise Server for Rate Limiting Bots Traffic. Below is the configuration done in jail.local file.
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
action = iptables-allports[name=apache-badbots, port="http,https" protocol=tcp]
logpath = /var/persistent/apache2/logs/site1-access.log
findtime = 60
bantime = 600
maxretry = 1
Below is the Regex Configuration.
failregex = <HOST> -.*(EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)
Below is the Log Format:
[14/Jul/2020:11:38:09 +0000] 192.168.1.14 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /sessionValueLink.action?crud=s&keyValue=JsMethodName&insertValue=submitShippingAddress();&dt=Tue%20Jul%2014%202020%2017:08:09%20GMT+0530%20(India%20Standard%20Time) HTTP/1.1" 200 44 [0/1894] "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/192.168.2.1 Safari/537.36"
When I ran the Regex Test, we are getting the below result.
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-badbots.conf
Results
Failregex: 2438 total
|- #) [# of hits] regular expression
| 1) [2438] <HOST> -.*(Googlebot/2\.1)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [113634] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 113634 lines, 0 ignored, 2438 matched, 111196 missed
Missed line(s): too many to print. Use --print-all-missed to print all 111196 lines
Issue is when we hit Google Bot continuously we are getting the IP Address is blocked in Log but it is not showing the Valid IPAddress and it is not blocking the bots traffic. Please find the below Log for your reference.
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,612 fail2ban.actions[431]: INFO [apache-badbots] 0.0.1.147 already banned
2020-07-14 14:27:03,274 fail2ban.actions[431]: WARNING [apache-badbots] Unban 0.0.1.147
2020-07-14 14:38:40,817 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 302 = ['0.0.1.46']
2020-07-14 14:38:41,073 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.46
2020-07-14 14:39:49,903 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:39:50,162 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.147
What mistake we have done here? How to solve the issue. I am new to fail2ban, any help will be appreciated.
We found the solution. Issue was with Regex, It was not fetching the proper IP Address from the log. We have changed to below Mentioned Regex and it is working fine.
failregex = (?:\[\]\s+)?\<HOST> [^"]*"[^"]*" \d+ \d+ [^"]*"[^"]*\b(?:EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)\b

scala apache access log regex not working

I have defined regex for apache access log as below:
val apacheLogPattern = """
^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+)\s?(\S+)?\s?(\S+)?" (\d{3}|-) (\d+|-)\s?"?([^"]*)"?\s?"?([^"]*)?"?$
""".r
And a function to parse the log:
def parse_log(line: String) = {
line match {
case apacheLogPattern(ipAddress, clientIdentity, userId, dateTime, method, endPoint,
protocol, responseCode, contentSize, browser, somethingElse) => "match"
}
}
val p = """66.249.69.97 - - [24/Sep/2014:22:25:44 +0000] "GET /071300/242153 HTTP/1.1" 404 514 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"""
parse_log(p)
Calling the parse function gives MatchError
scala.MatchError:
66.249.69.97 - - [24/Sep/2014:22:25:44 +0000] "GET /071300/242153 HTTP/1.1" 404 514 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
(of class java.lang.String)
at .parse_log(:13)
... 28 elided
Can someone help me where the scala regex is going wrong?
From The fourth bird's comment, the regex is lacking .r at the end, and has one too many capturing groups. The correct pattern is shown below.
val apacheLogPattern = """^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+)\s?(\S+)?\s?(\S+)?" (\d{3}|-) (\d+|-)\s?"?([^"]*)"?\s?"?([^"]*)?"?$""".r

fail2ban scan for 403 in nginx access logs

I have setup some specific rules on nginx, blocking some urls and some extensions (aspx, sh, jsp, etc..).
I have also enable a custom access log file only for 403|429|410 errors, so that in only 1 place i can have all my access denied log.
My goal is to have fail2ban read this log and for every GET/POST that ends in a 403 error, IP should be banned.
1) nginx.conf will be logging the custom error log file like this:
log_format limit '$time_local - $remote_addr "$request" $status';
and this is a log entry:
03/Jan/2017:15:53:01 +0100 - 1.2.3.4 "GET /aaa.jsp HTTP/1.1" 403
2) i have a fail2ban filter like this (taken from here)
^<HOST> .* "(GET|POST) [^"]+" 403
3) i have tried with fail2ban-regex
fail2ban-regex /var/log/nginx/access-live-limitbot-website.log /etc/fail2ban/filter.d/nginx-403.conf
and this is the output
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [1] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 2 lines, 0 ignored, 0 matched, 2 missed
|- Missed line(s):
| 217.19.158.242 "POST /wp-login.php HTTP/1.1" 403
| 03/Jan/2017:15:53:01 +0100 - 217.19.158.242 "GET /aaa.jsp HTTP/1.1" 403
`-
and i will never get the entry matching the error code.
Will someone please help me with the regex based on my custom log?
thank you
Fail2ban is picky about the date format. Also, for ease of matching, I suggest reordering the items in the log.
For date format, see documentation here:
https://www.fail2ban.org/wiki/index.php/MANUAL_0_8
In order for a log line to match your failregex, it actually has to match in two parts: the beginning of the line has to match a timestamp pattern or regex, and the remainder of the line has to match your failregex. If the failregex is anchored with a leading ^, then the anchor refers to the start of the remainder of the line, after the timestamp and intervening whitespace.
The pattern or regex to match the time stamp is currently not documented, and not available for users to read or set. See Debian bug #491253. This is a problem if your log has a timestamp format that fail2ban doesn't expect, since it will then fail to match any lines. Because of this, you should test any new failregex against a sample log line, as in the examples below, to be sure that it will match. If fail2ban doesn't recognize your log timestamp, then you have two options: either reconfigure your daemon to log with a timestamp in a more common format, such as in the example log line above; or file a bug report asking to have your timestamp format included.
For the reorder, something like datetime - status - host (- other stuff), would help create a simple pattern such as 403.
Therefore your log should look like:
03-01-2017 15:53:01 403 1.2.3.4 "GET /aaa.jsp HTTP/1.1"
and your pattern can be
403 <HOST>
You can run this from the command line to validate as:
fail2ban-regex '03-01-2017 15:53:01 403 1.2.3.4 "GET /aaa.jsp HTTP/1.1"' '403 <HOST>'
Which produces the output:
Running tests
=============
Use regex line : 403 <HOST>
Use single line: 03-01-2017 15:53:01 403 1.2.3.4 "GET /aaa.jsp HTTP...
Matched time template Day-Month-Year Hour:Minute:Second
Got time using template Day-Month-Year Hour:Minute:Second
Results
=======
Failregex: 1 total
|- #) [# of hits] regular expression
| 1) [1] 403 <HOST>
`-
Ignoreregex: 0 total
Summary
=======
Addresses found:
[1]
1.2.3.4 (Tue Jan 03 15:53:01 2017)
Date template hits:
2 hit(s): Day-Month-Year Hour:Minute:Second
Success, the total number of match is 1

Create fail2ban custom rule for Apache2

I am trying to create a custom rule to ban users trying to log in too many times. Trigger is the word "CheckLogin" in the apache log file.
Log extract:
[03/Mar/2016:19:38:24 -0600] 186.77.136.133 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "POST /CheckLogin HTTP/1.1" -
[03/Mar/2016:19:38:24 -0600] 186.77.136.133 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /Login?nok=badpassword HTTP/1.1" 10570
[03/Mar/2016:19:38:27 -0600] 186.77.136.133 TLSv1.2 ECDHE-RSA-AES128-
Current filter : /etc/fail2ban/filter.d/test.conf:
[INCLUDES]
[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD).*CheckLogin".*$
ignoreregex =
Current jail.local:
# detect password authentication failures
[test-auth-ssl]
enabled = true
port = https
filter = test
logpath = /var/log/apache2/ssl_request_log
maxretry = 3
bantime = 36000 ; 10 hrs
findtime = 360 ;
[test-auth]
enabled = true
port = http
filter = test
logpath = /var/log/apache2/access_log
maxretry = 3
bantime = 36000 ; 10 hrs
findtime = 360 ;
Must be a tricky detail in the filter failregex, but I tried various option and none worked. I can restart fail2ban without error, but external IP used for testing is never banned (trigger deos not work).
Status for the jail: test-auth-ssl
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- Journal matches:
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
Got it !
Playing around with fail2ban-regex I finally found the solution.
/etc/fail2ban/filter.d/test.conf
[INCLUDES]
[Definition]
failregex = <HOST> .*CheckLogin.*$
ignoreregex =
Also in jail.local I have had to add backend=auto since it was using systemd by default

Assistance needed with regular expressions [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 8 years ago.
Improve this question
The HTTP messages are listed below right after the questions.
I need a regular expression that finds the HTTP status codes within both messages.
Another one that finds the name of the requesting user in both messages.
A last one that finds the time stamp within both messages.
127.0.0.1 - Johny [17/Dec/2010:17:15:16 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
127.0.0.1 - debbie7 [19/Dec/2010:11:11:02 -0700] "GET /apache_pbs.gif
HTTP/1.0" 404 2336
Thanks!
Description
You can pull the values {username, date, and http code} in one pass using this regex:
^.*?-\s(\S*)\s+\[([^\]]*)\]\s"[^"]*"\s(\d+)\s\d+
Groups
Group 0 gets the entire line, while the other groups will individually get the respective matches.
gets the username
gets the date stamp
gets the http status code
PHP Code Example:
You didn't select a language so I present a php example to show how the regex works
Given input string, complete with link break in the middle of the message area
127.0.0.1 - Johny [17/Dec/2010:17:15:16 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
127.0.0.1 - debbie7 [19/Dec/2010:11:11:02 -0700] "GET /apache_pbs.gif
HTTP/1.0" 404 2336
Code Example
<?php
$sourcestring="your source string";
preg_match_all('/^.*?-\s(\S*)\s+\[([^\]]*)\]\s"[^"]*"\s(\d+)\s\d+/im',$sourcestring,$matches);
echo "<pre>".print_r($matches,true);
?>
$matches Array:
(
[0] => Array
(
[0] => 127.0.0.1 - Johny [17/Dec/2010:17:15:16 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
[1] => 127.0.0.1 - debbie7 [19/Dec/2010:11:11:02 -0700] "GET /apache_pbs.gif
HTTP/1.0" 404 2336
)
[1] => Array
(
[0] => Johny
[1] => debbie7
)
[2] => Array
(
[0] => 17/Dec/2010:17:15:16 -0700
[1] => 19/Dec/2010:11:11:02 -0700
)
[3] => Array
(
[0] => 200
[1] => 404
)
)
HTTP status:
(?<=HTTP/1.0" )\d+
Requesting user (works for any ip address):
(?<=(\d\d?\d?\.){3}\d\d?\d? - )\w+(?= \[)
Timestamp:
(?<=\[).*(?=\])
You can try with this Regex to achieve this:
^.* (\w*) \[([^\]]*)] \"[\w.\/ ]*\" ([\d]+)
Input:
127.0.0.1 - Johny [17/Dec/2010:17:15:16 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
Output:
Group 1: Johny
Group 2: 17/Dec/2010:17:15:16 -0700
Group 3: 200
You can test the Regex here.
In Perl:
!([a-zA-Z]+) \W+
(.* -) [\w\W]+
HTTP/1.0" \ ([\d]+)
!x
$1 -> username
$2 -> timestamp
$3 -> status