Rate Limit Bot Traffic Using Fail2ban - regex

Question

We have configured Fail2ban in Suse-Linux-Enterprise Server for Rate Limiting Bots Traffic. Below is the configuration done in jail.local file.
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
action = iptables-allports[name=apache-badbots, port="http,https" protocol=tcp]
logpath = /var/persistent/apache2/logs/site1-access.log
findtime = 60
bantime = 600
maxretry = 1
Below is the Regex Configuration.
failregex = <HOST> -.*(EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)
Below is the Log Format:
[14/Jul/2020:11:38:09 +0000] 192.168.1.14 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /sessionValueLink.action?crud=s&keyValue=JsMethodName&insertValue=submitShippingAddress();&dt=Tue%20Jul%2014%202020%2017:08:09%20GMT+0530%20(India%20Standard%20Time) HTTP/1.1" 200 44 [0/1894] "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/192.168.2.1 Safari/537.36"
When I ran the Regex Test, we are getting the below result.
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-badbots.conf
Results
Failregex: 2438 total
|- #) [# of hits] regular expression
| 1) [2438] <HOST> -.*(Googlebot/2\.1)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [113634] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 113634 lines, 0 ignored, 2438 matched, 111196 missed
Missed line(s): too many to print. Use --print-all-missed to print all 111196 lines
Issue is when we hit Google Bot continuously we are getting the IP Address is blocked in Log but it is not showing the Valid IPAddress and it is not blocking the bots traffic. Please find the below Log for your reference.
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,612 fail2ban.actions[431]: INFO [apache-badbots] 0.0.1.147 already banned
2020-07-14 14:27:03,274 fail2ban.actions[431]: WARNING [apache-badbots] Unban 0.0.1.147
2020-07-14 14:38:40,817 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 302 = ['0.0.1.46']
2020-07-14 14:38:41,073 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.46
2020-07-14 14:39:49,903 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:39:50,162 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.147
What mistake we have done here? How to solve the issue. I am new to fail2ban, any help will be appreciated.

Answer 1

We found the solution. Issue was with Regex, It was not fetching the proper IP Address from the log. We have changed to below Mentioned Regex and it is working fine.
failregex = (?:\[\]\s+)?\<HOST> [^"]*"[^"]*" \d+ \d+ [^"]*"[^"]*\b(?:EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)\b