I have this rule bellow, and I'd like to limit for only wp-login.php files. Is it possible?
SecRule REQUEST_HEADERS:User-Agent "#contains python-requests" "id:1000000,t:none,t:lowercase,deny,nolog,msg:'BAD BOT - Detected and Blocked. '"
Use chained rules:
SecRule REQUEST_URI "wp-admin.php" "chain,id:1000000,t:none,t:lowercase,deny,nolog,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#contains python-requests" ",t:none,t:lowercase"
Related
i would like to create a filter in fail2ban for searching and blocking bad request like "Controller class * could not be found."
For this problem i was create a cakephp.conf file in the filter.d directory in fail2ban. The Content:
[Definition]
failregex = ^[0-9]{4}\-[0-9]{2}\-[0-9]{2}.*Error:.*\nStack Trace:\n(\-.*|\n)*\n.*\n.*\nClient IP: <HOST>\n$
ignoreregex =
My example error log looks like this:
...
2020-10-08 19:59:46 Error: [Cake\Http\Exception\MissingControllerException] Controller class Webfig could not be found. in /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Controller/ControllerFactory.php on line 158
Stack Trace:
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Controller/ControllerFactory.php:46
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/BaseApplication.php:249
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:77
- /home/myapplication/htdocs/vendor/cakephp/authentication/src/Middleware/AuthenticationMiddleware.php:122
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:77
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Middleware/CsrfProtectionMiddleware.php:146
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:58
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Routing/Middleware/RoutingMiddleware.php:172
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Routing/Middleware/AssetMiddleware.php:68
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Error/Middleware/ErrorHandlerMiddleware.php:121
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:58
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Server.php:90
- /home/myapplication/htdocs/webroot/index.php:40
Request URL: /webfig/
Referer URL: http://X.X.X.X/webfig/
Client IP: X.X.X.X
...
X.X.X.X are replaced
But i can't match any ip adresses. The fail2ban tester says:
root#test:~# fail2ban-regex /home/myapplication/htdocs/logs/error.log /etc/fail2ban/filter.d/cakephp.conf
Running tests
=============
Use failregex filter file : cakephp, basedir: /etc/fail2ban
Use log file : /home/myapplication/htdocs/logs/error.log
Use encoding : UTF-8
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [719] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day(?:T| ?)24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
`-
Lines: 15447 lines, 0 ignored, 0 matched, 15447 missed
[processed in 10.02 sec]
Missed line(s): too many to print. Use --print-all-missed to print all 15447 lines
i can't see any problems. Can you help me? :)
Thanks
The issue is your log is poor suitable to parse - it is a multiline log-file (IP takes place in other line as the failure message).
Let alone the line with IP does not has any ID (common information with line of failure), it can be still worse if several messages are crossing (so Client IP from other message that is not a failure, coming after failure message).
If you can change the log-format better do that (so date, IP and failure sign are in the same line), e.g. if you use nginx, organize a conditional logging for access log from php-location in error case like this.
See Fail2ban :: wiki :: Best practice for more info.
If you cannot do that (well better would be to change it), you can use multi-line buffering and parsing using maxlines parameter and <SKIPLINES> regex.
Your filter would be something like that:
[Definition]
# we ignore stack trace, so don't need to hold buffer window too large,
# 5 would be enough, but to be sure (if some log-messages crossing):
maxlines = 10
ignoreregex = ^(?:Stack |- /)
failregex = ^\s+Error: \[[^\]]+\] Controller class \S+ could not be found\..*<SKIPLINES>^((?:Request|Referer) URL:.*<SKIPLINES>)*^Client IP: <HOST>
To test it directly use:
fail2ban-regex --maxlines=5 /path/to/log '^\s+Error: \[[^\]]+\] Controller class \S+ could not be found\..*<SKIPLINES>^((?:Request|Referer) URL:.*<SKIPLINES>)*^Client IP: <HOST>' '^(?:Stack |- /)'
But as already said, it is really ugly - better you find the way to log everything in a single line.
We have following rules that are not working and we wanted to white list this warning ( in event viewer ), which contains "testinguri" in URI.
SecRule REQUEST_URI "#contains testinguri\?op\=message" "id:200006,phase:1,nolog,allow,ctl:ruleEngine=DetectionOnly,msg:'Test 1'"
SecRule REQUEST_URI "#beginsWith /en-us/testinguri?op=message" "id:200007,phase:1,nolog,allow,ctl:ruleEngine=DetectionOnly,msg:'Test 2'"
SecRule REQUEST_URI "^/en-us/testinguri?op=message.*" "id:200008,phase:1,nolog,allow,ctl:ruleEngine=DetectionOnly,msg:'Test 3'"
SecRule REQUEST_URI "#contains testinguri" "id:200009,phase:1,nolog,allow,ctl:ruleEngine=DetectionOnly,msg:'Test 4'"
Above rules are for same purpose but we put them if any version of the rule works but no luck.
Below is the warning in the event viewer and we want to allow the URI that have "testinguri" in it. It is running in Detection mode right now.
ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "C:\Program Files\ModSecurity IIS\owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "86"]
[id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=5,PHPI=0,HTTP=0,SESS=0): Remote Command Execution: Windows Command Injection; individual paranoia level scores: 5, 0, 0, 0"] [tag "event-correlation"] [hostname "computerName"] [uri "/en-us/testinguri?op=message&to=FULL URI..."] [unique_id "454534234234234"]
Can you please help on this. Thanks.
We were able to figure it out. So created a conf file test.conf and put the rules in that that we wanted to white list.
Then in the modsecurity_iis.conf file added this files reference at last.
This works for us. Hope this will help someone. Thanks.
I am trying to build a series of rules for ModSecurity to stop the bad bots.
SecRule REQUEST_HEADERS:User-Agent "#pmFromFile /etc/apache2/conf.d/badbots.txt" "id:'444444',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
But this method it's not well, because ModSecurity use with contains instead begin.
360Spider
80legs
AIBOT
Aboundex
....
I think best way use a rule for every group
^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye)
^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck)
^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly)
^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon)
^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky)
^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet)
^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot)
^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar)
^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb)
^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch)
^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua)
^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite)
^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot)
^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer)
^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk)
^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker)
^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot)
^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload)
^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot)
^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport)
^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher)
^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker)
^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse)
^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack)
^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$
Try
SecRule REQUEST_HEADERS:User-Agent "#beginswith (black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye)" "id:'444440',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck)" "id:'444441',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly)" "id:'444442',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon)" "id:'444443',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky)" "id:'444444',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet)" "id:'444444',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot)" "id:'444445',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar)" "id:'444446',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "#beginswith (joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb)" "id:'444447',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
But it's not work.
Let’s start from the bottom up. The operator #beginsWith does not allow regular expressions. For that you would need to do the default regular expression match and start with the beginning regex syntax (^):
SecRule REQUEST_HEADERS:User-Agent "^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye)" "id:'444440',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
SecRule REQUEST_HEADERS:User-Agent "^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck)" "id:'444441',phase:2,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
...etc.
But, rather than creating massive rules like this you should just use #pmFromFile. You are right that you can’t use regex in #pmFromFile but you can add boundary matching by use if variables, and an example is given in the reference manual, so for you it would look like this:
# Prepare custom UA variable
SecAction "phase:1,id:444444,nolog,pass,setvar:tx.UA=/% REQUEST_HEADERS:User-Agent"
# Check if UA is blacklisted
SecRule TX:UA "#pmFromFile etc/apache2/conf.d/badbots.txt" "id:'444445',phase:1,t:none,deny,status:406,log,msg:'BAD BOT - Detected and Blocked. '"
And then just edit your badbots.txt file to have every line starting with / like this:
/360Spider
/80legs
/AIBOT
/Aboundex
/...etc.
Ive got an issue with modsecurity and Im wondering if anyone can help. I'm running into an issue with uploading files to my application, anytime the file in question has a quote in the filename. Eventually I will do client side validation which will alert a user to a quote in the filename they are trying to upload and tell them to rename it, but for now I need to amend my modsecurity settings to ignore that particular check.
The modsecurity rule is:
SecRule MULTIPART_STRICT_ERROR "!#eq 0" \
"phase:2,t:none,log,deny,msg:'Multipart request body \
failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_SEMICOLON_MISSING}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
IH %{MULTIPART_FILE_LIMIT_EXCEEDED}'"
The error Im getting is:
[2016-10-11T16:08:06.8336+01:00] [OHS] [ERROR:32] [OHS-9999] [blah.c] [host_id: blah-web-kc1d] [host_addr: 1.2.3.4] [tid: 1724] [user: SYSTEM] [ecid: 00ibIu6vODDF4ETzA8m3SD0000_^001B9G] [rid: 0] [VirtualHost: main] [client 1.2.3.4] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_STRICT_ERROR" required. [file "E:/blah/security/<span class="skimlinks-unlinked">blah_base_rules.conf</span>"] [line "65"] [msg "Multipart request body failed strict validation: PE 0, BQ 0, BW 0, DB 0, DA 0, HF 0, LF 0, SM , IQ 1, IH 0, IH 0"] [hostname "<span class="skimlinks-unlinked">www.dev.uk</span>"] [uri "/pls/dev/blah_details_form.process_blah"] [unique_id "ZOMG!<span class="skimlinks-unlinked">ROFL.TL;DR</span>"]
IQ 1 suggests its the invalid quoting which makes sense. How do I tell modsecurity, to not block when it detects invalid quoting, without disabling the rest of the rule?
Thanks
P.S. I know allowing quotes in a filename potentially introduces SQL injection, but we aren't worried about that for reasons I can't go into.
Just replace the current rule (which checks the overall MULTIPART_STRICT_ERROR variable) with separate rules for each individual variable instead, changing the deny to a warn for the one variable you don't want to deny:
SecRule REQBODY_PROCESSOR_ERROR "!#eq 0" \
"phase:2,t:none,log,deny,msg:'Multipart request body \ failed strict
validation: \ PE %{REQBODY_PROCESSOR_ERROR}'"
SecRule MULTIPART_BOUNDARY_QUOTED "!#eq 0" \
"phase:2,t:none,log,deny,msg:'Multipart request body \ failed strict
validation: \ BQ %{MULTIPART_BOUNDARY_QUOTED}'"
...etc.
SecRule MULTIPART_INVALID_QUOTING "!#eq 0" \
"phase:2,t:none,log,warn,msg:'Multipart request body \ failed strict
validation: \ IQ %{MULTIPART_INVALID_QUOTING}'"
SecRule MULTIPART_INVALID_HEADER_FOLDING "!#eq 0" \
"phase:2,t:none,log,deny,msg:'Multipart request body \ failed strict
validation: \ IH %{MULTIPART_INVALID_HEADER_FOLDING}'"
...etc.
Note newer versions of ModSecurity (since 2.7) require a unique id so if your rule has an id which you've not shown in your question then make sure you make it unique when creating the many rules.
Finally it is also possible to check all the variables in one rule or have the rules sum up the values (or have them as part of one large chained rule where values are similarly summed up) and then check sum = 0 but separate rules is probably just simpler and easier to follow in future.
I fixed this by renaming the file name to upload as it contains some unrecognized pattern.
How do I resolve the issue?
Simply put, rename the file to remove the offending special character from the file name
or (disable this security rule in /etc/{path}/mod_security.conf by commenting the line " SecRule MULTIPART_STRICT_ERROR "!#eq 0" \" or by .htaccess file - NOT RECOMMENDED AT ALL)
How is this error caused?
This error is caused by mod_security blocking a potentially malicious upload. While it may be completely harmless, mod_security has no way of knowing if it is harmless or not.
Typically, the content in question is a file being uploaded which contains a special character such as a single or double quote within the file name which is often used by attackers to inject malicious scripts into websites.
Many of webmasters use pingdom.com as a monitoring ping service.
But the problem is that /httpd/access_log is full of
208.64.28.194 - - [06/Aug/2015:12:20:22 -0500] "GET / HTTP/1.1" 200 2917 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
I set
CustomLog "logs/access_log" combined env=!dontlog
and tried to get rid of it using variations like
SetEnvIf Remote_Host "^pingdom\.com$" dontlog
SetEnvIFNoCase Remote_Host "pingdom.com$" dontlog
SetEnvIfNoCase Referer "www\.pingdom\.com" dontlog
SetEnvIFNoCase Host "^pingdom.com$" dontlog
but still no a success with any of them - so thanks for any else hint to try.
I'll put my comment here as answer so anyone will find this easier.
Since one can see from the log file, the host name is not Pingdom.com but a part of the user agent string.
Solutions to try:
First be sure you have enabled the setenvif-module. Write the command
sudo apache2ctl -M | grep setenv
It should return something like "setenvif_module (shared)"
Then you can try setting by remote address
SetEnvIf Remote_Addr "208\.64\.28\.194$" dontlog
The final working solution is this, dont log if the user agent string contains Pingdom string:
SetEnvIfNoCase User-Agent "^Pingdom" dontlog
Edit: enhanced some parts of the answer.