I'm creating my custom mod-security rules and I have a question about if rule ID affects the order by which rules are executed.
My current setup is:
I'm using mod-security version: 2.6.3-1ubuntu0.2
my mod-security config is based on http://mod-security.svn.sourceforge.net/viewvc/mod-security/m2/trunk/modsecurity.conf-recommended (it has 5 SecRule directives - ids: 200000 - 200005)
my rules are in a separated file /etc/apache2/conf.d/modsecurity-activated-rules.conf and they are loaded after the modsecurity config /etc/apache2/conf.d/modsecurity.conf
My rule is:
<IfModule mod_security2.c>
# block all GET requests - learning purposes only
SecRule REQUEST_METHOD "^(GET)$" \
"phase:1,t:none,auditlog,block,id:1001,rev:2,tag:HARDENING"
</IfModule>
Disclamer: it's a simple rule and probably not optimal since I'm still learning. Suggestions are welcome
So, my question came to me after I analyzed the debug file (level 9 activated):
[...]
[4] Recipe: Invoking rule 7f157a85da30; [file "/etc/apache2/conf.d/modsecurity-activated-rules.conf"] [line "11"] [id "1001"] [rev "2"].
[5] Rule 7f157a85da30: SecRule "REQUEST_METHOD" "#rx ^(GET)$" "phase:1,log,t:none,auditlog,block,id:1001,rev:2,tag:HARDENING"
[4] Transformation completed in 4 usec.
[4] Executing operator "rx" with param "^(GET)$" against REQUEST_METHOD.
[9] Target value: "GET"
[6] Ignoring regex captures since "capture" action is not enabled.
[4] Operator completed in 36 usec.
[2] Warning. Pattern match "^(GET)$" at REQUEST_METHOD. [file "/etc/apache2/conf.d/modsecurity-activated-rules.conf"] [line "11"] [id "1001"] [rev "2"] [tag "HARDENING"]
[4] Rule returned 1.[04/Sep/2012:09:30:27 +0000] [107.21.159.51/sid#7f157a854510][rid#7f1573fcf0a0][/poll/13456492248275482/vote/yes][9] Match -> mode NEXT_RULE.
[4] Recipe: Invoking rule 7f157a85e648; [file "/etc/apache2/conf.d/modsecurity.conf"] [line "24"] [id "200000"].
[5] Rule 7f157a85e648: SecRule "REQUEST_HEADERS:Content-Type" "#rx text/xml" "phase:1,auditlog,id:200000,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
[4] Rule returned 0.
[9] No match, not chained -> mode NEXT_RULE.
[...]
As the debug lines shows, my rule (id:1001) was executed first before rule (id:200000), even that my rule is read after the 20000x ones.
I though that IDs don't really matter in the order of rule execution since the OWASP ModSecurity Core Rule Set https://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project has rule ID between 950000 and 990000 and there is no info about rule ID ranges for specific attacks: SQLi, XSS, etc.
With the above, my questions are:
Are the rules execution determined by ID number (smallest number get executed first)?
Is there any documentation defining which rule ID ranges should be used for custom rules? I'm looking for something like:
Mod-security main rules: 200.000 - 200.xxx
OWASP ModSecurity Core Rule Set: 950.000 - 999.xxx
Custom rules: 1.000.000 - 9.xxx.xxx
Thanks for your time.
The RIGHT answer is that ModSecurity executes the rules in the order they are defined in your Apache config.
The only ordering ModSecurity applies by itself is the phase: ordering, which assures that rules will be execute by phases.
Example:
First execute all rules from phase:1 (in the order they are written
inside the config)
Next execute all rules from phase:2 (in the order they are written
inside the config)
etc...
A rule can has not id.But if has one,at the same pahse,just like your question
smallest number get executed first
Only at the same phase!
Maybe the bussiness rule define throse ids.https://ssl.trustwave.com/web-application-firewall
ModSecurity supports two types of Rule models that are positive security model and negative security model. Negative security model support signature based detection and ordering of rules matters when you want to skip rules using skip, skipafter keyword to avoid resource intensive regex based pattern patching.Secondly order of rule based on rule id is not absolute it can be changed by the rule engine dynamically i.e. rule with phase 1 will always be executed first. I recommend a read of ModSecurity handbook a good start for beginner.Second types of Rule is positive security model in which order does not matter as counters are maintained to detect anomaly and rule are triggered if threshold are exceeded.
Related
An image on our site is flagging a modsec rule I am trying to add a rule exception for only that occurrence. The number at the start of the flagged string is a session number, so I have added a regex to my rule.
I've tried various permutations but had no joy and would appreciate some advice.
Blocked URI:
https://www.website.com/application/login?0--preLoginHeaderPanel-companyLogo
Modsec log snippet:
[file "/usr/share/modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "65"] [id "942100"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: 1c found within ARGS_NAME:0--preLoginHeaderPanel-companyLogo: 0--preLoginHeaderPanel-companyLogo"]
Attempted exceptions (within apache.conf):
SecRuleUpdateTargetById 942100 !ARGS_NAMES:'[0-9][0-9]?--preLoginHeaderPanel-companyLogo'
Core Rule Set Dev on Duty here. Rule 942100 is one of our 'LibInjection' rules. LibInjection is quite opaque (it's a third party library/operator), so you're correct that a rule exclusion is the way to fix this issue.
The use of regular expressions in this context follows a specific form. They need to be sandwiched inside forward slashes, like so:
SecRuleUpdateTargetById 942100 "!ARGS_NAMES:/^[0-9][0-9]?--preLoginHeaderPanel-companyLogo/"
I added in a starting anchor at the beginning of the regular expression. You might want to think whether anchoring at the end is a good idea, as well.
For more examples and information, we have some great documentation on this here: https://coreruleset.org/docs/configuring/false_positives_tuning/#support-for-regular-expressions
Trying to set up a "catch-all" rule in Gmail, the rule is catching all incoming messages to the domain however I want to exclude mail sent to specific addresses in the catch-all rule.
Have followed this:
https://robbettis.blog/setting-a-catchall-email-for-g-suite-in-2018/
but with the variation of changing "1. Specify envelope recipients to match" from "All Recipients" to "Pattern Match"
and then used:
Match everything except for specified strings
to try something like:
^(?!(red|green|blue)$).+$
in the pattern to match but Google apparently uses a different standard of RegExp and says my regexp has invalid syntax.
I don't have a strong RegExp background, any advice is appreciated.
Can someone please help finding an expression google's system will accept to achieve this or an alternative to achieve the same outcome?
The flavour of regex used here is RE2 – https://github.com/google/re2/wiki/Syntax
which doesn't support (before) text not matching such as (?!example).
In terms of a solution, a rule with a lower numbered order will be enacted first. So perhaps you could capture the ^(red|green|blue).+$ items using that to do x, then with the next rule do the desired action y.
You can achieve this by excluding certain emails using an Address List. To access this feature, your catch-all rule needs to be defined in the parent organisational unit's Routing. Default Routing does not provide the "Address lists" option.
I recently had this explained to me step by step by a Google Support staff member so thought I'd share the process for others.
Navigate to "Routing", it's the one at the bottom:
|
Choose your parent organisational unit and click "ADD ANOTHER RULE". You may not have any child organisational units, in which case it will be the only one that appears here:
|
Under "Email messages to affect", choose "Inbound" and "Internal – Receiving":
|
Under "For the types of messages above, do the following" enter the preferred action to take. For example, "Modify message" > "Change envelope recipient":
|
Click "Show options" to reveal options for "Address lists" and "Account types to affect"
Choose "Apply address lists to recipients" ("correspondents" means senders)
Choose "Bypass this setting for specific addresses/domains"
Add or create a list of emails that you want to exclude from this catch-all rule. This is the magical part :). In the example below, I've created a list called "Exclude from catch-all rules". You can use this list across multiple catch-all rules for different domains.
Under "Account types to affect", untick "Users", tick "
Unrecognised/catch-all"
|
Under "Envelope filter", choose "Only affect specific envelope recipients" and select "Pattern match" to enter your catch-all regular expression:
I'm trying to re-write a URL such as
http://ourdomain.com/hotels/vegas?cf=0
to
http://ourdomain.com?d=vegas&cf=0
using haProxy.
We used to do it with Apache using
RewriteRule ^hotels/([^/]+)/?\??(.*)$ ?d=$1&$2 [QSA]
I've tried
reqrep ^([^\ :]*)\ /hotels/(.*) \1\ /?d=\2
But that gives me http://ourdomain.com?d=vegas?cf=0
And
reqrep ^([^\ :]*)\ /hotels/([^/]+)/?\??(.*) \1\ /?d=\2&\3
Just gives me a 400 error.
It would be nice to do it with acl's but I can't see how that would work.
reqrep ^([^\ :]*)\ /hotels/([^/]+)/?\??(.*) \1\ /?d=\2&\3
Just gives me a 400 error.
([^/]+) is too greedy when everything following it /?\??(.*) is optional. It's mangling the last part of the request, leading to the 400.
Remember what sort of data you're working with:
GET /path?query HTTP/1.(0|1)
Replace ([^/]+) with ([^/\ ]+) so that anything after and including the space will be captured by \3, not \2.
Update: it seems that the above is not quite perfect, since the alignment of the ? still doesn't work out. This -- and the original 400 error -- highlight some of the pitfalls with req[i]rep -- it's very low level request munging.
HAProxy 1.6 introduced several new capabilities that make request tweaking much cleaner, and this is actually a good case to illustrate several of them together. Note that these examples also use anonymous ACLs, wrapped in { }. The documentation seems to discourage these a little bit -- but this is only because they're unwieldy to maintain when you need to test the same set of conditions for multiple reasons (named ACLs can of course be more easily reused), but they're perfect for a case like this. Note that the braces must be surrounded by at least 1 whitespace character due to configuration parser limitations.
Variables, scoped to request (go out of scope as soon as a back-end is selected), response (go into scope only after the back-end responds), transaction (persistent from request to response, these can be used before the trip to the back-end and are still in scope when the response comes back), or session (in scope across multiple requests by this browser during this connection, if the browser reuses the connection), can be used to stash values.
The regsub() converter takes the preceding value as its input and returns that value passed through a simple regex replacement.
If the path starts with /hotels/, capture the path, scrub out ^/hotels/ (replacing it with the empty string that appears after the next comma), and stash it in a request variable called req.hotel.
http-request set-var(req.hotel) path,regsub(^/hotels/,) if { path_beg /hotels/ }
Processing of most http-request steps is done in configuration file order, so, at the next instruction, if (and only if) that variable has a value, we use http-request set-path with an argument of / in order to empty the path. Testing the variable is needed so that we don't do this with every request -- only the ones for /hotels/. It might be that you actually need something more like if { path_reg /hotels/.+ } since /hotels/ by itself might be a valid path we should leave alone.
http-request set-path / if { var(req.hotel) -m found }
Then, we use http-request set-query to set the query string to a value created by concatenating the value of the req.hotel variable with & and the original query string, which we obtain with using the query fetch.
http-request set-query d=%[var(req.hotel)]&%[query] if { var(req.hotel) -m found }
Note that the query fetch and http-request set-query both have some magical behavior -- they take care of the ? for you. The query fetch does not return it, and http-request set-query does not expect you to provide it. This is helpful because we may need to be able to handle requests correctly whether or not the ? is present in the original request, without having to manage it ourselves.
With the above configuration, GET /hotels/vegas?&cf=0 HTTP/1.1 becomes GET /?d=vegas&cf=0 HTTP/1.1.
If the initial query string is completely empty, GET /hotels/vegas HTTP/1.1 becomes GET /?d=vegas& HTTP/1.1. That looks a little strange, but it should be completely valid. A slightly more convoluted configuration to test for the presence of an intial query string could prevent that, but I don't see it being an issue.
So, we've turned 1 line of configuration into 3, but I would argue that those three lines are much more intuitive about what they are accomplishing and it's certainly a less delicate operation than massaging the entire start line of the request with a regex. Here they are, together, with some optional whitespace:
http-request set-var(req.hotel) path,regsub(^/hotels/,) if { path_beg /hotels/ }
http-request set-path / if { var(req.hotel) -m found }
http-request set-query d=%[var(req.hotel)]&%[query] if { var(req.hotel) -m found }
This is a working solution using reqrep
acl is_destination path_beg /hotels/
reqrep ^([^\ :]*)\ /hotels/([^/\ \?]+)/?\??([^\ ]*)(.*)$ \1\ /?d=\2&\3\4 if is_destination
I'm hoping that the acl will remove the need to run regex on everything (hence lightening the load a bit), but I'm not sure that's the case.
What does ARGS , ARGS_NAMES actually mean in mod_security core rule sets?
I have already referred to the Modsecurity2 Apache Reference but I didnt get any clear idea.
Can someone give me a specific idea , preferably with an explanation, what it actually is and how something actually triggers a rule positive like the one below.
The rule below is triggered positive for cases like " x and 6" etc, in general any "and" followed by a digit. I understand what the request filename is, in this case. what i dont understand is what are ARGS and ARGS_NAMES. I need a specific example with reference to the rule below.
SecRule REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "(?i)\b(?i:and)\b\s+(\d{1,10}|'[^=]{1,10}')\s*[=]|\b(?i:and)\b\s+(\d{1,10}|'[^=]{1,10}')\s*[<>]|\band\b ?(?:\d{1,10}|[\'\"][^=]{1,10}[\'\"]) ?[=<>]+|\b(?i:and)\b\s+(\d{1,10}|'[^=]{1,10}')" \
"phase:2,rev:'2.2.5',capture,t:none,t:urlDecodeUni,ctl:auditLogParts=+E,block,msg:'SQL Injection Attack',id:'959072',tag:'WEB_ATTACK/SQL_INJECTION',tag:'WASCTC/WASC-19',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE1',tag:'PCI/6.5.2',logdata:'%{TX.0}',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.sql_injection_score=+%{tx.critical_anomaly_score},setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}"
Example:
http://server.invalid/test.php?pretty_arg=test123&ugly_arg=345test
ARGS_NAMES = "pretty_arg","ugly_arg"
ARGS = "pretty_arg:test123","ugly_arg:345test"
See here:
Reference-Manual Variables
Reference-Manual args
If you want to remove the argument from a specific call, you could use
SecRule REQUEST_FILENAME "#streq /path/to/file.php" "phase:1,id:2001,t:none,nolog,pass,ctl:ruleRemoveTargetById=959072;ARGS:ugly_arg"
I need to match input strings (URLs) against a large set (anywhere from 1k-250k) of string rules with simple wildcard support.
Requirements for wildcard support are as follows:
Wildcard (*) can only substitute a "part" of a URL. That is fragments of a domain, path, and parameters. For example, "*.part.part/*/part?part=part&part=*". The only exception to this rule is in the path area where "/*" should match anything after the slash.
Examples:
*.site.com/* -- should match sub.site.com/home.html, sub2.site.com/path/home.html
sub.site.*/path/* -- should match sub.site.com/path/home.html, sub.site.net/path/home.html, but not sub.site.com/home.html
Additional requirements:
Fast lookup (I realize "fast" is a relative term. Given the max 250k rules, still fall within < 1.5s if possible.)
Work within the scope of a modern desktop (e.g. not a server implementation)
Ability to return 0:n matches given a input string
Matches will have rule data attached to them
What is the best system/algorithm for such as task? I will be developing the solution in C++ with the rules themselves stored in a SQLite database.
First of all, one of the worst performing searches you can do is with a wildcard at both ends of the string ".domain.com/path" -- and I think you're going to hit this case a lot. So my first recommendation is to reverse the order of the domains as they're stored in your DB: com.domain.example/path1/path2/page.html. That will allow you to keep things much more tidy and only use wildcards in "one direction" on the string, which will provide MUCH faster lookups.
I think John mentions some good points about how to do this all within your DB. If that doesn't work I would use a regex library in C++ against the list. I bet you'll get the best performance and most general regex syntax that way.
If I'm not mistaken, you can take string rule and break it up into domain, path, and query pieces, just like it's a URL. Then you can apply a standard wildcard matching algorithm with each of those pieces against the corresponding pieces from the URLs you want to test against. If all of the pieces match, the rule is a match.
Example
Rule: *.site.com/*
domain => *.site.com
path => /*
query => [empty]
URL: sub.site.com/path/home.html
domain => sub.site.com
path => /path/home.html
query => [empty]
Matching process:
domain => *.site.com matches sub.site.com? YES
path => /* matches /path/home.html? YES
query => [empty] matches [empty] YES
Result: MATCH
As you are storing the rules in a database I would store them already broken into those three pieces. And if you want uber-speed you could convert the *'s to %'s and then use the database's native LIKE operation to do the matching for you. Then you'd just have a query like
SELECT *
FROM ruleTable
WHERE #urlDomain LIKE ruleDomain
AND #urlPath LIKE rulePath
AND #urlQuery LIKE ruleQuery
where #urlDomain, #urlPath, and #urlQuery are variables in a prepared statement. The query would return the rules that match a URL, or an empty result set if nothing matches.