I am trying to match and ban certain patterns in my drupal logs (drupal 9).
I have taken the base drupal-auth regex, created a new conf and tried to amend it to my requirements but I seem to be failing at the first hurdle. This is the code that will give me anything that has the type 'user' and this is filtered by the user\ in the code below, just before the <HOST> block:
failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|<HOST>\|.+\|.+\|\d\|.*\|.+\.$
If I want to search exactly the same pattern, but with say 'page not found' or 'access denied' instead of 'user' what do I need? I cannot seem to get it to match the moment the type has a space in it. It seems such a simple thing to do!
I am using fail2ban-regex --print-all-matched to test.
Related
I'm using Google Data Studio to create a report analyzing specific referral sites. My data source is my site Google Analytics.
I want to replace the Full Referrer (e.g. of the format webaddress.com/page-name-one) with a text only value (i.e Page name one), so that it's clearer to see in the report which page is which in my charts and tables.
I've used the below formulae in the calculated fields, but none of them seem to change Full Referrer to match what I need it to. Data studio recognizes them all as valid formulae too.
I've anonymised my examples, but it has the same principles. I've tried:
REGEXP_REPLACE(Full Referrer,"[webaddress\\.com\\/page\\-name\\-one].*","Page name one")
REGEXP_REPLACE(Full Referrer, 'webaddress.com/page-name-one', 'Page name one')
REGEXP_REPLACE(Full Referrer, 'webaddress\\.com\\/page\\-name\\-one', 'Page name one')
REGEXP_REPLACE(Full Referrer, 'name', 'Page name one')
REGEXP_REPLACE(Full Referrer, 'page-name-one', 'Page name one')
REGEXP_REPLACE(Full Referrer, 'page\\-name\\-one', 'Page name one')
In testing this on one of my own GA data sources, I was able to achieve this using one of your patterns:
REGEXP_REPLACE(Full Referrer,'webaddress.com/page-name-one','Page name one')
It should be noted, however, that the . should be properly escaped (either by \ or wrapping it in a character class like [.]; see re2 syntax for details). Because you have to double-backslash, I also prefer to use something Data Studio borrowed from BigQuery (sort of an undocumented feature), which is the regular expression string type (r"" or r''). When using this, you only have to single-backslash (unless you want a literal backslash):
REGEXP_REPLACE(Full Referrer,r'webaddress\.com/page-name-one','Page name one')
Because you're using REGEXP_REPLACE, anything before or after your match string will still exist after the replacement—meaning that for a Full Referrer of "m.facebook.com/l", REGEXP_REPLACE(Full Referrer,r'facebook\.com','FB') would return "m.FB/l"
So your pattern above will match the value anywhere in the string, which likely isn't what you want. To anchor it to the beginning, use the ^ (start of string) assertion:
REGEXP_REPLACE(Full Referrer,r'^webaddress\.com/page-name-one','Page name one')
If you want to only match that exact value of Full Referrer (i.e. not including any additional path levels), make sure to use the $ (end of string) assertion as well:
REGEXP_REPLACE(Full Referrer,r'^webaddress\.com/page-name-one$','Page name one')
Keep in mind that if you're doing this in the data source as a calculated field, you aren't actually changing the original metric—you're working on a copy of it. So you need to replace Full Referrer with whatever you named your calculated field in the data source.
Often you're wanting to do this for a bunch of sites or pages, so you can use CASE and REGEXP_MATCH to handle all this logic in a single field:
CASE
WHEN REGEXP_MATCH(Full Referrer,r'^webaddress\.com/page-name-one$') THEN 'Page name one'
WHEN REGEXP_MATCH(Full Referrer,r'^site2\.com/example$') THEN 'S2 Example'
ELSE Full Referrer
END
These matches are done in order, so you can even match a specific page or pages, and then still provide a different value for anything on that domain that you didn't match:
CASE
WHEN REGEXP_MATCH(Full Referrer,r'^site\.com/$') THEN 'Site - Home'
WHEN REGEXP_MATCH(Full Referrer,r'^site\.com/about$') THEN 'Site - About'
WHEN REGEXP_MATCH(Full Referrer,r'^site\.com/') THEN 'Site - (other)'
ELSE Full Referrer
END
You can also use the ELSE if you want to bucket all of the unmatched values into an "other" grouping instead of just leaving the original value.
Another thing to remember is that due to shared fields in GA, things like Source (utm_source) also show up in Full Referrer, so you could be seeing values there that you wouldn't normally expect. Often you can get rid of these by also filtering to only the Default Channel Grouping of "Referral".
If your patterns still aren't matching, please update the question with some additional details such as what the output actually is, whether there's an error message, etc.—and also whether you're doing this as a calculated field in the data source or the "Create Field" button on a single chart.
I just tried the following in my AJAX update:
[Server]/secTypes/Update
This maps to the following url in URLS.py:
url(r'^secTypes/Update/', equity.views.updateSecTypes, name='updateSecTypes'),
This doesn't resolve to the following function in my view.
But when I change the URL expression to:
url(r'^su/', equity.views.updateSecTypes, name='updateSecTypes')
It works fine.
What in the URL resolver is not getting accurately mapped? Is it the forward slash?
I think it has to do with something related to the regex so if someone understands this better can help me that would be appreciated.
From the url patterns in your comments, it looks like you had another matching pattern before the one in your question.
There are two simple solution for this.
Move that first pattern down. Change this:
url(r'^secTypes/', equity.views.getSecTypes, name='getSecTypes'),
url(r'^secTypesAll/', equity.views.getSecTypesAll, name='getSecTypesAll'),
url(r'^secTypes/Update/', equity.views.updateSecTypes, name='updateSecTypes'),
url(r'^secTypes/Delete/', equity.views.deleteSecTypes, name='deleteSecTypes'),
url(r'^secTypes/Create/', equity.views.createSecTypes, name='createSecTypes'),
to this:
url(r'^secTypesAll/', equity.views.getSecTypesAll, name='getSecTypesAll'),
url(r'^secTypes/Update/', equity.views.updateSecTypes, name='updateSecTypes'),
url(r'^secTypes/Delete/', equity.views.deleteSecTypes, name='deleteSecTypes'),
url(r'^secTypes/Create/', equity.views.createSecTypes, name='createSecTypes'),
url(r'^secTypes/', equity.views.getSecTypes, name='getSecTypes'),
The order matters when resolving URL patterns and if an earlier one matches, the following ones are not processed.
Both r'^secTypes/' and r'^secTypes/Update/' matches the string 'secTypes/Update/' so you need to be careful to put the more specific one first and the more general one afterwards.
Update the regex to match the end of the URL string by adding a $ like this:
url(r'^secTypes/$', equity.views.getSecTypes, name='getSecTypes'),
url(r'^secTypesAll/$', equity.views.getSecTypesAll, name='getSecTypesAll'),
url(r'^secTypes/Update/$', equity.views.updateSecTypes, name='updateSecTypes'),
url(r'^secTypes/Delete/$', equity.views.deleteSecTypes, name='deleteSecTypes'),
url(r'^secTypes/Create/$', equity.views.createSecTypes, name='createSecTypes'),
This is the preferred solution since it would stop Django from matching a URL like secTypes/Update/foobar
However, if you have logic in the view that specifically uses the substring after the end of the URL pattern (i.e. foobar based on the above example), this wouldn't work.
I'm using Logstash to match Fortinet analyzer logs, and the problem is there are so many pattern without order of the fields.
e.g. one type of message would be:
service=DNS hostname="a.b.net" profile="Dns" action=blocked reqtype=direct url="/" sentbyte=0 rcvdbyte=0 direction=N/A msg="URL belongs to a denied category in policy" method=domain cat=61 catdesc="Phishing" crscore=60 crlevel=high
...and another is:
msg="File is infected." action=blocked service=HTTP sessionid=33137 direction=incoming filename="favicon.ico" quarskip=No-skip virus="MSWord/Agent.DD60!tr" dtype="Virus" ref="http://www.fortinet.com/ve?vn=MSWord%2FAgent.DD60%21tr" virusid=6920465 profile="AV"
As you can see both have msg, action, service and profile but with different order.
Is there anyway to build a pattern to match something like:
(.*?)=%{DATA:\1?}\s
...while giving the field the name of the first match?
Use the kv{} filter which can split it all apart and doesn't care about the order.
I'm already using Fiddler to intercept requests for specific remote files while I'm working on them (so I can tweak them locally without touching the published contents).
i.e. I use many rules like this
match: regex:(?insx).+/some_file([?a-z0-9-=&]+\.)*
respond: c:\somepath\some_file
This works perfectly.
What I'd like to do now is taking this a step further, with something like this
match: regex:http://some_dummy_domain/(anything)?(anything)
respond: c:\somepath\(anything)?(anything)
or, in plain text,
Intercept any http request to 'some_dummy_domain', go inside 'c:\somepath' and grab the file with the same path and name that was requested originally. Query string should pass through.
Some scenarios to further clarify:
http://some_domain/somefile --> c:\somepath\somefile
http://some_domain/path1/somefile --> c:\somepath\path1\somefile
http://some_domain/path1/somefile?querystring --> c:\somepath\path1\somefile?querystring
I tried to leverage what I already had:
match: regex:(?insx).+//some_dummy_domain/([?a-z0-9-=&]+\.)*
respond: ...
Basically, I'm looking for //some_dummy_domain/ in requests. This seems to match correctly when testing, but I'm missing how to respond.
Can Fiddler use matches in responses, and how could I set this up properly ?
I tried to respond c:\somepath\$1 but Fiddler seems to treat it verbatim:
match: regex:(?insx).+//some_domain/([?a-z0-9-=&]+\.)*
respond: c:\somepath\$1
request: http://some_domain/index.html
response: c:\somepath\$1html <-----------
The problem is your use of insx at the front of your expression; the n means that you want to require explicitly-named capture groups, meaning that a group $1 isn't automatically created. You can either omit the n or explicitly name the capture group.
From the Fiddler Book:
Use RegEx Replacements in Action Text
Fiddler’s AutoResponder permits you to use regular expression group replacements to map text from the Match Condition into the Action Text. For instance, the rule:
Match Text: REGEX:.+/assets/(.*)
Action Text: http://example.com/mockup/$1
...maps a request for http://example.com/assets/Test1.gif to http://example.com/mockup/Test1.gif.
The following rule:
Match Text: REGEX:.+example\.com.*
Action Text: http://proxy.webdbg.com/p.cgi?url=$0
...rewrites the inbound URL so that all URLs containing example.com are passed as a URL parameter to a page on proxy.webdbg.com.
Match Text: REGEX:(?insx).+/assets/(?'fname'[^?]*).*
Action Text C:\src\${fname}
...maps a request for http://example.com/assets/img/1.png?bunnies to C:\src\img\1.png.
I'm trying to figure out how to use Django Selenium Webdriver to test that the href URL's on a particular page are correctly formed.
e.g. http://www.domain.com/post/12/test_post
So I want to test a regex like:
'^post/(?P<id>\d+)/test_post)$'
While I know the name of the post(slug), I don't know what the ID will be.
How can I use Selenium to test that the href URL's are correctly formed?
I tried:
self._driver.find_element_by_xpath("//a[matches(#href='/post/(/?P<id>\d+)/test_post')]")
But I get this error:
*** InvalidSelectiorException: Message: u'The given selector //a[matches(#href=\'/post/(/?P<id>\\d+)/test_post\')] is either invalid or does not result in a WebElement. The following error occurred:\n[InvalidSelectorError] Unable to locate an element with the xpath expression //a[matches(#href=\'/post/(/?P<id>\\d+)/test_post\')] because of the following error:\n[Exception... "The expression is not a legal expression." code: "51" nsresult: "0x805b0033 (NS_ERROR_DOM_INVALID_EXPRESSION_ERR)" location: "file:///var/folders/m
But I kept getting, ""The expression is not a legal expression."
self._driver.find_element_by_xpath("//a[starts-with(#href, '/post']")
self._driver.find_element_by_xpath("//a[starts-with(#href, 'post/']")
self._driver.find_element_by_xpath("//a[starts-with(#href, '/post/']")
Here I also got ""The expression is not a legal expression."
Is there a way I can verify that the href tags are properly formed?
Hy,
It seems that you are not closing the parentheses of the method matches
Try:
self._driver.find_element_by_xpath("//a[matches(#href, '/post/(?P<id>\d+)/test_post')]")