I'm trying to parse some logs using split and regexes in powershell
Here's my code :
$string = "Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code. Test 229: Passed Test 260: Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1"
$string -Split '(Test (\d)+:)'
Result :
Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Test 229:
9
Passed
Test 260:
0
Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1
Expected result:
Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Test 229:
Passed
Test 260:
Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1
On this site : https://regexr.com/3c0lf I tried this regex and the groups captured were : Test 260: and Test 229: (which is exactly what I want)
I do not understand where the 0 and the 9 comes from.
Thanks a lot
Those are the last digits of the number. 0 from 26*0* and 9 from 22*9*.
You are seeing those because you've created an additional capturing group by putting parentheses around the digits. Just remove them like so:
$string -Split '(Test \d+:)
You probably don't even need those parentheses either, leaving just
$string -Split 'Test \d+:
Related
i would like to create a filter in fail2ban for searching and blocking bad request like "Controller class * could not be found."
For this problem i was create a cakephp.conf file in the filter.d directory in fail2ban. The Content:
[Definition]
failregex = ^[0-9]{4}\-[0-9]{2}\-[0-9]{2}.*Error:.*\nStack Trace:\n(\-.*|\n)*\n.*\n.*\nClient IP: <HOST>\n$
ignoreregex =
My example error log looks like this:
...
2020-10-08 19:59:46 Error: [Cake\Http\Exception\MissingControllerException] Controller class Webfig could not be found. in /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Controller/ControllerFactory.php on line 158
Stack Trace:
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Controller/ControllerFactory.php:46
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/BaseApplication.php:249
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:77
- /home/myapplication/htdocs/vendor/cakephp/authentication/src/Middleware/AuthenticationMiddleware.php:122
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:77
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Middleware/CsrfProtectionMiddleware.php:146
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:58
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Routing/Middleware/RoutingMiddleware.php:172
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Routing/Middleware/AssetMiddleware.php:68
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Error/Middleware/ErrorHandlerMiddleware.php:121
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:58
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Server.php:90
- /home/myapplication/htdocs/webroot/index.php:40
Request URL: /webfig/
Referer URL: http://X.X.X.X/webfig/
Client IP: X.X.X.X
...
X.X.X.X are replaced
But i can't match any ip adresses. The fail2ban tester says:
root#test:~# fail2ban-regex /home/myapplication/htdocs/logs/error.log /etc/fail2ban/filter.d/cakephp.conf
Running tests
=============
Use failregex filter file : cakephp, basedir: /etc/fail2ban
Use log file : /home/myapplication/htdocs/logs/error.log
Use encoding : UTF-8
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [719] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day(?:T| ?)24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
`-
Lines: 15447 lines, 0 ignored, 0 matched, 15447 missed
[processed in 10.02 sec]
Missed line(s): too many to print. Use --print-all-missed to print all 15447 lines
i can't see any problems. Can you help me? :)
Thanks
The issue is your log is poor suitable to parse - it is a multiline log-file (IP takes place in other line as the failure message).
Let alone the line with IP does not has any ID (common information with line of failure), it can be still worse if several messages are crossing (so Client IP from other message that is not a failure, coming after failure message).
If you can change the log-format better do that (so date, IP and failure sign are in the same line), e.g. if you use nginx, organize a conditional logging for access log from php-location in error case like this.
See Fail2ban :: wiki :: Best practice for more info.
If you cannot do that (well better would be to change it), you can use multi-line buffering and parsing using maxlines parameter and <SKIPLINES> regex.
Your filter would be something like that:
[Definition]
# we ignore stack trace, so don't need to hold buffer window too large,
# 5 would be enough, but to be sure (if some log-messages crossing):
maxlines = 10
ignoreregex = ^(?:Stack |- /)
failregex = ^\s+Error: \[[^\]]+\] Controller class \S+ could not be found\..*<SKIPLINES>^((?:Request|Referer) URL:.*<SKIPLINES>)*^Client IP: <HOST>
To test it directly use:
fail2ban-regex --maxlines=5 /path/to/log '^\s+Error: \[[^\]]+\] Controller class \S+ could not be found\..*<SKIPLINES>^((?:Request|Referer) URL:.*<SKIPLINES>)*^Client IP: <HOST>' '^(?:Stack |- /)'
But as already said, it is really ugly - better you find the way to log everything in a single line.
In package.json I have 2 script commands:
"test:unit": "jest --watch --testNamePattern='^(?!\\[functional\\]).+$'",
"test:functional": "jest --watch --testNamePattern='\\[functional\\]'",
copying ^(?!\\[functional\\]).+$ into https://regex101.com/, it does not match the test string below inside argument 1 of describe()
describe("[functional] live tests", () => {
When changed to ([functional]).+$, the pattern does match. I have to remove a pair of \ on each end to remove escapes for .json files (I think).
Here is what I see when running npm run test:unit in my project root:
// the functional test runs (not desired)
$ npm run test:unit
functions/src/classes/__tests__/Functional.test.ts:30:47 - error TS2339: Property 'submit' does not exist on type 'Element'.
30 await emailForm.evaluate(form => form.submit());
~~~~~~
RUNS ...s/__tests__/Functional.test.ts
Test Suites: 1 failed, 1 skipped, 3 passed, 4 of 5 total
Tests: 2 skipped, 16 passed, 18 total
Snapshots: 0 total
Time: 8.965s, estimated 27s
Ran all test suites with tests matching "^(?!\[functional\]).+$".
Active Filters: test name /^(?!\[functional\]).+$/
The functional tests are not built out which explains the syntax error, it's not important here. The key issue, is why the tests were not skipped.
I believe the problem has to do with the regex negative matcher. The positive matcher without the ! only matches tests that have, or are nested in a describe block with [functional]
$ npm run test:functional
Test Suites: 1 failed, 4 skipped, 1 of 5 total
Active Filters: test name /\[functional\]/
Anyone know why the negative regex pattern is failing during npm run test:unit ?
Instead of a regex fix I changed the flag on the unit testing script to an ignore, then copying the matching pattern for [functional]:
"test:unit": "jest --watch --testIgnorePattern='\\[functional\\]'",
"test:functional": "jest --watch --testNamePattern='\\[functional\\]'",
Okay, so I'm trying to find out if a ticket has been included in my release branch. The tickets are all built out of a project id and an id number, e.g. (PRO-123). I've tried this command:
git log --date=short --format="%h: %ad (%cn) %s" --abbrev-commit --grep='[A-Z]+-[0-9]+'
But it's not returning anything. If I take away the --grep part there's loads of matches to the pattern. For instance:
a6fdcd0: 2016-03-16 (ajfaraday) Merge remote-tracking branch 'origin/develop_5.2_customer' into release_5.2_customer
85d107a: 2016-03-16 (username) Merge pull request #477 from myapp/fix_CST-827_outline_method_in_use_check
6024bda: 2016-03-16 (Andrew Faraday) Merge pull request #473 from myapp/fix_CST-810_soap_container_create_bounds
eec2a61: 2016-03-16 (ajfaraday) added missing stubs
c03b3cb: 2016-03-15 (username) Merge pull request #472 from myapp/fix_CST-490_options_are_clickable_for_user_without_module_admin_rights
728539b: 2016-03-15 (username) Merge pull request #474 from myapp/fix_CST-873_hidden_error_on_pev_validation
4a11dd7: 2016-03-15 (username) Merge pull request #475 from myapp/fix_CST-854_copy_process_version_project_element_values
4a5af44: 2016-03-15 (ajfaraday) CST-854: fixed in-use check for methods
What am I doing wrong?
Okay, I think I've found the problem. It's some minor language difference in regexes (I'm usually writing them in my Ruby code).
For some reason [A-Z]+ wasn't matching but [A-Z]* is working fine. This line does what I wanted:
git log --date=short --format="%h: %ad (%cn) %s" --abbrev-commit --grep="[A-Z]*-[0-9]*"
Example:
info: 2014-10-28T22:39:46.593Z - info: an error occurred while trying
to handle command: PlaceMarketOrderCommand, xkkdAAGRIl. Error:
Insufficient Cash #userId=5 #orderId=Y5545
pattern:
> %{LOGLEVEL:stream_level}: %{TIMESTAMP_ISO8601:timestamp} -
> %{LOGLEVEL:log_level}: %{MESSAGE:message}
> (#userId=%{USER_ID:user_id})? (#orderId=%{ORDER_ID:order_id})?
extra patterns used:
USER_ID (\d+|None)
ORDER_ID .*
ORDER_ID_HASH \s*(#orderId=%{ORDER_ID:order_id})?
USER_ID_HASH \s*(#userId=%{USER_ID:user_id})?
MESSAGE (.*?)
Works fine:
removing the optional last orderId also works
info: 2014-10-28T22:39:46.593Z - info: an error occurred while trying
to handle command: PlaceMarketOrderCommand, xkkdAAGRIl. Error:
Insufficient Cash #userId=5
but if I keep the orderId and remove the userId then I get a "no match"
info: 2014-10-28T22:39:46.593Z - info: an error occurred while trying
to handle command: PlaceMarketOrderCommand, xkkdAAGRIl. Error:
Insufficient Cash #orderId=Y5545
Also the user_id group is ending with a ? as an optional group..
working with the grok debugger in heroku:
Is this a bug? (logstash 1.4.2) missing something with the regex? (more probable.. but what?)
I looked at the regex lib grok is using and looks this syntax supposed to work. It does work for the last group (orderId) but not for the one before..
Thanks for the help!
You are forcing a space to be before your optional last... you need to do ?:
%{LOGLEVEL:stream_level}: %{TIMESTAMP_ISO8601:timestamp} -> %{LOGLEVEL:log_level}: %{MESSAGE:message} ?(#userId=%{USER_ID:user_id})? ?(#orderId=%{ORDER_ID:order_id})?
I'm a Ruby on Rails newbie and writing tests. Some of these generate exceptions; I would like the "rake test" output to give me the exception error message but not the whole backtrace. (I'd like to write tests which exercise unimplemented functionality, which I'll then fill in.)
For example, actual output:
Started
E
Finished in 0.081054 seconds.
1) Error:
test_should_fail(VersioningTest):
ActiveRecord::StatementInvalid: PGError: ERROR: null value in column "client_ip" violates not-null constraint
: INSERT INTO "revisions" ("created_at", "id") VALUES ('2011-02-03 20:14:17', 980190962)
/Users/rpriedhorsky/.rvm/gems/ruby-1.9.2-p136/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:202:in `rescue in log'
/Users/rpriedhorsky/.rvm/gems/ruby-1.9.2-p136/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:194:in `log'
/Users/rpriedhorsky/.rvm/gems/ruby-1.9.2-p136/gems/activerecord-3.0.3/lib/active_record/connection_adapters/postgresql_adapter.rb:496:in `execute'
[... etc. etc. etc. ...]
1 tests, 0 assertions, 0 failures, 1 errors, 0 skips
Desired output:
Started
E
Finished in 0.081054 seconds.
1) Error:
test_should_fail(VersioningTest):
ActiveRecord::StatementInvalid: PGError: ERROR: null value in column "client_ip" violates not-null constraint
1 tests, 0 assertions, 0 failures, 1 errors, 0 skips
I found info (e.g.) on the opposite direction, but not on suppressing stack traces.
Edit:
It would be nice to turn them on and off easily; as pointed out below, sometimes they are useful for tracking down bugs.
You could take a look at "backtrace silencers" - for me (Rails 2.3.8), this is the file config/initializers/backtrace_silencers.rb:
# Be sure to restart your server when you modify this file.
# You can add backtrace silencers for libraries that you're using but
# don't wish to see in your backtraces.
# Rails.backtrace_cleaner.add_silencer { |line| line =~ /my_noisy_library/ }
# You can also remove all the silencers if you're trying do debug a
# problem that might steem from framework code.
# Rails.backtrace_cleaner.remove_silencers!
Rails.backtrace_cleaner.add_silencer {|line| line =~ /gems/}
Rails.backtrace_cleaner.add_silencer {|line| line =~ /passenger/}
It looks like you should be able to put a line like
Rails.backtrace_cleaner.add_silencer {|line| true}
In your config/environments/test.rb file, and that would wipe your backtraces clean away (though it might just apply to the logger - I'm not very familiar with the method).
But ask yourself - do you really want to do away with backtraces entirely? They can be pretty useful for tracking down bugs...