Only match unique string occurrences - regex

We are doing some Data Loss Prevention for emails, but the issue is when people reply to emails multiple times sometimes the credit card number or account number will appear multiple times.
How can we get Java Regex to only match strings once each.
So for example, we are using the following regex to catch account numbers that match 2 letters followed by 5 or 6 numbers. it will also omit CR in either case.
\b(?!CR)(?!cr)[A-Za-z]{2}[0-9]{5,6}\b
How can we have it find:
CX12345
CX14584
JB145888
JD748452
CX12345 (Ignore as its already found it above)
LM45855

Unique string occurrence can be matched with
<STRING_PATTERN>(?!.*<STRING_PATTERN>) // Find the last occurrence
(?<!<STRING_PATTERN>.*)<STRING_PATTERN> // Find the first occurrence, only works in regex
// that supports infinite-width lookbehind patterns
where <STRING_PATTERN> is the pattern the unique occurrence of which one searches for. Note that both will work with the .NET regex library, but the second one is not usually supported by the majority of other libraries (only PyPi Python regex library and the JavaScript ECMAScript 2018 regex support it). Note that . does not match line break chars by default, so you need to pass a modifier like DOTALL (in most libraries, you may add (?s) modifier inside the pattern (only in Ruby (?m) does the same), or use specific flags that you pass to the regex compile method. See more about this in How do I match any character across multiple lines in a regular expression?
You seem to need a regex like this:
/\b((?!CR|cr)[A-Za-z]{2}\d{5,6})\b(?![\s\S]*\b\1\b)/
The regex demo is available here
Details:
\b - a leading word boundary
((?!CR|cr)[A-Za-z]{2}\d{5,6}) - Group 1 capturing
(?!CR|cr) - the next two characters cannot be CR or cr, the negative lookahead check
[A-Za-z]{2} - 2 ASCII letters
\d{5,6} - 5 to 6 digits
\b - trailing word boundary
(?![\s\S]*\b\1\b) - a negative lookahead that fails the match if there are any 0+ chars ([\s\S]*) followed with a word boundary (\b), same value captured into Group 1 (with the \1 backreference), and a trailing word boundary.

I would use a Map of some sort here, to keep tally of the strings which you encounter. For example:
String ccNumber = "CX12345";
Map<String, Boolean> ccMap = new HashMap<>();
if (ccNumber.matches("^(?!CR)(?!cr)[A-Za-z]{2}[0-9]{5,6}$")) {
ccMap.put(ccNumber, null);
}
Then just iterate over the keyset of the map to get unique credit card numbers which matched the pattern in your regex:
for (String key : map.keySet()) {
System.out.println("Found a matching credit card: " + key);
}

Related

Notepad++ Search for and replace Underscore Characters in "GUIDs"

A colleague has written some C# code that outputs GUIDs to a CSV file. The code has been running for a while but it has been discovered that the GUIDs contain underscore characters, instead of hyphens :-(
There are several files which have been produced already and rather than regenerate these, I'm thinking that we could use the Search and Replace facility in Notepad++ to search across the files for "GUIDs" in this format:
{89695C16_C0FF_4E7C_9BB2_8B50FAC9D371}
and replace it with a properly formatted GUID like this:
{89695C16-C0FF-4E7C-9BB2-8B50FAC9D371}.
I have a RegEx to find the offending GUIDs (probably not very efficient):
(([A-Z]|[0-9]){8}_)(([A-Z]|[0-9]){4})_(([A-Z]|[0-9]){4})_(([A-Z]|[0-9]){4}_(([A-Z]|[0-9]){12}))
but I don't know what RegEx to use to replace the underscores with. Does anybody know how to do this?
You can use the following solution:
Find What: (?:\G(?!\A)|{(?=[a-f\d]{8}(?:_[a-f\d]{4}){4}[a-f\d]{8}\}))[a-f\d]*\K_
Replace with: -
Match case: OFF
See the settings and demo:
See the regex demo online. Details:
(?:\G(?!\A)|{(?=[a-f\d]{8}(?:_[a-f\d]{4}){4}[a-f\d]{8}\})) - either the end of the previous match or a { char immediately followed with eight alphanumeric chars, four repetitions of an underscore and then four alphanumeric chars and then eight alphanumeric chars and a } char
[a-f\d]* - zero or more alphanumeric chars
\K - match reset operator that discards the text matched so far from the overall match memory buffer
_ - an underscore.
You can match the pattern with 5 capture groups where you would match the underscores in between.
Then you can use the capture groups in the replacement with $1-$2-$3-$4-$5
{\K([A-Z0-9]{8})_([A-Z0-9]{4})_([A-Z0-9]{4})_([A-Z0-9]{4})_([A-Z0-9]{12})(?=})
{ Match {
\K Clear the match buffer (forget what is matched so far)
([A-Z0-9]{8})_ Capture group 1, match 8 times a char A-Z0-9
([A-Z0-9]{4})_ Capture 4 times a char A-Z0-9 in group 2
([A-Z0-9]{4})_ Same for group 3
([A-Z0-9]{4})_ Same for group 4
([A-Z0-9]{12}) Capture 12 times a char A-Z0-9 in group 5
(?=}) Positive lookahead, assert } to the right
Regex demo
If the pattern should also match without matching the curly's { and } you can append word boundaries
\b([A-Z0-9]{8})_([A-Z0-9]{4})_([A-Z0-9]{4})_([A-Z0-9]{4})_([A-Z0-9]{12})\b
Regex demo

Pattern to match everything except a string of 5 digits

I only have access to a function that can match a pattern and replace it with some text:
Syntax
regexReplace('text', 'pattern', 'new text'
And I need to return only the 5 digit string from text in the following format:
CRITICAL - 192.111.6.4: rta nan, lost 100%
Created Time Tue, 5 Jul 8:45
Integration Name CheckMK Integration
Node 192.111.6.4
Metric Name POS1
Metric Value DOWN
Resource 54871
Alert Tags 54871, POS1
So from this text, I want to replace everything with "" except the "54871".
I have come up with the following:
regexReplace("{{ticket.description}}", "\w*[^\d\W]\w*", "")
Which almost works but it doesn't match the symbols. How can I change this to match any word that includes a letter or symbol, essentially.
As you can see, the pattern I have is very close, I just need to include special characters and letters, whereas currently it is only letters:
You can match the whole string but capture the 5-digit number into a capturing group and replace with the backreference to the captured group:
regexReplace("{{ticket.description}}", "^(?:[\w\W]*\s)?(\d{5})(?:\s[\w\W]*)?$", "$1")
See the regex demo.
Details:
^ - start of string
(?:[\w\W]*\s)? - an optional substring of any zero or more chars as many as possible and then a whitespace char
(\d{5}) - Group 1 ($1 contains the text captured by this group pattern): five digits
(?:\s[\w\W]*)? - an optional substring of a whitespace char and then any zero or more chars as many as possible.
$ - end of string.
The easiest regex is probably:
^(.*\D)?(\d{5})(\D.*)?$
You can then replace the string with "$2" ("\2" in other languages) to only place the contents of the second capture group (\d{5}) back.
The only issue is that . doesn't match newline characters by default. Normally you can pass a flag to change . to match ALL characters. For most regex variants this is the s (single line) flag (PCRE, Java, C#, Python). Other variants use the m (multi line) flag (Ruby). Check the documentation of the regex variant you are using for verification.
However the question suggest that you're not able to pass flags separately, in which case you could pass them as part of the regex itself.
(?s)^(.*\D)?(\d{5})(\D.*)?$
regex101 demo
(?s) - Set the s (single line) flag for the remainder of the pattern. Which enables . to match newline characters ((?m) for Ruby).
^ - Match the start of the string (\A for Ruby).
(.*\D)? - [optional] Match anything followed by a non-digit and store it in capture group 1.
(\d{5}) - Match 5 digits and store it in capture group 2.
(\D.*)? - [optional] Match a non-digit followed by anything and store it in capture group 3.
$ - Match the end of the string (\z for Ruby).
This regex will result in the last 5-digit number being stored in capture group 2. If you want to use the first 5-digit number instead, you'll have to use a lazy quantifier in (.*\D)?. Meaning that it becomes (.*?\D)?.
(?s) is supported by most regex variants, but not all. Refer to the regex variant documentation to see if it's available for you.
An example where the inline flags are not available is JavaScript. In such scenario you need to replace . with something that matches ALL characters. In JavaScript [^] can be used. For other variants this might not work and you need to use [\s\S].
With all this out of the way. Assuming a language that can use "$2" as replacement, and where you do not need to escape backslashes, and a regex variant that supports an inline (?s) flag. The answer would be:
regexReplace("{{ticket.description}}", "(?s)^(.*\D)?(\d{5})(\D.*)?$", "$2")

Capture number if string contains "X", but limit match (cannot use groups)

I need to extract numbers like 2.268 out of strings that contain the word output:
Approxmiate output size of output: 2.268 kilobytes
But ignore it in strings that don't:
some entirely different string: 2.268 kilobytes
This regex:
(?:output.+?)([\d\.]+)
Gives me a match with 1 group, with the group being 2.268 for the target string. But since I'm not using a programming language but rather CloudWatch Log Insights, I need a way to only match the number itself without using groups.
I could use a positive lookbehind ?<= in order to not consume the string at all, but then I don't know how to throw away size of output: without using .+, which positive lookbehind doesn't allow.
With your shown samples, please try following regex.
output:\D+\K\d(?:\.\d+)?
Online demo for above regex
Explanation: Adding detailed explanation for above.
output:\D+ ##Matching output colon followed by non-digits(1 or more occurrences)
\K ##\K to forget previous matched values to make sure we get only further matched values in this expression.
\d(?:\.\d+)? ##Matching digit followed by optional dot digits.
Since you are using PCRE, you can use
output.*?\K\d[\d.]*
See the regex demo. This matches
output - a fixed string
.*? - any zero or more chars other than line break chars, as few as possible
\K - match reset operator that removes all text matched so far from the overall match memory buffer
\d - a digit
[\d.]* - zero or more digits or periods.

Regex Variable Size HexString Parsing

I am not familiar with Regex and I need to parse a spec using Regex.
I need to get the KEK, Key and Wrap hex values into a string/hex array but the hex string lengths can be variadic and have spaces. Please see an example below
The second example wraps 7 octets of key data with a 192-bit KEK.
KEK : 5840df6e29b02af1 ab493b705bf16ea1 ae8338f4dcc176a8
Key : 466f7250617369
Wrap : afbeb0f07dfbf541 9200f2ccb50bb24f
The explanation tells the length of the key which might be used; example "7 octets of key data"
The other problem is that online regex tools and online python interpreters to run python regex lib (re) behaves different so I can not be sure about the regex expression.
I tried to get a line using
(\w+)\s+:\s+([A-Fa-f\d][A-Fa-f\d]([A-Fa-f\d][A-Fa-f\d])*)
but it parse a line until space in hex string.
Any recommendation on that
In your pattern you are matching 2 chars and then optionally repeat per 2 chars. But you are repeating a capture group without matching spaces.
You can reuse that same mechanism optionally repeating per 2 chars with 1 or more whitespace chars prepended in a non capture group, and capture that whole repetition in an outer capture group.
(\w+)\s+:\s+((?:[A-Fa-f\d][A-Fa-f\d])+(?:\s+(?:[A-Fa-f\d][A-Fa-f\d])+)*)\b
Regex demo
The same mechanism as the above pattern to repeat 1 or more characters instead of per 2 characters:
(\w+)\s+:\s+([A-Fa-f\d]+(?:\s+[A-Fa-f\d]+)*)\b
(\w+) Capture group 1, match 1+ word chars
\s+:\s+ match : between 1 or more whitespace chars
( Capture group 2
[A-Fa-f\d]+ Match 1+ times any of the ranges
(?:\s+[A-Fa-f\d]+)* Match 1+ whitespace chars and 1+ times any of the ranges
) Close group 2
\b A word boundary to prevent a partial match
Regex demo
Thank you very much The fourth bird, it definitely parses the hex mentioned above.
But I have noticed that there are some irregular lines like below
The first example wraps 20 octets of key data with a 192-bit KEK.
KEK : 5840df6e29b02af1 ab493b705bf16ea1 ae8338f4dcc176a8
Key : c37b7e6492584340 bed1220780894115 5068f738
Wrap : 138bdeaa9b8fa7fc 61f97742e72248ee 5ae6ae5360d1ae6a
: 5f54f373fa543b6a
"Wrap" line is two lines and there is one more colon(:) symbol, I don't know is there anything to do here.

Regex to find a line with two capture groups that match the same regex but are still different

I am trying to analyse my source code (written in C) for not corresponding timer variable comparisons/allocations. I have a rage of timers with different timebases (2-250 milliseconds). Every timer variable contains its granularity in milliseconds in its name (e.g. timer10ms) as well as every timer-photo and define (e.g. fooTimer10ms, DOO_TIMEOUT_100MS).
Here are some example lines:
fooTimer10ms = timer10ms;
baaTimer20ms = timer10ms;
if (DIFF_100MS(dooTimer10ms) >= DOO_TIMEOUT_100MS)
if (DIFF_100MS(dooTimer10ms) < DOO_TIMEOUT_100MS)
I want to match those line where the timebases are not corresponding (in this case the second, third and fourth line). So far I have this regex:
(\d{1,3}(?i)ms(?-i)).*[^\d](\d{1,3}(?i)ms(?-i))
that is capable of finding every line where there are two of those granularities. So instead of just line 2, 3 and 4 it matches all of them. The only idea I had to narrow it down is to add a negative lookbehind with a back-reference, like so:
(\d{1,3}(?i)ms(?-i)).*[^\d](\d{1,3}(?i)ms(?-i))(?<!\1)
but this is not allowed because a negative lookbehind has to have a fixed length.
I found these two questions (one, two) but the fist does not have the restriction of having both capture groups being of the same kind and the second is looking for equal instances of the capture group.
If what I want can be achieved way easier, by using something else than regex, I would be happy to know. My mind is just stuck due to my believe that regex is capable of that and I am just not creative enough to use it properly.
One option is to match the timer part followed by the digits and use a negative lookahead with a backreference to assert that it does not occur at the right.
For the example data, a bit specific pattern using a range from 2-250 might be:
.*?(timer(?:2[0-4]\d|250|1?\d\d|[2-9])ms)\b\S*[^\S\r\n]*[<>]?=[^\S\r\n]*\b(?!\S*\1)\S+
The pattern matches
.*? Match any char except a newline, as least as possible (Non greedy)
( Capture group 1
timer Match literally
(?:2[0-4]\d|250|1?\d\d|[2-9]) Match a digit in the range of 2-250
ms Match literally
)\b Close group and a word boundary
\S*[^\S\r\n]* Match optional non whitespace chars and optional spaces without newlines
[<>]?= Match an optional < or > and =
[^\S\r\n]*\b Match optional whitespace chars without a newline and a word boundary
(?!\S*\1) Negative lookahead, assert no occurrence of what is captured in group 1 in the value
\S+ Match 1+ non whitespace chars
Regex demo
Or perhaps a broader pattern matching 1-3 digits and optional whitespace chars which might also match a newline:
.*?(timer\d{1,3}ms\b)\S*\s*[<>]?=\s*\b(?!.*\1)\S+
Regex demo
Note that {1-3} should be {1,3} and could also match 999