This is the second part of a series of educational regex articles. It shows how lookaheads and nested references can be used to match the non-regular languge anbn. Nested references are first introduced in: How does this regex find triangular numbers?
One of the archetypal non-regular languages is:
L = { anbn: n > 0 }
This is the language of all non-empty strings consisting of some number of a's followed by an equal number of b's. Examples of strings in this language are ab, aabb, aaabbb.
This language can be show to be non-regular by the pumping lemma. It is in fact an archetypal context-free language, which can be generated by the context-free grammar S → aSb | ab.
Nonetheless, modern day regex implementations clearly recognize more than just regular languages. That is, they are not "regular" by formal language theory definition. PCRE and Perl supports recursive regex, and .NET supports balancing groups definition. Even less "fancy" features, e.g. backreference matching, means that regex is not regular.
But just how powerful is this "basic" features? Can we recognize L with Java regex, for example? Can we perhaps combine lookarounds and nested references and have a pattern that works with e.g. String.matches to match strings like ab, aabb, aaabbb, etc?
References
perlfaq6: Can I use Perl regular expressions to match balanced text?
MSDN - Regular Expression Language Elements - Balancing Group Definitions
pcre.org - PCRE man page
regular-expressions.info - Lookarounds and Grouping and Backreferences
java.util.regex.Pattern
Linked questions
Does lookaround affect which languages can be matched by regular expressions?
.NET Regex Balancing Groups vs PCRE Recursive Patterns
The answer is, needless to say, YES! You can most certainly write a Java regex pattern to match anbn. It uses a positive lookahead for assertion, and one nested reference for "counting".
Rather than immediately giving out the pattern, this answer will guide readers through the process of deriving it. Various hints are given as the solution is slowly constructed. In this aspect, hopefully this answer will contain much more than just another neat regex pattern. Hopefully readers will also learn how to "think in regex", and how to put various constructs harmoniously together, so they can derive more patterns on their own in the future.
The language used to develop the solution will be PHP for its conciseness. The final test once the pattern is finalized will be done in Java.
Step 1: Lookahead for assertion
Let's start with a simpler problem: we want to match a+ at the beginning of a string, but only if it's followed immediately by b+. We can use ^ to anchor our match, and since we only want to match the a+ without the b+, we can use lookahead assertion (?=…).
Here is our pattern with a simple test harness:
function testAll($r, $tests) {
foreach ($tests as $test) {
$isMatch = preg_match($r, $test, $groups);
$groupsJoined = join('|', $groups);
print("$test $isMatch $groupsJoined\n");
}
}
$tests = array('aaa', 'aaab', 'aaaxb', 'xaaab', 'b', 'abbb');
$r1 = '/^a+(?=b+)/';
# └────┘
# lookahead
testAll($r1, $tests);
The output is (as seen on ideone.com):
aaa 0
aaab 1 aaa
aaaxb 0
xaaab 0
b 0
abbb 1 a
This is exactly the output we want: we match a+, only if it's at the beginning of the string, and only if it's immediately followed by b+.
Lesson: You can use patterns in lookarounds to make assertions.
Step 2: Capturing in a lookahead (and f r e e - s p a c i n g mode)
Now let's say that even though we don't want the b+ to be part of the match, we do want to capture it anyway into group 1. Also, as we anticipate having a more complicated pattern, let's use x modifier for free-spacing so we can make our regex more readable.
Building on our previous PHP snippet, we now have the following pattern:
$r2 = '/ ^ a+ (?= (b+) ) /x';
# │ └──┘ │
# │ 1 │
# └────────┘
# lookahead
testAll($r2, $tests);
The output is now (as seen on ideone.com):
aaa 0
aaab 1 aaa|b
aaaxb 0
xaaab 0
b 0
abbb 1 a|bbb
Note that e.g. aaa|b is the result of join-ing what each group captured with '|'. In this case, group 0 (i.e. what the pattern matched) captured aaa, and group 1 captured b.
Lesson: You can capture inside a lookaround. You can use free-spacing to enhance readability.
Step 3: Refactoring the lookahead into the "loop"
Before we can introduce our counting mechanism, we need to do one modification to our pattern. Currently, the lookahead is outside of the + repetition "loop". This is fine so far because we just wanted to assert that there's a b+ following our a+, but what we really want to do eventually is assert that for each a that we match inside the "loop", there's a corresponding b to go with it.
Let's not worry about the counting mechanism for now and just do the refactoring as follows:
First refactor a+ to (?: a )+ (note that (?:…) is a non-capturing group)
Then move the lookahead inside this non-capturing group
Note that we must now "skip" a* before we can "see" the b+, so modify the pattern accordingly
So we now have the following:
$r3 = '/ ^ (?: a (?= a* (b+) ) )+ /x';
# │ │ └──┘ │ │
# │ │ 1 │ │
# │ └───────────┘ │
# │ lookahead │
# └───────────────────┘
# non-capturing group
The output is the same as before (as seen on ideone.com), so there's no change in that regard. The important thing is that now we are making the assertion at every iteration of the + "loop". With our current pattern, this is not necessary, but next we'll make group 1 "count" for us using self-reference.
Lesson: You can capture inside a non-capturing group. Lookarounds can be repeated.
Step 4: This is the step where we start counting
Here's what we're going to do: we'll rewrite group 1 such that:
At the end of the first iteration of the +, when the first a is matched, it should capture b
At the end of the second iteration, when another a is matched, it should capture bb
At the end of the third iteration, it should capture bbb
...
At the end of the n-th iteration, group 1 should capture bn
If there aren't enough b to capture into group 1 then the assertion simply fails
So group 1, which is now (b+), will have to be rewritten to something like (\1 b). That is, we try to "add" a b to what group 1 captured in the previous iteration.
There's a slight problem here in that this pattern is missing the "base case", i.e. the case where it can match without the self-reference. A base case is required because group 1 starts "uninitialized"; it hasn't captured anything yet (not even an empty string), so a self-reference attempt will always fail.
There are many ways around this, but for now let's just make the self-reference matching optional, i.e. \1?. This may or may not work perfectly, but let's just see what that does, and if there's any problem then we'll cross that bridge when we come to it. Also, we'll add some more test cases while we're at it.
$tests = array(
'aaa', 'aaab', 'aaaxb', 'xaaab', 'b', 'abbb', 'aabb', 'aaabbbbb', 'aaaaabbb'
);
$r4 = '/ ^ (?: a (?= a* (\1? b) ) )+ /x';
# │ │ └─────┘ | │
# │ │ 1 | │
# │ └──────────────┘ │
# │ lookahead │
# └──────────────────────┘
# non-capturing group
The output is now (as seen on ideone.com):
aaa 0
aaab 1 aaa|b # (*gasp!*)
aaaxb 0
xaaab 0
b 0
abbb 1 a|b # yes!
aabb 1 aa|bb # YES!!
aaabbbbb 1 aaa|bbb # YESS!!!
aaaaabbb 1 aaaaa|bb # NOOOOOoooooo....
A-ha! It looks like we're really close to the solution now! We managed to get group 1 to "count" using self-reference! But wait... something is wrong with the second and the last test cases!! There aren't enough bs, and somehow it counted wrong! We'll examine why this happened in the next step.
Lesson: One way to "initialize" a self-referencing group is to make the self-reference matching optional.
Step 4½: Understanding what went wrong
The problem is that since we made the self-reference matching optional, the "counter" can "reset" back to 0 when there aren't enough b's. Let's closely examine what happens at every iteration of our pattern with aaaaabbb as input.
a a a a a b b b
↑
# Initial state: Group 1 is "uninitialized".
_
a a a a a b b b
↑
# 1st iteration: Group 1 couldn't match \1 since it was "uninitialized",
# so it matched and captured just b
___
a a a a a b b b
↑
# 2nd iteration: Group 1 matched \1b and captured bb
_____
a a a a a b b b
↑
# 3rd iteration: Group 1 matched \1b and captured bbb
_
a a a a a b b b
↑
# 4th iteration: Group 1 could still match \1, but not \1b,
# (!!!) so it matched and captured just b
___
a a a a a b b b
↑
# 5th iteration: Group 1 matched \1b and captured bb
#
# No more a, + "loop" terminates
A-ha! On our 4th iteration, we could still match \1, but we couldn't match \1b! Since we allow the self-reference matching to be optional with \1?, the engine backtracks and took the "no thanks" option, which then allows us to match and capture just b!
Do note, however, that except on the very first iteration, you could always match just the self-reference \1. This is obvious, of course, since it's what we just captured on our previous iteration, and in our setup we can always match it again (e.g. if we captured bbb last time, we're guaranteed that there will still be bbb, but there may or may not be bbbb this time).
Lesson: Beware of backtracking. The regex engine will do as much backtracking as you allow until the given pattern matches. This may impact performance (i.e. catastrophic backtracking) and/or correctness.
Step 5: Self-possession to the rescue!
The "fix" should now be obvious: combine optional repetition with possessive quantifier. That is, instead of simply ?, use ?+ instead (remember that a repetition that is quantified as possessive does not backtrack, even if such "cooperation" may result in a match of the overall pattern).
In very informal terms, this is what ?+, ? and ?? says:
?+
(optional) "It doesn't have to be there,"
(possessive) "but if it is there, you must take it and not let go!"
?
(optional) "It doesn't have to be there,"
(greedy) "but if it is you can take it for now,"
(backtracking) "but you may be asked to let it go later!"
??
(optional) "It doesn't have to be there,"
(reluctant) "and even if it is you don't have to take it just yet,"
(backtracking) "but you may be asked to take it later!"
In our setup, \1 will not be there the very first time, but it will always be there any time after that, and we always want to match it then. Thus, \1?+ would accomplish exactly what we want.
$r5 = '/ ^ (?: a (?= a* (\1?+ b) ) )+ /x';
# │ │ └──────┘ │ │
# │ │ 1 │ │
# │ └───────────────┘ │
# │ lookahead │
# └───────────────────────┘
# non-capturing group
Now the output is (as seen on ideone.com):
aaa 0
aaab 1 a|b # Yay! Fixed!
aaaxb 0
xaaab 0
b 0
abbb 1 a|b
aabb 1 aa|bb
aaabbbbb 1 aaa|bbb
aaaaabbb 1 aaa|bbb # Hurrahh!!!
Voilà!!! Problem solved!!! We are now counting properly, exactly the way we want it to!
Lesson: Learn the difference between greedy, reluctant, and possessive repetition. Optional-possessive can be a powerful combination.
Step 6: Finishing touches
So what we have right now is a pattern that matches a repeatedly, and for every a that was matched, there is a corresponding b captured in group 1. The + terminates when there are no more a, or if the assertion failed because there isn't a corresponding b for an a.
To finish the job, we simply need to append to our pattern \1 $. This is now a back reference to what group 1 matched, followed by the end of the line anchor. The anchor ensures that there aren't any extra b's in the string; in other words, that in fact we have anbn.
Here's the finalized pattern, with additional test cases, including one that's 10,000 characters long:
$tests = array(
'aaa', 'aaab', 'aaaxb', 'xaaab', 'b', 'abbb', 'aabb', 'aaabbbbb', 'aaaaabbb',
'', 'ab', 'abb', 'aab', 'aaaabb', 'aaabbb', 'bbbaaa', 'ababab', 'abc',
str_repeat('a', 5000).str_repeat('b', 5000)
);
$r6 = '/ ^ (?: a (?= a* (\1?+ b) ) )+ \1 $ /x';
# │ │ └──────┘ │ │
# │ │ 1 │ │
# │ └───────────────┘ │
# │ lookahead │
# └───────────────────────┘
# non-capturing group
It finds 4 matches: ab, aabb, aaabbb, and the a5000b5000. It takes only 0.06s to run on ideone.com.
Step 7: The Java test
So the pattern works in PHP, but the ultimate goal is to write a pattern that works in Java.
public static void main(String[] args) {
String aNbN = "(?x) (?: a (?= a* (\\1?+ b)) )+ \\1";
String[] tests = {
"", // false
"ab", // true
"abb", // false
"aab", // false
"aabb", // true
"abab", // false
"abc", // false
repeat('a', 5000) + repeat('b', 4999), // false
repeat('a', 5000) + repeat('b', 5000), // true
repeat('a', 5000) + repeat('b', 5001), // false
};
for (String test : tests) {
System.out.printf("[%s]%n %s%n%n", test, test.matches(aNbN));
}
}
static String repeat(char ch, int n) {
return new String(new char[n]).replace('\0', ch);
}
The pattern works as expected (as seen on ideone.com).
And now we come to the conclusion...
It needs to be said that the a* in the lookahead, and indeed the "main + loop", both permit backtracking. Readers are encouraged to confirm why this is not a problem in terms of correctness, and why at the same time making both possessive would also work (though perhaps mixing mandatory and non-mandatory possessive quantifier in the same pattern may lead to misperceptions).
It should also be said that while it's neat that there's a regex pattern that will match anbn, this is in not always the "best" solution in practice. A much better solution is to simply match ^(a+)(b+)$, and then compare the length of the strings captured by groups 1 and 2 in the hosting programming language.
In PHP, it may look something like this (as seen in ideone.com):
function is_anbn($s) {
return (preg_match('/^(a+)(b+)$/', $s, $groups)) &&
(strlen($groups[1]) == strlen($groups[2]));
}
The purpose of this article is NOT to convince readers that regex can do almost anything; it clearly can't, and even for the things it can do, at least partial delegation to the hosting language should be considered if it leads to a simpler solution.
As mentioned at the top, while this article is necessarily tagged [regex] for stackoverflow, it is perhaps about more than that. While certainly there's value in learning about assertions, nested references, possessive quantifier, etc, perhaps the bigger lesson here is the creative process by which one can try to solve problems, the determination and hard work that it often requires when you're subjected to various constraints, the systematic composition from various parts to build a working solution, etc.
Bonus material! PCRE recursive pattern!
Since we did bring up PHP, it needs to be said that PCRE supports recursive pattern and subroutines. Thus, following pattern works for preg_match (as seen on ideone.com):
$rRecursive = '/ ^ (a (?1)? b) $ /x';
Currently Java's regex does not support recursive pattern.
Even more bonus material! Matching anbncn !!
So we've seen how to match anbn which is non-regular, but still context-free, but can we also match anbncn, which isn't even context-free?
The answer is, of course, YES! Readers are encouraged to try to solve this on their own, but the solution is provided below (with implementation in Java on ideone.com).
^ (?: a (?= a* (\1?+ b) b* (\2?+ c) ) )+ \1 \2 $
Given that no mention has been made of PCRE supporting recursive patterns, I'd just like to point out the simplest and most efficient example of PCRE that describes the language in question:
/^(a(?1)?b)$/
As mentioned in the question — with .NET balancing group, the patterns of the type anbncndn…zn can be matched easily as
^
(?<A>a)+
(?<B-A>b)+ (?(A)(?!))
(?<C-B>c)+ (?(B)(?!))
...
(?<Z-Y>z)+ (?(Y)(?!))
$
For example: http://www.ideone.com/usuOE
Edit:
There is also a PCRE pattern for the generalized language with recursive pattern, but a lookahead is needed. I don't think this is a direct translation of the above.
^
(?=(a(?-1)?b)) a+
(?=(b(?-1)?c)) b+
...
(?=(x(?-1)?y)) x+
(y(?-1)?z)
$
For example: http://www.ideone.com/9gUwF
I'm using PowerShell to read output from an executable and needing to parse the output into an array. I've tried regex101 and I start to get close but not able to return everything.
Identity type: group
Group type: Generic
Project scope: PartsUnlimited
Display name: [PartsUnlimited]\Contributors
Description: {description}
5 member(s):
[?] test
[A] [PartsUnlimited]\PartsUnlimited-1
[A] [PartsUnlimited]\PartsUnlimited-2
[?] test2
[A] [PartsUnlimited]\PartsUnlimited 3
Member of 3 group(s):
e [A] [org]\Project Collection Valid Users
[A] [PartsUnlimited]\Endpoint Creators
e [A] [PartsUnlimited]\Project Valid Users
I need returned an array of:
test
[PartsUnlimited]\PartsUnlimited-1
[PartsUnlimited]\PartsUnlimited-2
test2
[PartsUnlimited]\PartsUnlimited 3
At first I tried:
$pattern = "(?<=\[A|\?\])(.*)"
$matches = ([Regex]$pattern).Matches(($output -join "`n")).Value
But that will return also the "Member of 3 group(s):" section which I don't want.
I also can only get the first value under 5 member(s) with (?<=member\(s\):\n).*?\n ([?] test).
No matches are returned when I add in a positive lookahead: (?<=member\(s\):\n).*?\n(?=Member).
I feel like I'm getting close, just not sure how to handle multiple \n and get strings in between strings if that's needed.
You could do it in two steps (not sure if \G is supported in PowerShell).
The first step would be to separate the block in question with
^\d+\s+member.+[\r\n]
(?:.+[\r\n])+
With the multiline and verbose flags, see a demo on regex101.com.
On this block we then need to perform another expression such as
^\s+\[[^][]+\]\s+(.+)
Again with the multiline flag enabled, see another demo on regex101.com.
The expressions explained:
^\d+\s+member.+[\r\n] # start of the line (^), digits,
# spaces, "member", anything else + newline
(?:.+[\r\n])+ # match any consecutive line that is not empty
The second would be
^\s+ # start of the string, whitespaces
\[[^][]+\]\s+ # [...] (anything allowed within the brackets),
# whitespaces
(.+) # capture the rest of the line into group 1
If \G was supported, you could do it in one rush:
(?:
\G(?!\A)
|
^\d+\s+member.+[\r\n]
)
^\s+\[[^][]*\]\s+
(.+)
[\r\n]
See a demo for the latter on regex101.com as well.
There is a problem that I need to do, but there are some caveats that make it hard.
Problem: Match on all non-empty strings over the alphabet {abc} that contain at most one a.
Examples
a
abc
bbca
bbcabb
Nonexample
aa
bbaa
Caveats: You cannot use a lookahead/lookbehind.
What I have is this:
^[bc]*a?[bc]*$
but it matches empty strings. Maybe a hint? Idk anything would help
(And if it matters, I'm using python).
As I understand your question, the only problem is, that your current pattern matches empty strings. To prevent this you can use a word boundary \b to require at least one word character.
^\b[bc]*a?[bc]*$
See demo at regex101
Another option would be to alternate in a group. Match an a surrounded by any amount of [bc] or one or more [bc] from start to end which could look like: ^(?:[bc]*a[bc]*|[bc]+)$
The way I understood the issue was that any character in the alphabet should match, just only one a character.
Match on all non-empty strings over the alphabet... at most one a
^[b-z]*a?[b-z]*$
If spaces can be included:
^([b-z]*\s?)*a?([b-z]*\s?)*$
You do not even need a regex here, you might as well use .count() and a list comprehension:
data = """a,abc,bbca,bbcabb,aa,bbaa,something without the bespoken letter,ooo"""
def filter(string, char):
return [word
for word in string.split(",")
for c in [word.count(char)]
if c in [0,1]]
print(filter(data, 'a'))
Yielding
['a', 'abc', 'bbca', 'bbcabb', 'something without the bespoken letter', 'ooo']
You've got to positively match something excluding the empty string,
using only a, b, or c letters. But can't use assertions.
Here is what you do.
The regex ^(?:[bc]*a[bc]*|[bc]+)$
The explanation
^ # BOS
(?: # Cluster choice
[bc]* a [bc]* # only 1 [a] allowed, arbitrary [bc]'s
| # or,
[bc]+ # no [a]'s only [bc]'s ( so must be some )
) # End cluster
$ # EOS
I'm trying to dig deeper into regexes and want to match a condition unless some substring is also found in the same string. I know I can use two grepl statements (as seen below) but am wanting to use a single regex to test for this condition as I'm pushing my understanding. Let's say I want to match the words "dog" and "man" using "(dog.*man|man.*dog)" (taken from here) but not if the string contains the substring "park". I figured I could use (*SKIP)(*FAIL) to negate the "park" but this does not cause the string to fail (shown below).
How can I match the logic of find "dog" & "man" but not "park" with 1 regex?
What is wrong with my understanding of (*SKIP)(*FAIL)|?
The code:
x <- c(
"The dog and the man play in the park.",
"The man plays with the dog.",
"That is the man's hat.",
"Man I love that dog!",
"I'm dog tired",
"The dog park is no place for man.",
"Park next to this dog's man."
)
# Could do this but want one regex
grepl("(dog.*man|man.*dog)", x, ignore.case=TRUE) & !grepl("park", x, ignore.case=TRUE)
# Thought this would work, it does not
grepl("park(*SKIP)(*FAIL)|(dog.*man|man.*dog)", x, ignore.case=TRUE, perl=TRUE)
You can use the anchored look-ahead solution (requiring Perl-style regexp):
grepl("^(?!.*park)(?=.*dog.*man|.*man.*dog)", x, ignore.case=TRUE, perl=T)
Here is an IDEONE demo
^ - anchors the pattern at the start of the string
(?!.*park) - fail the match if park is present
(?=.*dog.*man|.*man.*dog) - fail the match if man and dog are absent.
Another version (more scalable) with 3 look-aheads:
^(?!.*park)(?=.*dog)(?=.*man)
stribizhev has already answered this question as it should be approached: with a negative lookahead.
I'll contribute to this particular question:
What is wrong with my understanding of (*SKIP)(*FAIL)?
(*SKIP) and (*FAIL) are regex control verbs.
(*FAIL) or (*F)
This is the easiest to understand. (*FAIL) is exactly the same as a negative lookahead with an empty subpattern: (?!). As soon as the regex engine gets to that verb in the pattern it forces an immediate backtrack.
(*SKIP)
When the regex engine first encounters this verb, nothing happens, because it only acts when it's reached on backtracking. But if there is a later failure, and it reaches (*SKIP) from right to left, the backtracking can't pass (*SKIP). It causes:
A match failure.
The next match won't be attempted from the next character. Instead, it will start from the position in the text where the engine was when it reached (*SKIP).
That is why these two control verbs are usually together as (*SKIP)(*FAIL)
Let's consider the following example:
Pattern: .*park(*SKIP)(*FAIL)|.*dog
Subject: "That park has too many dogs"
Matches: " has too many dog"
Internals:
First attempt.
That park has too many dogs || .*park(*SKIP)(*FAIL)|.*dog
/\ /\
(here) we have a match for park
the engine passes (*SKIP) -no action
it then encounters (*FAIL) -backtrack
Now it reaches (*SKIP) from the right -FAIL!
Second attempt.
Normally, it should start from the second character in the subject. However, (*SKIP) has this particular behaviour. The 2nd attempt starts:
That park has too many dogs || .*park(*SKIP)(*FAIL)|.*dog
/\ /\
(here)
Now, there's no match for .*park
And off course it matches .*dog
That park has too many dogs || .*park(*SKIP)(*FAIL)|.*dog
^ ^ -----
| (MATCH!) |
+---------------+
DEMO
How can I match the logic of find "dog" & "man" but not "park" with 1 regex?
Use stribizhev's solution!! Try to avoid using control verbs for the sake of compatibility, they're not implemented in all regex flavours. But if you're interested in these regex oddities, there's another stronger control verb: (*COMMIT). It is similar to (*SKIP), acting only while on backtracking, except it causes the entire match to fail (there won't be any other attempt at all). For example:
+-----------------------------------------------+
|Pattern: |
|^.*park(*COMMIT)(*FAIL)|dog |
+-------------------------------------+---------+
|Subject | Matches |
+-----------------------------------------------+
|The dog and the man play in the park.| FALSE |
|Man I love that dog! | TRUE |
|I'm dog tired | TRUE |
|The dog park is no place for man. | FALSE |
|park next to this dog's man. | FALSE |
+-------------------------------------+---------+
IDEONE demo