I am looking through some old code bases and have come across two regular expression parts that I think are semantically identical. Wondering it the Stackoverflow community can confirm my understanding.
RegEx 1: (.+?) - One or more characters, but optional
RegEx 2: (.*) - Zero or more characters
I keep thinking of different scenarios but can't think of any input where both expressions wouldn't be the same.
(.+?) means match one or more character, but instead of the default greedy matching (match as much as possible), the ? after the quantifier makes the matching lazy (match as few as possible).
Conceptually, greedy matching will first try the longest possible sequence that can be formed by the pattern inside, then gradually reduce the length of the sequence as the engine backtracks. Lazy matching will first try the shortest possible sequence that can be formed by the pattern inside, then gradually increase the length of the sequence as the engine backtracks.
Therefore, (.+?) and (.*) are completely different. Given a string "abc", the pattern (.+?) will match "a" for the first match, while (.*) will match "abc" for the first match.
When you correct the pattern to the intended meaning: ((?:.+)?), it is exactly the same as (.*) in behavior. Since quantifier is greedy by default, ((?:.+)?) will first try the case of .+ before attempting the case of empty string. And .+ will try the longest sequence before the 1 character sequence. Therefore, the effect of ((?:.+)?) is exactly the same as (.*): it will find the longest sequence and backtrack gradually to the case of empty string.
First,
. is any character
Next
* is zero or more
+ is one or more
? is one or zero
You're thinking that .+? is one or more of any character and 0 or 1 of them I'm guessing?
You're missing this:
Lazy modifier
*? is zero or more getting as few as possible
+? is one or more getting as few as possible
See here for further discussion
Greedy vs. Reluctant vs. Possessive Quantifiers
Related
I noticed that there are 3 different classes of quantifiers: greedy, lazy (i.e. non-greedy) and possessive.
I know that, loosely speaking, greedy quantifiers try to get the longest match by first reading in the entire input string and then truncate the characters one by one if the attempts keep failing; lazy quantifiers try to get the shortest match by first reading in the empty string and then add in the characters one by one if the attempts keep failing; possessive quantifiers try the same way as greedy quantifiers while they will stop matching if the first attempt fails.
However, I'm not sure how exactly the aboves are being implemented 'internally', and would like to ask for clarification (hopefully with examples).
For example, say we have the input string as "fooaaafoooobbbfoo".
If the regex is "foo.*" (greedy), will the foo in the regex first match the foo in the input string, and then .* reads in aaafoooobbbfoo as 'the entire string'? Or will .* first read in fooaaafoooobbbfoo as 'the entire string', and then truncates fooaaafoooobbbfoo to try matching the foo in the regex? If it is the latter, will fooaaafoooobbbfoo be truncated from its left or from its right in each attempt?
Will the answers to the above questions change if I replace "foo.*" with ".*foo" or "foo.*foo" as my regex? What about if I change those greedy quantifiers to lazy ones and possessive ones?
And if there are more than one quantifiers in a regex, how will the engine deal with the priority (if that matters)?
Thanks in advance!
For your input string fooaaafoooobbbfoo.
Case 1: When you're using this regex:
foo.*
First remember this fact that engine traverses from left to right.
With that in mind above regex will match first foo which is at the start of input and then .* will greedily match longest possible match which is rest of the text after foo till end. At this point matching stops as there is nothing to match after .* in your pattern.
Case 2: When you're using this regex:
.*foo
Here again .* will greedily match longest possible match before matching last foo which is right the end of input.
Case 3: When you're using this regex:
foo.*foo
Which will match first foo found in input i.e. foo at the start then .* will greedily match longest possible match before matching last foo which is right the end of input.
Case 4: When you're using this regex with lazy quantifier:
foo.*?foo
Which will match first foo found in input i.e. foo at the start then .*? will lazily match shortest possible match before matching next foo which is second instance of foo starting at position 6 in input.
Case 5: When you're using this regex with possessive quantifier:
foo.*+foo
Which will match first foo found in input i.e. foo at the start then .*+ is using possessive quantifier which means match as many times as possible, without giving back. This will match greedily longest possible match till end and since possessive quantifier doesn't allow engine to backtrack hence presence of foo at the end of part will cause failure as engine will fail to match last foo.
I am trying to understand Tcl subexpression matches and "greediness" and am completely stumped as to what's going on. Referencing the example found at http://wiki.tcl.tk/396:
%regexp -inline (.*?)(n+)(.*) ennui
en e n {}
%regexp -inline ^(.*?)(n+)(.*)$ ennui
ennui e nn ui
Notwithstanding the fact that I don't completely understand "nested expressions" (that is what the parenthesis indicate, right?) matching, I decided to start small and just try the difference between * and + as greedy operators:
% regexp -inline (.*)(u*)(.*) ennui
ennui ennui {} {}
% regexp -inline (.*)(u+)(.*) ennui
ennui enn u i
If * matches zero or more, and + matches one or more, I don't understand the difference in the output between the two commands. Why do u* and u+ produce two different results on the same string?
I feel like this is an extremely important nuance - that if I can grasp what's going on in this simple pattern match/regex, my life will be made whole. Help!
Thanks in advance.
Regarding the non-greediness. Tcl regular expressions have a quirk: the first quantifier in the expression sets the greediness for the whole expression. (See the "Matching" section of the re_syntax manual page, paying close attention to the word "preference"):
A branch has the same preference as the first quantified atom in it which has a preference.
%regexp -inline (.*?)(n+)(.*) ennui
en e n {}
(.*?) grabs zero or more characters, preferring the shortest match
(n+) grabs one or more n, inheriting the shortest preference
(.*) grabs zero or more characters, inheriting the shortest preference
The first subexpression matches from the first character up to but not including the first n. The 2nd part matches one n. The 3rd part matches zero characters between the first and the second n.
I'm a bit surprised that the first subexpression captured an e instead of capturing zero characters before the first n, but that can be explained by the higher priority of "leftmost" matching to the regex engine:
In the event that an RE could match more than one substring of a given string, the RE matches the one starting earliest in the string.
The achored expression's results surprises me too: I would have expected e n nui instead of e nn ui. Adding the $ anchor seems to have discarded the expression's preference for shortest matching.
The reason for the (.*)(u*)(.*) and (.*)(u+)(.*) difference is that the second regex requires at least 1 u.
The ARE regex in Tcl uses backtracking (as most NFAs). With (.*), the engine grabs the whole string from the beginning to end, and starts backtracking to find if it can accommodate for the next subpattern.
In the first expression, u is optional (can be 0 due to *), thus, the greedy .* decides it won't yield any characters. Then, the last .* can also match 0 characters, again, no need to give any characters to that group.
In the second expression, the u is obligatory, must occur at least once. Thus, the engine grabs all the string with the first .*, then backtracks, and founds u. So, it puts the starting sequence to group 1, and matches and captures u with (u+). Since u is only 1, the last (.*) matches and captures the rest of the string.
#stribizhev answer pretty much explains everything. As for your non-greedy version — the question mark at end tells the engine that it shouldn't consume the whole string, but grab the least possible match and continue from there.
(.*?) for "ennui" matches 0 characters and it's ok, since we're not greedy
(n+) for "ennui" match fails, so the engine returns to matching (.*?) again
(.*?) for "ennui" now matches one character e
(n+) for "nnui" matches nn since its greedy
(.*) for "ui" matches whats left, ui
Can someone please illustrate it what is the difference between the following regular expressions?
/b\w+?/g
and
/b\w/g
based on the documentation of regexr for Lazy ?:Makes the preceding quantifier lazy, causing it to match as few characters as possible. By default, quantifiers are greedy, and will match as many characters as possible.
Both regexes will match the same thing (one "alphanumeric" character). The difference between them becomes noticeable only when the regex becomes more complex.
For example,
\b\w\.
will match "A." but not "Mr." because \w matches only one character.
\b\w+?\.
will match both. This hasn't yet to do with the laziness but with the lack of a quantifier in the first regex. But if you compare
\b\w+\d
and
\b\w+?\d
you'll notice an important difference: Both regexes will match the string "R2D2" differently - the first one matches "R2D2", the second one only matches "R2" because the ? tells the \w+ token to match as few characters as possible, so it stops the match after R (even though it could match more. It will only match more if the regex wouldn't match otherwise).
I'm trying to split up a string into two parts using regex. The string is formatted as follows:
text to extract<number>
I've been using (.*?)< and <(.*?)> which work fine but after reading into regex a little, I've just started to wonder why I need the ? in the expressions. I've only done it like that after finding them through this site so I'm not exactly sure what the difference is.
On greedy vs non-greedy
Repetition in regex by default is greedy: they try to match as many reps as possible, and when this doesn't work and they have to backtrack, they try to match one fewer rep at a time, until a match of the whole pattern is found. As a result, when a match finally happens, a greedy repetition would match as many reps as possible.
The ? as a repetition quantifier changes this behavior into non-greedy, also called reluctant (in e.g. Java) (and sometimes "lazy"). In contrast, this repetition will first try to match as few reps as possible, and when this doesn't work and they have to backtrack, they start matching one more rept a time. As a result, when a match finally happens, a reluctant repetition would match as few reps as possible.
References
regular-expressions.info/Repetition - Laziness instead of Greediness
Example 1: From A to Z
Let's compare these two patterns: A.*Z and A.*?Z.
Given the following input:
eeeAiiZuuuuAoooZeeee
The patterns yield the following matches:
A.*Z yields 1 match: AiiZuuuuAoooZ (see on rubular.com)
A.*?Z yields 2 matches: AiiZ and AoooZ (see on rubular.com)
Let's first focus on what A.*Z does. When it matched the first A, the .*, being greedy, first tries to match as many . as possible.
eeeAiiZuuuuAoooZeeee
\_______________/
A.* matched, Z can't match
Since the Z doesn't match, the engine backtracks, and .* must then match one fewer .:
eeeAiiZuuuuAoooZeeee
\______________/
A.* matched, Z still can't match
This happens a few more times, until finally we come to this:
eeeAiiZuuuuAoooZeeee
\__________/
A.* matched, Z can now match
Now Z can match, so the overall pattern matches:
eeeAiiZuuuuAoooZeeee
\___________/
A.*Z matched
By contrast, the reluctant repetition in A.*?Z first matches as few . as possible, and then taking more . as necessary. This explains why it finds two matches in the input.
Here's a visual representation of what the two patterns matched:
eeeAiiZuuuuAoooZeeee
\__/r \___/r r = reluctant
\____g____/ g = greedy
Example: An alternative
In many applications, the two matches in the above input is what is desired, thus a reluctant .*? is used instead of the greedy .* to prevent overmatching. For this particular pattern, however, there is a better alternative, using negated character class.
The pattern A[^Z]*Z also finds the same two matches as the A.*?Z pattern for the above input (as seen on ideone.com). [^Z] is what is called a negated character class: it matches anything but Z.
The main difference between the two patterns is in performance: being more strict, the negated character class can only match one way for a given input. It doesn't matter if you use greedy or reluctant modifier for this pattern. In fact, in some flavors, you can do even better and use what is called possessive quantifier, which doesn't backtrack at all.
References
regular-expressions.info/Repetition - An Alternative to Laziness, Negated Character Classes and Possessive Quantifiers
Example 2: From A to ZZ
This example should be illustrative: it shows how the greedy, reluctant, and negated character class patterns match differently given the same input.
eeAiiZooAuuZZeeeZZfff
These are the matches for the above input:
A[^Z]*ZZ yields 1 match: AuuZZ (as seen on ideone.com)
A.*?ZZ yields 1 match: AiiZooAuuZZ (as seen on ideone.com)
A.*ZZ yields 1 match: AiiZooAuuZZeeeZZ (as seen on ideone.com)
Here's a visual representation of what they matched:
___n
/ \ n = negated character class
eeAiiZooAuuZZeeeZZfff r = reluctant
\_________/r / g = greedy
\____________/g
Related topics
These are links to questions and answers on stackoverflow that cover some topics that may be of interest.
One greedy repetition can outgreed another
Regex not being greedy enough
Regular expression: who's greedier
It is the difference between greedy and non-greedy quantifiers.
Consider the input 101000000000100.
Using 1.*1, * is greedy - it will match all the way to the end, and then backtrack until it can match 1, leaving you with 1010000000001.
.*? is non-greedy. * will match nothing, but then will try to match extra characters until it matches 1, eventually matching 101.
All quantifiers have a non-greedy mode: .*?, .+?, .{2,6}?, and even .??.
In your case, a similar pattern could be <([^>]*)> - matching anything but a greater-than sign (strictly speaking, it matches zero or more characters other than > in-between < and >).
See Quantifier Cheat Sheet.
Let's say you have:
<a></a>
<(.*)> would match a></a where as <(.*?)> would match a.
The latter stops after the first match of >. It checks for one
or 0 matches of .* followed by the next expression.
The first expression <(.*)> doesn't stop when matching the first >. It will continue until the last match of >.
How would the '.+?' regular expression work? Is the .+ part matching anything written, and the ? part saying it can either be there or not? So, for example, this regular expression would match:
'cat'
'' (ie, nothing written, just the empty string)
The "+?" is not a "+" quantifier followed by a "?" quantifier. Instead the "?" modifies the "+" to perform a "lazy" or "non greedy" match, meaning that the least number of characters that match is already sufficient.
So a "a+?" regex would match just a single "a" in "caaat".
Besides what Hans Kesting already said, a lazy multiplier will do the exact oposite of the normal greedy multipliers: The possible match is kept as small as possible and the rest of the regular expression is tested.
So if you’re having the string aaba and test the regular expression a.*b on it, the internal processing steps would be as follows:
a in a.*b matches aaba
.* in a.*b matches aaba, and since .* is greedy
.* then matches aaba
.* then matches aaba
b in a.*b fails as there is no letter left
backtracking goes one step back and .* will now only match bb in aaba
b in a.*b still fails on aaba
backtracking goes one step back and .* now matches only b in aaba
b in a.*b now matches b in aaba and we’re done.
So the full match is aaba.
If we do the same with a lazy multiplier (a.*?b), the processing will do the oposite, try to match the least possible characters as possible:
a in a.*?b matches aaba
.* in a.*?b matches nothing (* = zero or more repetitions), and since .* is declared as lazy (.*?), the rest of the regular expression is tested
b in a.*?b fails on aaba
backtracking will try to increase the match of .*
.* matches now aaba
b in a.*?b matches aaba and we’re done.
So the full match if aaba.
+? (lazy plus)
Repeats the previous item once or
more. Lazy, so the engine first
matches the previous item only once,
before trying permutations with ever
increasing matches of the preceding
item.
/".+?"/ matches "def" (and "ghi") in abc "def" "ghi" jkl, while /".+"/ matches "def" "ghi".
You can find more info here
There is documentation on how Perl handles these quantifiers perldoc perlre.
By default, a quantified subpattern is "greedy", that is, it will match as many times as possible (given a particular starting location) while still allowing the rest of the pattern to match. If you want it to match the minimum number of times possible, follow the quantifier with a "?". Note that the meanings don't change, just the "greediness":
*? Match 0 or more times, not greedily
+? Match 1 or more times, not greedily
?? Match 0 or 1 time, not greedily
{n}? Match exactly n times, not greedily
{n,}? Match at least n times, not greedily
{n,m}? Match at least n but not more than m times, not greedily
By default, when a quantified subpattern does not allow the rest of the overall pattern to match, Perl will backtrack. However, this behaviour is sometimes undesirable. Thus Perl provides the "possessive" quantifier form as well.
*+ Match 0 or more times and give nothing back
++ Match 1 or more times and give nothing back
?+ Match 0 or 1 time and give nothing back
{n}+ Match exactly n times and give nothing back (redundant)
{n,}+ Match at least n times and give nothing back
{n,m}+ Match at least n but not more than m times and give nothing back
For instance,
'aaaa' =~ /a++a/
will never match, as the a++ will gobble up all the a 's in the string and won't leave any for the remaining part of the pattern. This feature can be extremely useful to give perl hints about where it shouldn't backtrack. For instance, the typical "match a double-quoted string" problem can be most efficiently performed when written as:
/"(?:[^"\\]++|\\.)*+"/
as we know that if the final quote does not match, backtracking will not help. See the independent subexpression (?>...) for more details; possessive quantifiers are just syntactic sugar for that construct. For instance the above example could also be written as follows:
/"(?>(?:(?>[^"\\]+)|\\.)*)"/
link
inevitably, the regex is going to look for at least one character. I've come across case where an empty string wouldn't pass that test already, it would be better to use .*? or (.*)? instead, sometimes you have to specify the part of the string which may be null in braces before the question mark, it helps. E.g. \d{6}? will yield a wrong result, whereas if I had said (\d{6})? in a string say for example:
preg_match("/shu\.(\d{6})?/", "shu.321456")
this will yield true and so will the string "shu." without any int after the period