how do the regular expressions * and ? metacharacter work? - regex

Hi I'm going through regular expressions but I'm confused about metacharacters, particularly '*' and '?'.
'*' is supposed to match the preceding character 0 or more times.
For example, 'ta*k' supposedly matches 'tak' and 'tk'.
But I wouldn't have thought this to be true at all - here's my reasoning:
for tak:
regexp: I need a 't'
string: I have 't'
regexp: okay, your next character needs to be an 'a'
string: yes it is
regexp: okay, keep giving me characters until your character isn't an 'a'
string: okay. I've just given you 'k'
regexp: okay, your next character needs to be a 'k'
string: I don't have any more characters left!
regexp: fail
for tk:
regexp: I need a 't'
string: I have 't'
regexp: okay, your next character needs to be an 'a'
string: no, it's a 'k'
regexp: fail
Can someone clarify for me why 'tak' and 'tk' matches 'ta*k'?

* does not mean to match a character zero or more times, but an atom zero or more times. A single character is an atom, but so is any grouping.
And * means zero or more. When the regex cursor has "swallowed" the t, the positions are:
in the regex: t|a*k
in the string: t|ak
The regex engine then tries and eats as as much as possible. Here there is one. After it has swallowed it, the positions are:
in the regex: ta*|k
in the string: ta|k
Then the k is swallowed:
in the regex: ta*k|
in the string: tak|
End of regex, match. Note that the string may have other characters behind, the regex engine doesn't care: it has a match.
In the case where the string is tk, before a* the positions are:
in the regex: t|a*k
in the string: t|k
But * can match an empty set of as, therefore a* is satisfied! Which means the positions then become:
in the regex: ta*|k
in the string: t|k
Rinse, repeat. Now, let's take taak as an input and ta?k as a regex: this will fail, but let's see how...
# before first character
regex: |ta?k
input: |taak
# t
regex: t|a?k
input: t|aak
# a?
regex: ta?|k
input: ta|ak
# k? Oops! No...
regex: |ta?k
input: t|aak
# t? Oops! No...
regex: |ta?k
input: ta|ak
# t? Oops! No...
regex: |ta?k
input: taa|k
# t? Oops! No...
regex: |ta?k
input: taak|
# t? Oops! No... And nothing to read anymore
# FAIL
Which is why it is VERY important to make regexes fail FAST.

Because a* means "zero or more instances of a".
When "it" asks for all characters that aren't "a", once it has one, it (roughly) pushes it back into the input stream. (Or it peeks ahead, or it just keeps it, etc.)
First sequence: here's your first non-"a", I'll hold on to that. You need a "k" next, that's what I have.
Second sequence: the next character doesn't need to be an "a"--it may be one or more "a". In this case it's none. I'll hold on to that non-"a". You need a "k"? I got your "k" right here still.

You are one character ahead:
regexp: okay, keep giving me characters until your character isn't an
'a'
string: next character is not an 'a'
regexp: okay, your next character needs to be a 'k'
string: next char is a 'k'
So it works. Note that 'a*' means "0 or more occourrences of 'a'", and not "1 or more occources of 'a'". For the latter one there's the '+' sign, like in 'a+'.

ta*k means, one 't', followed by 0 or more 'a's, followed by one 'k'. So 0 'a' characters, would make 'tk` a possible match.
If you want "1 or more" instead of "0 or more", use the + instead of *. That is, ta+k will match 'tak' but not 'tk'.
Let me know if there's anything I didn't explain.
By the way, RegEx doesn't always go left to right. The engine often backtracks, peeks ahead and studies the input. It's really complicated, which is why it's so powerful. If you looks at sites such as this one, they sometimes explain what the engine is doing. I recommend their tutorials because that's where I learned about RegEx!

The fundamental thing to remember is that a regular expression is a convenient shorthand for typing out a set of strings. a{1,5} is simply shorthand for the set of strings (a, aa, aaa, aaaa, aaaaa). a* is shorthand for ([empty], a, aa, aaa, ...).
Thus, in effect, when you feed a regular expression to a search algorithm, you are telling it the list of strings to search for.
Consequently, when you feed ta*k to your search algorithm, you are actually feeding it the set of strings (tk, tak, taak, taaak, taaaak, ...).
So, yes, it is useful to understand how the search algorithm will work, so that you can offer the most efficient regular expression, but don't let the tail wag the dog.

Related

Shorten Regular Expression (\n) [duplicate]

I'd like to match three-character sequences of letters (only letters 'a', 'b', 'c' are allowed) separated by comma (last group is not ended with comma).
Examples:
abc,bca,cbb
ccc,abc,aab,baa
bcb
I have written following regular expression:
re.match('([abc][abc][abc],)+', "abc,defx,df")
However it doesn't work correctly, because for above example:
>>> print bool(re.match('([abc][abc][abc],)+', "abc,defx,df")) # defx in second group
True
>>> print bool(re.match('([abc][abc][abc],)+', "axc,defx,df")) # 'x' in first group
False
It seems only to check first group of three letters but it ignores the rest. How to write this regular expression correctly?
Try following regex:
^[abc]{3}(,[abc]{3})*$
^...$ from the start till the end of the string
[...] one of the given character
...{3} three time of the phrase before
(...)* 0 till n times of the characters in the brackets
What you're asking it to find with your regex is "at least one triple of letters a, b, c" - that's what "+" gives you. Whatever follows after that doesn't really matter to the regex. You might want to include "$", which means "end of the line", to be sure that the line must all consist of allowed triples. However in the current form your regex would also demand that the last triple ends in a comma, so you should explicitly code that it's not so.
Try this:
re.match('([abc][abc][abc],)*([abc][abc][abc])$'
This finds any number of allowed triples followed by a comma (maybe zero), then a triple without a comma, then the end of the line.
Edit: including the "^" (start of string) symbol is not necessary, because the match method already checks for a match only at the beginning of the string.
The obligatory "you don't need a regex" solution:
all(letter in 'abc,' for letter in data) and all(len(item) == 3 for item in data.split(','))
You need to iterate over sequence of found values.
data_string = "abc,bca,df"
imatch = re.finditer(r'(?P<value>[abc]{3})(,|$)', data_string)
for match in imatch:
print match.group('value')
So the regex to check if the string matches pattern will be
data_string = "abc,bca,df"
match = re.match(r'^([abc]{3}(,|$))+', data_string)
if match:
print "data string is correct"
Your result is not surprising since the regular expression
([abc][abc][abc],)+
tries to match a string containing three characters of [abc] followed by a comma one ore more times anywhere in the string. So the most important part is to make sure that there is nothing more in the string - as scessor suggests with adding ^ (start of string) and $ (end of string) to the regular expression.
An alternative without using regex (albeit a brute force way):
>>> def matcher(x):
total = ["".join(p) for p in itertools.product(('a','b','c'),repeat=3)]
for i in x.split(','):
if i not in total:
return False
return True
>>> matcher("abc,bca,aaa")
True
>>> matcher("abc,bca,xyz")
False
>>> matcher("abc,aaa,bb")
False
If your aim is to validate a string as being composed of triplet of letters a,b,and c:
for ss in ("abc,bbc,abb,baa,bbb",
"acc",
"abc,bbc,abb,bXa,bbb",
"abc,bbc,ab,baa,bbb"):
print ss,' ',bool(re.match('([abc]{3},?)+\Z',ss))
result
abc,bbc,abb,baa,bbb True
acc True
abc,bbc,abb,bXa,bbb False
abc,bbc,ab,baa,bbb False
\Z means: the end of the string. Its presence obliges the match to be until the very end of the string
By the way, I like the form of Sonya too, in a way it is clearer:
bool(re.match('([abc]{3},)*[abc]{3}\Z',ss))
To just repeat a sequence of patterns, you need to use a non-capturing group, a (?:...) like contruct, and apply a quantifier right after the closing parenthesis. The question mark and the colon after the opening parenthesis are the syntax that creates a non-capturing group (SO post).
For example:
(?:abc)+ matches strings like abc, abcabc, abcabcabc, etc.
(?:\d+\.){3} matches strings like 1.12.2., 000.00000.0., etc.
Here, you can use
^[abc]{3}(?:,[abc]{3})*$
^^
Note that using a capturing group is fraught with unwelcome effects in a lot of Python regex methods. See a classical issue described at re.findall behaves weird post, for example, where re.findall and all other regex methods using this function behind the scenes only return captured substrings if there is a capturing group in the pattern.
In Pandas, it is also important to use non-capturing groups when you just need to group a pattern sequence: Series.str.contains will complain that this pattern has match groups. To actually get the groups, use str.extract. and
the Series.str.extract, Series.str.extractall and Series.str.findall will behave as re.findall.

Is there a more efficient RegEx for solving Wordle?

So I have a list of all 5 letter words in the English language that I can interrogate when I'm really stuck at Wordle. I found this an excellent exercise for brushing up on my Regular Expressions in BBEDIT, which is what I tell myself I'm doing.
The way wordle works, I can have three conditions.
A letter that is somewhere in the word (and must be present)
A letter that is not present in the word
A letter that is correct in presence and position
Condition 3 is easy. If my start word "crone" has the n in the right place, my pattern is
...n.
And I can add condition 2 fairly easily with
^(?!.*[croe])...n.
If my next guess is "burns" I'll know there's an "s"
^(?!.*[croebur])^(?=.*s)...n.
And that it's not in the last position:
^(?!.*[croebur])^(?=.*s)...n[^s]
If my next (very poor) guess is 'stone' I'll know there's a 't'.
^(?!.*[croebur])^(?=.*s)^(?=.*t)sa.n.
So that's a workable formula.
But if my next guess were "wimpy" I'd know there was an 'i' in the answer, but I have to add an additional ^(?=.*i) which just feels inefficient. I tried grouping the letters that must be in the word by using a bracket set, ^(?=.*[ist]) but of course that will match targets that contain any one of those characters rather than all.
Is there a more efficient way to express the phrase "the word must contain all of the following letters to match" than a series of "start at the beginning, scan for occurence of this single character until the end" phrases?
If you enter a word into Wordle, it displays all the matched characters in your word. It also shows the characters which exist in the word but not in the correct order.
Considering these requirements, I think you should create different rules for each letter's place. This way, your regex pattern keeps simple, and you get the search results quickly. Let me give an example:
Input word: crone
Matched Characters: ...n.
Characters in the wrong place: -
Next regex search pattern: ^[^crone][^crone][^crone]n[^crone]$
Input word: burns
Matched Characters: ...n.
Characters in the wrong place: s
Next regex search pattern: ^(?=\S*[s]\S*)[^bucrone][^bucrone][^bucrone]n[^bucrones]$ (Be careful, there is an "s" character in the last parenthesis because we know its place isn't there.)
Input word: stone
Matched Characters: s..n.
Characters in the wrong place: t
Next regex search pattern: ^(?=\S*[t]\S*)s[^tsbucrone][^sbucrone]n[^sbucrones]$ (Be careful, there is a "t" character in the first parenthesis because we know its place isn't there.)
^ => Start of the line
[^abc] => Any character except "a" and "b" and "c"
(?=\S*[t]\S*)=> There must be a "t" character in the given string
(?=\S*[t]\S*)(?=\S*[u]\S*)=> There must be "t" and "u" characters in the given string
$ => End of the line
When we look at performance tests of the regex patterns with a seven-word sample, my regex pattern found the result in 130 steps, whereas your pattern in 175 steps. The performance difference will increase as the word-list increase. You can review it from the following links:
Suggested pattern: https://regex101.com/r/mvHL3J/1
Your pattern: https://regex101.com/r/Nn8EwL/1
Note: You need to click the "Regex Debugger" link in the left sidebar to see the steps.
Note 2: I updated my response to fix the bug in the following comment.

Substitute every character after a certain position with a regex

I'm trying to use a regex replace each character after a given position (say, 3) with a placeholder character, for an arbitrary-length string (the output length should be the same as that of the input). I think a lookahead (lookbehind?) can do it, but I can't get it to work.
What I have right now is:
regex: /.(?=.{0,2}$)/
input string: 'hello there'
replace string: '_'
current output: 'hello th___' (last 3 substituted)
The output I'm looking for would be 'hel________' (everything but the first 3 substituted).
I'm doing this in Typescript, to replace some old javascript that is using ugly split/concatenate logic. However, I know how to make the regex calls, so the answer should be pretty language agnostic.
If you know the string is longer than given position n, the start-part can be optionally captured
(^.{3})?.
and replaced with e.g. $1_ (capture of first group and _). Won't work if string length is <= n.
See this demo at regex101
Another option is to use a lookehind as far as supported to check if preceded by n characters.
(?<=.{3}).
See other demo at regex101 (replace just with underscore) - String length does not matter here.
To mention in PHP/PCRE the start-part could simply be skipped like this: ^.{1,3}(*SKIP)(*F)|.

Why the * regular expression indicates what can or cannot be it's previous character

Take this for an example which I found in some blog,
"How about searching for apple word which was spelled wrong in a given file where apple is misspelled as ale, aple, appple, apppple, apppppple etc. To find all patterns
grep 'ap*le' filename
Readers should observe that the above pattern will match even ale word as * indicates 0 or more of previous character occurrence."
Now it's saying that "ale" will be accept when we are having ap*le, isn't the "ap" and "le" fixed?
The * is a quantifier meaning 0 or more times for the previous pattern -- in this case a single literal p. You can also state the same as * with a quantifier:
ap{0,}le
The interesting question sometimes is 'what is the previous pattern?' It is often helpful to put a pattern in a group to aid understand of what the 'previous pattern' is.
Consider wanting to find any of:
ale, aple, appple, apppple, apppppple, able, abbbbbbble
Your first try might be:
/ap|b*le/
^ literal 'p' is the first alternative #WRONG regex will use 'ap'
^ or
^ literal 'b'
Demo
What you want in this case is:
/a(?:p|b)*le/
Demo
If you do not want to match ale and only match aple, appple, apppple, apppppple, use the + instead of the * which means one or more:
/ap+le/
And is equivalent to /ap{1,}le/
Demo
And if you want to only match aple, appple and leave out the variants with more than 3 'p's use the additional max quantifier:
/ap{1,3}le/
All the variants above will match apple correctly spelled. If you what only aple, appple, and not match apple, use alteration:
/a(?:p|p{3})le/
Demo
No its not.
"*" in your case means zero or any occurrence of p. While a and le is fixed. If you need fixed ap and le then this is what you need:
ap+le
"+" means at least once but no limit on number of occurrences.
This means now any number of p after a but before l. So it wont select ale now.

regular expression is chopping off last character of filename

Anyone know why this is happening:
Filename: 031\_Lobby.jpg
RegExp: (\d+)\_(.*)[^\_e|\_i]\.jpg
Replacement: \1\_\2\_i.jpg
That produces this:
031\_Lobb\_i.jpg
For some reason it's chopping the last character from the second back-
reference (the "y" in "Lobby". It doesn't do that when I remove the [^_e|_i] so I must be doing something wrong that's related to that.
Thanks!
You force it to chop off the last character with this part of your regex:
[^_e|_i]
Which translates as: Any single character except "_", "e", "|", "i".
The "y" in "Lobby" matches this criterion.
You mean "not _e" and "not _i", obviously, but that's not the way to express it. This would be right:
(\d+)_(.+)(?<!_[ei])\.jpg
Note that the dot needs to be escaped in regular expressions.
it is removing the "y" because [^_e|_i] matches the y, and the .* matches everything before the y.
You're forcing it to have a last character different from _e and _i. You should use this instead (note the last *):
(\d+)_(.*)[^_e|_i]*.jpg