Not understanding a RegEx with Non-Capturing Group - regex

Question

I need help in understanding why a specific regular expression works and another one doesn't .. here's the background:
I want to use the ansible lineinfile module to add/modify the -u ntpd:ntpd option to the OPTIONS="" field in /etc/sysconfig/ntpd.
I see three cases:
No line with OPTIONS at all --> add OPTIONS="-u ntpd:ntpd"
OPTIONS line already there, but not with that specific option --> add the option to the existing options
OPTIONS line existing, -u option existing but with wrong parameters --> change the parameters
My first shot was this:
- name: Make sure ntpd runs as user:group ntpd:ntpd
lineinfile:
path: /etc/sysconfig/ntpd
regexp: 'OPTIONS=\"(.*)(-u\s[A-Za-z0-9\-_]+:[A-Za-z0-9\-_]+)?(.*)\"'
line: 'OPTIONS="\1 \3 -u ntpd:ntpd'
backrefs: yes
But the first group contains everything between "" Example at regex101
Changing the 1st and 3rd group to lazy makes everything between "" belong to the third group Example at regex101
After trying around and looking at it with a colleague we came up with this regular expression which does what we want:
OPTIONS=\"(?:(.*)(-u [a-z]+:[a-z]+))?(.*?)\"
Example at regex101
But honestly, we do not understand why. Anyone out there who can shed some light on this?

Answer 1

Your culprits are the wildcards in your first and third matching groups! Let's have a look at each one. We'll use OPTIONS="-x -u wrong:wrong -c blah" as the string to test.
OPTIONS=\"(.*)(-u\s[A-Za-z0-9\-_]+:[A-Za-z0-9\-_]+)?(.*)\"
This regex starts with a greedy wildcard, and in a way, that's where it ends, too. Between your two quote marks must appear:
First capture group: as much of anything as it wants. This group can, will, and does eat your entire string, leaving nothing else behind. It'll give back if it has to, but...
Second capture group: the interesting part of the regex. Normally, this would capture what you want to see. However, since it's marked with the ? quantifier, it's allowed to exist zero or one times - and the greedy quantifier is happy to keep its catch if the second capture group is not required to exist.
Third capture group: same issue here. Since .* means "zero or more" of any character, and the first greedy wildcard ate them all, it's happy to match nothing, which is why your first regex example has the dotted red line just before the closing quote - it signifies group 3's empty match.
Result: the first greedy wildcard ate everything, and none of the other capture groups made it a requirement to give something back.
OPTIONS=\"(.*?)(-u\s[A-Za-z0-9\-_]+:[A-Za-z0-9\-_]+)?(.*?)\"
In contrast, your second regex contains lazy wildcards. These won't eat anything if they don't have to.
First CG: doesn't capture anything if it doesn't specifically have to. It won't capture anything unless we get to the very end of the string and find no matches, at which point it wakes up and starts eating things until a match can be found.
Second CG: again, it can exist, but because of the ? quantifier, it doesn't have to. Since it can't instantly match the -x at the start of the string, it decides to try the final CG.
Third CG: doesn't capture anything if it doesn't specifically have to.
Except... now there's no match. At this point, the regex engine starts working backwards.
Third CG: starts capturing things, increasing until it hits a match. In this case, since your second group doesn't immediately match, it assumes it won't match at all (instead of checking to see if it can make something work with the first wildcard CG) and instead eats all the text. This counts as a match, and the engine is satisfied.
Second CG - tries to match, but doesn't if it can't immediately do so. Note that on the second line, where the first text after OPTIONS=" is a match, this group does activate.
First CG: never reached, the third one had it handled.
Result: you're getting closer, but because the lazy wildcard doesn't capture unless it has to, and the engine is more willing to let your third CG eat everything than it is to try and force a match between the second and first groups, the third group gets it all.
OPTIONS=\"(?:(.*)(-u [a-z]+:[a-z]+))?(.*?)\"
Now you got it. The non-capture group at the start treats the first two capture groups as the same entity. Let's look at what happens now:
First NCG:
First nested CG: matches the entire string.
Second nested CG: forces the first nested CG to surrender some text so it can match. There's no ? quantifier this time, so it's not an option: the second CG must match. Because the entire NCG must try to find a match before the rest of the regex can continue matching, if a match exists, it's guaranteed to be properly found.
Second CG: cleans up whatever the first NCG left behind.
Result: you get the string. Woo!
Let's take a closer look at your non-capturing group and see why it works.
(?:(.*)(-u [a-z]+:[a-z]+))?
(.*)(-u [a-z]+:[a-z]+)
Notice something? Your ? quantifier is outside the non capture group. This means that inside the NCG, your second capture group is no longer optional. The regex is forced to try to match your second CG, and if it can't, the ? quantifier outside the NCG cancels the entire thing, including the first wildcard CG. This means that the first wildcard CG can't be used to gobble text if the engine feels like ignoring your second CG - it can now only be used to assist the second CG in matching, as was likely your original intent.
The ? quantifier on the second CG was necessary, because there was no guarantee the command would exist. However, this gave the engine the option of being lazy and just ignoring it altogether - which it will gladly do, especially if you place it right next to a wildcard CG.
As an aside, if you change your regex like so, and put the ? quantifier back where it was, next to your second CG...
OPTIONS=\"(?:(.*)(-u [a-z]+:[a-z]+))?(.*?)\" works
OPTIONS=\"(?:(.*)(-u [a-z]+:[a-z]+)?)(.*?)\" uh oh
^^
...It ignores your second group again, and you're right back where you started.
Good luck ;)

Answer 2

First regex : the first capturing group's .* is greedy, it will match as much as possible, and it is possible to match everything up to the double-quote since everything else that follows is optional.
Second regex : the first capturing group's .*? is lazy, it will match as little as possible. It first tries matching nothing, which is possible since the last capturing group will be able to match the rest of the string.
Third regex : the non-capturing group is greedy and will try to match if possible, and its -u group is mandatory. The regex engine will match everything with .*, then backtrack until it's possible to match the -u group. If there's no -u, it leaves the whole thing to match to the third group.
Note that the laziness of the third group isn't necessary in your third regex, and that making the first .* lazy will improve performances : https://regex101.com/r/xvYOoF/7
As an alternative the following regex that makes sure the first capturing group can't match a -u option might be more understandable and will be more performant especially on long options strings (not that it matters I guess) :
OPTIONS="((?:-[^u]|[^-])*)(-u \w+:\w+)?(.*)"