Regular Expression to match groups that may not exist - regex

I'm trying to capture some data from logs in an application. The logs look like so:
*junk* [{count=240.0, state=STATE1}, {count=1.0, state=STATE2}, {count=93.0, state=STATE3}, {count=1.0, state=STATE4}, {count=1147.0, state=STATE5}, etc. ] *junk*
If the count for a particular state is ever 0, it actually won't be in the log at all, so I can't guarantee the ordering of the objects in the log (The only ordering is that they are sorted alphabetically by state name)
So, this is also a potential log:
*junk* [{count=240.0, state=STATE1}, {count=1.0, state=STATE4}, {count=1147.0, state=STATE5}, etc. ] *junk*
I'm somewhat new to using regular expressions, and I think I'm overdoing it, but this is what I've tried.
^[^=\n]*=(?:(?P<STATE1>\d+)(?=\.0,\s+\w+=STATE1))*.*?=(?P<STATE2>\d+)(?=\.0,\s+\w+=STATE2)*.*?=(?P<STATE3>\d+)(?=\.0,\s+\w+=STATE3)
The idea being that I'll loook for the '=' and then look ahead to see if this is for the state that I want, and it may or may not be there. Then skip all the junk after the count until the next state that I'm interested in(this is the part that I'm having issues with I believe). Sometimes it matches too far, and skips the state I'm interested in, giving me a bad value. If I use the lazy operator(as above), sometimes it doesn't go far enough and gets the count for a state that is before the one I want in the log.

See if this approach works for you:
Regex: (?<=count=)\d+(?:\.\d+)?(?=, state=(STATE\d+))
Demo
The group will be your State# and Full match will be the count value

You might use 2 capturing groups to capture the count and the state.
To capture for example STATE1, STATE2, STATE3 and STATE5, you could specify the numbers using a character class with ranges and / or an alternation.
{count=(\d+(?:\.\d+)?), state=(STATE(?:[123]|5))}
Explanation
{count= Match literally
( Capture group 1
\d+(?:\.\d+)? Match 1+ digits with an optional decimal part
) Close group
, state= Match literally
( Capture group 2
STATE(?:[123]|5) Match STATE and specify the allowed numbers
)} Close group and match }
Regex demo
If you want to match all states and digits:
{count=(\d+(?:\.\d+)?), state=(STATE\d+)}
Regex demo

After some experimentation, this is what I've come up with:
The answers provided here, although good answers, don't quite work if your state names don't end with a number (mine don't, I just changed them to make the question easier to read and to remove business information from the question).
Here's a completely tile-able regex where you can add on as many matches as needed
count=(?P<GROUP_NAME_HERE>\d+(?=\.0, state=STATE_NAME_HERE))?
This can be copied and appended with the new state name and group name.
Additionally, if any of the states do not appear in the string, it will still match the following states. For example:
count=(?P<G1>\d+(?=\.0, state=STATE_ONE))?(?P<G2>\d+(?=\.0, state=STATE_TWO))?(?P<G3>\d+(?=\.0, state=STATE_THREE))?
will match states STATE_ONE and STATE_THREE with named groups G1 & G3 in the following string even though STATE_TWO is missing:
[{count=55.0, state=STATE_ONE}, {count=10.0, state=STATE_THREE}]
I'm sure this could be improved, but it's fast enough for me, and with 11 groups, regex101 shows 803 steps with a time of ~1ms
Here's a regex101 playground to mess with: https://regex101.com/r/3a3iQf/1
Notice how groups 1,2,3,4,5,6,7,9, & 11 match. 8 & 10 are missing and the following groups still match.

Related

Possible to limit to scope/range of a lookahead

We can check to see if a digit is in a password, for example, by doing something like:
(?=.*\d)
Or if there's a digit and lowercase with:
(?=.*\d)(?=.*[a-z])
This will basically go on "until the end" to check whether there's a letter in the string.
However, I was wondering if it's possible in some sort of generic way to limit the scope of a lookahead. Here's a basic example which I'm hoping will demonstrate the point:
start_of_string;
middle_of_string;
end_of_string;
I want to use a single regular expression to match against start_of_string + middle_of_string + end_of_string.
Is it possible to use a lookahead/lookbehind in the middle_of_string section WITHOUT KNOWING WHAT COMES BEFORE OR AFTER IT? That is, not knowing the size or contents of the preceding/succeeding string component. And limit the scope of the lookahead to only what is contained in that portion of the string?
Let's take one example:
start_of_string = 'start'
middle_of_string = '123'
end_of_string = 'ABC'
Would it be possible to check the contents of each part but limit it's scope like this?
string = 'start123ABC'
# Check to make sure the first part has a letter, the second part has a number and the third part has a capital
((?=.*[a-z]).*) # limit scope to the first part only!!
((?=.*[0-9]).*) # limit scope to only the second part.
((?=.*[A-Z]).*) # limit scope to only the last part.
In other words, can lookaheads/lookbehinds be "chained" with other components of a regex without it screwing up the entire regex?
UPDATE:
Here would be an example, hopefully this is more helpful to the question:
START_OF_STRING = 'abc'
Does 'x' exist in it? (?=.*x) ==> False
END_OF_STRING = 'cdxoy'
Does 'y' exist in it? (?=.*y) ==> True
FULL_STRING = START_OF_STRING + END_OF_STRING
'abcdxoy'
Is it possible to chain the two regexes together in any sort of way to only wok on its 'substring' component?
For example, now (?=.*x) in the first part of the string would return True, but it should not.
`((?=.*x)(?=.*y)).*`
I think the short answer to this is "No, it's not possible.", but am looking to hear from someone who understands this to tell why it is or isn't.
In .NET and javascript you could use a positive lookahead at the start of your string component and a negative lookbehind at the end of it to "constrain" the match. Example:
.*(?=.*arrow)(?<middle>.*)(?<=.*arrow).*
helloarrowxyz
{'middle': 'arrow'}
If in pcre, python, or other you would need to either have a fixed width lookahead to constraint it from going too far forward, such as what Wiktor Stribiżew says above:
.*(?=.{0,5}arrow)(?<middle>.{0,5}).*
Otherwise, it wouldn't be possible to do without either a fixed-width lookahead or a variable width look-behind.

Regex: Capture any (dynamic) amount of lines

I've been trying to match the following:
First Group:Line1,
Line2,
..
LineX
Second Group:Some_Sample_text
With this query:
First Group:(?<first_group>.+\n*\n)Second Group:(?<second_group>.*)
My main goal is to capture any amount of lines between Line1 and LineX (because I can't anticipate how many there'll be), but since there's no option to match the end of files I'll probably need to use the "\n" tokens. I've also tried with IF and THEN statements but I just can't get it to work.
Any ideas appreciated.
Here, we might want to design an expression that'd just pass newlines, such as
First Group:([\s\S]*)Second Group:(.*)
First Group:([\d\D]*)Second Group:(.*)
First Group:([\w\W]*)Second Group:(.*)
Demo 1
and we'd expand it to,
First Group:([\s\S]*)Second Group:([\s\S]*)
First Group:([\d\D]*)Second Group:([\d\D]*)
First Group:([\w\W]*)Second Group:([\w\W]*)
If our second group would have had multiple lines.
Demo 2
Advice
The fourth bird advises that:
You could make the charachter class non greedy to prevent over matching ([\s\S]*?)
which then the expression would become,
First Group:([\s\S]*?)Second Group:([\s\S]*)
for instance.
Demo 3

How to Match in a Strict Order Data that comes in a Random Order?

I'm quite new to regular expressions and I have the following target string resource which can sometimes differ slightly. For example, the string might be:
<TITLE>SomeTitle</TITLE>
<ITEM1>Item 1 text</ITEM>
<ITEM2>Item 2 text</ITEM2>
<ITEM3>Item 3 text</ITEM3>
And the next time the resource is requested, it's output might be:
<ITEM1>Item 1 text</ITEM>
<ITEM2>Item 2 text</ITEM2>
<ITEM3>Item 3 text</ITEM3>
<TITLE>SomeTitle</TITLE>
I want to capture the data between the two tags in order of the first example, so that the match would always match "SomeTitle" first, followed by the items. So if the search string was the second example, I need an expression that can first match "SomeTitle" and then somehow "reset" the position of the match to start from the beginning so I can then match the items.
I can achieve this with two different pattern searches, but was wondering if there is a way to do this in a single search pattern? Perhaps using lookaheads/lookbehinds and conditionals?
Capture Groups inside Lookaheads
Use this:
(?s)(?=.*<TITLE>(.*?)</)(?=.*<ITEM1>(.*?)</)(?=.*<ITEM2>(.*?)</)(?=.*<ITEM3>(.*?)</)
Even when the tokens are in a random order, you can see them in the right order by examining Capture Groups 1, 2, 3 and 4.
For instance, in the online regex demo, see how the input is in a random order, but the capture groups in the right pane are in the right order.
PCRE: How to use in a programming language
The PCRE library is used in several programming languages: for instance PHP, R, Delphi, and often C. Regardless of the language, the idea is the same: retrieve the capture groups.
As an example, here is how to do it in PHP:
$regex = '~(?s)(?=.*<TITLE>(.*?)</)(?=.*<ITEM1>(.*?)</)(?=.*<ITEM2>(.*?)</)(?=.*<ITEM3>(.*?)</)~';
if (preg_match($regex, $yourdata, $m)) {
$title = $m[1];
$item1 = $m[2];
$item2 = $m[3];
$item3 = $m[4];
}
else { // sorry, no match...
}

Complex regex situation

I have a results list that looks like this:
1lemon_king9mumu (2-1), YearofHell (2-0), kriswithak (2-1)0.44440.75000.4444
2mumu6lemon_king (1-2), MogwaiAC (2-0), Dathanja (2-1)0.66670.62500.5655
3MogwaiAC6Dathanja (2-0), mumu (0-2), Jebnarf (2-1)0.55560.57140.5417
4Jebnarf6YearofHell (2-1), kriswithak (2-0), MogwaiAC (1-2)0.44440.62500.4266
5YearofHell3Jebnarf (1-2), lemon_king (0-2), Mig82 (2-1)0.66670.37500.6012
6Dathanja3MogwaiAC (0-2), Mig82 (2-1), mumu (1-2)0.55560.37500.5417
7Mig823Bye, Dathanja (1-2), YearofHell (1-2)0.33330.42860.3750
8kriswithak0Jebnarf (0-2), lemon_king (1-2)0.83330.20000.6875
I want to be able to pull the username of the person AFTER the rank (first number) but it is mashed together with points gained by the player, as well as their first opponent.
For example, the first persons name is "Lemon_king", and his opponents were "Mumu", "YearofHell" and "Kriswithak". The numbers on the right are irrelevant for me, but the major problem I have is that the number of points won by the player is there. Lemon_King wins 9 points for first place. I would normally just get the name by looking for the string between 1 and 9, but players usernames can have a 9 in it as well.
Can anyone think of a good solution to this problem to be able to grab the persons username?
Thanks
I think you'd need a list of the usernames to compare against; it doesn't look like the results list is "regular" enough for a regular expression.
For example the line
7Mig823Bye, Dathanja
Could be "Mig82" 3 points vs "Bye, Dathanja", but it could also be "Mig8", 23 points, "Bye, Dathanja" or "Mig8", 2 points, "3Bye, Dathanja".
Is that correct? Because if it is, you aren't going to get away with a simple solution.
Edit: Wilson commented that getting the list of usernames might be an option. In that case, something like the following might work:
/^\d+?(username1|username2|username3)\d+?(username1|username2|username3)/
It will probably take some fiddling to get right.
Here's a plnkr demonstrating it with the data you provided: http://plnkr.co/edit/nJeGfbfHgvh5zJcTWRXS?p=preview
That said, a regex might not be the right tool for this job.
As far as I can tell, you want something like
(?x) # allow whitespace and comments just like
# any real programming language
^ # beginning of line
( \d+ ) # starts with one or more digits: CAPTURE 1
(?= \D ) # must have a non-digit following
( \w+ ) # capture one or more "word" characters: CAPTURE 2
( \d ) # next is a single digit: CAPTURE 3
(?= \D ) # must have a non-digit following
( \w+ ) # capture one or more "word" characters: CAPTURE 4
# now add things for the rest of the line if you want
Your username should now be in the second capture. I’ve been a tad more careful than strictly necessary, but if you end up munging this, you may need that. I’ve alos put all the captures in case you want to move stuff around or pull more stuff out.
Please provide a bit more information, if you want the thing between the first number and second number:
[0-9]+([^0-9])
The first group will contain the first username.
Please comment on this (so I check) an edit your question with more detail though.
I wouldnt use regex. It will be a pain to debug it, and you'll never be 100% certain you've covered all the edge cases.
Try doing 'manual' parsing using your language of choice's built in string manipulation functions.

Regex named capture group with multiple values

I seem to be having a tough regex week. Anyone that can save me from throwing my laptop out the window gets a virtual beer. I have some data in the form of:
... f=something group="First Group,Group2" foo=val ...
where the number of groups can vary. I need to capture each group entry to a named capture. Based on a previous post, The difference here is that I don't have a constant to key off of within the values (i.e. ID-1-1, ID-2-2 allows me to say ID-\d+-\d+ whereas these values could be pretty much anything). I've been trying a ton of stuff, but I tend to get matches that are far too greedy, or I (often) get these 2 values:
First Group
First Group,Group2
What I need is:
First Group
Group2
...
I'm currently trying regex such as this where I'm trying to anchor to the group=" portion, and not exceed the ending ":
(?:(?:group=\")|(?:\"))(?<group>(?:(.+)+?)
Hopefully someone can make my day a lot better...
Here's the PHP solution. Once again, regex doesn't like capturing the multiple values so we need to break it in to two searches. One extracts the group value, the next extracts each value from the group
$test = 'f=something group="First Group,Group2" foo=val';
$re = '/(?:group=)?\x22(?<group>(?:[^\x2C]+\x2C*)+)\x22/';
$_ = null;
if (preg_match($re,$test,$_))
echo "Group Contents: ".$_['group']."\r\n";
$__ = null;
$re = '/(?:^|\x2C)(?<value>(?:[^\x2C]+)+)/';
if (preg_match_All($re,$_['group'],$__))
echo "Group Values: ".print_r($__['value'],true);
Should be pretty easy to port in to another language, just extract the regexes out and manage them the way you normally would.