If I have some HTML like this:
<b>1<i>2</i>3</b>
And the following regex:
\<[^\>\/]+\>(.*?)\<\/[^\>]+\>
Then it will match:
<b>1<i>2</i>
I want it to only match HTML where the start and end tags are the same. Is there a way to do this?
Thanks,
Joe
Is there a way to do this?
Yes, certainly. Ignore those flippant non-answers that tell you it can’t be done. It most certainly can. You just may not wish to do so, as I explain below.
Numbered Captures
Pretending for the nonce that HTML <i> and <b> tags are always denude of attributes, and moreover, neither overlap nor nest, we have this simple solution:
#!/usr/bin/env perl
#
# solution A: numbered captures
#
use v5.10;
while (<>) {
say "$1: $2" while m{
< ( [ib] ) >
(
(?:
(?! < /? \1 > ) .
) *
)
</ \1 >
}gsix;
}
Which when run, produces this:
$ echo 'i got <i>foo</i> and <b>bar</b> bits go here' | perl solution-A
i: foo
b: bar
Named Captures
It would be better to use named captures, which leads to this equivalent solution:
#!/usr/bin/env perl
#
# Solution B: named captures
#
use v5.10;
while (<>) {
say "$+{name}: $+{contents}" while m{
< (?<name> [ib] ) >
(?<contents>
(?:
(?! < /? \k<name> > ) .
) *
)
</ \k<name> >
}gsix;
}
Recursive Captures
Of course, it is not reasonable to assume that such tags neither overlap nor nest. Since this is recursive data, it therefore requires a recursive pattern to solve. Remembering that the trival pattern to parse nested parens recursively is simply:
( \( (?: [^()]++ | (?-1) )*+ \) )
I’ll build that sort of recursive matching into the previous solution, and I’ll further toss in a bit interative processing to unwrap the inner bits, too.
#!/usr/bin/perl
use v5.10;
# Solution C: recursive captures, plus bonus iteration
while (my $line = <>) {
my #input = ( $line );
while (#input) {
my $cur = shift #input;
while ($cur =~ m{
< (?<name> [ib] ) >
(?<contents>
(?:
[^<]++
| (?0)
| (?! </ \k<name> > )
.
) *+
)
</ \k<name> >
}gsix)
{
say "$+{name}: $+{contents}";
push #input, $+{contents};
}
}
}
Which when demo’d produces this:
$ echo 'i got <i>foo <i>nested</i> and <b>bar</b> bits</i> go here' | perl Solution-C
i: foo <i>nested</i> and <b>bar</b> bits
i: nested
b: bar
That’s still fairly simple, so if it works on your data, go for it.
Grammatical Patterns
However, it doesn’t actually know about proper HTML syntax, which admits tag attributes to things like <i> and <b>.
As explained in this answer, one can certainly use regexes to parse markup languages, provided one is careful about it.
For example, this knows the attributes germane to the <i> (or <b>) tag. Here we defined regex subroutines used to build up a grammatical regex. These are definitions only, just like defining regular subs but now for regexes:
(?(DEFINE) # begin regex subroutine defs for grammatical regex
(?<i_tag_end> < / i > )
(?<i_tag_start> < i (?&attributes) > )
(?<attributes> (?: \s* (?&one_attribute) ) *)
(?<one_attribute>
\b
(?&legal_attribute)
\s* = \s*
(?:
(?"ed_value)
| (?&unquoted_value)
)
)
(?<legal_attribute>
(?&standard_attribute)
| (?&event_attribute)
)
(?<standard_attribute>
class
| dir
| ltr
| id
| lang
| style
| title
| xml:lang
)
# NB: The white space in string literals
# below DOES NOT COUNT! It's just
# there for legibility.
(?<event_attribute>
on click
| on dbl click
| on mouse down
| on mouse move
| on mouse out
| on mouse over
| on mouse up
| on key down
| on key press
| on key up
)
(?<nv_pair> (?&name) (?&equals) (?&value) )
(?<name> \b (?= \pL ) [\w\-] + (?<= \pL ) \b )
(?<equals> (?&might_white) = (?&might_white) )
(?<value> (?"ed_value) | (?&unquoted_value) )
(?<unwhite_chunk> (?: (?! > ) \S ) + )
(?<unquoted_value> [\w\-] * )
(?<might_white> \s * )
(?<quoted_value>
(?<quote> ["'] )
(?: (?! \k<quote> ) . ) *
\k<quote>
)
(?<start_tag> < (?&might_white) )
(?<end_tag>
(?&might_white)
(?: (?&html_end_tag)
| (?&xhtml_end_tag)
)
)
(?<html_end_tag> > )
(?<xhtml_end_tag> / > )
)
Once you have the pieces of your grammar assembled, you could incorporate those definitions into the recursive solution already given to do a much better job.
However, there are still things that haven’t been considered, and which in the more general case must be. Those are demonstrated in the longer solution already provided.
SUMMARY
I can think of only three possible reasons why you might not care to use regexes for parsing general HTML:
You are using an impoverished regex language, not a modern one, and so you have to recourse to essential modern conveniences like recursive matching or grammatical patterns.
You might such concepts as recursive and grammatical patterns too complicated for you to easily understand.
You prefer for someone else to do all the heavy lifting for you, including the heavy testing, and so you would rather use a separate HTML parsing module instead of rolling your own.
Any one or more of those might well apply. In which case, don’t do it this way.
For simple canned examples, this route is easy. The more robust you want this to work on things you’ve never seen before, the harder this route becomes.
Certainly you can’t do any of it if you are using the inferior, impoverished pattern matching bolted onto the side of languages like Python or even worse, Javascript. Those are barely any better than the Unix grep program, and in some ways, are even worse. No, you need a modern pattern matching engine such as found in Perl or PHP to even start down this road.
But honestly, it’s probably easier just to get somebody else to do it for you, by which I mean that you should probably use an already-written parsing module.
Still, understanding why not to bother with these regex-based approaches (at least, not more than once) requires that you first correctly implement proper HTML parsing using regexes. You need to understand what it is all about. Therefore, little exercises like this are useful for improving your overall understanding of the problem-space, and of modern pattern matching in general.
This forum isn’t really in the right format for explaining all these things about modern pattern-matching. There are books, though, that do so equitably well.
You probably don't want to use regular expressions with HTML.
But if you still want to do this you need to take a look at backreferences.
Basically it's a way to capture a group (such as "b" or "i") to use it later in the same regular expression.
Related issues:
RegEx match open tags except XHTML self-contained tags
Related
I am trying to modify some LaTeX Beamer code, and want to do a quick regex find a certain pattern that defines a block of code. For example, in
\only{
\begin{equation*}
\psi_{w}(B)(1-B)^{d}y_{t,w} = x^{T}_{t,w}\gamma_{w} + \eta_{w}(B)z_{t,w},
\label{eq:gen_arima}
\end{equation*}
}
I want to match just the \only{ and the final }, and nothing else, to remove them. Is that even possible?
Regexes cannot count, as expressed in this famous SO answer: RegEx match open tags except XHTML self-contained tags
In addition, LaTeX itself has a fairly complicated grammar (i.e. macro expansions?). Ideally, you'd use a parser of some kind, but perhaps that's a little overkill. If you're looking for a really simple, quick and dirty hack which will only work for a certain class of inputs you can:
Search for \only.
Increment a counter every time you see a { and decrement it every time you see a }. If there is a \ preceding the {, ignore it. (Fancy points if you look for an odd number of \.)
When counter gets to 0, you've found the closing }.
Again, this is not reliable.
I want to remove \only{ and }, and keep everything within it
On a PCRE (php), Perl, Notepad++ it's done like this:
For something as simple as this, all you need is
Find \\only\s*({((?:[^{}]++|(?1))*)})
Replace $2
https://regex101.com/r/U3QxGa/1
Explained
\\ only \s*
( # (1 start)
{
( # (2 start), Inner core to keep
(?: # Cluster group
[^{}]++ # Possesive, not {}
| # or,
(?1) # Recurse to group 1
)* # End cluster, do 0 to many times
) # (2 end)
}
) # (1 end)
I am trying to parse a csv file, and I am trying to access names regex in proto regex in Perl6. It turns out to be Nil. What is the proper way to do it?
grammar rsCSV {
regex TOP { ( \s* <oneCSV> \s* \, \s* )* }
proto regex oneCSV {*}
regex oneCSV:sym<noQuote> { <-[\"]>*? }
regex oneCSV:sym<quoted> { \" .*? \" } # use non-greedy match
}
my $input = prompt("Enter csv line: ");
my $m1 = rsCSV.parse($input);
say "===========================";
say $m1;
say "===========================";
say "1 " ~ $m1<oneCSV><quoted>; # this fails; it is "Nil"
say "2 " ~ $m1[0];
say "3 " ~ $m1[0][2];
Detailed discussion complementing Christoph's answer
I am trying to parse a csv file
Perhaps you are focused on learning Raku parsing and are writing some throwaway code. But if you want industrial strength CSV parsing out of the box, please be aware of the Text::CSV modules[1].
I am trying to access a named regex
If you are learning Raku parsing, please take advantage of the awesome related (free) developer tools[2].
in proto regex in Raku
Your issue is unrelated to it being a proto regex.
Instead the issue is that, while the match object corresponding to your named capture is stored in the overall match object you stored in $m1, it is not stored precisely where you are looking for it.
Where do match objects corresponding to captures appear?
To see what's going on, I'll start by simulating what you were trying to do. I'll use a regex that declares just one capture, a "named" (aka "Associative") capture that matches the string ab.
given 'ab'
{
my $m1 = m/ $<named-capture> = ( ab ) /;
say $m1<named-capture>;
# 「ab」
}
The match object corresponding to the named capture is stored where you'd presumably expect it to appear within $m1, at $m1<named-capture>.
But you were getting Nil with $m1<oneCSV>. What gives?
Why your $m1<oneCSV> did not work
There are two types of capture: named (aka "Associative") and numbered (aka "Positional"). The parens you wrote in your regex that surrounded <oneCSV> introduced a numbered capture:
given 'ab'
{
my $m1 = m/ ( $<named-capture> = ( ab ) ) /; # extra parens added
say $m1[0]<named-capture>;
# 「ab」
}
The parens in / ( ... ) / declare a single top level numbered capture. If it matches, then the corresponding match object is stored in $m1[0]. (If your regex looked like / ... ( ... ) ... ( ... ) ... ( ... ) ... / then another match object corresponding to what matches the second pair of parentheses would be stored in $m1[1], another in $m1[2] for the third, and so on.)
The match result for $<named-capture> = ( ab ) is then stored inside $m1[0]. That's why say $m1[0]<named-capture> works.
So far so good. But this is only half the story...
Why $m1[0]<oneCSV> in your code would not work either
While $m1[0]<named-capture> in the immediately above code is working, you would still not get a match object in $m1[0]<oneCSV> in your original code. This is because you also asked for multiple matches of the zeroth capture because you used a * quantifier:
given 'ab'
{
my $m1 = m/ ( $<named-capture> = ( ab ) )* /; # * is a quantifier
say $m1[0][0]<named-capture>;
# 「ab」
}
Because the * quantifier asks for multiple matches, Raku writes a list of match objects into $m1[0]. (In this case there's only one such match so you end up with a list of length 1, i.e. just $m1[0][0] (and not $m1[0][1], $m1[0][2], etc.).)
Summary
Captures nest;
A capture quantified by either * or + corresponds to two levels of nesting not just one.
In your original code, you'd have to write say $m1[0][0]<oneCSV>; to get to the match object you're looking for.
[1] Install relevant modules and write use Text::CSV; (for a pure Raku implementation) or use Text::CSV:from<Perl5>; (for a Perl plus XS implementation) at the start of your code. (talk slides (click on top word, eg. "csv", to advance through slides), video, Raku module, Perl XS module.)
[2] Install CommaIDE and have fun with its awesome grammar/regex development/debugging/analysis features. Or install the Grammar::Tracer; and/or Grammar::Debugger modules and write use Grammar::Tracer; or use Grammar::Debugger; at the start of your code (talk slides, video, modules.)
The match for <oneCSV> lives within the scope of the capture group, which you get via $m1[0].
As the group is quantified with *, the results will again be a list, ie you need another indexing operation to get at a match object, eg $m1[0][0] for the first one.
The named capture can then be accessed by name, eg $m1[0][0]<oneCSV>. This will already contain the match result of the appropriate branch of the protoregex.
If you want the whole list of matches instead of a specific one, you can use >> or map, eg $m1[0]>>.<oneCSV>.
I have the following string:
<Multicast ID="0/m1" Feed="EUREX-EMDI" IPPort="224.0.50.128:59098" State="CHECK" IsTainted="0" UncrossAfterGap="0" ManualUncrosses="0" AutoUncrosses="0" ExpectedSeqNo="-" />
I need to strip everything in this string apart from:
Feed="EUREX-EMDI"
State="CLOSED"
IsTainted="0"
I have managed to get "Feed="EUREX-EMDI"" with the following code:
s/^[^Feed]*(?=Feed)//;
So it now looks like:
Feed="EUREX-EMDI" IPPort="224.0.50.0:59098" State="CLOSED" IsTainted="0" UncrossAfterGap="0" ManualUncrosses="0" AutoUncrosses="0" ExpectedSeqNo="2191840" />
However I now don't know how to look for the next part "State="CLOSED"" in the string whilst ignoring my already found "Feed="EUREX-EMDI"" match
The perl idiom for this type of thing is a multiple assignment from regex capture groups. Assuming you can always count on the items of interest being in the same order and format (quoting):
($feed, $state, $istainted) = /.*(Feed="[^"]*").*(State="[^"]*").*(IsTainted="[^"]*")/;
Or if you only want to capture the (unquoted) values themselves, change the parentheses (capture groups):
($feed, $state, $istainted) = /.*Feed="([^"]*)".*State="([^"]*)".*(IsTainted="([^"]*)"/;
Please, don't try and parse XML with a regex. It's brittle. XML is contextual, and regular expression aren't. So at best, it's a dirty hack, and one that may one day break without warning for the most inane reasons.
See: RegEx match open tags except XHTML self-contained tags for more.
However, XML is structured, and it's actually quite easy to work with - provided you use something well suited to the job: A parser.
I like XML::Twig. XML::LibXML is also excellent, but has a bit of a steeper learning curve. (You also get XPath which is like regular expressions, but much more well suited for XML)
#!/usr/bin/env perl
use strict;
use warnings;
use XML::Twig;
#create a list of what we want to keep. This map just turns it
#into a hash.
my %keep = map { $_ => 1 } qw ( IsTainted State Feed );
#parse the XML. If it's a file, you may want "parsefile" instead.
my $twig = XML::Twig->parse( \*DATA );
#iterate the attributes.
foreach my $att ( keys %{ $twig->root->atts } ) {
#delete the attribute unless it's in our 'keep' list.
$twig->root->del_att($att) unless $keep{$att};
}
#print it. You may find set_pretty_print useful for formatting XML.
$twig->print;
__DATA__
<Multicast ID="0/m1" Feed="EUREX-EMDI" IPPort="224.0.50.128:59098" State="CHECK" IsTainted="0" UncrossAfterGap="0" ManualUncrosses="0" AutoUncrosses="0" ExpectedSeqNo="-" />
Outputs:
<Multicast Feed="EUREX-EMDI" IsTainted="0" State="CHECK"/>
That preserves the attributes, and gives you valid XML. But if you just want the values:
foreach my $att ( qw ( Feed State IsTainted ) ) {
print $att, "=", $twig->root->att($att),"\n";
}
This will strip all but those strings.
$str =~ s/(?s)(?:(?!(?:Feed|State|IsTainted)\s*=\s*".*?").)*(?:((?:Feed|State|IsTainted)\s*=\s*".*?")|$)/$1/g;
If you want to include a space separator, make the replacement ' $1'.
Explained
(?s) # Dot - all
(?: # To be removed
(?!
(?: Feed | State | IsTainted )
\s* = \s* " .*? "
)
.
)*
(?: # To be saved
( # (1 start)
(?: Feed | State | IsTainted )
\s* = \s* " .*? "
) # (1 end)
| $
)
If I have a lot of matches, for example in multi line mode, and I want to replace them with part of the match as well as a counter number that increments.
I was wondering if any regex flavor has such a variable. I couldn't find one, but I seem to remember something like that exists...
I'm not talking about scripting languages in which you can use callbacks for replacement. It's about being able to do this in tools like RegexBuddy, sublime text, gskinner.com/RegExr, ... much in the same way you can refer to captured substrings with \1 or $1.
FMTEYEWTK about Fancy Regexes
Ok, I’m going to go from the simple to the sublime. Enjoy!
Simple s///e Solution
Given this:
#!/usr/bin/perl
$_ = <<"End_of_G&S";
This particularly rapid,
unintelligible patter
isn't generally heard,
and if it is it doesn't matter!
End_of_G&S
my $count = 0;
Then this:
s{
\b ( [\w']+ ) \b
}{
sprintf "(%s)[%d]", $1, ++$count;
}gsex;
produces this
(This)[1] (particularly)[2] (rapid)[3],
(unintelligible)[4] (patter)[5]
(isn't)[6] (generally)[7] (heard)[8],
(and)[9] (if)[10] (it)[11] (is)[12] (it)[13] (doesn't)[14] (matter)[15]!
Interpolated Code in Anon Array Solution
Whereas this:
s/\b([\w']+)\b/##{[++$count]}=$1/g;
produces this:
#1=This #2=particularly #3=rapid,
#4=unintelligible #5=patter
#6=isn't #7=generally #8=heard,
#9=and #10=if #11=it #12=is #13=it #14=doesn't #15=matter!
Solution with code in LHS instead of RHS
This puts the incrementation within the match itself:
s/ \b ( [\w']+ ) \b (?{ $count++ }) /#$count=$1/gx;
yields this:
#1=This #2=particularly #3=rapid,
#4=unintelligible #5=patter
#6=isn't #7=generally #8=heard,
#9=and #10=if #11=it #12=is #13=it #14=doesn't #15=matter!
A Stuttering Stuttering Solution Solution Solution
This
s{ \b ( [\w'] + ) \b }
{ join " " => ($1) x ++$count }gsex;
generates this delightful answer:
This particularly particularly rapid rapid rapid,
unintelligible unintelligible unintelligible unintelligible patter patter patter patter patter
isn't isn't isn't isn't isn't isn't generally generally generally generally generally generally generally heard heard heard heard heard heard heard heard,
and and and and and and and and and if if if if if if if if if if it it it it it it it it it it it is is is is is is is is is is is is it it it it it it it it it it it it it doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't doesn't matter matter matter matter matter matter matter matter matter matter matter matter matter matter matter!
Exploring Boundaries
There are more robust approaches to word boundaries that work for plural possessives (the previous approaches don’t), but I suspect your mystery lies in getting the ++$count to fire, not with the subtleties of \b behavior.
I really wish people understood that \b isn’t what they think it is.
They always think it means there's white space or the edge of the string
there. They never think of it as \w\W or \W\w transitions.
# same as using a \b before:
(?(?=\w) (?<!\w) | (?<!\W) )
# same as using a \b after:
(?(?<=\w) (?!\w) | (?!\W) )
As you see, it's conditional depending on what it's touching. That’s what the (?(COND)THEN|ELSE) clause is for.
This becomes an issue with things like:
$_ = qq('Tis Paul's parents' summer-house, isn't it?\n);
my $count = 0;
s{
(?(?=[\-\w']) (?<![\-\w']) | (?<![^\-\w']) )
( [\-\w'] + )
(?(?<=[\-\w']) (?![\-\w']) | (?![^\-\w']) )
}{
sprintf "(%s)[%d]", $1, ++$count
}gsex;
print;
which correctly prints
('Tis)[1] (Paul's)[2] (parents')[3] (summer-house)[4], (isn't)[5] (it)[6]?
Worrying about Unicode
1960s-style ASCII is about 50 years out of date. Just as whenever you see anyone write [a-z], it’s nearly always wrong, it turns out that things like dashes and quotation marks shouldn’t show up as literals in patterns, either. While we’re at it, you probably don’t want to use \w, because that includes numbers and underscores as well, not just alphabetics.
Imagine this string:
$_ = qq(\x{2019}Tis Ren\x{E9}e\x{2019}s great\x{2010}grandparents\x{2019} summer\x{2010}house, isn\x{2019}t it?\n);
which you could have as a literal with use utf8:
use utf8;
$_ = qq(’Tis Renée’s great‐grandparents’ summer‐house, isn’t it?\n);
This time I’ll go at the pattern a bit differently, separating out my definition of terms from their execution to try to make it more readable and thence maintainable:
#!/usr/bin/perl -l
use 5.10.0;
use utf8;
use open qw< :std :utf8 >;
use strict;
use warnings qw< FATAL all >;
use autodie;
$_ = q(’Tis Renée’s great‐grandparents’ summer‐house, isn’t it?);
my $count = 0;
s{ (?<WORD> (?&full_word) )
# the rest is just definition
(?(DEFINE)
(?<word_char> [\p{Alphabetic}\p{Quotation_Mark}] )
(?<full_word>
# next line won't compile cause
# fears variable-width lookbehind
#### (?<! (?&word_char) ) )
# so must inline it
(?<! [\p{Alphabetic}\p{Quotation_Mark}] )
(?&word_char)
(?:
\p{Dash}
| (?&word_char)
) *
(?! (?&word_char) )
)
) # end DEFINE declaration block
}{
sprintf "(%s)[%d]", $+{WORD}, ++$count;
}gsex;
print;
That code when run produces this:
(’Tis)[1] (Renée’s)[2] (great‐grandparents’)[3] (summer‐house)[4], (isn’t)[5] (it)[6]?
Ok, so that may have beeen FMTEYEWTK about fancy regexes, but aren’t you glad you asked? ☺
In plain regular expressions there isn't as far as I know.
On the other hand, there are several tools which offer it as an extension, for example grepWin. In the tool's help (press F1):
Internally it uses Boost's Perl Regular Expression engine but the ${count} is implemented within (as with other extensions).
How should I handle regex-features labeled with "warning" like "(?{ code })", "(??{ code })" or "Special Backtracking Control Verbs"? How serious should I take the warnings?
I kinda think they’re here to stay, one way or the other — especially code escapes. Code escapes have been with us for more than a decade.
The scariness of them — that they can call code in unforeseen ways — is taken care of by use re "eval". Also, the regex matcher hasn’t been reëntrant until 5.12 IIRC, which could limit their usefulness.
The string-eval version, (??{ code }), used to be the only way to do recursion, but since 5.10 we have a much better way to do that; benchmarking the speed differences shows the eval way is way slower in most cases.
I mostly use the block-eval version, (?{ code}), for adding debugging, which happens at a different granualarity than use re "debug". It used to vaguely bother me that the return value from the block-eval version’s wasn’t usable, until I realized that it was. You just had to use it as the test part of a conditional pattern, like this pattern for testing whether a number was made up of digits that were decreasing by one each position to the right:
qr{
^ (
( \p{Decimal_Number} )
(?(?= ( \d )) | $)
(?(?{ ord $3 == 1 + ord $2 }) (?1) | $)
) $
}x
Before I figured out conditionals, I would have written that this way:
qr{
^ (
( \p{Decimal_Number} )
(?= $ | (??{ chr(1+ord($2)) }) )
(?: (?1) | $ )
) $
}x
which is much less efficient.
The backtracking control verbs are newer. I use them mostly for getting all possible permutations of a match, and that requires only (*FAIL). I believe it is the (*ACCEPT) feature that is especially marked “highly experimental”. These have only been with us since 5.10.