How to reconstruct regex matched part - regex

I have simplify some latex math formula within text, for example
This is ${\text{BaFe}}_{2}{\text{As}}_{2}$ crystal
I want to transform this into
This is BaFe2As2 crystal
That is to concatenate only content within inner most bracket.
I figure out that I can use regex pattern
\{[^\{\}]*\}
to match those inner most bracket. But the problem is how to concatenate them together?
I don't know if this could be done in notepad++ regex replacement. If notepad++ is not capable, I can also accept perl one-liner solution.

There may clearly be multiple such equations (the markup between two $s) in the document. So while you need to assemble text between all {}, this also need be constrained within a $ pair. Then all such equations need be processed.
Matching that in a single pattern results in a complex regex. Instead, we can first extract everything within a pair of $s and then gather text within {}s from that, simplifying the regex a lot. This makes two passes over each equation but a Latex document is small for computational purposes and the loss of efficiency can't be noticed.
use warnings;
use strict;
use feature 'say';
my $text = q(This is ${\text{BaFe}}_{2}{\text{As}}_{2}$ crystal,)
. q( and ${\text{Some}}{\mathbf{More}}$ text);
my #results;
while ($text =~ /\$(.*?)\$/g) {
my $eq = $1;
push #results, join('', $eq =~ /\{([^{}]+)\}/g);
}
say for #results;
This prints lines BaFe2As2 and SomeMore.
The regex in the while condition captures all chars between two $s. After the body of the loop executes and the condition is checked again, the regex continues searching the string from the position of the previous match. This is due to the "global" modifier /g in scalar context, imposed on regex since it is in the loop condition. Once there are no more matches the loop terminates.
In the body we match between {}, and again due to /g this is done for all {}s in the equation. Here, however, the regex is in the list context (as it is assigned to an array) and then /g makes it return all matches. They are joined into a string, which is added to the array.
In order to replace the processed equation, use this in a substitution instead
$text =~ s{ \$(.*?)\$ }{ join('', $1 =~ /\{([^{}]+)\}/g) }egx;
where the modifier e makes it so that the replacement part is evaluated as Perl code, and the result of that used to replace the matched part. Then in it we can run our regex to match content of all {} and join it into the string, as explained above. I use s{}{} delimiters, and x modifier so to be able to space things in the matching part as well.
Since the whole substitution has the g modifier the regex keeps going through $text, as long as there are equations to match, replacing them with what's evaluated in the replacement part.
I use a hard-coded string (extended) from the question, for an easy demo. In reality you'd read a file into a scalar variable ("slurp" it) and process that.
This relies on the question's premise that text of interest in an equation is cleanly between {}.
Missed the part that a one-liner is sought
perl -0777 -wnE'say join("", $1=~/\{([^{}]+)\}/g) while /\$(.*?)\$/g' file.tex
With -0777 the file is read whole ("slurped"), and as -n provides a loop over input lines it is in the $_ variable; the regex in the while condition works by default on $_. In each interation of while the contents of the captured equation, in $1, is directly matched for {}s.
Then to replace each equation and print out the whole processed file
perl -0777 -wne's{\$(.*?)\$}{join "", $1=~/\{([^{}]+)\}/g}eg; print' file.tex
where I've removed extra spaces and (unnecessary) parens on join.

Use this regex in Notepad++. I have tried to match everything which is NOT present between the innermost curly brackets and then replaced the match with a blank string.
[^{}]*\{|\}[^{}]*
Click for Demo
Explanation:
[^{}]*\{ - matches 0+ occurrences of any character that is neither { nor } followed by {
| - OR
\}[^{}]* - matches } followed by 0+ occurrences of any character that is neither { nor }
Before Replacement:
After Replacement:
UPDATE:
Try this updated regex:
\$?(?=[^$]*\$[^$]*$)(?:[^{}]*{|}[^{}]*)(?=[^$]*\$[^$]*$)\$?
Click for Demo

Related

Why does perl not keep the match variable around in this situation?

I just struggled a long time to come up with a working little perl one-liner like this:
perl -pe 'if (/^(".*",").*, /) { $a = $1; s/, /"\n$a/g}'
My input data looks something like this:
"foo","bar a"
"baz","bar a, bar b, bar c"
And I'm transforming it to this:
"foo","bar a"
"baz","bar a"
"baz","bar b"
"baz","bar c"
Basically I wanted to only match certain lines (if (/, /)...) and on those lines replace all instances of that match with a part of the original line. A s///g with a match group would not work because it would not recurse properly, the replacement string has to be figured out before replacements start happening.
if (/^(".*",").*, /) { s/, /"\n$1/g}
Yet it did not. The var $1 was never anything but empty. Given what the perl docs I read said about persistence, this was a surprise to me:
These match variables generally stay around until the next successful pattern match.
Only when I started stashing the result in a variable of my own could I access the result from the substitution expression:
if (/^(".*",").*, /) { $a = $1; s/, /"\n$a/g}
Why was $1 being cleared when not only was there no successful match, there was no request for a match at all in my search and replace? And would there have been a better way to approach this problem?
The values of match variables do indeed stay around until the next successful pattern match (or until the scope in which the match occurred is exited).
In your case, they changed because there was a successful pattern match. You successfully matched against the pattern , . The capture variables will therefore reflect the text captured by the captures of that match. $1 returns the text matched by the non-existent first capture, so it returned undef.
$ perl -e'
$_ = "a";
s/(a)/a/; CORE::say $1 // "[undef]"; # Successful match
s/(c)/c/; CORE::say $1 // "[undef]"; # Unsuccessful match
s/a/a/; CORE::say $1 // "[undef]"; # Successful match
'
a
a
undef
You ask:
Why was $1 being cleared when not only was there no successful match, there was no request for a match at all in my search and replace?
Are you perhaps conflating matching and capturing?
For s/PATTERN/REPLACEMENT/ to do anything, the PATTERN must match. So if there is any substitution at all as a result of an s/// operation, you know that its PATTERN regex-matched successfully. REPLACEMENT is then evaluated.
(In your case, the s/, /.../ PATTERN matches at least once on the comma and space after the text bar a in your second input line.)
Of course, when that happens, the interpreter will reset all the capture elements ($1, $2, etc.) to whatever PATTERN captured. Again, this is before REPLACEMENT is evaluated. Since your PATTERN doesn't capture anything, those elements are undefined, just as they would be if you had explicitly done a non-capturing m/, / match.

Regex to find(/replace) multiple instances of character in string

I have a (probably very basic) question about how to construct a (perl) regex, perl -pe 's///g;', that would find/replace multiple instances of a given character/set of characters in a specified string. Initially, I thought the g "global" flag would do this, but I'm clearly misunderstanding something very central here. :/
For example, I want to eliminate any non-alphanumeric characters in a specific string (within a larger text corpus). Just by way of example, the string is identified by starting with [ followed by #, possibly with some characters in between.
[abc#def"ghi"jkl'123]
The following regex
s/(\[[^\[\]]*?#[^\[\]]*?)[^a-zA-Z0-9]+?([^\[\]]*?)/$1$2/g;
will find the first " and if I run it three times I have all three.
Similarly, what if I want to replace the non-alphanumeric characters with something else, let's say an X.
s/(\[[^\[\]]*?#[^\[\]]*?)[^a-zA-Z0-9]+?([^\[\]]*?)/$1X$2/g;
does the trick for one instance. But how can I find all of them in one go?
The reason your code doesn't work is that /g doesn't rescan the string after a substitution. It finds all non-overlapping matches of the given regex and then substitutes the replacement part in.
In [abc#def"ghi"jkl'123], there is only a single match (which is the [abc#def" part of the string, with $1 = '[abc#def' and $2 = ''), so only the first " is removed.
After the first match, Perl scans the remaining string (ghi"jkl'123]) for another match, but it doesn't find another [ (or #).
I think the most straightforward solution is to use a nested search/replace operation. The outer match identifies the string within which to substitute, and the inner match does the actual replacement.
In code:
s{ \[ [^\[\]\#]* \# \K ([^\[\]]*) (?= \] ) }{ $1 =~ tr/a-zA-Z0-9//cdr }xe;
Or to replace each match by X:
s{ \[ [^\[\]\#]* \# \K ([^\[\]]*) (?= \] ) }{ $1 =~ tr/a-zA-Z0-9/X/cr }xe;
We match a prefix of [, followed by 0 or more characters that are not [ or ] or #, followed by #.
\K is used to mark the virtual beginning of the match (i.e. everything matched so far is not included in the matched string, which simplifies the substitution).
We match and capture 0 or more characters that are not [ or ].
Finally we match a suffix of ] in a look-ahead (so it's not part of the matched string either).
The replacement part is executed as a piece of code, not a string (as indicated by the /e flag). Here we could have used $1 =~ s/[^a-zA-Z0-9]//gr or $1 =~ s/[^a-zA-Z0-9]/X/gr, respectively, but since each inner match is just a single character, it's also possible to use a transliteration.
We return the modified string (as indicated by the /r flag) and use it as the replacement in the outer s operation.
So...I'm going to suggest a marvelously computationally inefficient approach to this. Marvelously inefficient, but possibly still faster than a variable-length lookbehind would be...and also easy (for you):
The \K causes everything before it to be dropped....so only the character after it is actually replaced.
perl -pe 'while (s/\[[^]]*#[^]]*\K[^]a-zA-Z0-9]//){}' file
Basically we just have an empty loop that executes until the search and replace replaces nothing.
Slightly improved version:
perl -pe 'while (s/\[[^]]*?#[^]]*?\K[^]a-zA-Z0-9](?=[^]]*?])//){}' file
The (?=) verifies that its content exists after the match without being part of the match. This is a variable-length lookahead (what we're missing going the other direction). I also made the *s lazy with the ? so we get the shortest match possible.
Here is another approach. Capture precisely the substring that needs work, and in the replacement part run a regex on it that cleans it of non-alphanumeric characters
use warnings;
use strict;
use feature 'say';
my $var = q(ah [abc#def"ghi"jkl'123] oh); #'
say $var;
$var =~ s{ \[ [^\[\]]*? \#\K ([^\]]+) }{
(my $v = $1) =~ s{[^0-9a-zA-Z]}{}g;
$v
}ex;
say $var;
where the lone $v is needed so to return that and not the number of matches, what s/ operator itself returns. This can be improved by using the /r modifier, which returns the changed string and doesn't change the original (so it doesn't attempt to change $1, what isn't allowed)
$var =~ s{ \[ [^\[\]]*? \#\K ([^\]]+) }{
$1 =~ s/[^0-9a-zA-Z]//gr;
}ex;
The \K is there so that all matches before it are "dropped" -- they are not consumed so we don't need to capture them in order to put them back. The /e modifier makes the replacement part be evaluated as code.
The code in the question doesn't work because everything matched is consumed, and (under /g) the search continues from the position after the last match, attempting to find that whole pattern again further down the string. That fails and only that first occurrence is replaced.
The problem with matches that we want to leave in the string can often be remedied by \K (used in all current answers), which makes it so that all matches before it are not consumed.

Regular Expression required for matching a string that should not be followed by another specific string

I am using the below code for matching a string(EX: <jdgdt\s+mdy=.*?>\s*) which should not be followed by another specific string (<jdg>). But i am unable to get the desired output as per the below code. Can anyone help me regarding this ?
Input file :
<dckt>Docket No. 7677-12.</dckt>
<jdgdt mdy='02/25/2014'>
<jdg>Opinion by Marvel, <e>J.</e></jdg>
<taxyr></taxyr>
<disp></disp>
</tcpar>
<dckt>Docket No. 7237-13.</dckt>
<jdgdt mdy='02/24/2014'>
</tcpar>
Desired Output:
<dckt>Docket No. 7677-12.</dckt>
<jdgdt mdy='02/25/2014'>
<jdg>Opinion by Marvel, <e>J.</e></jdg>
<taxyr></taxyr>
<disp></disp>
</tcpar>
<dckt>Docket No. 7237-13.</dckt>
<jdgdt mdy='02/24/2014'>
<jdg>Opinion by Marvel, <e>J.</e></jdg>
<taxyr></taxyr>
<disp></disp>
</tcpar>
Code:
#/usr/bin/perl
my $filename = $ARGV[0];
my $ext = $ARGV[1];
my $InputFile = "$filename" . "\." . "$ext";
my $document = do {
local $/ = undef;
open my $fh, "<", $InputFile or die "Error: Could Not Open File $InputFile: $!";
<$fh>;
};
$document =~ s/(<jdgdt\s+mdy=.*?>\s*)(?!<jdg>)/$1<jdg>Opinion by Marvel,<e>J.<\/e><\/jdg>\n<taxyr><\/taxyr>\n<disp><\/disp>/isg;
print $document;
I had to make two minor adjustments to your regex to get the desired output:
$document =~ s{(<jdgdt\s+mdy\=[^>]*>\s*)(?!\s*<jdg>)}{$1<jdg>Opinion by Marvel,<e>J.</e></jdg>\n<taxyr></taxyr>\n<disp></disp>}isg;
Also, to clean up the code, I switched from using / to using {} to delimit the regex; that way, you don't need to backslash all the slashes that you actually want there in your replacement.
Explanation of what I changed:
First off, negative lookahead is tricky. What you have to remember is that perl will try to match your expression the maximum amount of times possible. Because you had this initially:
/(<jdgdt\s+mdy\=.*?>\s*)(?!<jdg>)/
What would happen is that in that first clause you'd get this match:
<jdgdt mdy='02/25/2014'>\n<jdg>Opinion by Marvel, <e>J.</e></jdg>
^^^^^^^^^^^^^^^^^^^^^^^^
(this part matched by paren. Note the \n is not matched!)
Perl would consider this a match because after the first parenthesized expression, you have "\n<jdg>". Well, that doesn't match the expression "<jdg>" (because of the initial newline), so yay! found a match.
In other words, initially, perl would have the \s* that you end your parenthesized expression with match the empty string, and therefore it would find a match and you'd end up stuffing things into the first clause that you didn't want. Another way to put it is that because of the freedom to choose what went into \s*, perl would choose the amount that allowed the expression as a whole to match. (and would fill \s* with the empty string for the first docket record, and newline for the second docket record)
To get perl to never find a match on the first docket record, I repeated the \s* in the negative lookahead as well. That way, no choice of what to put in \s* could make the expression as a whole match on the initial docket record, and perl had to give up and move to the second docket record.
But then there was a second problem! Remember how I said perl was really aggressive about finding matches anywhere it could? Well, next perl would expand your mdy\=.*?> bit to still find a result in the first docket record. After I added \s* to the negative lookahead, the first docket was still matching (but in a different spot) with:
<jdgdt mdy='02/25/2014'>\n<jdg>Opinion by Marvel, <e>J.</e></jdg>
^^^^^^^^^^^???????????????????^
(Underlined part matched by paren. ? denotes the bit matched by .*?)
See how perl expanded your .*? way beyond what you had intended? You'd intended that bit to match only stuff up to the first > character, but perl will stretch your non-greedy matches as far as necessary so that the whole pattern matches. This time, it stretched your .*? to cover the > that closed the <jdg> tag so that it could find a spot where the negative lookahead didn't block the match.
To keep perl from stretching your .*? pattern that far, I replaced .*? with [^>]*, which is really what you meant.
After these two changes, we then only found a match in the second docket record, as initially desired.
Use positive lookahead. (?!<jdg>) or something similar, look it up.

Perl regex with grouping in LHS

How to can match the next lines?
sometext_TEXT1.yyy-TEXT1.yyy
anothertext_OTHER.yyy-MAX.yyy
want remove the - repetative.text from the end, but only if it repeats.
sometext_TEXT1.yyy
anothertext_OTHER.yyy-MAX.yyy
my trying
use strictures;
my $text="sometext_TEXT1.xxx-TEXT1.xxx";
$text =~ s/(.*?)(.*)(\s*-\s*$2)/$1$2/;
print "$text\n";
prints
Use of uninitialized value $2 in regexp compilation at a line 3.
with other words, looking for better solution for the next split + match...
while(<DATA>) {
chomp;
my($first, $second) = split /\s*-\s*/;
s/\s*-\s*$second$// if ( $first =~ /$second$/ );
print "$_\n";
}
__DATA__
sometext_TEXT1.yyy-TEXT1.yyy
anothertext_OTHER.yyy-MAX.yyy
$text =~ s/(.*?)(.*)(\s*-\s*$2)/$1$2/;
This regex has various issues, but is on the right path.
Use \2 (or better: \g2 or \g{-1}) or something to reference the contents of a capture group. The $2 variable is interpolated when the Perl statement is executed. At that time, $2 is undefined, as there was no previous match. You get a warning as it is uninitialized. Even if it were defined, the pattern would be fixed during compilation.
You define three capture groups, but only need one. There is a trick with the \Keep directive: It let's the regex engine forget the previously matched text, so that it won't be affected by the substitution. That is, s/(foo)b/$1/ is equivalent to s/foo\Kb//. The effect is similar to a variable-length lookbehind.
The (.*?)(.*) part is a bit of an backtracking nightmare. We can reduce the cost of your match by adding further conditions, e.g. by anchoring the pattern at start and end of line. Using above modifications, we now have s/^.*?(.*)\K\s*-\s*\g1$//. But on second thought, we can just remove the ^.*? because this describes something the regex engine does anyway!
A short test:
while(<DATA>) {
s/(.*)\K\s*-\s*\g1$//;
print;
}
__DATA__
sometext_TEXT1.yyy-TEXT1.yyy
anothertext_OTHER.yyy-MAX.yyy
Output:
sometext_TEXT1.yyy
anothertext_OTHER.yyy-MAX.yyy
A few words regarding your splitting solution: This will also shorten the line
sometext_TEXT1xyyy - 1.xyyy
because when you interpolate a variable into a regex, the contents aren't matched literally. Instead, they are interpreted as a pattern (where . matches any non-newline codepoint)! You can avoid this by quoting all metacharacters with the \Q...\E escape:
s/\s*-\s*\Q$second\E$// if $first =~ /\Q$second\E$/;
When you use $2 Perl will try to interpolate that variable, but the variable will only be set after the match has completed. What you want, is a backreference, for which you need to use \2:
$text =~ s/(.*?)(.*)(\s*-\s*\2)/$1$2/;
Note that, when the replacement part is evaluated, $1 and $2 have been set and can be interpolated as expected. Also you could make the pattern a bit more concise (and probably more efficient), by using:
$text =~ s/(.*)\s*-\s*\2/$1/;
There is no need to match the initial part (.*?) if it's arbitrary and you just write it back anyway. What you might want to do though, is anchor the pattern to the end of the string:
$text =~ s/(.*)\s*-\s*\1$/$1/;
Otherwise (with your initial attempt or mine), you'd turn something-thingelse into somethingelse.

How can I extract a substring enclosed in double quotes in Perl?

I'm new to Perl and regular expressions and I am having a hard time extracting a string enclosed by double quotes. Like for example,
"Stackoverflow is
awesome"
Before I extract the strings, I want to check if it is the end of the line of the whole text was in the variable:
if($wholeText =~ /\"$/) #check the last character if " which is the end of the string
{
$wholeText =~ s/\"(.*)\"/$1/; #extract the string, removed the quotes
}
My code didn't work; it is not getting inside of the if condition.
You need to do:
if($wholeText =~ /"$/)
{
$wholeText =~ s/"(.*?)"/$1/s;
}
. doesn't match newlines unless you apply the /s modifier.
There's no need to escape the quotes like you're doing.
The above poster who recommended using the "m" flag in the regular expression is correct, however the regex provided won't quite work. When you say:
$wholeText =~ s/\"(.*)\"/$1/m; #extract the string, removed the quotes
...the regular expression is too "greedy", which means the (.*) part will gobble up too much of the text. If you have a sample like this:
"The quick brown fox," he said, "jumped over the lazy dog."
...then the above regex will capture everything from "The" through "dog.", which is probably not what you intend. There are two ways to make the regex less greedy. Which one is better has everything to do with how you choose to handle extra " marks inside your string.
One:
$wholeText =~ s/\"([^"]*)\"/$1/m;
Two:
$wholeText =~ s/\"(.*?)\"/$1/m;
In One, the regex says "start with quote, then find everything that is not a quote and remember it, until you see another quote." In Two, the regex says "Start with quote, then find everything until you find another quote." The extra ? inside the ( ) tells the regex processor to not be greedy. Without considering quote escaping within the string, both regular expressions should behave the same.
By the way, this is a classic problem when parsing a CSV ("Comma Separated Values") file, by the way, so looking up some references on that may help you out.
If you want to anchor a match to the very end of the string (not line, entire string), use the \z anchor:
if( $wholeText =~ /"\z/ ) { ... }
You don't need a guard condition for this. Just use the right regex in the substitution. If it doesn't match the regex, nothing happens:
$wholeText =~ s/"(.*?)"\z/$1/s;
I think you really have a different question though. Why are you trying to anchor it to the end of the string? What problems are you trying to avoid?
For multi-line strings, you need to include the 'm' modifier with the search pattern.
if ($wholeText =~ m/\"$/m) # First m for match operator; second multi-line modifier
{
$wholeText =~ s/\"(.*?)\"/$1/s; #extract the string, removed the quotes
}
You will also need to consider whether you allow double quotes inside the string and if so, which convention to use. The primary ones are backslash and double quote (also backslash backslash), or double quote double quote in the string. These slightly complicate your regex.
The answer by #chaos uses 's' as a multi-line modifier. There's a small difference between the two:
m
Treat string as multiple lines. That is, change "^" and "$" from matching the start or end of the string to matching the start or end of any line anywhere within the string.
s
Treat string as single line. That is, change "." to match any character whatsoever, even a newline, which normally it would not match.
Used together, as /ms, they let the "." match any character whatsoever, while still allowing "^" and "$" to match, respectively, just after and just before newlines within the string.
Assuming you have a single substring in quotes, this will extract it:
s/."(.?)".*/$1/
And the answer above (s/"(.*?)"/$1/s) will just remove quotes.
Test code:
my $text = "no \"need this\" again, no\n";
my $text2 = $text;
print $text;
$text2 =~ s/.*\"(.*?)\".*/$1/;
print $text2;
$text =~ s/"(.*?)"/$1/s;
print $text;
Output:
no "need this" again, no
need this
no need this again, no