This question already has an answer here:
Reference - What does this regex mean?
(1 answer)
Closed 8 years ago.
I have been reading perl regular expression with modifier s m and g. I understand that //g is a global matching where it will be a greedy search.
But I am confused with the modifier s and m. Can anyone explain the difference between s and m with code example to show how it can be different? I have tried to search online and it only gives explanation as in the link http://perldoc.perl.org/perlre.html#Modifiers. In stackoverflow I have even seen people using s and m together. Isn't s is the opposite of m?
//s
//m
//g
I am not able to match multiple line using using m.
use warnings;
use strict;
use 5.012;
my $file;
{
local $/ = undef;
$file = <DATA>;
};
my #strings = $file =~ /".*"/mg; #returns all except the last string across multiple lines
#/"String"/mg; tried with this as well and returns nothing except String
say for #strings;
__DATA__
"This is string"
"1!=2"
"This is \"string\""
"string1"."string2"
"String"
"S
t
r
i
n
g"
The documentation that you link to yourself seems very clear to me. It would help if you would explain what problem you had with understanding it, and how you came to think that /s and /m were opposites.
Very briefly, /s changes the behaviour of the dot metacharacter . so that it matches any character at all. Normally it matches anything except a newline "\n", and so treats the string as a single line even if it contains newlines.
/m modifies the caret ^ and dollar $ metacharacters so that they match at newlines within the string, treating it as a multi-line string. Normally they will match only at the beginning and end of the string.
You shouldn't get confused with the /g modifier being "greedy". It is for global matches which will find all occurrences of the pattern within the string. The term greedy is usually user for the behaviour of quantifiers within the pattern. For instance .* is said to be greedy because it will match as many characters as possible, as opposed to .*? which will match as few characters as possible.
Update
In your modified question you are using /".*"/mg, in which the /m is irrelevant because, as noted above, that modifier alters only the behaviour of the $ and ^ metacharacters, and there are none in your pattern.
Changing it to /".*"/sg improves things a little in that the . can now match the newline at the end of each line and so the pattern can match multi-line strings. (Note that it is the object string that is considered to be "single line" here - i.e. the match behaves just as if there were no newlines in it as far as . is concerned.) Hower here is the conventional meaning of greedy, because the pattern now matches everything from the first double-quote in the first line to the last double-quote at the end of the last line. I assume that isn't what you want.
There are a few ways to fix this. I recommend changing your pattern so that the string you want is a double-quote, followed by any sequence of characters except double-quotes, followed by another double quote. This is written /"[^"]*"/g (note that the /s modifier is no longer necessary as there are now no dots in the pattern) and very nearly does what you want except that the escaped double-quotes are seen as ending the pattern.
Take a look at this program and its output, noting that I have put a chevron >> at the start of each match so that they can be distinguished
use strict;
use warnings;
my $file = do {
local $/;
<DATA>;
};
my #strings = $file =~ /"[^"]*"/g;
print ">> $_\n\n", for #strings;
__DATA__
"This is string"
"1!=2"
"This is \"string\""
"string1"."string2"
"String"
"S
t
r
i
n
g"
output
>> "This is string"
>> "1!=2"
>> "This is \"
>> ""
>> "string1"
>> "string2"
>> "String"
>> "S
t
r
i
n
g"
As you can see everything is now in order except that in "This is \"string\"" it has found two matches, "This is \", and "". Fixing that may be more complicated than you want to go but it's perfectly possible. Please say so if you need that fixed too.
Update
I may as well finish this off. To ignore escaped double-quotes and treat them as just part of the string, we need to accept either \" or any character except double-quote. That is done using the regex alternation operator | and must be grouped inside non-capturing parentheses (?: ... ). The end result is /"(?:\\"|[^"])*"/g (the backslash itself must be escaped so it is doubled up) which, when put into the above program, produces this output, which I assume is what you wanted.
>> "This is string"
>> "1!=2"
>> "This is \"string\""
>> "string1"
>> "string2"
>> "String"
>> "S
t
r
i
n
g"
/m and /s both affect how the match operator treats multi-line strings.
With the /m modifier, ^ and $ match the beginning and end of any line within the string. Without the /m modifier, ^ and $ just match the beginning and end of the string.
Example:
$_ = "foo\nbar\n";
/foo$/, /^bar/ do not match
/foo$/m, /^bar/m match
With the /s modifier, the special character . matches all characters including newlines. Without the /s modifier, . matches all characters except newlines.
$_ = "cat\ndog\ngoldfish";
/cat.*fish/ does not match
/cat.*fish/s matches
It is possible to use /sm modifiers together.
$_ = "100\n101\n102\n103\n104\n105\n";
/^102.*104$/ does not match
/^102.*104$/s does not match
/^102.*104$/m does not match
/^102.*104$/sm matches
With /".*"/mg your match
starts with "
and then .*" matches every character (except \n) as much as possible till "
since you use /g and match stopped at second ", regex will try to repeat first two steps
/m doesn't make difference here as you're not using ^ or $ anchors
Since you have escaped quotes in your example, regex is not the best tool to do what you want.
If that wasn't the case and you wanted everything between two quotes, /".*?"/gs would do the job.
Borodin's regex will work for the examples from this lab assignment.
However, it's also possible for a backslash to escape itself. This comes up when one includes windows paths in a string, so the following regex would catch that case:
use warnings;
use strict;
use 5.012;
my $file = do { local $/; <DATA>};
my #strings = $file =~ /"(?:(?>[^"\\]+)|\\.)*"/g;
say "<$_>" for #strings;
__DATA__
"This is string"
"1!=2"
"This is \"string\""
"string1"."string2"
"String"
"S
t
r
i
n
g"
"C:\\windows\\style\\path\\"
"another string"
Outputs:
<"This is string">
<"1!=2">
<"This is \"string\"">
<"string1">
<"string2">
<"String">
<"S
t
r
i
n
g">
<"C:\\windows\\style\\path\\">
<"another string">
For a quick explanation of the pattern:
my #strings = $file =~ m{
"
(?:
(?> # Independent subexpression (reduces backtracking)
[^"\\]+ # Gobble all non double quotes and backslashes
)
|
\\. # Backslash followed by any character
)*
"
}xg; # /x modifier allows whitespace and comments.
Related
I have a (probably very basic) question about how to construct a (perl) regex, perl -pe 's///g;', that would find/replace multiple instances of a given character/set of characters in a specified string. Initially, I thought the g "global" flag would do this, but I'm clearly misunderstanding something very central here. :/
For example, I want to eliminate any non-alphanumeric characters in a specific string (within a larger text corpus). Just by way of example, the string is identified by starting with [ followed by #, possibly with some characters in between.
[abc#def"ghi"jkl'123]
The following regex
s/(\[[^\[\]]*?#[^\[\]]*?)[^a-zA-Z0-9]+?([^\[\]]*?)/$1$2/g;
will find the first " and if I run it three times I have all three.
Similarly, what if I want to replace the non-alphanumeric characters with something else, let's say an X.
s/(\[[^\[\]]*?#[^\[\]]*?)[^a-zA-Z0-9]+?([^\[\]]*?)/$1X$2/g;
does the trick for one instance. But how can I find all of them in one go?
The reason your code doesn't work is that /g doesn't rescan the string after a substitution. It finds all non-overlapping matches of the given regex and then substitutes the replacement part in.
In [abc#def"ghi"jkl'123], there is only a single match (which is the [abc#def" part of the string, with $1 = '[abc#def' and $2 = ''), so only the first " is removed.
After the first match, Perl scans the remaining string (ghi"jkl'123]) for another match, but it doesn't find another [ (or #).
I think the most straightforward solution is to use a nested search/replace operation. The outer match identifies the string within which to substitute, and the inner match does the actual replacement.
In code:
s{ \[ [^\[\]\#]* \# \K ([^\[\]]*) (?= \] ) }{ $1 =~ tr/a-zA-Z0-9//cdr }xe;
Or to replace each match by X:
s{ \[ [^\[\]\#]* \# \K ([^\[\]]*) (?= \] ) }{ $1 =~ tr/a-zA-Z0-9/X/cr }xe;
We match a prefix of [, followed by 0 or more characters that are not [ or ] or #, followed by #.
\K is used to mark the virtual beginning of the match (i.e. everything matched so far is not included in the matched string, which simplifies the substitution).
We match and capture 0 or more characters that are not [ or ].
Finally we match a suffix of ] in a look-ahead (so it's not part of the matched string either).
The replacement part is executed as a piece of code, not a string (as indicated by the /e flag). Here we could have used $1 =~ s/[^a-zA-Z0-9]//gr or $1 =~ s/[^a-zA-Z0-9]/X/gr, respectively, but since each inner match is just a single character, it's also possible to use a transliteration.
We return the modified string (as indicated by the /r flag) and use it as the replacement in the outer s operation.
So...I'm going to suggest a marvelously computationally inefficient approach to this. Marvelously inefficient, but possibly still faster than a variable-length lookbehind would be...and also easy (for you):
The \K causes everything before it to be dropped....so only the character after it is actually replaced.
perl -pe 'while (s/\[[^]]*#[^]]*\K[^]a-zA-Z0-9]//){}' file
Basically we just have an empty loop that executes until the search and replace replaces nothing.
Slightly improved version:
perl -pe 'while (s/\[[^]]*?#[^]]*?\K[^]a-zA-Z0-9](?=[^]]*?])//){}' file
The (?=) verifies that its content exists after the match without being part of the match. This is a variable-length lookahead (what we're missing going the other direction). I also made the *s lazy with the ? so we get the shortest match possible.
Here is another approach. Capture precisely the substring that needs work, and in the replacement part run a regex on it that cleans it of non-alphanumeric characters
use warnings;
use strict;
use feature 'say';
my $var = q(ah [abc#def"ghi"jkl'123] oh); #'
say $var;
$var =~ s{ \[ [^\[\]]*? \#\K ([^\]]+) }{
(my $v = $1) =~ s{[^0-9a-zA-Z]}{}g;
$v
}ex;
say $var;
where the lone $v is needed so to return that and not the number of matches, what s/ operator itself returns. This can be improved by using the /r modifier, which returns the changed string and doesn't change the original (so it doesn't attempt to change $1, what isn't allowed)
$var =~ s{ \[ [^\[\]]*? \#\K ([^\]]+) }{
$1 =~ s/[^0-9a-zA-Z]//gr;
}ex;
The \K is there so that all matches before it are "dropped" -- they are not consumed so we don't need to capture them in order to put them back. The /e modifier makes the replacement part be evaluated as code.
The code in the question doesn't work because everything matched is consumed, and (under /g) the search continues from the position after the last match, attempting to find that whole pattern again further down the string. That fails and only that first occurrence is replaced.
The problem with matches that we want to leave in the string can often be remedied by \K (used in all current answers), which makes it so that all matches before it are not consumed.
I am new to regex and I am using Perl.
I have below tag:
<CFSC>cfsc_service=TRUE
SEC=1
licenses=10
expires=20170511
</CFSC>
I want to match anything between <CFSC> and </CFSC> tags.
I tried /<CFSC>.*?\n.*?\n.*?\n.*?\n<\/CFSC>/
and /<CFSC>(.*)<\/CFSC>/ but had no luck.
You need the /s single line modifier to make the regex engine include line breaks in ..
Treat string as single line. That is, change "." to match any character whatsoever, even a newline, which normally it would not match.
See this example.
my $foo = qq{<CFSC>cfsc_service=TRUE
SEC=1
licenses=10
expires=20170511
</CFSC>};
$foo =~ m{>(.*)</CFSC>}s;
print $1;
You also need to use a different delimiter than /, or escape it.
Try
/<CFSC>(.*)<\/CFSC>/s
The final s makes the . match newline chars (\n = 0x0a) which is usually doesn't match:
Treat string as single line. That is, change "." to match any
character whatsoever, even a newline, which normally it would not
match.
from http://perldoc.perl.org/perlre.html#Modifiers
Try this:
$foo =~ m/<CFSC>((?:(?!<\/CFSC>).)*)<\/CFSC>/gs;
Modifiers:
g - Matches global
s - newline
i - case sensitive
\ - escape sequence
I am using Perl Regular expressions.
How would i go about ignoring white space and still perform a test to see if a string match.
For example.
$var = " hello "; #I want var to igonore whitespace and still match
if($var =~ m/hello/)
{
}
what you have there should match just fine. the regex will match any occurance of the pattern hello, so as long as it sees "hello" somewhere in $var it will match
On the other hand, if you want to be strict about what you ignore, you should anchor your string from start to end
if($var =~ m/^\s*hello\s*$/) {
}
and if you have multiple words in your pattern
if($var =~ m/^\s*hello\s+world\s*$/) {
}
\s* matches 0 or more whitespace, \s+ matches 1 or more white space. ^ matches the beginning of a line, and $ matches the end of a line.
As other have said, Perl matches anywhere in the string, not the whole string. I found this confusing when I first started and I still get caught out. I try to teach myself to think about whether I need to look at the start of the line / whole string etc.
Another useful tip is use \b. This looks for word breaks so /\bbook\b/ matches
"book. "
"book "
"-book"
but not
"booking"
"ebook"
This regex is a little unrelated but if you wanted to concatenate all of the whitespaces from your string before passing it through the if.
s/[\h\v]+/ /g;
/^\shello\s$/
I'm new to Perl and regular expressions and I am having a hard time extracting a string enclosed by double quotes. Like for example,
"Stackoverflow is
awesome"
Before I extract the strings, I want to check if it is the end of the line of the whole text was in the variable:
if($wholeText =~ /\"$/) #check the last character if " which is the end of the string
{
$wholeText =~ s/\"(.*)\"/$1/; #extract the string, removed the quotes
}
My code didn't work; it is not getting inside of the if condition.
You need to do:
if($wholeText =~ /"$/)
{
$wholeText =~ s/"(.*?)"/$1/s;
}
. doesn't match newlines unless you apply the /s modifier.
There's no need to escape the quotes like you're doing.
The above poster who recommended using the "m" flag in the regular expression is correct, however the regex provided won't quite work. When you say:
$wholeText =~ s/\"(.*)\"/$1/m; #extract the string, removed the quotes
...the regular expression is too "greedy", which means the (.*) part will gobble up too much of the text. If you have a sample like this:
"The quick brown fox," he said, "jumped over the lazy dog."
...then the above regex will capture everything from "The" through "dog.", which is probably not what you intend. There are two ways to make the regex less greedy. Which one is better has everything to do with how you choose to handle extra " marks inside your string.
One:
$wholeText =~ s/\"([^"]*)\"/$1/m;
Two:
$wholeText =~ s/\"(.*?)\"/$1/m;
In One, the regex says "start with quote, then find everything that is not a quote and remember it, until you see another quote." In Two, the regex says "Start with quote, then find everything until you find another quote." The extra ? inside the ( ) tells the regex processor to not be greedy. Without considering quote escaping within the string, both regular expressions should behave the same.
By the way, this is a classic problem when parsing a CSV ("Comma Separated Values") file, by the way, so looking up some references on that may help you out.
If you want to anchor a match to the very end of the string (not line, entire string), use the \z anchor:
if( $wholeText =~ /"\z/ ) { ... }
You don't need a guard condition for this. Just use the right regex in the substitution. If it doesn't match the regex, nothing happens:
$wholeText =~ s/"(.*?)"\z/$1/s;
I think you really have a different question though. Why are you trying to anchor it to the end of the string? What problems are you trying to avoid?
For multi-line strings, you need to include the 'm' modifier with the search pattern.
if ($wholeText =~ m/\"$/m) # First m for match operator; second multi-line modifier
{
$wholeText =~ s/\"(.*?)\"/$1/s; #extract the string, removed the quotes
}
You will also need to consider whether you allow double quotes inside the string and if so, which convention to use. The primary ones are backslash and double quote (also backslash backslash), or double quote double quote in the string. These slightly complicate your regex.
The answer by #chaos uses 's' as a multi-line modifier. There's a small difference between the two:
m
Treat string as multiple lines. That is, change "^" and "$" from matching the start or end of the string to matching the start or end of any line anywhere within the string.
s
Treat string as single line. That is, change "." to match any character whatsoever, even a newline, which normally it would not match.
Used together, as /ms, they let the "." match any character whatsoever, while still allowing "^" and "$" to match, respectively, just after and just before newlines within the string.
Assuming you have a single substring in quotes, this will extract it:
s/."(.?)".*/$1/
And the answer above (s/"(.*?)"/$1/s) will just remove quotes.
Test code:
my $text = "no \"need this\" again, no\n";
my $text2 = $text;
print $text;
$text2 =~ s/.*\"(.*?)\".*/$1/;
print $text2;
$text =~ s/"(.*?)"/$1/s;
print $text;
Output:
no "need this" again, no
need this
no need this again, no
Let's say I have some original text:
here is some text that has a substring that I'm interested in embedded in it.
I need the text to match a part of it, say: "has a substring".
However, the original text and the matching string may have whitespace differences. For example the match text might be:
has a
substring
or
has a substring
and/or the original text might be:
here is some
text that has
a substring that I'm interested in embedded in it.
What I need my program to output is:
here is some text that [match starts here]has a substring[match ends here] that I'm interested in embedded in it.
I also need to preserve the whitespace pattern in the original and just add the start and end markers to it.
Any ideas about a way of using Perl regexes to get this to happen? I tried, but ended up getting horribly confused.
Been some time since I've used perl regular expressions, but what about:
$match = s/(has\s+a\s+substring)/[$1]/ig
This would capture zero or more whitespace and newline characters between the words. It will wrap the entire match with brackets while maintaining the original separation. It ain't automatic, but it does work.
You could play games with this, like taking the string "has a substring" and doing a transform on it to make it "has\s*a\s*substring" to make this a little less painful.
EDIT: Incorporated ysth's comments that the \s metacharacter matches newlines and hobbs corrections to my \s usage.
This pattern will match the string that you're looking to find:
(has\s+a\s+substring)
So, when the user enters a search string, replace any whitespace in the search string with \s+ and you have your pattern. The, just replace every match with [match starts here]$1[match ends here] where $1 is the matched text.
In regexes, you can use + to mean "one or more." So something like this
/has\s+a\s+substring/
matches has followed by one or more whitespace chars, followed by a followed by one or more whitespace chars, followed by substring.
Putting it together with a substitution operator, you can say:
my $str = "here is some text that has a substring that I'm interested in embedded in it.";
$str =~ s/(has\s+a\s+substring)/\[match starts here]$1\[match ends here]/gs;
print $str;
And the output is:
here is some text that [match starts here]has a substring[match ends here] that I'm interested in embedded in it.
A many has suggested, use \s+ to match whitespace. Here is how you do it automaticly:
my $original = "here is some text that has a substring that I'm interested in embedded in it.";
my $search = "has a\nsubstring";
my $re = $search;
$re =~ s/\s+/\\s+/g;
$original =~ s/\b$re\b/[match starts here]$&[match ends here]/g;
print $original;
Output:
here is some text that [match starts here]has a substring[match ends here] that I'm interested in embedded in it.
You might want to escape any meta-characters in the string. If someone is interested, I could add it.
This is an example of how you could do that.
#! /opt/perl/bin/perl
use strict;
use warnings;
my $submatch = "has a\nsubstring";
my $str = "
here is some
text that has
a substring that I'm interested in, embedded in it.
";
print substr_match($str, $submatch), "\n";
sub substr_match{
my($string,$match) = #_;
$match =~ s/\s+/\\s+/g;
# This isn't safe the way it is now, you will need to sanitize $match
$string =~ /\b$match\b/;
}
This currently does anything to check the $match variable for unsafe characters.