Perl - how to get values of tokens - regex

I am searching how to get tokens values in properties file with Perl.
Given the source property:
my $source="application.1.hostname={{DNS_APP}}:{{PORT_APP}}/WHATEVER";
And given the target property:
my $target="application.1.hostname=test.test.com:8080/WHATEVER";
I would like to get the following result:
{{DNS_APP}}=test.test.com
{{PORT_APP}}=8080
I have no trouble to get the tokens with :
my #matches= ( $source =~ /({{.*?}})/g );
But then, how to match with their values ?
Is there an easy way, with perl regexps to get these substitutions ?
Another difficulty (but they are execption, so it is not a big deal if this problem is not addressed) is that, sometimes, $target can be
my $target="application.1.hostname=test.test.com/WHATEVER";
Or
my $target="application.1.hostname=test.test.com:8080/SOMETHINGELSE";
Or even
my $target="application.1.hostname=test.test.com/SOMETHINGELSE";
How to deal with that ?
I thank you in advance for you answers.
Regards.

OK, at a basic level, you can turn your thing into a named capture for a regex. There's a caveat though - you might need to restrict character sets.
But something like this might work:
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
my $source = "application.1.hostname={{DNS_APP}}:{{PORT_APP}}/WHATEVER";
my $target = "application.1.hostname=test.test.com:8080/WHATEVER";
$source =~ s|\Q{{\E(\w+)\Q}}\E|(?<$1>.*)|g;
$source = qr/$source/;
print "Using Regex:", $source,"\n";
$target =~ m/$source/;
#%+ is the special named-capture hash. You can access $+{DNS_APP} for example
print Dumper \%+;
Note though - that .* is a greedy match, and that will mean without delimitors/anchors between patterns, this will break. You could perhaps define a more narrow character class - I would think \w normally, but you also have . so perhaps [\w.]+ - or maybe even .*? for non greedy matching instead. This depends rather on what would 'fit' with the types of patterns you're trying to match. If you need to do so with arbitrary patterns, I think you're going to need to need ... something like regex to define the match criteria in the first place.
If your 'targets' are purely that pattern - e.g. trailing static words - you can trim you initial pattern with s/\w+$// which will reduce it to:
application.1.hostname={{DNS_APP}}:{{PORT_APP}}/
Which you then regex transform to:
(?^:application.1.hostname=(?<DNS_APP>.*):(?<PORT_APP>.*)/)
And then get %+ of:
$VAR1 = {
'DNS_APP' => 'test.test.com',
'PORT_APP' => '8080'
};
As you're on 5.8.8 - my first advice is upgrade it, because it's 7 year old software, and is long since end of life.
This variable was added in Perl v5.10.0.
However you should be able to work around by:
my #match_names = $source =~ m|\Q{{\E(\w+)\Q}}\E|g; #capture 'names' of matches
$source =~ s|\Q{{\E(\w+)\Q}}\E|(.*)|g;
$source = qr/$source/;
print "Using Regex:", $source, "\n";
my %results;
my #matches = $target =~ m/$source/;
#results{#match_names} = #matches;
print Dumper \%results;
I'm pretty sure there's a way of capturing what matched from the s pattern replacement. If I figure out what it was, I'll update.
(As it stands:
my ( #match_names ) = $source =~ s|\Q{{\E(\w+)\Q}}\E|\(.*\)|g;
doesn't seem to work as I want - #match_names contains the number of replacements. )

Related

Remove certain characters from a regex group

I have a string that looks like this (key":["value","value","value"])
"emailDomains":["google.co.uk","google.com","google.com","google.com","google.co.uk"]
and I use the following regex to select from the string. (the regex is setup in a way where it wont select a string that looks like this "key":[{"key":"value","key":"value"}] )
(?<=:\[").*?(?="])
Resulting Selection:
google.co.uk","google.com","google.com","google.com","google.co.uk
I want to remove the " in that select string, and i was wondering if there was an easy way to do this using the replace command. Desired result...
"emailDomains":["google.co.uk, google.com, google.com, google.com, google.co.uk"]
How do I solve this problem?
If your string indeed has the form "key":["v1", "v2", ... "vN"], you can split off the part that needs to be changed, replace "," by a space in it, and re-assemble:
my #parts = split / (\["\s* | \s*\"]) /x, $string; #"
$parts[2] =~ s/",\s*"/ /g;
my $processed = join '', #parts;
The regex pattern for the separator in split is captured since in that case the separators are also in the returned list, what is helpful here for putting the string back together. Then, we need to change the third element of the array.
In this approach, we have to change a specific element in the array so if your format varies, even a little, this may not (or still may) be suitable.
This should of course be processed as JSON, using a module. If the format isn't sure, as indicated in a comment, it would be best to try to ensure that you have JSON. Picking bits and pieces like above (or below) is a road to madness once requirements slowly start evolving.
The same approach can be used in a regex, and this may in fact have an advantage to be able to scoop up and ignore everything preceding the : (with split that part may end up with multiple elements if the format isn't exactly as shown, what then affects everything)
$string =~ s{ :\["\s*\K (.*?) ( "\] ) }{
my $e = $2;
my $n = $1 =~ s/",\s*"/ /gr;
$n.$e
}ex;
Here /e modifier makes it so that the replacement side is evaluated as code, where we do the same as with the split above. Notes on regex
Have to save away $2 first, since it gets reset in the next regex
The /r modifier†, which doesn't change its target but rather returns the changed string, is what allows us to use substitution operator on the read-only $1
If nothing gets captured for $2, and perhaps for $1, that means that there was no match and the outcome is simply that $string doesn't change, quietly. So if this substitution should always work then you may want to add handling of such unexpected data
Don't need a $n above, but can return ($1 =~ s/",\s*"/ /gr) . $e
Or, using lookarounds as attempted
$string =~ s{ (?<=:\[") (.+?) (?="\]) }{ $1 =~ s/",\s*"/ /gr }egx;
what does reduce the amount of code, but may be trickier to work with later.
While this is a direct answer to the question I think it's least maintainable.
†  This useful modifier, for "non-destructive substitution," appeared in v5.14. In earlier Perl versions we would copy the string and run regex on that, with an idiom
(my $n = $1) =~ s/",\s*"/ /g;
In the lookarounds-example we then need a little more
$string =~ s{...}{ (my $n = $1) =~ s/",\s*"/ /g; $n }gr
since s/ operator returns the number of substitutions made while we need $n to be returned from that whole piece of code in {} (the replacement side), to be used as the replacement.
You can use this \G based regex to start the match with :[" and further captures the values appropriately and replaces matched text so that only comma is retained and doublequotes are removed.
(:\[")|(?!^)\G([^"]+)"(,)"
Regex Demo
Your text is almost proper JSON, so it's really easy to go the final inch and make it so, and then process that:
#!/usr/bin/perl
use warnings;
use strict;
use feature qw/say postderef/;
no warnings qw/experimental::postderef/;
use JSON::XS; # Install through your OS package manager or a CPAN client
my $str = q/"emailDomains":["google.co.uk","google.com","google.com","google.com","google.co.uk"]/;
my $json = JSON::XS->new();
my $obj = $json->decode("{$str}");
my $fixed = $json->ascii->encode({emailDomains =>
join(', ', $obj->{'emailDomains'}->#*)});
$fixed =~ s/^\{|\}$//g;
say $fixed;
Try Regex: " *, *"
Replace with: ,
Demo

Best way to deal with "Unescaped braces in regex" inside Perl regex

I recently started learning Perl to automate some mindless data tasks. I work on windows machines, but prefer to use Cygwin. Wrote a Perl script that did everything I wanted fine in Cygwin, but when I tried to run it with Strawberry Perl on Windows via CMD I got the "Unescaped left brace in regex is illegal here in regex," error.
After some reading, I am guessing my Cygwin has an earlier version of Perl and modern versions of Perl which Strawberry is using don't allow for this. I am familiar with escaping characters in regex, but I am getting this error when using a capture group from a previous regex match to do a substitution.
open(my $fh, '<:encoding(UTF-8)', $file)
or die "Could not open file '$file' $!";
my $fileContents = do { local $/; <$fh> };
my $i = 0;
while ($fileContents =~ /(.*Part[^\}]*\})/) {
$defParts[$i] = $1;
$i = $i + 1;
$fileContents =~ s/$1//;
}
Basically I am searching through a file for matches that look like:
Part
{
Somedata
}
Then storing those matches in an array. Then purging the match from the $fileContents so I avoid repeats.
I am certain there are better and more efficient ways of doing any number of these things, but I am surprised that when using a capture group it's complaining about unescaped characters.
I can imagine storing the capture group, manually escaping the braces, then using that for the substitution, but is there a quicker or more efficient way to avoid this error without rewriting the whole block? (I'd like to avoid special packages if possible so that this script is easily portable.)
All of the answers I found related to this error were with specific cases where it was more straightforward or practical to edit the source with the curly braces.
Thank you!
I would just bypass the whole problem and at the same time simplify the code:
my $i = 0;
while ($fileContents =~ s/(.*Part[^\}]*\})//) {
$defParts[$i] = $1;
$i = $i + 1;
}
Here we simply do the substitution first. If it succeeds, it will still set $1 and return true (just like plain /.../), so there's no need to mess around with s/$1// later.
Using $1 (or any variable) as the pattern would mean you have to escape all regex metacharacters (e.g. *, +, {, (, |, etc.) if you want it to match literally. You can do that pretty easily with quotemeta or inline (s/\Q$1//), but it's still an extra step and thus error prone.
Alternatively, you could keep your original code and not use s///. I mean, you already found the match. Why use s/// to search for it again?
while ($fileContents =~ /(.*Part[^\}]*\})/) {
...
substr($fileContents, $-[0], $+[0] - $-[0], "");
}
We already know where the match is in the string. $-[0] is the position of the start and $+[0] the position of the end of the last regex match (thus $+[0] - $-[0] is the length of the matched string). We can then use substr to replace that chunk by "".
But let's keep going with s///:
my $i = 0;
while ($fileContents =~ s/(.*Part[^\}]*\})//) {
$defParts[$i] = $1;
$i++;
}
$i = $i + 1; can be reduced to $i++; ("increment $i").
my #defParts;
while ($fileContents =~ s/(.*Part[^\}]*\})//) {
push #defParts, $1;
}
The only reason we need $i is to add elements to the #defParts array. We can do that by using push, so there's no need for maintaining an extra variable. This saves us another line.
Now we probably don't need to destroy $fileContents. If the substitution exists only for the benefit of this loop (so I doesn't re-match already extracted content), we can do better:
my #defParts;
while ($fileContents =~ /(.*Part[^\}]*\})/g) {
push #defParts, $1;
}
Using /g in scalar context attaches a "current position" to $fileContents, so the next match attempt starts where the previous match left off. This is probably more efficient because it doesn't have to keep rewriting $fileContents.
my #defParts = $fileContents =~ /(.*Part[^\}]*\})/g;
... Or we could just use //g in list context, where it returns a list of all captured groups of all matches, and assign that to #defParts.
my #defParts = $fileContents =~ /.*Part[^\}]*\}/g;
If there are no capture groups in the regex, //g in list context returns the list of all matched strings (as if there had been ( ) around the whole regex).
Feel free to choose any of these. :-)
As for the question of escaping, that's what quotemeta is for,
my $needs_escaping = q(some { data } here);
say quotemeta $needs_escaping;
what prints (on v5.16)
some\ \{\ data\ \}\ here
and works on $1 as well. See linked docs for details. Also see \Q in perlre (search for \Q), which is how this is used inside a regex, say s/\Q$1//;. The \E stops escaping (what you don't need).
Some comments.
Relying on deletion so that the regex keeps finding further such patterns may be a risky design. If it isn't and you do use it there is no need for indices, since we have push
my #defParts;
while ($fileContents =~ /($pattern)/) {
push #defParts, $1;
$fileContents =~ s/\Q$1//;
}
where \Q is added in the regex. Better yet, as explained in melpomene's answer the substitution can be done in the while condition itself
push #defParts, $1 while $fileContents =~ s/($pattern)//;
where I used the statement modifier form (postfix syntax) for conciseness.
With the /g modifier in scalar context, as in while (/($pattern)/g) { .. }, the search continues from the position of the previous match in each iteration, and this is a usual way to iterate over all instances of a pattern in a string. Please read up on use of /g in scalar context as there are details in its behavior that one should be aware of.
However, this is tricky here (even as it works) as the string changes underneath the regex. If efficiency is not a concern, you can capture all matches with /g in list context and then remove them
my #all_matches = $fileContents =~ /$patt/g;
$fileContents =~ s/$patt//g;
While inefficient, as it makes two passes, this is much simpler and clearer.
I expect that Somedata cannot possibly, ever, contain }, for instance as nested { ... }, correct? If it does you have a problem of balanced delimiters, which is far more rounded. One approach is to use the core Text::Balanced module. Search for SO posts with examples.

How can I interpolate a variable into a Perl 6 regex?

Synopsis 05 mentions that Perl 6 doesn't interpolate variables into a regex, but you can associate an external variable with a pattern. The docs don't mention this feature as far as I can tell. I think people are still going to want to build up a pattern from a string somehow, so I'm curious how that would work.
Here's a program that demonstrates what happens now. I don't know if that's what is supposed to happen or what anyone intended. I insert a variable into a pattern. If you look at $r with .perl, you see the variable name. Then, I apply the pattern and it matches. I change the variable's value. Now the pattern doesn't match. Change it to something else that would work, and it matches again:
my $target = 'abcdef';
my $n = 'abc';
my $r = rx/ ( <$n> ) /;
# the smart match like this doesn't return a Match object
# https://rt.perl.org/Ticket/Display.html?id=126969
put 'rx// directly: ',
$target ~~ $r
?? "Matched $0" !! 'Misssed';
# now, change $n. The same $r won't match.
$n = 'xyz';
put 'rx// directly: ',
$target ~~ $r
?? "Matched $0" !! 'Misssed';
# now, change back $n. The same $r does match.
$n = 'ab';
put 'rx// directly: ',
$target ~~ $r
?? "Matched $0" !! 'Misssed';
If that's what it's supposed to do, fine. The docs are light here, and the tests (the de facto spec) aren't sophisticated for long range behavior like this.
I could do extra work to close over a copy (and maybe more work than I show depending on what is in $n), which I find unperly:
my $r = do {
my $m = $n;
rx/ <$m> /;
};
But, I'd still like to have a way to "finalize" a pattern (oh my god, I just asked for /o to come back). A method in Regex perhaps. I think people will look for this feature.
my $r = rx/ .... /.finalize; # I wish!
Or, Perl 6 has a much better way to do this sort of thing and I'm just full of old school thinking. Perl 6 has rules instead of regexes. There's actually a parser behind all this. I thought defining a token or rule might be the way to go, but I think I run into the same problem. I don't see a what to have a subrule factory.
Is there some other way I could do this?
As an alternative to using a closure, you can of course build the regex via EVAL.
Besides these two arguably subpar solutions, I'm drawing a blank as well. Note that you can do more complex interpolations via the <{...}> syntax, eg
/ <{ BEGIN compute-string-once-at-compile-time }> /
but I don't see how that can be used to solve the problem...

Perl regex substitution using external parameters

Consider the following example:
my $text = "some_strange_thing";
$text =~ s/some_(\w+)_thing/no_$1_stuff/;
print "Result: $text\n";
It prints
"Result: no_strange_stuff"
So far so good.
Now, I need to get both the match and replacement patterns from external sources (user input, config file, etc).
Naive solution appears to be like this:
my $match = "some_(\\w+)_thing";
my $repl = "no_\$1_stuff";
my $text = "some_strange_thing";
$text =~ s/$match/$repl/;
print "Result: $text\n";
However:
"Result: no_$1_stuff".
What's wrong? How can I get the same outcome with externally supplied patterns?
Solution 1: String::Substitution
Use String::Substitution package:
use String::Substitution qw(gsub_modify);
my $find = 'some_(\w+)_thing';
my $repl = 'no_$1_stuff';
my $text = "some_strange_thing";
gsub_modify($text, $find, $repl);
print $text,"\n";
The replacement string only interpolates (term used loosely) numbered match vars (like $1 or ${12}). See "interpolate_match_vars" for more information.
This module does not save or interpolate $& to avoid the "considerable performance penalty" (see perlvar).
Solution 2: Data::Munge
This is a solution mentioned by Grinnz in the comments below.
The Data::Munge can be used the following way:
use Data::Munge;
my $find = qr/some_(\w+)_thing/;
my $repl = 'no_$1_stuff';
my $text = 'some_strange_thing';
my $flags = 'g';
print replace($text, $find, $repl, $flags);
# => no_strange_stuff
Solution 3: A quick'n'dirty way (if replacement won't contain double quotes and security is not considered)
DISCLAIMER: I provide this solution as this approach can be found online, but its caveats are not explained. Do not use it in production.
With this approach, you can't have a replacement string that includes a " double quotation mark and, since this is equivalent to handing whoever is writing the configuration file direct code access, it should not be exposed to Web users (as mentioned by Daniel Martin).
You can use the following code:
#!/usr/bin/perl
my $match = qr"some_(\w+)_thing";
my $repl = '"no_$1_stuff"';
my $text = "some_strange_thing";
$text =~ s/$match/$repl/ee;
print "Result: $text\n";
See IDEONE demo
Result:
Result: no_strange_stuff
You have to
Declare the replacement in '"..."' so as $1 could be later evaluated
Use /ee to force the double evaluation of the variables in the replacement.
A modifier available specifically to search and replace is the s///e evaluation modifier. s///e treats the replacement text as Perl code, rather than a double-quoted string. The value that the code returns is substituted for the matched substring. s///e is useful if you need to do a bit of computation in the process of replacing text.
You can use qr to instantiate pattern for the regex (qr"some_(\w+)_thing").
Essentially the same approach as the accepted solution, but I kept the initial lines the same as the problem statement, since I thought that might make it easier to fit into more situations:
my $match = "some_(\\w+)_thing";
my $repl = "no_\$1_stuff";
my $qrmatch = qr($match);
my $code = $repl;
$code =~ s/([^"\\]*)(["\\])/$1\\$2/g;
$code = qq["$code"];
if (!defined($code)) {
die "Couldn't find appropriate quote marks";
}
my $text = "some_strange_thing";
$text =~ s/$qrmatch/$code/ee;
print "Result: $text\n";
Note that this works no matter what is in $repl, whereas the naive solution has issues if $repl contains a double quote character itself, or ends in a backslash.
Also, assuming that you're going to run the three lines at the end (or something like it) in a loop, do make sure that you don't skip the qr line. It will make a huge performance difference if you skip the qr and just use s/$match/$code/ee.
Also, even though it's not as trivial to get arbitrary code execution with this solution as it is with the accepted one, it wouldn't surprise me if it's still possible. In general, I'd avoid solutions based on s///ee if the $match or $repl come from untrusted users. (e.g., don't build a web service out of this)
Doing this kind of replacement securely when $match and $repl are supplied by untrusted users should be asked as a different question if your use case includes that.

In Perl, how can I correctly extract URLs that are enclosed in parentheses?

I've got two question about Regexp::Common qw/URI/ and Regex in Perl.
I use Regexp::Common qw/URI/ to parse URI in the strings and delete them. But I've got an error when a URI is between parentheses.
For example: (http://www.example.com)
The error is caused by ')', and when it try to parse the URI, the app crash. So I've thought two fixes:
Do a simple (or I thought so) that writes a whitespace between parentheses and ) characters
The Regexp::Common qw/URI/ has a function that implement a fix.
In my code I've tried to implement the Regex but the app freezes. The code that I've tried is this:
use strict;
use Regexp::Common qw/URI/;
my $str = "Hello!!, I love (http://www.example.com)";
while ($str =~ m/\)/){
$str =~ s/\)/ \)/;
}
my ($uri) = $str =~ /$RE{URI}{-keep}/;
print "$uri\n";
print $str;
The output that I want is: (http://www.example.com )
I'm not sure, but I think that the problem is in $str =~ s/\)/ \)/;
BTW, I've got a question about Regexp::Common qw/URI/. I've got two string type:
ablalbalblalblalbal http://www.example.com
asfasdfasdf http://www.example.com aasdfasdfasdf
I want to remove the URI if it is the last component (and save it). And, if not, save it without removing it from the text.
You don't have to first test for a match to be able to use the s/// operator correctly: If the string does not match the search pattern, it will not do anything.
#!/usr/bin/perl
use strict; use warnings;
my $str = "Hello!!, I love (GOOGLE)";
$str =~ s/\)/ )/g;
print "$str\n";
The general problem of detecting URLs correctly in text is error-prone. See for example Jeff's thoughts on this.
my $str = "Hello!!, I love (GOOGLE)";
while ($str =~ m/)/){
$str =~ s/)/ )/;
}
Your program goes into an infinite loop at this point. To see why, try printing the value of $str each time round the loop.
my $str = "Hello!!, I love (GOOGLE)";
while ($str =~ m/)/){
$str =~ s/)/ )/;
print $str, "\n";
}
The first time it prints "Hello!!, I love (GOOGLE )". The while loop condition is then evaluated again. Your string still matches your regular expression (it still contains a closing parenthesis) so the replacement is run again and this time it prints out "Hello!!, I love (GOOGLE )" with two spaces.
And so it goes on. Each time round the loop another space is added, but each time you still have a closing parenthesis, so another substitution is run.
The simplest solution I can see is to only match the closing parenthesis if it is preceded by a non-whitespace character (using \S).
my $str = "Hello!!, I love (GOOGLE)";
while ($str =~ m/\S)/){
$str =~ s/)/ )/;
print $str, "\n";
}
In this case the loop is only executed once.
Why not just include the parentheses in the search? If the URLs will always be bracketed, then something like this:
#!/usr/bin/perl
use warnings;
use strict;
use Regexp::Common qw/URI/;
my $str = "Hello!!, I love (http://www.google.com)";
my ($uri) = $str =~ / \( ( $RE{URI} ) \) /x;
print "$uri\n";
The regex from Regex::Common can be used as part of a longer regex, it doesn't have to be used on its own. Also I've used the 'x' modifier on the regex to allow whitespace so you can see more clearly what is going on - the brackets with the backslashes are treated as characters to match, those without define what is to matched (presumably like the {-keep} - I've not used that before).
You could also make the brackets optional, with something like:
/ (?: \( ( $RE{URI} ) \) | ( $RE{URI} ) ) /
although that would result in two match variables, one undefined - so something like following would be needed:
my $uri = $1 || $2 || die "Didn't match a URL!";
There's probably a better way to do this, and also if you're not bothered about matching parentheses then you could simply make the brackets optional (via a '?') in the first regex...
To answer your second question about only matching URLs at the end of the line - have a look at Regex 'anchors' which can force a match against the beginning or end of a line: ^ and $ (or \A and \Z if you prefer). e.g. matching a URL at the end of a line only:
/$RE{URI}\Z/