When I run this code:
$_='xaxbxc';
if(/(x(?<foo>.))+/) {
say "&: ", $&;
say "0: ", $-{foo}[0];
say "1: ", $-{foo}[1];
}
I get:
&: xaxbxc
0: c
1:
I understand that this is how it's supposed to work, but I would like to be able to somehow get the list of all matches ('a', 'b', 'c') instead of just the last match (c). How can I do this?
In situations like these, using embeded code blocks provides an easy way out:
my #match;
$_='xaxbxc';
if(/((?:x(.)(?{push #match, $^N}))+)/) {
say "\$1: ", $1;
say "#match"
}
which prints:
$1: xaxbxc
a b c
I don't think there is a way to do this in general (please correct me if I am wrong), but there is likely to be a way to accomplish the same end-goal in specific situations. For example, this would work for your specific code sample:
$_='xaxbxc';
while (/x(?<foo>.)/g) {
say "foo: ", $+{foo};
}
What exactly are you trying to accomplish? Perhaps we could find a solution for your actual problem even if there is no way to do repeating captures.
Perl allows a regular expression to match multiple times with the "g" switch past the end. Each individual match can then be looped over, as described in the Global Matching subsection of the Using Regular Expressions in Perl section of the Perl Regex Tutorial:
while(/(x(?<foo>.))+/g){
say "&: ", $&;
say "foo: ", $+{foo};
}
This will produce an iterated list:
&: xa
foo: a
&: xb
foo: b
&: xc
foo: c
Which still isn't what you want, but it's really close. Combining a global regex (/g) with you previous local regex probably will do it. Generally, make a capturing group around your repeated group, then re-parse just that group with a global regex that represents just a single iteration of that group, and iterate over it or use it as a list.
It looks like a question fairly similar to this one- at least in answer, if not forumlation- has been answered by someone much more competent at Perl than I: "Is there a Perl equivalent of Python's re.findall/re.finditer (iterative regex results)?" You might want to check the answers for that as well, with more details about the proper use of global regexes. (Perl isn't my language, I just have an unhealthy appreciation for regular expressions.)
The %- variable is used when you have more than one of the same named group in the same pattern, not when the a given group happens to be iterated.
That’s why /(.)+/ doesn’t load up $1 with each separate character, just with the last one. Same with /(<x>.)+/. However, with /(<x>.)(<x>.)/ you have two different <x> groups, so $-{x}. Consider:
% perl -le '"foobar" =~ /(?<x>.)(?<x>.)/; print "x#1 is $-{x}[0], x#2 is $-{x}[1]"'
x#1 is f, x#2 is o
% perl -le '"foobar" =~ /(?:(?<x>.)(?<x>.))+/; print "x#1 is $-{x}[0], x#2 is $-{x}[1]"'
x#1 is a, x#2 is r
I'm not sure that is exactly what you're looking for, but the following code should do the trick.
$_='xaxbxc';
#l = /x(?<foo>.)/g;
print join(", ", #l)."\n";
But, I'm not sure this would work with overlapping strings.
Related
I have two arrays #Mister and #Mrs and need to add prefix based on the values.
#Mister = qw(Parasuram Raghavan Srivatsan);
#Mrs = qw(Kalai Padmini Maha);
my $str = "I was invited the doctor Parasuram and Kalai and civil Engineer Raghavan and Padmini and finally Advocate Srivatsan and Maha";
#Mr. Parasuram Mr. Raghavan Mr. Srivatsan
if(grep ($_ eq $str), #Mister)
{ $str=~s/($_)/Mr. $1/g; }
#Mrs. Kalai Mrs. Padmini Mrs. Maha`
if(grep ($_ eq $str), #Mrs)
{ $str=~s/($_)/Mrs. $1/g; }
Output Should be:
I was invited the doctor Mr. Parasuram and Mrs. Kalai and civil Engineer Mr. Raghavan and Mrs. Padmini and finally Advocate Mr. Srivatsan and Mrs. Maha
Could someone simplify the way I am doing and whats wrong in this code.
A simple take
my $mr_re = join '|', #Mister;
my $mrs_re = join '|', #Mrs;
$str =~ s/\b($mr_re)\b/Mr. $1/g;
$str =~ s/\b($mrs_re)\b/Ms. $1/g;
(note that I used the neutral Ms above instead of Mrs)
However, when we consider the bewildering complexity of names, the \b doesn't take care of all ways for a name to contain another. An easy example: the - is readily found in names and \b is an anchor between \w and \W, where \w does not include -.
Thus Name-Another would be matched by Name alone as well.
If there are characters other than alphanumeric (plus _) that can be inside names consider
my $w_re = /[a-z-]/i; # list all characters that can be in a name
$str =~ s/(?<!$w_re)($mr_re)(?!$w_re)/Mr. $1/g; # same for Ms.
where negative lookarounds ?<! and ?! are assertions that match your non-name characters (those not listed in $w_re) but do not consume them. Thus they delimit acceptable names.
The same holds for accents, and yet many other characters used in names in various cultures. The task of forming a satisfactory $w_re may be a tricky one even for one particular use case.
If names can come in multiple words (with spaces), in order to handle names within others you would have to parse them in general. That is a complex task; seek modules as little regex won't cut it.
A simple fix would be to preprocess lists to check for names with multiple words that contain other names from your lists, and to handle that case by case.
For your example with hard coded and verifiable names the above works. However, in general, when assembling a regex from strings make sure that all (ascii) non-word chars are escaped so that you actually have the intended literal characters without a special meaning
my $mr_re = join '|', map { quotemeta } #Mister;
my $mrs_re = join '|', map { quotemata } #Mrs;
See quotemeta; inside a regex use \Q, see it in perlbackslash and in perlre.
Note that this problem critically relies on sensible input.
If names are duplicated in lists the problem is ill-posed: If they repeat in the sentence it is unknown which is which, if they don't it is unknown whether it is Mr. or Ms. Thus the name lists should be first checked for duplicates.
"Could someone simplify the way I am doing and whats wrong in this code."
The first part is addressed by zdim in a way I would do it too, but the "what's wrong" part could get some more addressing, in my opinion (just nitpicking, but maybe useful for someone):
if(grep ($_ eq $str), #Mister)
{ $str =~ s/($_)/Mr. $1/g; }
Your list entries will never equal the $str, I think you meant $str =~/$_/
Either use an additional pair of parenthesis around both condition and #list or use the block form of grep (grep { $str =~ /$_/ } #Mister) - otherwise grep will miss the list as argument, since it takes the one existing pair as limiter for it's argument list right now.
the $_ used in the grep command is not available outside of the command, so the $str-substitution would use whatever the value of $_ is currently. In the example it would most likely be undef, so that between each character in the former $str 'Mr. ' is inserted.
Like I said: A perfectly good solution to your problem is given in zdim's answer, but you also asked "what's wrong in this code".
#ssr1012 and other readers: Be careful! It's tempting to think there is a universal solution for this problem. But, unfortunately, even #zdim's approach will give undesirable results if the same name appears in both arrays, and it is still tricky if a name in one array is the same as a name in the other array except for a few additional characters at the start or end.
Here's your example using slightly different names:
my #Mister = qw(Parasuram Mahan Srivatsan);
my #Mrs = qw(Kalai Padmini Maha);
...
# I was invited the doctor Mr. Parasuram and Ms. Kalai and civil Engineer Mr. Ms. Mahan and Ms. Padmini and finally Advocate Mr. Srivatsan and Ms. Maha
See the "Mr. Ms. Mahan"? You don't have enough information for a universal solution. This is only reliable if your names are hard-coded and checked first to avoid collisions.
Even if you added first names, you might not have enough information - guessing gender from first names is unreliable in many language cultures.
I am trying to find out if basket has apple [simplified version of a big problem]
$check_fruit = "\$fruit =~ \/has\/apple\/";
$fruit="basket/has/mango/";
if ($check_fruit) {
print "apple found\n";
}
check_fruit variable is holding the statement of evaluating the regexp.
However it check_fruit variable always becomes true and shows apple found :(
Can somebody help me here If I am missing something.
Goal to accomplish:
Okay so let me explain:
I have a file with a pattern clause defined on eachline similar to:
Line1: $fruit_origin=~/europe\\/finland/ && $fruit_taste=~/sweet/
Line2: similar stuff that can contain ~10 pattern checks seprated by && or || with metacharacters too
2.I have another a list of fruit attributes from a perl hash containing many such fruits
3 I want to categorize each fruit to see how many fruits fall into category defined by each line of the file seprately.
Sort of fruit count /profile per line Is there an easier way to accomplish this ? Thanks a lot
if ($check_fruit) returns true because $check_fruit is defined, not empty and not zero. If you want to evaluate its content, use eval. But a subroutine would serve better:
sub check_fruit {
my $fruit = shift;
return $fruit =~ m(has/apple);
}
if (check_fruit($fruit)) {
print "Apple found\n";
}
Why is there a need to store the statement in a variable? If you're sure the value isn't set by a user, then you can do
if (eval $check_fruit) {
but this isn't safe if the user can set anything in that expression.
Put the pattern (and only the pattern) into the variable, use the variable inside the regular expression matching delimiters m/.../. If you don't know the pattern in advance then use quotemeta for escaping any meta characters.
It should look like this:
my $check_fruit = '/has/apple/'; # here no quotemeta is needed
my $fruit = 'basket/has/mango/';
if ($fruit =~ m/$check_fruit/) {
# do stuff!
}
$check_fruit is nothing but a variable holding string data. If you want to execute the code it contains, you have to use eval.
There were also some other errors in your code related to string quoting/escaping. This fixes that as well:
use strict;
use warnings;
my $check_fruit = '$apple =~ m|/has/mango|';
my $apple="basket/has/mango/";
if (eval $check_fruit) {
print "apple found\n";
}
However, this is not usually a good design. At the very least, it makes for confusing code. It is also a huge security hole if $check_fruit is coming from the user. You can put a regex into a variable, which is preferable:
Edit: note that a regex that comes from user input can be a security problem as well, but it is more limited in scope.
my $check_fruit = qr|/has/mango|;
my $apple="basket/has/mango/";
if ($apple =~ /$check_fruit/) {
print "apple found\n";
}
There are other things you can do to make your Perl code more dynamic, as well. The best approach would depend on what you are trying to accomplish.
I am trying to write a common regular expression for the below 3 cases:
Supernatural_S07E23_720p_HDTV_X264-DIMENSION.mkv
the.listener.313.480p.hdtv.x264-2hd.mkv
How.I.met.your.mother.s02e07.hdtv.x264-xor.avi
Now my regular exoression should remove the series name from the original string i,e the output of above string will be:
S07E23_720p_HDTV_X264-DIMENSION.mkv
313.480p.hdtv.x264-2hd.mkv
s02e07.hdtv.x264-xor.avi
Now for the basic case of supernatural string I wrote the below regex and it worked fine but as soon as the series name got multiple words it fails.
$string =~ s/^(.*?)[\.\_\- ]//i; #delimiter can be (. - _ )
So, I have no idea how to proceed for the aboves cases I was thinking along the lines of \w+{1,6} but it also failed to do the required.
PS: Explanation of what the regular expression is doing will be appreciated.
you can detect if the .'s next token contains digit, if not, consider it as part of the name.
HOWEVER, I personally think there is no perfect solution for this. it'd still meet problem for something like:
24.313.480p.hdtv.x264-2hd.mkv // 24
Warehouse.13.s02e07.hdtv.x264-xor.avi // warehouse 13
As StanleyZ said, you'll always get into trouble with names containing numbers.
But, if you take these special cases appart, you can try :
#perl
$\=$/;
map {
if (/^([\w\.]+)[\.\_]([SE\d]+[\.\_].*)$/i) {
print "Match : Name='$1' Suffix='$2'";
} else {
print "Did not match $_";
}
}
qw!
Supernatural_S07E23_720p_HDTV_X264-DIMENSION.mkv
the.listener.313.480p.hdtv.x264-2hd.mkv
How.I.met.your.mother.s02e07.hdtv.x264-xor.avi
!;
which outputs :
Match : Name='Supernatural' Suffix='S07E23_720p_HDTV_X264-DIMENSION.mkv'
Match : Name='the.listener' Suffix='313.480p.hdtv.x264-2hd.mkv'
Match : Name='How.I.met.your.mother' Suffix='s02e07.hdtv.x264-xor.avi'
note : aren't you doing something illegal ? ;)
I'm trying to create a PCRE that will match only the commas used as delimiters in a line from a CSV file. Assuming the format of a line is this:
1,"abcd",2,"de,fg",3,"hijk"
I want to match all of the commas except for the one between the 'e' and 'f'. Alternatively, matching just that one is acceptable, if that is the easier or more sensible solution. I have the sense that I need to use a negative lookahead assertion to handle this, but I'm finding it a bit too difficult to figure out.
See my post that solves this problem for more detail.
^(?:(?:"((?:""|[^"])+)"|([^,]*))(?:$|,))+$ Will match the whole line, then you can use match.Groups[1 ].Captures to get your data out (without the quotes). Also, I let "My name is ""in quotes""" be a valid string.
CSV parsing is a difficult problem, and has been well-solved. Whatever language you are using doubtless has a complete solution that takes care of it, without you having to go down the road of writing your own regex.
What language are you using?
As you've already been told, a regular expression is really not appropriate; it is tricky to deal with the general case (doubly so if newlines are allowed in fields, and triply so if you might have to deal with malformed CSV data.
I suggest the tool CSVFIX as likely to do what you need.
To see how bad CSV can be, consider this data (with 5 clean fields, two of them empty):
"""",,"",a,"a,b"
Note that the first field contains just one double quote. Getting the two double quotes squished to one is really rather tough; you probably have to do it with a second pass after you've captured both with the regex. And consider this ill-formed data too:
"",,"",a",b c",
The problem there is that the field that starts with a contains a double quote; how to interpret it? Stop at the comma? Then the field that starts with b is similarly ill-formed. Stop at the next quote? So the field is a",b c" (or should the quotes be removed)? Etc...yuck!
This Perl gets pretty close to handling correctly both the above lines of data with a ghastly regex:
use strict;
use warnings;
my #list = ( q{"""",,"",a,"a,b"}, q{"",,"",a",b c",} );
foreach my $string (#list)
{
print "Pattern: <<$string>>\n";
while ($string =~ m/ (?: " ( (?:""|[^"])* ) " | ( [^,"] [^,]* ) | ( .? ) )
(?: $ | , ) /gx)
{
print "Found QF: <<$1>>\n" if defined $1;
print "Found PF: <<$2>>\n" if defined $2;
print "Found EF: <<$3>>\n" if defined $3;
}
}
Note that as written, you have to identify which of the three captures was actually used. With two stage processing, you could just deal with one capture and then strip out enclosing double quotes and nested doubled up double quotes. This regex assumes that if the field does not start with a double quote, then there double quote has no special meaning within the field. Have fun ringing the changes!
Output:
Pattern: <<"""",,"",a,"a,b">>
Found QF: <<"">>
Found EF: <<>>
Found QF: <<>>
Found PF: <<a>>
Found QF: <<a,b>>
Found EF: <<>>
Pattern: <<"",,"",a",b c",>>
Found QF: <<>>
Found EF: <<>>
Found QF: <<>>
Found PF: <<a">>
Found PF: <<b c">>
Found EF: <<>>
We can debate whether the empty field (EF) at the end of the first pattern is correct; it probably isn't, which is why I said 'pretty close'. OTOH, the EF at the end of the second pattern is correct.
Also, the extraction of two double quotes from the field """" is not the final result you want; you'd have to post-process the field to eliminate one of each adjacent pair of double quotes.
Without thinking to hard, I would do something like [0-9]+|"[^"]*" to match everything except the comma delimiters. Would that do the trick?
Without context it's impossible to give a more specific solution.
Andy's right: correctly parsing CSV is a lot harder than you probably realise, and has all kinds of ugly edge cases. I suspect that it's mathematically impossible to correctly parse CSV with regexes, particularly those understood by sed.
Instead of sed, use a Perl script that uses the Text::CSV module from CPAN (or the equivalent in your preferred scripting language). Something like this should do it:
use Text::CSV;
use feature 'say';
my $csv = Text::CSV->new ( { binary => 1, eol => $/ } )
or die "Cannot use CSV: ".Text::CSV->error_diag ();
my $rows = $csv->getline_all(STDIN);
for my $row (#$rows) {
say join("\t", #$row);
}
That assumes that you don't have any tab characters embedded in your data, of course - perhaps it would be better to do the subsequent stages in a Real Scripting Language as well, so you could take advantage of proper lists?
I know this is old, but this RegEx works for me:
/(\"[^\"]+\")|[^,]+/g
It could be use potentially with any language. I tested it in JavaScript, so the g is just a global modifier. It works even with messed up lines (extra quotes), but empty is not dealt with.
Just sharing, maybe this will help someone.
I'm spending my weekend analyzing Campaign Finance Contribution records. Fun!
One of the annoying things I've noticed is that entity names are entered differently:
For example, i see stuff like this: 'llc', 'llc.', 'l l c', 'l.l.c', 'l. l. c.', 'llc,', etc.
I'm trying to catch all these variants.
So it would be something like:
"l([,\.\ ]*)l([,\.\ ]*)c([,\.\ ]*)"
Which isn't so bad... except there are about 40 entity suffixes that I can think of.
The best thing I can think of is programmatically building up this pattern , based on my list of suffixes.
I'm wondering if there's a better way to handle this within a single regex that is human readable/writable.
You could just strip out excess crap. Using Perl:
my $suffix = "l. lc.."; # the worst case imaginable!
$suffix =~ s/[.\s]//g;
# no matter what variation $suffix was, it's now just "llc"
Obviously this may maul your input if you use it on the full company name, but getting too in-depth with how to do that would require knowing what language we're working with. A possible regex solution is to copy the company name and strip out a few common words and any words with more than (about) 4 characters:
my $suffix = $full_name;
$suffix =~ s/\w{4,}//g; # strip words of more than 4 characters
$suffix =~ s/(a|the|an|of)//ig; # strip a few common cases
# now we can mangle $suffix all we want
# and be relatively sure of what we're doing
It's not perfect, but it should be fairly effective, and more readable than using a single "monster regex" to try to match all of them. As a rule, don't use a monster regex to match all cases, use a series of specialized regexes to narrow many cases down to a few. It will be easier to understand.
Regexes (other than relatively simple ones) and readability rarely go hand-in-hand. Don't misunderstand me, I love them for the simplicity they usually bring, but they're not fit for all purposes.
If you want readability, just create an array of possible values and iterate through them, checking your field against them to see if there's a match.
Unless you're doing gene sequencing, the speed difference shouldn't matter. And it will be a lot easier to add a new one when you discover it. Adding an element to an array is substantially easier than reverse-engineering a regex.
The first two "l" parts can be simplified by [the first "l" part here]{2}.
You can squish periods and whitespace first, before matching: for instance, in perl:
while (<>) {
$Sq = $_;
$Sq =~ s/[.\s]//g; # squish away . and " " in the temporary save version
$Sq = lc($Sq);
/^llc$/ and $_ = 'L.L.C.'; # try to match, if so save the canonical version
/^ibm/ and $_ = 'IBM'; # a different match
print $_;
}
Don't use regexes, instead build up a map of all discovered (so far) entries and their 'canonical' (favourite) versions.
Also build a tool to discover possible new variants of postfixes by identifying common prefixes to a certain number of characters and printing them on the screen so you can add new rules.
In Perl you can build up regular expressions inside your program using strings. Here's some example code:
#!/usr/bin/perl
use strict;
use warnings;
my #strings = (
"l.l.c",
"llc",
"LLC",
"lLc",
"l,l,c",
"L . L C ",
"l W c"
);
my #seps = ('.',',','\s');
my $sep_regex = '[' . join('', #seps) . ']*';
my $regex_def = join '', (
'[lL]',
$sep_regex,
'[lL]',
$sep_regex,
'[cC]'
);
print "definition: $regex_def\n";
foreach my $str (#strings) {
if ( $str =~ /$regex_def/ ) {
print "$str matches\n";
} else {
print "$str doesn't match\n";
}
}
This regular expression could also be simplified by using case-insensitive matching (which means $match =~ /$regex/i ). If you run this a few times on the strings that you define, you can easily see cases that don't validate according to your regular expression. Building up your regular expression this way can be useful in only defining your separator symbols once, and I think that people are likely to use the same separators for a wide variety of abbreviations (like IRS, I.R.S, irs, etc).
You also might think about looking into approximate string matching algorithms, which are popular in a large number of areas. The idea behind these is that you define a scoring system for comparing strings, and then you can measure how similar input strings are to your canonical string, so that you can recognize that "LLC" and "lLc" are very similar strings.
Alternatively, as other people have suggested you could write an input sanitizer that removes unwanted characters like whitespace, commas, and periods. In the context of the program above, you could do this:
my $sep_regex = '[' . join('', #seps) . ']*';
foreach my $str (#strings) {
my $copy = $str;
$copy =~ s/$sep_regex//g;
$copy = lc $copy;
print "$str -> $copy\n";
}
If you have control of how the data is entered originally, you could use such a sanitizer to validate input from the users and other programs, which will make your analysis much easier.