Team
I have written a Perl program to validate the accuracy of formatting (punctuation and the like) of surnames, forenames, and years.
If a particular entry doesn't follow a specified pattern, that entry is highlighted to be fixed.
For example, my input file has lines of similar text:
<bibliomixed id="bkrmbib5">Abdo, C., Afif-Abdo, J., Otani, F., & Machado, A. (2008). Sexual satisfaction among patients with erectile dysfunction treated with counseling, sildenafil, or both. <emphasis>Journal of Sexual Medicine</emphasis>, <emphasis>5</emphasis>, 1720–1726.</bibliomixed>
My programs works just fine, that is, if any entry doesn't follow the pattern, the script generates an error. The above input text doesn't generate any error. But the one below is an example of an error because Rose A. J. is missing a comma after Rose:
NOT FOUND: <bibliomixed id="bkrmbib120">Asher, S. R., & Rose A. J. (1997). Promoting children’s social-emotional adjustment with peers. In P. Salovey & D. Sluyter, (Eds). <emphasis>Emotional development and emotional intelligence: Educational implications.</emphasis> New York: Basic Books.</bibliomixed>
From my regex search pattern, is it possible to capture all the surnames and the year, so I can generate a text prefixed to each line as shown below?
<BIB>Abdo, Afif-Abdo, Otani, Machado, 2008</BIB><bibliomixed id="bkrmbib5">Abdo, C., Afif-Abdo, J., Otani, F., & Machado, A. (2008). Sexual satisfaction among patients with erectile dysfunction treated with counseling, sildenafil, or both. <emphasis>Journal of Sexual Medicine</emphasis>, <emphasis>5</emphasis>, 1720–1726.</bibliomixed>
My regex search script is as follows:
while(<$INPUT_REF_XML_FH>){
$line_count += 1;
chomp;
if(/
# bibliomixed XML ID tag and attribute----<START>
<bibliomixed
\s+
id=".*?">
# bibliomixed XML ID tag and attribute----<END>
# --------2 OR MORE AUTHOR GROUP--------<START>
(?:
(?:
# pattern for surname----<START>
(?:(?:[\w\x{2019}|\x{0027}]+\s)+)? # surnames with spaces
(?:(?:[\w\x{2019}|\x{0027}]+-)+)? # surnames with hyphens
(?:[A-Z](?:\x{2019}|\x{0027}))? # surnames with closing single quote or apostrophe O’Leary
(?:St\.\s)? # pattern for St.
(?:\w+-\w+\s)?# pattern for McGillicuddy-De Lisi
(?:[\w\x{2019}|\x{0027}]+) # final surname pattern----REQUIRED
# pattern for surname----<END>
,\s
# pattern for forename----<START>
(?:
(?:(?:[A-Z]\.\s)+)? #initials with periods
(?:[A-Z]\.-)? #initials with hyphens and periods <<Y.-C. L.>>
(?:(?:[A-Z]\.\s)+)? #initials with periods
[A-Z]\. #----REQUIRED
# pattern for titles....<START>
(?:,\s(?:Jr\.|Sr\.|II|III|IV))?
# pattern for titles....<END>
)
# pattern for forename----<END>
,\s)+
#---------------FINAL AUTHOR GROUP SEPATOR----<START>
&\s
#---------------FINAL AUTHOR GROUP SEPATOR----<END>
# --------2 OR MORE AUTHOR GROUP--------<END>
)?
# --------LAST AUTHOR GROUP--------<START>
# pattern for surname----<START>
(?:(?:[\w\x{2019}|\x{0027}]+\s)+)? # surnames with spaces
(?:(?:[\w\x{2019}|\x{0027}]+-)+)? # surnames with hyphens
(?:[A-Z](?:\x{2019}|\x{0027}))? # surnames with closing single quote or apostrophe O’Leary
(?:St\.\s)? # pattern for St.
(?:\w+-\w+\s)?# pattern for McGillicuddy-De Lisi
(?:[\w\x{2019}|\x{0027}]+) # final surname pattern----REQUIRED
# pattern for surname----<END>
,\s
# pattern for forename----<START>
(?:
(?:(?:[A-Z]\.\s)+)? #initials with periods
(?:[A-Z]\.-)? #initials with hyphens and periods <<Y.-C. L.>>
(?:(?:[A-Z]\.\s)+)? #initials with periods
[A-Z]\. #----REQUIRED
# pattern for titles....<START>
(?:,\s(?:Jr\.|Sr\.|II|III|IV))?
# pattern for titles....<END>
)
# pattern for forename----<END>
(?: # pattern for editor notation----<START>
\s\(Ed(?:s)?\.\)\.
)? # pattern for editor notation----<END>
# --------LAST AUTHOR GROUP--------<END>
\s
\(
# pattern for a year----<START>
(?:[A-Za-z]+,\s)? # July, 1999
(?:[A-Za-z]+\s)? # July 1999
(?:[0-9]{4}\/)? # 1999\/2000
(?:\w+\s\d+,\s)?# August 18, 2003
(?:[0-9]{4}|in\spress|manuscript\sin\spreparation) # (1999) (in press) (manuscript in preparation)----REQUIRED
(?:[A-Za-z])? # 1999a
(?:,\s[A-Za-z]+\s[0-9]+)? # 1999, July 2
(?:,\s[A-Za-z]+\s[0-9]+\x{2013}[0-9]+)? # 2002, June 19–25
(?:,\s[A-Za-z]+)? # 1999, Spring
(?:,\s[A-Za-z]+\/[A-Za-z]+)? # 1999, Spring\/Winter
(?:,\s[A-Za-z]+-[A-Za-z]+)? # 2003, Mid-Winter
(?:,\s[A-Za-z]+\s[A-Za-z]+)? # 2007, Anniversary Issue
# pattern for a year----<END>
\)\.
/six){
print $FOUND_REPORT_FH "$line_count\tFOUND: $&\n";
$found_count += 1;
} else{
print $ERROR_REPORT_FH "$line_count\tNOT FOUND: $_\n";
$not_found_count += 1;
}
Thanks for your help,
Prem
Alter this bit
# pattern for surname----<END>
,?\s
This now means an optional , followed by white space. If the Persons surname is "Bunga Bunga" it won't work
All of your subpatterns are non-capturing groups, starting with (?:. This reduces compilation times by a number of factors, one of which being that the subpattern is not captured.
To capture a pattern you merely need to place parenthesis around the part you require to capture. So you could remove the non-capturing assertion ?: or place parens () where you need them. http://perldoc.perl.org/perlretut.html#Non-capturing-groupings
I'm not sure but, from your code I think you may be attempting to use lookahead assertions as, for example, you test for surnames with spaces, if none then test for surnames with hyphens. This will not start from the same point every time, it will either match the first example or not, then move forward to test the next position with the second surname pattern, whether the regex will then test the second name for the first subpattern is what I am unsure of. http://perldoc.perl.org/perlretut.html#Looking-ahead-and-looking-behind
#!usr/bin/perl
use warnings;
use strict;
my $line = '123 456 7antelope89';
$line =~ /^(\d+\s\d+\s)?(\d+\w+\d+)?/;
my ($ay,$be) = ($1 ? $1:'nocapture ', $2 ? $2:'nocapture ');
print 'a: ',$ay,'b: ',$be,$/;
undef for ($ay,$be,$1,$2);
$line = '123 456 7bealzelope89';
$line =~ /(?:\d+\s\d+\s)?(?:\d+\w+\d+)?/;
($ay,$be) = ($1 ? $1:'nocapture ', $2 ? $2:'nocapture ');
print 'a: ',$ay,'b: ',$be,$/;
undef for ($ay,$be,$1,$2);
$line = '123 456 7canteloupe89';
$line =~ /((?:\d+\s\d+\s))?(?:\d+(\w+)\d+)?/;
($ay,$be) = ($1 ? $1:'nocapture ', $2 ? $2:'nocapture ');
print 'a: ',$ay,'b: ',$be,$/;
undef for ($ay,$be,$1,$2);
exit 0;
For capturing the whole pattern the first pattern of the third example does not make sense, as this tells the regex to not capture the pattern group while also capturing the pattern group. Where this is useful is in the second pattern which is a fine grained pattern capture, in that the pattern captured is part of a non-capturing group.
a: 123 456 b: 7antelope89
a: nocapture b: nocapture
a: 123 456 b: canteloupe
One little nitpic
id=".*?"
may be better as
id="\w*?"
id names requiring to be _alphanumeric iirc.
Related
I am sure this is a simple thing to do but I cannot seem to find any examples or make it past the numerous documentation sources I have been using.
I have a variable in a table (called location) such as: OH_DRT HOME_G4-T7 77 Cafe Entrance
I want to be able to parse this into several columns based on some delimiters. There is variability in my data set so I thought using perl expressions for pattern matching would be the way to go. I am trying to take that string and break it up into something like this:
State
Building
Name
Desc
OH
DRT HOME
G4
T7 Cafe Entrance
FL
Cleveland
RG
03 Back Entry
I am able to split the first part out
Data Mydata;
Set Int_Data;
retain re;
if _N_ = 1 Then re = prxparse("/(\D{2})/");
if prxmatch(re, location) Then Do
State= prxposn(re,1,location);
end;
It is parsing out any of the other sections I am at a loss for. The only one I have been able to get to work correctly is the State. I assume I should be able to pull anything between two characters.
In my head I should be able to split something like this:
Anything before the first _, anything between the first _ and second _, anything second _ to first -, and then finally anything after the -
Are all records exactly the same? If so:
use warnings;
use strict;
my $data = 'OH_DRT HOME_G4-T7 77 Cafe entrance';
my ($state, $building, $name, $desc);
if ($data =~ /^([A-Z]{2})_(.*)_(\w{2})-\w{2}\s+(.*)$/) {
$state = $1;
$building = $2;
$name = $3;
$desc = $4;
}
print "$state, $building, $name, $desc\n";
The regex works as follows:
Capture two upper-cased letters at the start of the string and put it into $1
Skip an underscore and capture everything until the next underscore and put it into $2
Capture the following two word characters and put them into $3
Skip a hyphen and the following two word characters along with any amount of whitespace, and put the remaining portion of the string into $4
Assign the numbered matches into the more descriptive named variables
Note that if any of the matches/captures fail, all of the named variables will be undefined.
The output of the above is:
OH, DRT HOME, G4, 77 Cafe entrance
You can use a pattern with 4 capture groups, but note that when keeping the following remark into account, it will give T7 77 Cafe entrance in the last group.
and then finally anything after the -
If you want to match anything between the underscores and the - you can use a negated character class excluding characters to match that you specify.
To not cross newlines, you can add a newline and a carriage return [^_\r\n]+
^([^_]+)_([^_]+)_([^-]+)-(.*)
Explanation
^ Start of string
([^_]+)_ Capture 1+ chars other than _ in group 1 and then match it
([^_]+)_ Capture 1+ chars other than _ in group 2 and then match it
([^-]+)- Capture 1+ chars other than - in group 3 and then match it
(.*) Match all after the underscore in group 4
Regex demo
If you want 77 Cafe entrance in group 4:
^([^_]+)_([^_]+)_([^-]+)-[^\s-]*\s*(.*)
Regex demo
I'm sure the regex solution works fine. If you wanted a SCAN solution.
Data WANT(Keep STATE BUILDING NAME DESC);
Length State $2 Building $50 Name $2 Desc $100;
TEST="OH_DRT HOME_G4-T7 77 Cafe Entrance";
State=scan(test,1,"_");
Building=scan(test,2,"_");
temp=scan(test,3,"_");
Name=scan(temp,1,"-");
Desc=scan(temp,2,"-");
Run;
I have strings in this form:
"""00.000000 00.000000; X-XX000-0000-0; France; Paris; Street 12a;
00.000000 00.000000; X-XX000-0000-0; Spain; Barcelona; Street 123;"""
I want to get specific data towns above string. How do I get this data??
If you just want to get the city for your given example, you could use a positive lookahead:
\b[^;]+(?=;[^;]+;$)
Explanation
\b # Word boundary
[^;]+ # Match NOT ; one or more times
(?= # Positive lookahead that asserts what follows is
; # Match semicolon
[^;]+ # Match NOT ; one or more times
; # Match ;
$ # Match end of the string
) # Close lookahead
Assuming Python (three quotes-string):
string = """00.000000 00.000000; X-XX000-0000-0; France; Paris; Street 12a;
00.000000 00.000000; X-XX000-0000-0; Spain; Barcelona; Street 123;"""
towns = [part[3] for line in string.split("\n") for part in [line.split("; ")]]
print(towns)
Which yields
['Paris', 'Barcelona']
No regex needed, really.
If you have the city on the 4th field, you can match it using this pattern:
/(?:[^;]*;){3}([^;]*);/
See the demo
[^;]*; you find a field consisting in non-semicolons and ending with a semicolon
(?:...){3} you find it 3 times, but you do not capture it
([^;]*); then you get 4th column matching its content (not the semicolon)
I have a simple receipt which I typed out. I need to be able to read the items purchased on the receipt. The sample receipt is below.
Tim Hortons
Alwasy Fresh
1 Brek Wrap Combo /A ($0.76)
1 Bacon-wrap $3.79
1 Grilled $0.00
1 5 Pieces Bacon-wrap $0.00
1 Orange $1.40
1 Deposit $0.10
Subtotal: $55.84
GST: $0.29
Debit: $55.84
Take out
Thanks for stopping by!!
Tell us how we did
I came up with the following regex string to find the items.
\d(\s){1,10}(.)*\s{1,}\$\d\.[0-9]{2}
It works for the most part but there are a few incorrect lines like
4
GST: $0.29
Can someone come up with a better pattern. Below is a link to see it in action.
http://regexr.com/3cnk9
I see a number of problems with this original regex:
\d(\s){1,10}(.)*\s{1,}\$\d\.[0-9]{2}
First, parentheses both group and match, though when you quantify your match, only the last iteration is captured, so matching like (.)* will only store the last character; you wanted (.*) for that. Since it's greedy, that will be the character before the space preceding a dollar sign, which given your data will always be a space. Similarly, you're quantifying a group at the beginning with (\s){1,10}, which captures only the last whitespace character. In this case, you don't need the group since \s is a single space character, so you can simply use \s{1,10}.
Here is a piece-by-piece explanation of what that regular expression does.
Capturing solution
The following regex captures the quantity ($1), item description ($2), whether the price is parenthesized ($3), and the price ($4):
^\s*(\d+)\s+(.*\S)\s+(\(?)\$([0-9.]+)\)?\s*$
Explained and matched to your sample at regex101.
Separated out and commented (assumes the /x flag is supported):
/ # begin regex
^\s* # start of line, ignore leading spaces if present
(\d+) # $1 = quantity
\s+ # spacing as a delimiter
(.*\S) # $2 = item: contains anything, must end in a non-space char
\s+ # spacing as a delimiter
(\(?) # $3 = negation, an optional open parenthesis
\$ # dollar sign
([0-9.]+) # $4 = price
\)?\s*$ # trailing characters: optional end-paren and space(s)
/x # end regex, multi-line regex flag
with sample perl code executed from a command line:
perl -ne '
my ($quantity, $item, $neg, $price)
= /^\s*(\d+)\s+(.*\S)\s+(\(?)\$([0-9.]+)\)?\s*$/;
if ($item) {
if ($neg) { $price *= -1; }
print "<$quantity><$item><$price>\n"
}' RECEIPT_FILE
(If you want that as a perl script, wrap the code with while(<>) { } and you're done.)
This assigns the variables $quantity, $item, and $price to the itemized lines on your receipt. I am assuming that a parenthesized item is to be subtracted (but I can't verify that since the totals are nonsensical), so $neg notes the existence of a parenthesis so the $price can be negated.
I set the output to use angle brackets (< and >) to indicate what each variable stores.
The output of your given sample receipt would therefore be:
<1><Brek Wrap Combo /A><-0.76>
<1><Bacon-wrap><3.79>
<1><Grilled><0.00>
<1><5 Pieces Bacon-wrap><0.00>
<1><Orange><1.40>
<1><Deposit><0.10>
Prices only solution
You didn't say what you wanted to match. If you don't care about anything but the prices and there are no negative values, you don't need matchers if you have negative look-behind or \K:
grep -Po '^\s*[0-9].*\$\K[0-9.]+' RECEIPT_FILE
Grep's -P flag invokes libpcre (which may not be available if you're on an old or embedded system) and -o displays only the matching text. \K denotes the start of the match. Put the \$ after the \K if you want to capture it. (See also the regex101 description and matches.)
Output from that grep command:
0.76
3.79
0.00
0.00
1.40
0.10
Prices only – with awk
There aren't great ways to handle this regex with efficiency. If you're processing through a mountain of content, you'll feel the hurt. Here's a solution using awk that should be significantly faster. (The difference won't be noticeable with a small input.)
awk '$1 / 1 > 0 && $NF ~ /\$/ { gsub(/[()]/, "", $0); print $NF; }' RECEIPT_FILE
Commented version with explanation:
awk '
# if the quantity is indeed a number and the last field has a dollar sign
$1 / 1 > 0 && $NF ~ /\$/ {
gsub(/[()]/, "", $NF); # remove all parentheses from the last field
print $NF; # print the contents of the last field
}' RECEIPT_FILE
Prices only – with awk, supporting negative prices
awk '
# if the quantity is indeed a number and the last field has a dollar sign
$1 / 1 > 0 && $NF ~ /\$/ {
neg = 1;
if ( $NF ~ /\(/ ) { # the last field has an open parenthesis
gsub(/[()]/, "", $NF); # remove all parentheses from the last field
neg = -1;
}
print $NF * neg; # print the last field, negated if parenthesized
}' RECEIPT_FILE
Here's my attempt:
^(\d+)\s+(.*)\s+\(?(\$.+)\)?$
Stub. Remember to turn the multiline option on. Components:
^ - beginning of line
(\d+) - capture the quantity at the beginning of each line item
\s+ - one or more space
(.*) - capture the item description
\s+ - one or more space
\(? - optional open bracket `(` character
($.+) - capture anything including and after the dollar sign
\)? - optional close bracket `)` character
$ - end of line
You can use
^(\d+)\s+(.*?)\s+\(?\$(\d+\.\d+)
See the regex demo
This regex should be used with the /m modifier to match data on different lines. In JS, the /g modifier is also required.
Explanation:
^ - start of a line
(\d+) - Group 1 capturing one or more digits
\s+ - one or more whitespaces
(.*?) - Group 2 capturing zero or more any characters but a newline up to the closest
\s+ - one or more whitespaces
\(? - an optional ( (on the first line)
\$ - a literal $
(\d+\.\d+) - Group 3 capturing one or more digits followed with . and one or more digits.
JS demo:
var re = /^(\d+)\s+(.*?)\s+\(?\$(\d+\.\d+)/gm;
var str = ' Tim Hortons\n Alwasy Fresh\n\n1 Brek Wrap Combo /A ($0.76)\n1 Bacon-wrap $3.79\n1 Grilled $0.00\n1 5 Pieces Bacon-wrap $0.00\n1 Orange $1.40\n1 Deposit $0.10\nSubtotal: $55.84\nGST: $0.29\nDebit: $55.84\nTake out\n\n Thanks for stopping by!!\n Tell us how we did';
while ((m = re.exec(str)) !== null) {
document.body.innerHTML += "Pcs: <b>" + m[1] + "</b>, item: <b>" + m[2] + "</b>, paid: <b>" + m[3] + "</b><br/>";
}
Adam Katz's answer should be the accepted one! I used this variation of his answer for an implementation in JavaScript:
const receiptRegex = /^\s*(\d+)\s+(.*\S)\s+(\(?)\$([0-9.]+)\)?\s*$/gm
let items = [];
const matches = inputStr.matchAll(receiptRegex);
for (const matchedGroup of matches) {
const [
fullString, //[0] -> matched string "1 Blue gatorade $2.00"
quantity, //[1] -> quantity "1"
item, //[2] -> item description "Blue gatorade"
ignoredSymbol, //[3] -> "$" (should probably always ignore)
price //[4] -> amount "2.00"
] = matchedGroup;
items.push({
quantity,
item,
price,
});
}
I'm looking to do some simple partial redaction for addresses. Basically I'd like to replace the street name before the street suffix with ### while keeping the street suffix.
Examples:
Cherry Street -> ### Street
America Lane -> ### Lane
The full list of suffixes will be known at some point soon, and I could end up replacing the entirety of the address (getting rid of Street and Lane as well as the street name) with something like
regexp_replace(col1, '(\w) (Street|Lane)', '###', 'g')
but I can't figure out how to just replace just the word before the street suffix.
What you need is a positive look ahead (?= )
try the regex
(\w)* (?=Street|Lane)
see how the regex works http://regex101.com/r/pS9oV3/2
Explanation
(\w)* matches anything followed by a space
(?=Street|Lane) asserts if the following regex can be matched. successfull if it can be matched
regexp_replace(col1, '(\w)* (?=Street|Lane)', '###', 'g')
would produce
### Street
### Lane
I'm not familiar with PostgreSQL's exact syntax, but you need to reference the capture group you wanna keep. Something like:
regexp_replace(col1, '(\w) (Street|Lane)', '### $2', 'g')
or
regexp_replace(col1, '(\w) (Street|Lane)', '### \2', 'g')
Where $2 or \2 references the second capture group, so whatever was in the 2nd pair of parentheses will be placed there.
I wouldn't expect all street names to be that simple. Therefore:
SELECT col1
, regexp_replace(col1, '.+(?=\m(Street|Lane)\M)', '### ', 'i')
FROM (
VALUES
('Cherry Street'::text)
, ('America Lane')
, ('Weird name Lane')
, ('Lanex Lane') -- 'Lane' included in street name
, ('Buster-Keaton-Lane')
, ('White Space street') -- with tab
) t(col1);
SQL Fiddle.
Explain
.+ ... any string of one or more characters (including white space)
(?=\m(Street|Lane)\M) ... that is followed by 'Street' or 'Lane'
- (?=) ... positive look-ahead
- \m\M ... begin and end of word
The additional parameter 'i' switches to case-insensitive matching. No point in adding 'g' (replace "globally"), since only a single replacement should happen.
Details in the manual.
I have inherited a perl script that pulls data out of some files. The whole script works fine but recently some engineers have been putting in more than one number for a certain spot that usually took one number, so the output is not showing all of what is expected.
Sample input:
CRXXXX: "Then some text"
CRs XXXX, XXXX, XX, XXX
CRXXX "Some Text"
Currently this regex statement I have pulls out the number after the CR, but if given then second line of sample input it prints "s XXXX, XXXX, XX, XXX" instead of the wanted "XXXX XXXX XX XXX"
I am very new to perl and am struggling to figure out how to alter this regex to work on all of the inputs.
$temp_comment =~ s/\s[cC][rR][-\s:;]*([\d])/\n$1/mg;
Thanks in advance!
Brock
For sample data like:
my $temp_comment =
'CR1234: "Then some text"
CRs 2345, 3456, 45, 567
CR678 "Some Text"';
try:
$temp_comment =~ s/(,)|[^\d\n]+/$1?' ':''/semg;
or, if you want to stay close to the string templates:
$temp_comment =~ s/ ^ # multi-line mode, line start
\s* # leading blanks?
CR # CR tag
\D* # non-number stuff
( # start capture group
(?:\d+ [,\s]*)+ # find (number, comma, space) groups
) # end capture group
\D* # skip remaining non-number stuff
$ # multi-line mode, line end
/$1/mxg; # set multi-line mode + regex comments "x"
but you'd have to remove the commas in the number group in a subsequent step.
$temp_comment =~ tr/,//d; # remove commas in the whole string
or
$temp_comment =~ s/(?<=\d),(?=\s\d)//g; # remove commas between numbers '11, 22'
For "single step", you have to use the /e modifier:
$temp_comment =~ s{ ^ # line start
\s* # leading blanks?
CR # CR tag
\D* # non-number stuff
((?:\d+ [,\s]*)+) # single or group of numbers
\D* # non number stuff
$ # line end
}
{do{(local$_=$1)=~y/,//d;$_}}mxeg;
This will, on the data above, result in:
1234
2345 3456 45 567
678
But really, please use, if possible, the simpler two step approach. The latter regex might be a maintainance nightmare for your successors.
You may be better off doing this in two steps:
1) Create your regular expression
s/\s[cC][rR][-\s:;]*([\d\ ]+)/\n$1/mg (note the new way to capture all of the numbers, you're only capturing the first number above)
2) Then just strip out the commas in the string with find/replace.
my ($v) = /CR[s ]*((?:\d+[\s,]*)*)/ig;
$v =~ s/,//g;
print $v,"\n";
Perhaps the following will work for you:
use Modern::Perl;
say join ' ', (/(\d+)/g) for <DATA>;
__DATA__
CR1234: "Then some text"
CRs 1111, 2222, 33, 444
CR567 "Some Text"
Output:
1234
1111 2222 33 444
567
Hope this helps!