I am trying to make a regex that matches proper nouns including numbers (if there are any) i.e. Fifa 2017
I have this:
(?:\s*\b([A-Z][a-z]+)\b)+
...which gets the string without numbers.
Test at: http://regexr.com/3dmuo
I've fiddled around with so many approaches but Regex is dare I say slightly beyond my ability.
Thanks in advance for any advice.
This solution shows how to match a single-word resembling a "proper noun" followed by a number. This explicitly matches a word-like string starting with a capital letter, followed by any number of letters or digits until a space is reached, and then any number of digits.
data = [
"I am reviewing Fifa 2017",
"I am reviewing Mighty No 9",
"I am writing about Unreal Engine",
"Are you interested in MotoGP 2017?",
"When does NASCAR 2017 start?",
"Can Team Ferrari win Formula1 2017?",
"Or will Red Bull take the Formula 1 2017 win?",
"I plan to see F-1 2019, so I best start planning now!",
"Have you used an Apple Mac Book Pro lately?",
"Microsoft makes consumer operating systems"
];
for (var i in data) {
var match = data[i].match(/(?:\b[A-Z][A-Za-z0-9]+\b)(?:\s*\b[A-Z][A-Za-z0-9]+\b)*(?:\s*\d+)?/g);
if (match) {
console.log(data[i], " match: ", match)
} else {
console.log(data[i], " doesn't match!")
}
}
The data used is taken as a riff on the original example of "Fifa 2017", and other major sporting seasons are also represented. There are a variety of requirements represented here.
One failing examples is presented for "F-1 2019", since it fails to meet the original specification. Matching that case, as well, would not be difficult, but the specification would need to be expanded to suit.
There are also a few false matches, due to the specification. These matches are either due to matching text that looks like a "proper noun" (e.g. "When", "Or", "Have") or numbers within the "proper noun", but separated by space (e.g. "Formula 1 2017" matches "Formula 1", but not the "2017"). These may or may not be able to be handled strictly by a regex, and might even be too complex for solving in the general case.
If the input text is suitably constrained, this sort of searching can work, but there may be exceptions that occur unexpectedly.
I looked at the rules for proper nouns in Wikipedia: Letter Case to create a fairly comprehensive english language proper-noun-finder. I have no formal training in regex so please point out any mistakes (still don't know how \b works, ha ha).
(\b(the\b\s\b)?((([A-Z]('[A-Z])?[a-z]+)-*)+\b((\s\b(of|the|de|los|e|van|der|von|zu|d|di|ibn)\b)*(\s\b([A-Z]('[A-Z])?[a-z]+)-*)+\b)*)+)+
The major issue with this parser is that it recognizes the beginning of each sentence as a proper noun.
This may be desirable to you, because the alternative is to sometimes recognize broken words. For example, Jade Smith is swell. Tim van Smythe isn't will recognize Smith and Smythe as the only proper nouns, if you implement my below solution.
If your parser support negative lookbehind, you can prepend (?<!([.!?;]['"]?\s\b)|^) to the regex string.
Some parsers (such as python's re module) will treat the ^ (beginning of string) as a non-fixed width, and reject your search. My solution to this problem was to remove it (making the prepend be (?<![.!?;]['"]?\s\b) instead), and prepend . to the input string.
This matches all(most) words that begin with a single capital letter. It does allow for complicated names, but obviously doesn't take everything into account. I was fairly rigid in only matching correctly-capitalized proper nouns, but regex has limitations and I'm not very good at it in the first place.
For example, here's a list of potential matches:
Tam O'Shanter or Tam-O-Shanter but not Tam o'Shanter or Tam-o-Shanter
Rio di Janeiro or Rio Di Janeiro (both correct as far as I know)
Ludvig van Halen
Shea D'Angelo and Shea d Angelo but not Shea d'Angelo
It does not match acronyms, such as NBA, FIFA, or NHL. Importantly, this means that it will not match Jonah J. Jamieson as a full proper noun (it will match Jonah and Jamieson as two separate nouns). It cannot handle single-letter proper nouns.
Try this:
(?:\s*\b([A-Z][a-z]+)\b)+\s?(\d+)?
Related
I have been trying to create a parser for Law texts.
I need to find a way to find "external links" like : art. 45 alin. (1) din Lege nr. 54/2000
But the problem is that my country law writing style is so, soooo lacking uniformity and that means sometimes the links might look like this : articolul 45 alineatul (1) din Legeea nr. 30/2000
The fact that my language has forms for words for days. (articol, articolului, articolelor....)
That means that i need to generalize that first thing... (art.) as to catch as many forms as possible and pray that the last thing is a law number & year (54/2000).
Now here comes the hard part... The problem is that every section that starts with Articol N starts the regex and it goes on and on until it finds a law number & year that have absolutely no relation between them.
This is how it looks \b(((A|a)rt.*?) \(?\d*?\)??)( \w*? )*?nr\.? (\d+\/\d\d\d\d|\d+\/\d\d\d\d)\b
My question is there a way to limit the words between the two capturing groups?
Link to a Docs to determine what should pass and what not:
https://docs.google.com/document/d/1vn2HwYaCq8UB1felY1GvfmbTI2w8o5RgW4efD9fsvQM/edit?usp=sharing
As Cary and James answered in comments above, I used (?:\S+\s*){0,15}. I used \S instead of \w to include punctuation and thus, abbreviated forms of the names of the Law (e.g. Const. for Constitution). That was the reason why my original regex wasn't working even when using {m,n}.
Context
Hi, earlier I was browsing the web in order to find a quick answer about telephone number validation in one regex formula : for emergency, short, international, french, spanish and north american numbers (normal, fancy and extended versions).
Strangely, I couldn't find better than "A comprehensive regex for phone number formula", since it seems to be the best topic about this, or I missed it, which is totally possible.
So I'm new to the site and actually writing this very first question (yeah!), since that other thread is currently on hold of some sort : seems the author didn't get what he and I were seeking.
That makes at least three of us who would like to have a good solution, as I know at least my pal, the one who asked me first about finding one to be used in simple integrations like his Google Forms.
Hence my current question(s) and own answer to begin with, since I took some night time to build my own based on advices and tests patterns from the best replies on the other thread. If you're interested by the topic, there are some interesting elements.
Questions
What is the best way to optimize and improve this regex (without resorting to coding) which is dedicated to validation of international and most national phone numbers (along the recommendations of RFC 3966 at least)?
Not sure if I can add a related question as well (since it is still on purpose to improve the usefulness of the regex pattern), no harm asking I guess.
Are there other commonly-used formats that this regex should match (and not)?
If you can add them (or a link) here for me to update my test bundles, I would be thankful. Equally useful would be phone numbers that should definitely not be validated (the unwanted).
My initial solution
My current regex solution (version 4) on Regular Expressions 101
An earlier version was matching results despite leading and trailing whitespaces, not that useful to the point (a bit too fancy for the exceution time).
The latest version at the time of writing took into consideration the other posts on the subject RFC 3966 (from the IETF standards) and the wikipedia article on "Natural conventions for writing telephone numbers".
Another potentially side dish is to isolate matching groups for country code, area code and extended code... and things work relatively dandy to a certain point : it only works well when there are some separators (or the parenthesis) to distinguish those groups of digits.
Matching goals
Emergency and short numbers : 112 or 911
Spanish international : +34 987 654 321
French extended +33 (0)1 23 45 67 89
French national : 01 23 45 67 89
American extended : 001-(123)-456-7890 ext-4321
German (Microsoft style) : +49 (1234) 567890
Mexican national : (01 55) 1234 5678
Hypothetical international number (max length?) : 00321-(4321)-567.89 ext-4321
Another matching goal is to have a regex that do not under-perform too much, not really picky since it is not to be used in critical parts of code.
Still, how could we optimize those best regex(es) people will find/propose without changing their results?
Goals from the main thread
+1(234)/567.8901 x1234 and the like (with different permutations of separators : ., /, - and horizontal whitespaces.
2345678901 : same US number dialed in the states I guess.
Not sure how it should work since I though that + (or its equivalent the double zero 00) was required in front of any international number... always done it that way. The other thread had a list of positive matches without.
Could someone confirm that + or 00 is not mandatory to US numbers? Thank you again.
Best of unwanted formats
12(34567890 and 123)456789012345 : unmatched parenthesis.
)123(34567890 : parenthesis are wrongly matched.
++34123456789 : double + is a typo.
+9-123/456.7890 x12345 : ext has 4 numbers top.
1-234-567-8901 : missing 00 or + at the beginning of an international number.
1234 to 12345678 : not a short number, yet not a normal one (between 9 and 12? as far as i know).
1234567890123 : over max length (since without international features).
0012312345678901 : over max length (as international number).
Regex101.com was a big plus to rewrite and test the regex to this point, I couldn't have progressed so far without its help. Yet, I'm no expert so I can only scratch the surface here and I need your help to improve this.
Thank you for reading, it was very educating to write the question (but not something I would do every day, very time-consuming at my pace), hope it will find its answers as well. Have a nice day (or night... ;) ).
Before I forgot, here's the post of the latest version of the regex I put together and its code :
^(?=(?:\+|0{2})?(?:(?:[\(\-\)\.\/ \t\f]*\d){7,10})?(?:[\-\.\/ \t\f]?\d{2,3})(?:[\-\s]?[ext]{1,3}[\-\.\/ \t\f]?\d{1,4})?$)((?:\+|0{2})\d{0,3})?(?:[\-\.\/ \t\f]?)(\(0\d[ ]?\d{0,4}\)|\(\d{0,4}\)|\d{0,4})(?:[\-\.\/ \t\f]{0,2}\d){3,8}(?:[\-\s]?(?:x|ext)[\-\t\f ]?(\d{1,4}))?$
As far as I know, it pass the tests I put in the question and some more that I added on that Regex101.com page. You can even fork it, very useful feature indeed, I'm a new fan. :)
The code seems to work, as is, with PHP (pcre), Python and Javascript (but not Golang) with different performance that are not awesome but good enough for our purpose.
For instance, I wanted to use \h for horizontal whitespaces (instead of \t, \f and space, but it is less compatible with the different platforms.
It still need a lot of improvements, and I'm eager to see what you will be cooking to answer this little problem of ours, but I'm spent... already a sunny morning here. Good night folks.
I need to search my corpus for words such as game or shame but I would like to specify the search to exclude three strings a game/a shame or , A game/A shame and a/an/A/An WORD game or a/an/A/An WORD shame , where WORD is a modifier, e.g., a great game or a great shame.
If someone could help me out, that would be great, thanks!
In my corpus, the optional WORD between the indefinite article a/an and game or a/an and shame is most commonly great and real. So even excluding these two, would already help me a lot.
The lookbehind below works perfectly to exclude a/A
(?<!a\s|A\s)\bshame\b
To exclude the modifying WORD, I was trying to use ?\w in the lookbehind grep, but it just wouldn't work - the grep below without ? runs and it still excludes examples such as a shame, but it still returns the undesired examples such as a great shame or a crying shame - see concordance lines (3) and (4) in the sample text below:
(?<!a\s|A\s|a\b\w\b|A\b\w\b)\bshame\b
The tool I'm using to implement regex is AntConc, which supports Perl regular expressions.
Sample text with two irrelevant examples (3 & 4) after using the search string below
(?<!a\s|A\s)\bshame\b
1 (match shame)
, people ogling from the sidelines. If you want a closer look, you have to ring for entry and wait to be admitted. I guess me and Saul just have no shame (or just know the benefits of our bank accounts being in hard currencies), because we wandered into plenty. Lots and lots of little boutiques and edgily designed fashion stores with music blaring.& abbutterflie.txt 47 1
2 (match shame)
last twenty years and I've experienced all sorts of biggotry but I seriously thought that anti black nazism in football wass a thing of the past. You should all hang your heads in shame, bunch of [badword]s. adamdphillips.txt 57 1
3 (don't match shame)
me monetarily as I wasn't that close to her, but she was really good friends with the other girl and it's messed that up for them a bit, which is a great shame. Anyway, Holly and I have since found somewhere to move in just the two of us. It's going to cost an absolute fortune and I'm going to be eating basics beans on aderyn.txt 60 1
4 (don't match shame)
are loads of amazingly good bands out there, gigging up and down the country who will never get signed because no-one can figure out how to market them, and this is a crying shame. There are artists out there like Thea Gilmore and <a href="http://blog.amandapalmer.net/" rel="nofollow"> Amanda Palmer& aderyn.txt 60 2
5 (match shame)
/><br />"There is no better time to show these terrorists that we have no fear of them. Instead we are forced, through the cowardly acts of our superiors, to hide in shame."<br /><br />But Herb Wiseman, high school consultant for Lee County, Florida, pointed to the July 7 London bombings.<br /><br />"What happens if kids get on aggy91.txt 64 1
Because variable length negative lookbehinds are not allowed, the approach in your previous question's answer won't transfer to this one.
I've gone with a (*SKIP)(*FAIL) pattern. This will match and discard the disqualified matches, and only retain qualifying matches:
/[Aa]n?( \w+)? shame(*SKIP)(*FAIL)|shame/ 3844 steps (Demo)
Or if you wish to include word boundary metacharacters:
/\b[Aa]n?( \w+)? shame\b(*SKIP)(*FAIL)|\bshame\b/ 4762 steps (Demo)
It seems hard to detect a sentence boundary in a text. Quotation marks like .!? may be used to delimite sentences but not so accurate as there may be ambiguous words and quotations such as U.S.A or Prof. or Dr. I am studying Tperlregex library and Regular Expression Cookbook by Jan Goyvaerts but I do not know how to write the expression that detects sentence?
What may be comparatively accurate expression using Tperlregex in delphi?
Thanks
First, you probably need to arrive at your own definition of what a "sentence" is, then implement that definition. For example, how about:
He said: "It's OK!"
Is it one sentence or two? A general answer is irrelevant. Decide whether you want it to interpret it as one or two sentences, and proceed accordingly.
Second, I don't think I'd be using regular expressions for this. Instead, I would scan each character and try to detect sequences. A period by itself may not be enough to delimit a sentence, but a period followed by whitespace or carriage return (or end of string) probably does. This immediately lets you weed out U.S.A (periods not followed by whitespace).
For common abbreviations like Prof. an Dr. it may be a good idea to create a dictionary - perhaps editable by your users, since each language will have its own set of common abbreviations.
Each language will have its own set of punctuation rules too, which may affect how you interpret punctuation characters. For example, English tends to put a period inside the parentheses (like this.) while Polish does the opposite (like this). The same difference will apply to double quotes, single quotes (some languages don't use them at all, sometimes they are indistinguishable from apostrophes etc.). Your rules may well have to be language-specific, at least in part.
In the end, you may approximate the human way of delimiting sentences, but there will always be cases that can throw the analysis off. For example, assuming that you have a dictionary that recognizes "Prof." as an abbreviation, what are you going to do about
Most people called him Professor Jones, but to me he was simply The Prof.
Even if you have another sentence that follows and starts with a capital letter, that still won't help you know where the sentence ends, because it might as well be
Most people called him Professor Jones, but to me he was simply Prof. Bill.
Check my tutorial here http://code.google.com/p/graph-expression/wiki/SentenceSplitting. This concrete example can be easily rewritten to regular expressions and some imperative code.
It will be wise to use a NLP processor with a pre-trained model. EnglishSD.nbin is one such model that is available for OpenNLP and it can be used in Visual Studio with SharpNLP.
The advantage of using this method is numerous. For example consider the input
Prof. Jessica is a wonderful woman. She is a native of U.S.A. She is married to Mr. Jacob Jr.
If you are using a regex split, for example
string[] sentences = Regex.Split(text, #"(?<=['""A-Za-z0-9][\.\!\?])\s+(?=[A-Z])");
Then the above input will be split as
Prof.
Jessica is a wonderful woman.
She is a native of U.
S.
A.
She is married to Mr.
Jacob Jr.
However the desired output is
Prof. Jessica is a wonderful woman.
She is a native of U.S.A. She is married to Mr. Jacob Jr.
This kind of logical sentence split can be achieved only using trained models from OpenNLP project. The method is as simple as this.
private string mModelPath = #"C:\Users\ATS\Documents\Visual Studio 2012\Projects\Google_page_speed_json\Google_page_speed_json\bin\Release\";
private OpenNLP.Tools.SentenceDetect.MaximumEntropySentenceDetector mSentenceDetector;
private string[] SplitSentences(string paragraph)
{
if (mSentenceDetector == null)
{
mSentenceDetector = new OpenNLP.Tools.SentenceDetect.EnglishMaximumEntropySentenceDetector(mModelPath + "EnglishSD.nbin");
}
return mSentenceDetector.SentenceDetect(paragraph);
}
where mModelPath is the path of the directory containing the nbin file.
The mSentenceDetector is derived from the OpenNLP dll.
You can get the desired output by
string[] sentences = SplitSentences(text);
Kindly read through this article I have written for integrating SharpNLP with your Application in Visual Studio to make use of the NLP tools
I am fairly experienced with regular expressions, but I am having some difficulty with a current application involving disjunction.
My situation is this: I need to separate an address into its component parts based on a regular expression match on the "Identifier elements" of the address -- A comparable English example would be words like "state", "road", or "boulevard"--IF, for example, we wrote these out in our addresses. Imagine we have an address like the following, where (and this would never happen in English), we specified the identifier type after each name
United States COUNTRY California STATE San Francisco CITY Mission STREET 345 NUMBER
(Where the words in CAPS are what I have called "identifiers").
We want to parse it into:
United States COUNTRY
California STATE
San Francisco CITY
Mission STREET
245 NUMBER
OK, this is certainly contrived for English, but here's the catch: I am working with Chinese data, where in fact this style of identifier specification happens all the time. An example below:
云南-省 ; 丽江-市 ; 古城-区 ; 西安-街 ; 杨春-巷 ;
Yunnan-Province ; LiJiang-City ; GuCheng-District ; Xi'An-Street ; Yangchun-Alley
This is easy enough--a lazy match on a potential candidate identifier names, separated into a disjunctive list.
For China, the following are the "province-level" entities:
省 (Province) ,
自治区 (Autonomous Region) ,
市 (Municipality)
So my regex so far looks like this:
(.+?(?:(?:省)|(?:自治区)|(?:市)))
I have a series of these, in order to account for different portions of the address. The next level, corresponding to cities, for instance, is:
(.+?(?:(?:地区)|(?:自治州)|(?:市)|(?:盟)))
So to match a province entity followed by a city entity:
(.+?(?:(?:省)|(?:自治区)|(?:市)))(.+?(?:(?:地区)|(?:自治州)|(?:市)|(?:盟)))
With named capture groups:
(?<Province>.+?(?:(?:省)|(?:自治区)|(?:市)))(?<City>.+?(?:(?:地区)|(?:自治州)|(?:市)|(?:盟)))
For the above, this yields:
$+{Province} = 云南省
$+{City} = 丽江市
This is all good and well, and gets me pretty far. The problem, however, is when I try to account for identifiers that can be a substring of other identifiers. A common street-level entity, for instance, is "村委会", which means village organizing committee. In the set of addresses I wish to separate, not every address has this written out in full. In fact, I find "村委" and just plain "村" as well.
The problem? If I have a pure disjunction of these elements, we have the following:
(?<Street>.+?(?:(?:村委会)|(?:村委)|(?:村)))
What happens, though, is that if you have an entity 保定-村委会 (Baoding Village organizing committee), this lazy regex stops at 村 and calls it a day, orphaning our poor 委会 because 村 is one of the potential disjunctive elements.
Imagine an English equivalent like the following:
(?<Animal>.+?(?:(?:Cat)|(?:Elephant)|(?:CatElephant)|(?:City)))
We have two input strings:
1. "crap catelephant crap city", where we wanted "Crap catelephant" and "crap city"
2. "crap catelephant city" , where we wanted "crap cat" "elephant city"
Ah, the solution, you say, is to make the pre-identifier capture greedy. But! There are entities have the same identifier that are not at the same level.
Take 市 for example. It means simply "city". But in China, there are county-level, province-level, and municipality-level cities. If this character occurred twice in the string, especially in two adjacent entities, the greedy search would incorrectly tag the greedy match as the first entity. As in the following:
广东-省 ; 江门-市 ; 开平-市 ; 三埠-区 石海管-区
Guangdong-province ; Jiangmen-City ; Kaiping-City ; Sanbu-District ; Shihaiguan-District
(Note, as above, this has been hand-segmented. The raw data would simply have a string of concatenated characters)
The match for a greedy search would be
江门市开平市
This is wrong, as the two adjacent entities should be separated into their constituent parts. Once is at the level of provincial city, one is a county-level city.
Back to the original point, and I thank you for reading this far, is there a way to put a weighting on disjunctive entities? I would want the regex to find the highest "weighted" identifier first. 村委会 instead of simple 村 for example, "catelephant" instead of just "cat". In preliminary experiments, the regex parser apparently proceeds left to right in finding disjunctive matches. Is this a valid assumption to make? Should I put the most frequently-occurring identifiers first in the disjunctive list?
If I have lost anyone with Chinese-related details, I apologize, and can further clarify if needed. The example really doesn't have to be Chinese--I think more generally it is a question about the mechanics of the regex disjunctive match -- in what order does it preference the disjunctive entities, and how does it decide when to "call it a day" in the context of a lazy search?
In a way, is there some sort of middle ground between lazy and greedy searches? Find the smallest bit you can find before the longest / highest weighted disjunctive entity? Be lazy, but put in that little bit of extra effort if you can for the sake of thoroughness?
(Incidentally, my work philosophy in college?)
How alternations are handled depends on the particular regular expression engine. For almost all engines (including Perl's regular expression engine) the alternation matches eagerly - that is, it matches the left-most choice first and only tries another alternative if this fails. For example, if you have /(cat|catelephant)/ it will never match catelephant. The solution is to reorder the choices so that the most specific comes first.