Parsing out a number after a specific word appears - regex

I am trying to collect the forecasted high temperature from the National Weather Service from their text-based website. The website I am trying to pull information from can be found here.
So far I have been able to pull the first number that appears after each day. Most of the time this is the high temperature, but occasionally they will put a precipitation amount before the forecated temperature for the day. I want to find a way to pull the digits that follow the word "high". It should also be noted that sometimes they use "high near", "high around", or any other variation so it wouldn't necessarily be the next string following high.
Below is my code. I intend to run this everyday at a certain time, so I will get the current days forecast up to six days later. If you were to run this code in the evening, you would get the next seven days of forecasted temperatures, with the first temperature actually referring to next weeks forecast.
My end goal is to put this onto trendy, so I'm sure this would be easier to accomplish in other formats, but I want to stick straight to Matlab.
url = 'http://forecast.weather.gov/MapClick.php?lat=40.48622&lon=-74.45181587699966&unit=0&lg=english&FcstType=text&TextType=1';
html = urlread(url);
DayForm = 'long';
today = clock;
today = today(:,3);
nvalue = zeros(6,1);
for i = 0:6
[~, getDay] = weekday(today+i,DayForm);
target = ['<b>' getDay ':'];
[a,b] = regexp(html,'\d');
strPos = find( a > strfind(html,target),1,'first');
nvalue(i+1) = str2double(html(a(strPos):b(strPos)+1));
end
EDIT: after implenting the answer, here is my updated code:
url = 'http://forecast.weather.gov/MapClick.php?lat=40.48622&lon=-74.45181587699966&unit=0&lg=english&FcstType=text&TextType=1';
html = urlread(url);
fcast = zeros(7,1);
target = 'with\sa\shigh\s\w*\s?([0-9]+)';
[~,b] = regexp(html,target);
for i = 1:7
fcast(i) = str2double(html(b(i)-1:b(i)));
end

well it seems matlab supports gnu extended regex which is limiting which means MrAzzaman answer may not work. Though he accounts for mph that has the word high before, the following regex should match and capture the digits you want into capture group $1.
with\sa\shigh\s\w*\s?([0-9]+)
find with a high, then a space then possible word, then another space followed by the captured group that contains temp.
It should work

This is slightly complicated by the fact that they also occasionally say things like "winds as high as 32 mph". The following works, though there may be more edge cases that aren't accounted for:
high\D+(\d+)\D(?!mph)
This searches for the word 'high', and then slurps all of the characters until it reaches a digit. It grabs the digits in a group, and then grabs the next non-digit character (this ensures it grabs all of the digits). It then uses a negative lookahead to make sure the next 3 letters aren't 'mph' (which would suggest that the number indicates a wind, rather than a temperature).
As I said, there may be more edge cases, but it seems to work for the present web page.

Related

How to get a string before the last occurrence of a specific character before a maximum character count?

I have some long but variable-length texts that are divided into sections marked by ********************. I need to post those texts into a field that only accepts 2048 characters, so I will need to divide that text into groups of no more than 2048 characters but which do not contain an incomplete section.
My regex so far is ^([\s\S]{1,2048})([\s\S]{1,2048})([\s\S]{1,2048})
However, this has two problems:
1) It divides the text into groups that can include an incomplete section. What I want is a complete section, even if it is not a full 2048 characters. Assume the example below is at the end of 2048 characters.
Here's my actual result. Notice that the "7 Minute Workout" section is cut off mid-section
********************
Maybe Baby™ Period & Fertility (📱)
Popular app for tracking your periods and predicting times of fertility; recommended; avg 4.5/5 stars (3,500+ ratings); 50% off, $3.99 ↘️ $1.99!
https://example.com/2019/07/29/maybe-baby-period-fertility-7-29-19/
********************
7 Minute Workout: Lose Weight (📱)
Scientifically-proven and featured by the New York Times, a 7-minute high intensity workout proven to lose weig
Here's my desired result. Notice that the "7 Minute Workout" section is entirely omitted because it could not be included in its entirety while staying under the 2048 character limit.
********************
Maybe Baby™ Period & Fertility (📱)
Popular app for tracking your periods and predicting times of fertility; recommended; avg 4.5/5 stars (3,500+ ratings); 50% off, $3.99 ↘️ $1.99!
https://example.com/2019/07/29/maybe-baby-period-fertility-7-29-19/
2) The second problem with this regex is that the text I need to input varies greatly in length; it may be less than 2048 or it could be 10,000+ characters. My regex obviously only works for texts up to 6,144 characters long. Do I just keep duplicating the regex a crazy number of times to get longer than the longest text I could enter, or is there a way to get it to repeat?
Addendum: Several asked about the use case/environment for this question. No, it’s not a spambot 🙂. Rather, I’m trying to use Apple’s Shortcuts app to cross-post items from my website to followers on Kik. Unfortunately, Kik has a 2048 character limit, so I can’t post it all at once. I’m trying to use regex to split the text into appropriate sections so I can copy them from Shortcuts and paste them one at a time into Kik.
Couple Notes:
No need to use groups at all, just use match results directly as each match represent one section.
Use lazy quantifier instead of greedy by adding ? after {1,2048} to make the match cut in the right place.
In my regex, I used only Global g without the multiline m.
The code below will work only with sections that have 2048 characters or less. If the section has more than 2048 characters, it will be skipped.
The regex below uses Positive Lookahead to signal the end of the section without matching.
Here is the regex:
^|\*[\s\S]{1,2048}?(?=\n\*|$)
Example: https://regex101.com/r/hezvu5/1/
==== Update ====
To make the results greedy, to match as many sections as possible without splitting the last section, use this regex:
^|\*[\s\S]{1,2048}(?=\n\*|$)

Excluding % from a Regex number search

I'm attempting to create a Regex that finds only 2-digit integers or numbers with a precision of 2 decimal points.
In the example string at the bottom, I want to find only the following:
21 and 10.50
Using this expression, 100% is getting captured, in addition to the strings I desire to capture:
(\d){1,2}(\.?)([0-9]?[0-9]?){1,2}
I know I need to use ^% somewhere, but I can't figure out where it goes. Any suggestions are greatly appreciated.
Here's my sample string:
Earn Up to $21 Per Hour - Deliver Food with !!
Delivery Drivers work when they want and make great money when they do.
All orders are prepaid, just pick them up and deliver them to hungry diners. No waiting in line or fumbling with receipts and prepaid cards.
It's fast and easy to start working. Get started today.
Apply Now
Why choose ?
More orders than any other takeout platform
100% of our restaurants are official partners
Competitive pay: Per order fee + mileage + tips
We guarantee an hourly minimum of $10.50/hour*
Create your own schedule & work the hours you want
Word boundaries in your regular expression will grant you a bit more control.
Since word boundaries are a bit strict, we need to introduce an OR condition to address both cases which will satisfy your regex.
(\b[\d]{2}\.[\d]{2}\b)|(\b[\d]{2}\b)
Edit: Try this one,
\b[\d]{2}\b(\.[\d]{2})?
The first example has a chance to fail as it is order dependent due to the way it short-circuits. This I believe should address multiple cases properly.
I think this should work:
(?<!\d)((\d+\.\d\d)|(\d\d))(?!%|\d)
Demo (and explanation)
EDIT:
Improved version:
(?<!\d)(\d{1,2}(?:\.\d{1,2})?)(?!%|\d)
Demo (and explanation)
You can try this variant: (\d{1,}|[\d.])\b(?!%)
It uses negative lookahead (?!%) to exclude digits following by % sign.
Details at regex101

Regular expression to expand to sentence

I'm trying to extract regions around keywords from longer passages of text. They should include complete sentences, based on the following conditions:
n=250 Charactars before / after keyword should be included if existing (the keyword can be closer then this to the start / end of the text)
from there it should expand further to include the complete sentence (let's assume here we can define sentence borders with ".?! or :" knowing it's not completely accurate)
I already achieved the expanding to the end of the last sentence, but not to start of the first in the following example, where vitamin is the keyword and the italic is captured by the regex. However, it should capture from "An extra 24 hours..."
Apparently I don't get the corresponding group up front, neither using lazy nor using lookbehind.
((.{0,250}(vitamin)\b.{0,250})(.+?(\.|\!|\?|\:))?)/ig
Well, this year you’re getting an extra day to get ahead on your taxes or (finally) clean out the garage. (Hey, we’re not trying to tell you what do but you might as well be productive.) February 29 is back on the calendar this year because it’s a leap year. Whether you love or loathe the extra winter day, you’re probably wondering why it happens in the first place. An extra 24 hours — or day — is built into the calen dar every four years to ensure it aligns with the Earth’s movement around the sun. There’s 365 days in a calendar year, but it actually takes longer for the Earth’s annual journey — about 365.2421 days — around the star that gives us light, life and vitamin D. The difference may seem like no big deal to us, but over time, it adds up. “To ensure consistency with the true astronomical year, it is necessary to periodically add in an extra day to make up the lost time and get the calendar back in sync with the heavens,” according the history. com.
Acknowledgement of the need for a leap year happened around the time of Julius Caesar. In 46 B.C., Caesar enlisted the help of astronomer Sosigenes to update the calendar so that it had 12 months and 365 days, including a leap year every four years.,
You can try something like this:
(([.?!:][^.?!:]*.{250}\bvitamin\b.{250})[^.?!:]*[.?!:])
It works by consuming 250 characters of text before and after the keyword "vitamin". From that point it finds the first punctuation point (.?!:) before/after the 250 characters of text.
Here's a sample of it in action.
You can you use extra parentheses () to strategically group what exact output you want. For example, the above answer includes the ending period from the preceding sentence in the output. So you could use
(([.?!:]([^.?!:]*.{250}\bvitamin\b.{250})[^.?!:]*[.?!:]))
and use group 3 from the result set which doesn't have this ending period.
I do not see how the specification in the question can be matched by a regex. It boils down to the following logic problem:
to match as many characters as possible but no more than 250 before/after the keyword, .{0,250} needs to be greedy and can neither be lazy .{0,250}? nor possessive .{0,250}+
if this part is greedy, you will miss the occurrences of the keyword that start before the .{0,250} part is matched.
The same logic applies to my understanding to the 'match back to the start of the sentnence as well.
I played around with the following more or less meaningful regex:
[.?!:]?([^.?!:]*?(.{0,250}\byear\b.{0,250})[^.?!:]*[.?!:]?) misses first 'year'
[.?!:]?([^.?!:]*?(.{0,250}?\byear\b.{0,250})[^.?!:]*[.?!:]?) gets the first 'year' but fails on others.
I suggest you write your on extraction logic in a function, eihter using regex or not, to achieve the extraction you want.
You could for example find the index of the start of the keyword \bkeyword\b and the full stops (\.[^\d]|[.?!:]$) and then with this information extract the part of the text you want.

Regex to match time

I want my users to be able to enter a time form.
If more info necessary, users use this to express how much time is needed to complete a task, and it will be saved in a database if filled.
here is what I have:
/^$|^([0-1]?[0-9]|2[0-4]):([0-5][0-9])(:[0-5][0-9])?$/
It matches an empty form or 01:30 and 01:30:00 formatted times. I really won't need the seconds as every task takes a minute at least, but I tried removing it and it just crashed my code and removed support for empty string.. I really don't understand regex at all.
What I'd like, is for it to also match simple minutes and simple hours, like for instance 3:30, 3:00, 5. Is this possible? It would greatly improve the user experience and limit waste typing. But I'd like to keep the zero optional in case some users find it natural to type it.
I think the following pattern does what you want:
p="((([01]?\d)|(2[0-4])):)?([0-5]\d)?(:[0-5]\d)?"
The first part:
(([01]?\d)|(2[0-3])):)?
is an optional group which deals with hours in format 00-24.
The second part:
([0-5]\d)?
is an optional group which deals with minutes if hours or seconds are present in your expression. The group also deals with expressions containing only minutes or only hours.
The third part:
(:[0-5]\d)?
is an optional group dealing with seconds.
The following samples show the pattern at work:
In [180]: re.match(p,'14:25:30').string
Out[180]: '14:25:30'
In [182]: re.match(p,'2:34:05').string
Out[182]: '2:34:05'
In [184]: re.match(p,'02:34').string
Out[184]: '02:34'
In [186]: re.match(p,'59:59').string
Out[186]: '59:59'
In [188]: re.match(p,'59').string
Out[188]: '59'
In [189]: re.match(p,'').string
Out[189]: ''
As every group is optional the pattern matches also the empty string. I've tested it with Python but I think it will work with other languages too with minimal changes.

SQL Server Regular Expression Workaround in T-SQL?

I have some SQLCLR code for working with Regular Expresions. But now that it is getting migrated into Azure, which does not allow SQLCLR, that's out. I need to find a way to do regex in pure T-SQL.
Master Data Services are not available because the dev edition of MSSQL we have is not R2.
All ideas appreciated, thanks.
Regular expression match samples that need handling
(culled from regexlib and other places over the past few years)
email address
^[\w-]+(\.[\w-]+)*#([a-z0-9-]+(\.[a-z0-9-]+)*?\.[a-z]{2,6}|(\d{1,3}\.){3}\d{1,3})(:\d{4})?$
dollars
^(\$)?(([1-9]\d{0,2}(\,\d{3})*)|([1-9]\d*)|(0))(\.\d{2})?$
uri
^(http|https|ftp)\://([a-zA-Z0-9\.\-]+(\:[a-zA-Z0-9\.&%\$\-]+)*#)*((25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[1-9])\.(25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[1-9]|0)\.(25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[1-9]|0)\.(25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[0-9])|localhost|([a-zA-Z0-9\-]+\.)*[a-zA-Z0-9\-]+\.(com|edu|gov|int|mil|net|org|biz|arpa|info|name|pro|aero|coop|museum|[a-zA-Z]{2}))(\:[0-9]+)*(/($|[a-zA-Z0-9\.\,\?\'\\\+&%\$#\=~_\-]+))*$
one numeric digit
^\d$
percentage
^-?[0-9]{0,2}(\.[0-9]{1,2})?$|^-?(100)(\.[0]{1,2})?$
height notation
^\d?\d'(\d|1[01])"$
numbers between 1 1000
^([1-9]|[1-9]\d|1000)$
credit card numbers
^((4\d{3})|(5[1-5]\d{2})|(6011))-?\d{4}-?\d{4}-?\d{4}|3[4,7]\d{13}$
list of years
^([1-9]{1}[0-9]{3}[,]?)*([1-9]{1}[0-9]{3})$
days of the week
^(Sun|Mon|(T(ues|hurs))|Fri)(day|\.)?$|Wed(\.|nesday)?$|Sat(\.|urday)?$|T((ue?)|(hu?r?))\.?$
time on 12 hour clock
(?<Time>^(?:0?[1-9]:[0-5]|1(?=[012])\d:[0-5])\d(?:[ap]m)?)
time on 24 hour clock
^(?:(?:(?:0?[13578]|1[02])(\/|-|\.)31)\1|(?:(?:0?[13-9]|1[0-2])(\/|-|\.)(?:29|30)\2))(?:(?:1[6-9]|[2-9]\d)?\d{2})$|^(?:0?2(\/|-|\.)29\3(?:(?:(?:1[6-9]|[2-9]\d)?(?:0[48]|[2468][048]|[13579][26])|(?:(?:16|[2468][048]|[3579][26])00))))$|^(?:(?:0?[1-9])|(?:1[0-2]))(\/|-|\.)(?:0?[1-9]|1\d|2[0-8])\4(?:(?:1[6-9]|[2-9]\d)?\d{2})$
usa phone numbers
^\(?[\d]{3}\)?[\s-]?[\d]{3}[\s-]?[\d]{4}$
Unfortunately, you will not be able to move your CLR function(s) to SQL Azure. You will need to either use the normal string functions (PATINDEX, CHARINDEX, LIKE, and so on) or perform these operations outside of the database.
EDIT Adding some information for the examples added to the question.
Email address
This one is always controversial because people disagree about which version of the RFC they want to support. The original didn't support apostrophes, for example (or at least people insist that it didn't - I haven't dug it up from the archives and read it myself, admittedly), and it has to be expanded quite often for new TLDs (once for 4-letter TLDs like .info, then again for 6-letter TLDs like .museum). I've often heard quite knowledgeable people state that perfect e-mail validation is impossible, and having previously worked for an e-mail service provider, I can tell you that it was a constantly moving target. But for the simplest approaches, see the question TSQL Email Validation (without regex).
One numeric digit
Probably the easiest one of the bunch:
WHERE #s LIKE '[0-9]';
Credit card numbers
Assuming you strip out dashes and spaces, which you should do in any case. Note that this isn't an actual check of the credit card number algorithm to ensure that the number itself is actually valid, just that it conforms to the general format (AmEx = 15 digits starting with a 3, the rest are 16 digits - Visa starts with a 4, MasterCard starts with a 5, Discover starts with 6 and I think there's one that starts with a 7 (though that may just be gift cards of some kind)):
WHERE #s + ' ' LIKE '[3-7]'+ REPLICATE('[0-9]', 14) + '[0-9 ]';
If you want to be a little more precise at the cost of being long-winded, you can say:
WHERE (LEN(#s) = 15 AND #s LIKE '3' + REPLICATE('[0-9]', 14))
OR (LEN(#s) = 16 AND #s LIKE '[4-7]' + REPLICATE('[0-9]', 15));
USA phone numbers
Again, assuming you're going to strip out parentheses, dashes and spaces first. Pretty sure a US area code can't start with a 1; if there are other rules, I am not aware of them.
WHERE #s LIKE '[2-9]' + REPLICATE('[0-9]', 9);
-----
I'm not going to go further, because a lot of the other expressions you've defined can be extrapolated from the above. Hopefully this gives you a start. You should be able to Google for some of the others to see how other people have replicated the patterns with T-SQL. Some of them (like days of the week) can probably just be checked against a table - seems overkill to do an invasie pattern matching for a set of 7 possible values. Similarly with a list of 1000 numbers or years, these are things that will be much easier (and probably more efficient) to check if the numeric value is in a table rather than convert it to a string and see if it matches some pattern.
I'll state again that a lot of this will be much better if you can cleanse and validate the data before it gets into the database in the first place. You should strive to do this wherever possible, because without CLR, you just can't do powerful RegEx inside SQL Server.
Ken Henderson wrote about ways to replicate RegEx without CLR, but they require sp_OA* procedures, which are even less likely to ever see the light of day in Azure than CLR. Most of the other articles you'll find online use an approach similar to Ken's or use complex use of built-in string functions.
Which portions of RegEx specifically are you trying to replicate? Can you show an example of the input/output of one of your functions? Perhaps it will be easy to convert to get similar results using the built-in string functions like PATINDEX.