Ok what I'm trying to do is to check for the presence of
"TestItem-1"
a number greater then 1
one of the possible words in the list of "KG. Kg, kg, Kilo(s) or Kilogram(s)"
Where any of the items could be in any order and within a 6 word limit of each other.
Has to be done in regex as there is no access to the underlying scripting engine
This is what I've got as there a way of checking greater then I decided to use a range of 1-999 for the number check.
\b(?:[T|t]estItem-1\W+(?:\w+\W+){1,6}(^[0-9]|[1-9][0-9]|[1-9][0-9][0-9])$)\W+(?:\w+\W+){1,6}[K|k]il[o|os]|[K|k][[G|GS]|[g|gs]]|[|K|k]ilogra[m|ms]\b
Examples of what I need to find would be like -
"TestItem-1 is unstable in quanties above 12 Kilograms"
"1 Kilogram of TestItem-1"
While I wouldn't want to find -
"15 units of TestItem-1"
I know that what I got isn't working each section appears to work independently of each other but not together.
I pass this over to far greater minds then mine :)
You can try something like this:
\b(?:[2-9]|\d\d+)\b\s\b(?:KG.|Kg,|kg,|Kilos?|Kilograms?)\b(?:\S+\s){0,6}\bTestItem-1\b|\bTestItem-1\b(?:\S+\s){0,6}\b(?:[2-9]|\d\d+)\b\s\b(?:KG.|Kg,|kg,|Kilos?|Kilograms?)\b
Not ideal with the duplication but without lookarounds that's the best I could think of. I'll try and improve it in a bit.
Related
Got an interesting one, and can't come up with any solid ideas, so thought maybe someone else may have done something similar.
I want to be able to identify strings of letters in a longer sentence that are not words and remove them. Essentially things like kuashdixbkjshakd
Everything annoyingly is in lowercase which makes it more difficult, but since I only care about English, I'm essentially looking for the opposite of consonant clusters, groups of them that don't make phonetically pronounceable sounds.
Has anyone heard of/done something like this before?
EDIT: this is what ChatGpt tells me
It is difficult to provide a comprehensive list of combinations of consonants that have never appeared in a word in the English language. The English language is a dynamic and evolving language, and new words are being created all the time. Additionally, there are many regional and dialectal variations of the language, which can result in different sets of words being used in different parts of the world.
It is also worth noting that the frequency of use of a particular combination of consonants in the English language is difficult to quantify, as the existing literature on the subject is limited. The best way to determine the frequency of use of a particular combination of consonants would be to analyze a large corpus of written or spoken English.
In general, most combinations of consonants are used in some words in the English language, but some combinations of consonants may be relatively rare. Some examples of relatively rare combinations of consonants in English include "xh", "xw", "ckq", and "cqu". However, it is still possible that some words with these combinations of consonants exist.
You could try to pass every single word inside the sentence to a function that checks wether the word is listed inside a dictionary. There is a good number of dictionary text files on GitHub. To speed up the process: use a hash map :)
You could also use an auto-corretion API or a library.
Algorithm to combine both methods:
Run sentence through auto correction
Run every word through dictionary
Delete words that aren't listed in the dictionary
This could remove typos and words that are non-existent.
You could train a simple model on sequences of characters which are permitted in the language(s) you want to support, and then flag any which contain sequences which are not in the training data.
The LangId language detector in SpamAssassin implements the Cavnar & Trenkle language-identification algorithm which basically uses a sliding window over the text and examines the adjacent 1 to 5 characters at each position. So from the training data "abracadabra" you would get
a 5
ab 2
abr 2
abra 2
abrac 1
b 2
br 2
bra 2
brac 1
braca 1
:
With enough data, you could build a model which identifies unusual patterns (my suggestion would be to try a window size of 3 or smaller for a start, and train it on several human languages from, say, Wikipedia) but it's hard to predict how precise exactly this will be.
SpamAssassin is written in Perl and it should not be hard to extract the language identification module.
As an alternative, there is a library called libtextcat which you can run standalone from C code if you like. The language identification in LibreOffice uses a fork which they adapted to use Unicode specifically, I believe (though it's been a while since I last looked at that).
Following Cavnar & Trenkle, all of these truncate the collected data to a few hundred patterns; you would probably want to extend this to cover up to all the 3-grams you find in your training data at least.
Perhaps see also Gertjan van Noord's link collection: https://www.let.rug.nl/vannoord/TextCat/
Depending on your test data, you could still get false positives e.g. on peculiar Internet domain names and long abbreviations. Tweak the limits for what you want to flag - I would think that GmbH should be okay even if you didn't train on German, but something like 7 or more letters long should probably be flagged and manually inspected.
This will match words with more than 5 consonants (you probably want "y" to not be considered a consonant, but it's up to you):
\b[a-z]*[b-z&&[^aeiouy]]{6}[a-z]*\b
See live demo.
5 was chosen because I believe witchcraft has the longest chain of consonants of any English word. You could dial back "6" in the regex to say 5 or even 4 if you don't mind matching some outliers.
Not 100% if this is possible but I would like to convert any outbound call that does not match my DID range to a set phone number.
With our carrier in Australia if the ANI is not from their supplied range the call is blocked as part of new regulations.
What I am looking for is something like this.
if not +61 2 XXXX XXXX - +61 2 XXXX XXXX then send as +612XXXX XXXX
I apologise I have no true understanding of regex and do not know even where to begin.
I am starting to work on my knowledge of it though. please be kind. If anyone can point me to an "idiots guide" link I would be appreciative as I am just getting into this.
Of course it's possible. It's just a matter of how much work you want to do. I'm not quite sure what you want to mask and what you want to pass on unmutilated. A couple of particular examples would help. How many different formats, countries, and so on do you need to support?
With these problems, I tend to follow this approach:
Normalize the data. Make them all look the same. So, remove all non-digits, for example. +61 2 XXXX XXXX turns into 612XXXXXXXX. In this step, you'd also fill in implicit information, like a local number that does not include the country code. Number::Phone may be interesting, but, also note is was the largest distro on CPAN for awhile.
Now it should be easier to recognize the number and it's components (because if it isn't, you didn't do Step 1 right). Instead of a regex, you might use a parser. That is, get the country code, and then from that, decide what has to happen next. That's the sort of thing I have to do with ISBNs in Business::ISBN, which have a group code then a publisher code (both of which are variable length.
Once you can recognize the number, it's easy to select a range. If it's in the range, you know what to replace.
I'm attempting to create a Regex that finds only 2-digit integers or numbers with a precision of 2 decimal points.
In the example string at the bottom, I want to find only the following:
21 and 10.50
Using this expression, 100% is getting captured, in addition to the strings I desire to capture:
(\d){1,2}(\.?)([0-9]?[0-9]?){1,2}
I know I need to use ^% somewhere, but I can't figure out where it goes. Any suggestions are greatly appreciated.
Here's my sample string:
Earn Up to $21 Per Hour - Deliver Food with !!
Delivery Drivers work when they want and make great money when they do.
All orders are prepaid, just pick them up and deliver them to hungry diners. No waiting in line or fumbling with receipts and prepaid cards.
It's fast and easy to start working. Get started today.
Apply Now
Why choose ?
More orders than any other takeout platform
100% of our restaurants are official partners
Competitive pay: Per order fee + mileage + tips
We guarantee an hourly minimum of $10.50/hour*
Create your own schedule & work the hours you want
Word boundaries in your regular expression will grant you a bit more control.
Since word boundaries are a bit strict, we need to introduce an OR condition to address both cases which will satisfy your regex.
(\b[\d]{2}\.[\d]{2}\b)|(\b[\d]{2}\b)
Edit: Try this one,
\b[\d]{2}\b(\.[\d]{2})?
The first example has a chance to fail as it is order dependent due to the way it short-circuits. This I believe should address multiple cases properly.
I think this should work:
(?<!\d)((\d+\.\d\d)|(\d\d))(?!%|\d)
Demo (and explanation)
EDIT:
Improved version:
(?<!\d)(\d{1,2}(?:\.\d{1,2})?)(?!%|\d)
Demo (and explanation)
You can try this variant: (\d{1,}|[\d.])\b(?!%)
It uses negative lookahead (?!%) to exclude digits following by % sign.
Details at regex101
I'm in need of a regex, which takes a minimum and a maximum number to determine valid input, And I want the maximum and minimum to be dynamic.
I have been trying to get this done using this link
https://stackoverflow.com/a/13473595/1866676
But couldn't get it to work. Can someone please let me know how to do this.
Let's say I want to make a html5 input box, and I Want it to only receive numbers from 100 to 1999
What would a regex for this like this look like?
First off, while it is possible to do this, I think if there is a simpler way to choose a number range such as <input type="number" min="1" max="100">, that way would be preferred.
Having said that, here's how the kind of regex you requested works:
ones: ^[0-9]$ // just set the numbers -- matches 0 to 9
tens: ^[1-3]?[0-9]$ //set max tens and max ones -- matches 0 to 39
tens where max does not end in 9 ^[1-2]?[0-9]$|^[3][0-4]$ // 0 to 34
only tens: ^[1][5-9]$|^[2-3][0-9]$|^[4][0-5]$ // 15 to 45
Here, lets pick an arbitrary number 1234 to 2345
^[1][2][3][4-9]$|
^[1][2][4-9][0-9]$|
^[1][3-9][0-9][0-9]$|
^[2][0-2][0-9][0-9]$|
^[2][3][0-3][0-9]$|
^[2][3][4][0-5]$
https://regex101.com/r/pP8rQ7/4
Basically the ending of the middle series always needs to be a straight range that can reach 9 unless we are dealing with the ones place, and if it cant, you have to build it upwards toward the middle each time we have a value that can't start in 0 and then once we reach a value that cant end in 9 break early and set it in the next condition.
Notice the pattern, as each place solidifies. Also keep in mind that when dealing with going from lower to higher places, optional operators ? should be used.
Its a bit complex, but its nowhere near impossible to design a custom range with a bit of thought.
If you are more specific, we can craft an exact example, but this is generally how it is done:beginning-range|middle-range|end-range
You should only need beginning or end-ranges in certain cases like if the min or max does not end in 9. the ? means that the range that comes after it is optional. (so for example in the first case it lets us have both single and double numbers.
so for 100 - 1999 it's quite simple actually because you have lots of 9's and 0's
/^[1-9][0-9][0-9]$|^[1][0-9][0-9][0-9]$/
https://regex101.com/r/pP8rQ7/1
Note: Single values don't need ranges [n] I just added them for readability.
Edit: There used to be a regex range generator at: http://gamon.webfactional.com/regexnumericrangegenerator/. It appears to be offline now.
Essentially, you can't.
For every numeric range, there exists a regex that will match numbers in that range, therefore it is possible to write code that can generate a such regex. But such a regex is not a simple reformatting of the range ends.
However, such code would require colossal effort and complexity to write compared to code that simply checked the number using numeric methods.
With HTML 5 simply put a range input...
<form>
Quantity (between 100 and 1999):
<input type="number" name="quantity" min="100" max="1999">
</form>
with regex:
^([12345679])(\d)(\d)|^(1)(\d)(\d)(\d)
So if you need to create the regex dinamically it's possible but a bit tricky and complex
I have some SQLCLR code for working with Regular Expresions. But now that it is getting migrated into Azure, which does not allow SQLCLR, that's out. I need to find a way to do regex in pure T-SQL.
Master Data Services are not available because the dev edition of MSSQL we have is not R2.
All ideas appreciated, thanks.
Regular expression match samples that need handling
(culled from regexlib and other places over the past few years)
email address
^[\w-]+(\.[\w-]+)*#([a-z0-9-]+(\.[a-z0-9-]+)*?\.[a-z]{2,6}|(\d{1,3}\.){3}\d{1,3})(:\d{4})?$
dollars
^(\$)?(([1-9]\d{0,2}(\,\d{3})*)|([1-9]\d*)|(0))(\.\d{2})?$
uri
^(http|https|ftp)\://([a-zA-Z0-9\.\-]+(\:[a-zA-Z0-9\.&%\$\-]+)*#)*((25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[1-9])\.(25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[1-9]|0)\.(25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[1-9]|0)\.(25[0-5]|2[0-4][0-9]|[0-1]{1}[0-9]{2}|[1-9]{1}[0-9]{1}|[0-9])|localhost|([a-zA-Z0-9\-]+\.)*[a-zA-Z0-9\-]+\.(com|edu|gov|int|mil|net|org|biz|arpa|info|name|pro|aero|coop|museum|[a-zA-Z]{2}))(\:[0-9]+)*(/($|[a-zA-Z0-9\.\,\?\'\\\+&%\$#\=~_\-]+))*$
one numeric digit
^\d$
percentage
^-?[0-9]{0,2}(\.[0-9]{1,2})?$|^-?(100)(\.[0]{1,2})?$
height notation
^\d?\d'(\d|1[01])"$
numbers between 1 1000
^([1-9]|[1-9]\d|1000)$
credit card numbers
^((4\d{3})|(5[1-5]\d{2})|(6011))-?\d{4}-?\d{4}-?\d{4}|3[4,7]\d{13}$
list of years
^([1-9]{1}[0-9]{3}[,]?)*([1-9]{1}[0-9]{3})$
days of the week
^(Sun|Mon|(T(ues|hurs))|Fri)(day|\.)?$|Wed(\.|nesday)?$|Sat(\.|urday)?$|T((ue?)|(hu?r?))\.?$
time on 12 hour clock
(?<Time>^(?:0?[1-9]:[0-5]|1(?=[012])\d:[0-5])\d(?:[ap]m)?)
time on 24 hour clock
^(?:(?:(?:0?[13578]|1[02])(\/|-|\.)31)\1|(?:(?:0?[13-9]|1[0-2])(\/|-|\.)(?:29|30)\2))(?:(?:1[6-9]|[2-9]\d)?\d{2})$|^(?:0?2(\/|-|\.)29\3(?:(?:(?:1[6-9]|[2-9]\d)?(?:0[48]|[2468][048]|[13579][26])|(?:(?:16|[2468][048]|[3579][26])00))))$|^(?:(?:0?[1-9])|(?:1[0-2]))(\/|-|\.)(?:0?[1-9]|1\d|2[0-8])\4(?:(?:1[6-9]|[2-9]\d)?\d{2})$
usa phone numbers
^\(?[\d]{3}\)?[\s-]?[\d]{3}[\s-]?[\d]{4}$
Unfortunately, you will not be able to move your CLR function(s) to SQL Azure. You will need to either use the normal string functions (PATINDEX, CHARINDEX, LIKE, and so on) or perform these operations outside of the database.
EDIT Adding some information for the examples added to the question.
Email address
This one is always controversial because people disagree about which version of the RFC they want to support. The original didn't support apostrophes, for example (or at least people insist that it didn't - I haven't dug it up from the archives and read it myself, admittedly), and it has to be expanded quite often for new TLDs (once for 4-letter TLDs like .info, then again for 6-letter TLDs like .museum). I've often heard quite knowledgeable people state that perfect e-mail validation is impossible, and having previously worked for an e-mail service provider, I can tell you that it was a constantly moving target. But for the simplest approaches, see the question TSQL Email Validation (without regex).
One numeric digit
Probably the easiest one of the bunch:
WHERE #s LIKE '[0-9]';
Credit card numbers
Assuming you strip out dashes and spaces, which you should do in any case. Note that this isn't an actual check of the credit card number algorithm to ensure that the number itself is actually valid, just that it conforms to the general format (AmEx = 15 digits starting with a 3, the rest are 16 digits - Visa starts with a 4, MasterCard starts with a 5, Discover starts with 6 and I think there's one that starts with a 7 (though that may just be gift cards of some kind)):
WHERE #s + ' ' LIKE '[3-7]'+ REPLICATE('[0-9]', 14) + '[0-9 ]';
If you want to be a little more precise at the cost of being long-winded, you can say:
WHERE (LEN(#s) = 15 AND #s LIKE '3' + REPLICATE('[0-9]', 14))
OR (LEN(#s) = 16 AND #s LIKE '[4-7]' + REPLICATE('[0-9]', 15));
USA phone numbers
Again, assuming you're going to strip out parentheses, dashes and spaces first. Pretty sure a US area code can't start with a 1; if there are other rules, I am not aware of them.
WHERE #s LIKE '[2-9]' + REPLICATE('[0-9]', 9);
-----
I'm not going to go further, because a lot of the other expressions you've defined can be extrapolated from the above. Hopefully this gives you a start. You should be able to Google for some of the others to see how other people have replicated the patterns with T-SQL. Some of them (like days of the week) can probably just be checked against a table - seems overkill to do an invasie pattern matching for a set of 7 possible values. Similarly with a list of 1000 numbers or years, these are things that will be much easier (and probably more efficient) to check if the numeric value is in a table rather than convert it to a string and see if it matches some pattern.
I'll state again that a lot of this will be much better if you can cleanse and validate the data before it gets into the database in the first place. You should strive to do this wherever possible, because without CLR, you just can't do powerful RegEx inside SQL Server.
Ken Henderson wrote about ways to replicate RegEx without CLR, but they require sp_OA* procedures, which are even less likely to ever see the light of day in Azure than CLR. Most of the other articles you'll find online use an approach similar to Ken's or use complex use of built-in string functions.
Which portions of RegEx specifically are you trying to replicate? Can you show an example of the input/output of one of your functions? Perhaps it will be easy to convert to get similar results using the built-in string functions like PATINDEX.