Regex checks wouldn't be a strong point of mine. This is trivial but after playing around with it for 15 minutes already I think it would be quicker posting here. Ultimately I want to filter out any results of a table where a certain text column value ends with S(01 -99), i.e. the letter S followed by 2 digits. Consider the following test query
select x.* from (
select
unnest(array['kjkjkj','jhjs01','kjkj11','kjhkjh','uusus','iiosis99']::text[])
as tests ) x
where RIGHT(x.tests,3) !~ 'S[0-9]{1,2}$'
This returns everything in the unnested array, whereas I'm hoping to return everything excluding the second and last values. Any pointers in the right direction would be much appreciated. I'm using PostgreSQL v11.9
You may actually use SIMILAR TO here since your pattern is not that complex:
SELECT * FROM table
WHERE column_name NOT SIMILAR TO '%S[0-9]{2}'
SIMILAR TO patterns require a full string match, so here, % matches any text from the start of the string, then S matches S and [0-9]{2} matches two digits that must be at the end of the string.
If you were to use a regex, you could use
WHERE column_name !~ 'S[0-9]{2}$'
Or, 'S[0-9]{1,2}$' if there can be one or two digits. Since the regex search in PostgreSQL does not require a full string match, it just matches S, two (or one or two with {1,2}) digits at the end of string ($).
Related
Introduction:
I have the following scenario in PostgreSQL whereby I want to perform some data validation on a .csv string prior to inserting it into a table (see the fiddle here).
I've managed to get a regex (in a CHECK constraint) which disallows spaces within strings (e.g. "12 34") and also disallows preceding zeros ("00343").
Now, the icing on the cake would be if I could use regular expressions to disallow strings which contain a repeat of an integer - i.e. if a sequence \d+ matched another \d+ within the same string.
Is this beyond the capacities of regular expressions?
My table is as follows:
CREATE TABLE test
(
data TEXT NOT NULL,
CONSTRAINT d_csv_only_ck
CHECK (data ~ '^([ ]*([1-9]\d*)+[ ]*)(,[ ]*([1-9]\d*)+[ ]*)*$')
);
And I can populate it as follows:
INSERT INTO test VALUES
('992,1005,1007,992,456,456,1008'), -- want to make this line unnacceptable - repeats!
('44,1005,1110'),
('13, 44 , 1005, 10078 '), -- acceptable - spaces before and after integers
('11,1203,6666'),
('1,11,99,2222'),
('3435'),
(' 1234 '); -- acceptable
But:
INSERT INTO test VALUES ('23432, 3433 ,00343, 567'); -- leading 0 - unnacceptable
fails (as it should), and also fails (again, as it should)
INSERT INTO test VALUES ('12 34'); -- spaces within numbers - unnacceptable
The question:
However, if you notice the first string, it has repeats of 992and 456.
I would like to be able to match these.
All of these rules do not have to be in the same regex - I can use a second CHECK constraint.
I would like to know if what I am asking is possible using Regular Expressions?
I did find this post which appears to go some (all?) of the way to solving my issue, but I'm afraid it's beyond my skillset to get it to work - I've included a small test at the bottom of the fiddle.
Please let me know should you require any further information.
p.s. as an aside, I'm not very experienced with regexes and I would welcome any input on my basic one above.
Since PostegreSQL regex does not support backreferences, you cannot apply this restriction because you would need a negative lookahead with a backreference in it.
Have a look at this PCRE regex:
^(?!.*\b(\d+)\b.*\b\1\b) *[1-9]\d* *(?:, *[1-9]\d* *)*$
See this regex demo.
Details:
^ - start of string
(?!.*\b(\d+)\b.*\b\1\b) - no same two numbers as whole word allowed anywhere in the string
* - zero or more spaces
[1-9]\d* - a non-zero digit and then any zero or more digits
* - zero or more spaces
(?:, *[1-9]\d* *)* - zero or more occurrences of
, * - comma and zero or more spaces
[1-9]\d* - a non-zero digit and then any zero or more digits
* - zero or more spaces
$ - end of string.
Even if you replace \b with \y (PostgreSQL regex word boundaries) in the PostgreSQL code, it won't work due to the drawback mentioned at the top of the answer.
Why the following code returns just empty brackets - {''}. How to make it return matching strings?
SELECT regexp_matches('ATGCATGCATGCCAACAACAACCTGTCAAGTGAGT','(?=..CAA)','g');
Expected output is:
regexp_matches
----------------
{GCCAA}
{AACAA}
{AACAA}
{GTCAA}
(4 rows)
but instead it returns the following:
regexp_matches
----------------
{""}
{""}
{""}
{""}
(4 rows)
I actually have a bit more complicated query, which requires positive lookahead in order to cover all occurrences of patterns in the string even if they overlap.
Well, it's not pretty, but you can do it without regular expressions or custom functions.
WITH data(d) as (
SELECT * FROM (VALUES ('ATGCATGCATGCCAACAACAACCTGTCAAGTGAGT')) v
)
SELECT substr(d, x, 5) AS match
FROM data
JOIN LATERAL (SELECT generate_series(1, length(d))) g(x) ON TRUE
WHERE substr(d, x, 5) LIKE '__CAA'
;
match
-------
GCCAA
AACAA
AACAA
GTCAA
(4 rows)
Basically, get each five letter slice of the string and see if it matches __CAA.
You could change generate_series(1, length(d)) to generate_series(1, length(d)-4) because the last ones will never match, but you would have to remember to update this if the length of your matching string changes.
Using a lookahead has the problem that the lookahead itself is not part of the match but it allows overlapping searches
Without using a lookahead, you lose the ability for overlapping searches.
Using Powershell, you can loop over the indexes returned from the lookaheads and use that as an index into your searchstring to get the matches
$string = 'ATGCATGCATGCCAACAACAACCTGTCAAGTGAGT'
$r = [regex]::new('(?=..CAA)')
$r.Matches($string) | % {$string.Substring($_.Index, 5)}
returns
GCCAA
AACAA
AACAA
GTCAA
I don't know how to translate this to PostgreSQL (or if that's even possible)
update:
Aparently it won't capture inside of an assertion, that's ok because
what you really need is the first 2 characters, which can safely be
consumed. It will only give you the first 2 characters per row, but
since you know the last 3, you can easily join the set elements
with the CAA constant.
Try this
..(?=CAA)
and you're done.
If I knew the bizarre sql language, I could show you how to do the join.
Output should now be
match
-------
GC
AA
AA
GT
(4 rows)
This is the regex you need for overlapped matches.
(?=(..CAA))
https://regex101.com/r/eJ36zb/1
I think you just need this sql statement which captures group 1:
SELECT regexp_matches('ATGCATGCATGCCAACAACAACCTGTCAAGTGAGT','(?=(..CAA))','g');
Formatted regex
(?=
( . . CAA ) # (1)
)
The reason you got empty strings in your result is that
you didn't give the expression anything to consume and
nothing to capture.
I.e., it matched at the right places, but nothing was consumed or captured.
So, doing it this way allows the overlap and the capture so it
should show up on the output now.
Lookahead is a zero-width assertion. It doesn't match anything. If you change your regular expression to just a regular match/capture, you'll get a result. For matching any two characters that are followed by CAA in your case, lookahead probably isn't necessary.
I have data like this
Giftsbirth;;Basket7;CC
Giftswedding;;Cake4;COD
I am trying to find a regex that will only select the second data (Basket7, Cake4).
From past help I tried something like
^(\w+ [^\v;;]+;;[^\v;]+)?.*
But I know that is not right
Please assist with the regex if you can
You could use a positive lookbehind (?<= to assert what is before is ;; and a positive lookahead (?= to assert that what follows is ;
Use a negative character class [^;]+ to match not a ; to match your values.
(?<=;;)[^;]+(?=;)
You may use
(?:.*;)?([^;\n\r]+);[^;\n\r]+$
Or,
.*?;;([^;\r\n]+)(?:;.*)?
and replace with $1.
Details
(?:.*;)? - an optional substring having 0+ chars other than line break chars, as many as possible, up to the ;
([^;\n\r]+) - Group 1: any one or more chars other than CR, LF and ;
; - a semi-colon
[^;\n\r]+ - any one or more chars other than CR, LF and ;
$ - end of line.
The second regex matches
.*?;; - any 0+ chars as few as possible up to (and including) the first ;;
([^;\r\n]+) - Group 1: any one or more chars other than CR, LF and ;
(?:;.*)? - an optional group matching 1 or 0 occurrences of a ; and then any 0+ chars up to the end of line
The $1 in the replacement is the value you need to keep.
You need to specify more precisely what "the second data (Basket7, Cake4)" means. This looks like CSV data with the ; set as separator, but that would place Basket7 and Cake4 in the third column, since the second column is empty. In order to write a regex that solves this problem in the general case, you need to take into account the full domain of possible lines, and you've only given two examples and let everyone guess what the underlying format and total possible variations might be.
For example, is it always reasonable to assume that that which you're looking for is always preceded by ;; and ends with a ;, and that ;; never occurs in other places than immediately before that which you're looking for? In that case, (?<=;;)([^;]*) captures this. But what if you encounter one of the following lines?
Giftsbirth;;;CC # Here, the thing matched is empty
Giftsbirth;1600;Basket7;CC # Here, the second column isn't empty
;;Basket7;CC # Here, the first column is empty
;;;CC # Here, all but the last column are empty
;;; # Here, all columns are empty
You may experience that various suggestions will give you "the right text", but if you test this on a limited subset that does not account for all variations that can reasonably be expected in the input, you will inevitably have to revise your regex.
Assuming this is a CSV where the fields don't contain literal ;s, and that you don't know anything about the length of any of the fields (and consequently that the second column isn't always empty), but that there are at least three columns, you could consider the regex:
^[^;]*;[^;]*;([^;]*)
(See demo at https://regex101.com/r/vhPNEj/1)
These assumptions may not be correct, but my ability to guess are much worse than yours, since you're sitting with a larger sample size of data. In order to succeed at automating your tasks, it is critical that you learn to modify code to fit your assumptions.
For example, you may want to disregard the cases where the third column is empty:
^[^;]*;[^;]*;([^;]+)
Here the difference is [^;]* changed into [^;]+.
Or you may want to take into account that the first column could contain semicolons when they are wrapped in double quotes, e.g. like "Giftsbirth; Holiday";;Basket7;CC:
^(?:[^;"]*|"[^"]*");[^;]*;([^;]*)
Here the difference is [^;]* changed into (?:[^;"]*|"[^"]*") being either [^;"]* (being all but ; and ") or "[^"]*" (being " followed by anything but ", which includes ;, followed by ").
I am trying to extract R7080075 and X1234567 from the sample data below. The format is always a single upper case character followed by 7 digit number. This ID is also always preceded by an underscore. Since it's user generated data, sometimes it's the first underscore in the record and sometimes all preceding spaces have been replaced with underscores.
I'm querying HDP Hive with this in the select statement:
REGEXP_EXTRACT(column_name,'[(?:(^_A-Z))](\d{7})',0)
I've tried addressing positions 0-2 and none return an error or any data. I tested the code on regextester.com and it highlighted the data I want to extract. When I then run it in Zepplin, it returns NULLs.
My regex experience is limited so I have reviewed the articles here on regexp_extract (+hive) and talked with a colleague. Thanks in advance for your help.
Sample data:
Sept Wk 5 Sunny Sailing_R7080075_12345
Holiday_Wk2_Smiles_X1234567_ABC
The Hive manual says this:
Note that some care is necessary in using predefined character classes: using '\s' as the second argument will match the letter s; '\\s' is necessary to match whitespace, etc.
Also, your expression includes unnecessary characters in the character class.
Try this:
REGEXP_EXTRACT(column_name,'_[A-Z](\\d{7})',0)
Since you want only the part without underscore, use this:
REGEXP_EXTRACT(column_name,'_([A-Z]\\d{7})',1)
It matches the entire pattern, but extracts only the second group instead of the entire match.
Or alternatively:
REGEXP_EXTRACT(column_name,'(?<=_)[A-Z]\\d{7}', 0)
This uses a regexp technique called "positive lookbehind". It translates to : "find me an upper case alphabet followed by 7 digits, but only if they are preceded by an _". It uses the _ for matching but doesn't consider it part of the extracted match.
I need a regex to get numeric values that can be
111.111,11
111,111.11
111,111
And separate the integer and decimal portions so I can store in a DB with the correct syntax
I tried ([0-9]{1,3}[,.]?)+([,.][0-9]{2})? With no success since it doesn't detect the second part :(
The result should look like:
111.111,11 -> $1 = 111111; $2 = 11
First Answer:
This matches #,###,##0.00:
^[+-]?[0-9]{1,3}(?:\,?[0-9]{3})*(?:\.[0-9]{2})?$
And this matches #.###.##0,00:
^[+-]?[0-9]{1,3}(?:\.?[0-9]{3})*(?:\,[0-9]{2})?$
Joining the two (there are smarter/shorter ways to write it, but it works):
(?:^[+-]?[0-9]{1,3}(?:\,?[0-9]{3})*(?:\.[0-9]{2})?$)
|(?:^[+-]?[0-9]{1,3}(?:\.?[0-9]{3})*(?:\,[0-9]{2})?$)
You can also, add a capturing group to the last comma (or dot) to check which one was used.
Second Answer:
As pointed by Alan M, my previous solution could fail to reject a value like 11,111111.00 where a comma is missing, but the other isn't. After some tests I reached the following regex that avoids this problem:
^[+-]?[0-9]{1,3}
(?:(?<comma>\,?)[0-9]{3})?
(?:\k<comma>[0-9]{3})*
(?:\.[0-9]{2})?$
This deserves some explanation:
^[+-]?[0-9]{1,3} matches the first (1 to 3) digits;
(?:(?<comma>\,?)[0-9]{3})? matches on optional comma followed by more 3 digits, and captures the comma (or the inexistence of one) in a group called 'comma';
(?:\k<comma>[0-9]{3})* matches zero-to-any repetitions of the comma used before (if any) followed by 3 digits;
(?:\.[0-9]{2})?$ matches optional "cents" at the end of the string.
Of course, that will only cover #,###,##0.00 (not #.###.##0,00), but you can always join the regexes like I did above.
Final Answer:
Now, a complete solution. Indentations and line breaks are there for readability only.
^[+-]?[0-9]{1,3}
(?:
(?:\,[0-9]{3})*
(?:.[0-9]{2})?
|
(?:\.[0-9]{3})*
(?:\,[0-9]{2})?
|
[0-9]*
(?:[\.\,][0-9]{2})?
)$
And this variation captures the separators used:
^[+-]?[0-9]{1,3}
(?:
(?:(?<thousand>\,)[0-9]{3})*
(?:(?<decimal>\.)[0-9]{2})?
|
(?:(?<thousand>\.)[0-9]{3})*
(?:(?<decimal>\,)[0-9]{2})?
|
[0-9]*
(?:(?<decimal>[\.\,])[0-9]{2})?
)$
edit 1: "cents" are now optional;
edit 2: text added;
edit 3: second solution added;
edit 4: complete solution added;
edit 5: headings added;
edit 6: capturing added;
edit 7: last answer broke in two versions;
I would at first use this regex to determine wether a comma or a dot is used as a comma delimiter (It fetches the last of the two):
[0-9,\.]*([,\.])[0-9]*
I would then strip all of the other sign (which the previous didn't match). If there were no matches, you already have an integer and can skip the next steps. The removal of the chosen sign can easily be done with a regex, but there are also many other functions which can do this faster/better.
You are then left with a number in the form of an integer possible followed by a comma or a dot and then the decimals, where the integer- and decimal-part easily can be separated from eachother with the following regex.
([0-9]+)[,\.]?([0-9]*)
Good luck!
Edit:
Here is an example made in python, I assume the code should be self-explaining, if it is not, just ask.
import re
input = str(raw_input())
delimiterRegex = re.compile('[0-9,\.]*([,\.])[0-9]*')
splitRegex = re.compile('([0-9]+)[,\.]?([0-9]*)')
delimiter = re.findall(delimiterRegex, input)
if (delimiter[0] == ','):
input = re.sub('[\.]*','', input)
elif (delimiter[0] == '.'):
input = re.sub('[,]*','', input)
print input
With this code, the following inputs gives this:
111.111,11
111111,11
111,111.11
111111.11
111,111
111,111
After this step, one can now easily modify the string to match your needs.
How about
/(\d{1,3}(?:,\d{3})*)(\.\d{2})?/
if you care about validating that the commas separate every 3 digits exactly,
or
/(\d[\d,]*)(\.\d{2})?/
if you don't.
If I'm interpreting your question correctly so that you are saying the result SHOULD look like what you say is "would" look like, then I think you just need to leave the comma out of the character class, since it is used as a separator and not a part of what is to be matched.
So get rid of the "." first, then match the two parts.
$value = "111,111.11";
$value =~ s/\.//g;
$value =~ m/(\d+)(?:,(\d+))?/;
$1 = leading integers with periods removed
$2 = either undef if it didn't exist, or the post-comma digits if they do exist.
See Perl's Regexp::Common::number.