I have a need to execute a metadata query which will dump a list of tables into a file. However, I need a way to eliminate all formatting besides the tableId itself. Can this be done through a regex? Appreciate all help in advance.
+-------------------------------------+-------+
| tableId | Type |
+-------------------------------------+-------+
| t_margins | TABLE |
| t_rev_test | TABLE |
| t_rev_share | TABLE |
You have some options, but I would suggest something like this:
^\| (\S+)
It will match on the line from the start, a pipe, a space and then all non-spaces. The non-spaces will be your tableId. Here is a little example in Python:
import re
my_string = '''| t_margins | TABLE |
| t_rev_test | TABLE |
| t_rev_share | TABLE |'''
my_list = my_string.split('\n')
for line in my_list:
match = re.search("^\| (\S+)", line)
print (match.group(1))
This will give you:
t_margins
t_rev_test
t_rev_share
The following regexp captures just the column values of the first column:
^\| (\w+)
https://regex101.com/r/gODhra/3
Related
I have a table of rows with cells containing multiple strings. Like this:
K1111=V1111;K1=V1;kv13_key4=--xxxxxsomething;id5=true;impid=23123123;location=domain_co_uk
I need to extract a substring that begins with kv13_key4= and ends with anything after but the lengths all vary and the substrings are all separated by a semicolon ; . I tried
REGEXP_EXTRACT(customtargeting,'%in2w_key4%;') As contains_key_Value
but didn't work. I need something like this:
| Original Cell | Extracted |
| key88=1811111;id89=9990string;K1=V1;23234234234tttttttt13_key4=--x;id5=true;impid=23123;url=domain_co_uk | kv13_key4=--x |
| K1111=V1111;K1=V1;kv13_key4=--xsomething;id5=true;impid=23123123;location=domain_co_uk | kv13_key4=--xsomething |
| ;id5=true;T6791=V1111;K1=V1;kv13_key4=--xxxxxsomething123;impid=23123 | kv13_key4=--xxxxxsomething123 |
Consider below
select *, regexp_extract(customtargeting, r'kv13_key4=[^;]+') as Extracted
from your_table
if applied to sample data in your question - output is
Does this regex work:
(?<=kv13_key4=)[^;]+(?=;)
It captures everything between 'kv13_key4=' and the nearest ';'
Your REGEX_EXTRACT would look like:
REGEXP_EXTRACT(customtargeting,r'(?<=kv13_key4=)[^;]+(?=;)')
I need to print a string array along with one field in my json object.
The data:
{ "key1":"val1", "key2":"value2", "codes":["apple","mango","banana","orange"], "key3_conditional":"yes"}
My Search query:
<My search query>
| rex "\|(?<payload>[^\|]*)$"
| spath input=payload
| rex "\"codes\":\"(?<codes>[^\"]*)"
| eval is_unknown=if(isnotnull(key3_conditional), key3_conditional, "no")
| table codes, is_unknown
Desired result
codes | is_unknown
--------------------------------------------------
apple, mango, banana, orange | yes
Currently, this only displays the 1st value in codes i.e. apple and I need all values of codes as comma separated. I'm supposing there is some issue with my regex. Please suggest.
If this data is being brought-in a JSON, you won't have to rex it out
If not, though, the issue is your regular expression
Try it out on regex101.com - you'll see you're only grabbing the first value because you're stopping at a literal "
Try this instead:
...
| rex field=_raw "codes\":\[(?<codes>[^\]]+)"
| eval codes=split(replace(codes,"\"",""),",")
That will make codes into a multivalue field
If you don't care about it being multivalue, you can just do:
| eval codes=replace(codes,"\"","")
to pull the quote marks
I am trying to match a param string but exclude any matches when a substring is present.
From my limited regex knowledge this should work to exlude any string containing "porcupine", but it's not. What am I doing wrong?
(\/animal\?.*(?!porcupine).*color=white)
Expected Outcome
| string | matches? |
| ----------------------------------------------- | -------- |
| /animal?nose=wrinkly&type=porcupine&color=white | false |
| /animal?nose=wrinkly&type=puppy&color=white | true |
Actual Outcome
| string | matches? |
| ----------------------------------------------- | -------- |
| /animal?nose=wrinkly&type=porcupine&color=white | true |
| /animal?nose=wrinkly&type=puppy&color=white | true |
Use a Tempered Greedy Token:
/animal\?(?:(?!porcupine).)*color=white
Demo & explanation
The .* searches anything for any number of times, greedily. So you could replace it with a literal search:
(\/animal\?nose=wrinkly\&type=(?!porcupine).*color=white)
See example here: https://regex101.com/r/HJiM2N/1
This may seem overly verbose but it is actually relatively efficient in the number of steps:
(?!\/animal.*?porcupine.*?color)\/animal\?.*color=white
See Regex Demo
If the input string consists of only one and only one occurrence of what you are trying to match and nothing else, then just use the following to ensure that porcupine does not occur anywhere in the input string:
(?!.*porcupine)\/animal\?.*color=white
The code:
import re
tests = [
'/animal?nose=wrinkly&type=porcupine&color=white',
'/animal?nose=wrinkly&type=puppy&color=white'
]
rex = r'(?!\/animal.*?porcupine.*?color)\/animal\?.*color=white'
for test in tests:
m = re.search(rex, test)
print(test, 'True' if m else 'False')
Prints:
/animal?nose=wrinkly&type=porcupine&color=white False
/animal?nose=wrinkly&type=puppy&color=white True
I need to extract domain name from url using Redshift PostgreSQL. Example : extract 'google.com' from 'www.google.com'. Each url in my dataset has different top level domain (TLD). My approach was to first join the matching TLD to the dataset and use regex to extract 'first_string.TLD'. In Redshift, I'm getting error 'The pattern must be a valid UTF-8 literal character expression'. Is there a way around this?
A sample of my dataset:
+---+------------------------+--------------+
| id| trimmed_domain | tld |
+---+------------------------+--------------+
| 1 | sample.co.uk | co.uk |
| 2 | www.sample.co.uk | co.uk |
| 3 | www3.sample.co.uk | co.uk |
| 4 | biz.sample.co.uk | co.uk |
| 5 | digital.testing.sam.co | co |
| 6 | sam.co | co |
| 7 | www.google.com | com |
| 8 | 1.11.220 | |
+---+------------------------+--------------+
My code:
SELECT t1.extracted_domain, COUNT(DISTINCT(t1.id))
FROM(
SELECT
d.id,
d.trimmed_domain,
CASE
WHEN d.tld IS null THEN d.trimmed_domain ELSE
regexp_replace(d.trimmed_domain,'(.*\.)((.[a-z]*).*'||replace(tld,'.','\.')||')','\2')
END AS "extracted_domain"
FROM dataset d
)t1
GROUP BY 1
ORDER BY 2;
Expected output:
+------------------------+--------------+
| extracted_domain | count |
+------------------------+--------------+
| sample.co.uk | 4 |
| sam.co | 2 |
| google.com | 1 |
| 1.11.220 | 1 |
+------------------------+--------------+
I'm so sure about the query. However, you can use this tool and design any expression that you wish to modify your query.
My guess is that maybe this would help:
^(?!d|b|www3).*
You can list any domain that you wish to exclude in the list using OR (?!d|b|www3).
RegEx Circuit
You can visualize your expressions in this link:
You maybe want to add your desired URLs to an expression similar to:
^(sam|www.google|1.11|www.sample|www3.sample).*
So, I've found a solution. Redshift does not support column based regex so the alternative is to use Python UDF.
Change the tld column to regex pattern.
Go row by row and extract the domain name using the regex pattern column.
Group by the extracted_domain and count the users.
The SQL query is as below:
CREATE OR REPLACE function extractor(col_domain varchar)
RETURNS varchar
IMMUTABLE AS $$
import re
_regex = ''
for domain in col_domain:
if domain is None:
continue
else:
_regex += r'{}'.format(domain)
domain_regex = r'([^/.]+\.({}))'.format(_regex)
return domain_regex
$$ LANGUAGE plpythonu;
CREATE OR REPLACE FUNCTION regex_match(in_pattern varchar, input_str varchar)
RETURNS varchar
IMMUTABLE AS $$
import re
if in_pattern == '':
a = str(input_str)
else:
a= str(re.search(in_pattern, input_str).group())
return a
$$ LANGUAGE plpythonu;
SELECT
t2.extracted_domain,
COUNT(DISTINCT(t2.id)) AS "Unique Users"
FROM(
SELECT
t1.id,
t1.trimmed_domain,
regex_match(t1.regex_pattern, t1.trimmed_domain) AS "extracted_domain"
FROM(
SELECT
id,
trimmed_domain,
CASE WHEN tld is null THEN '' ELSE extractor(tld) END AS "regex_pattern"
FROM dataset
)t1
)t2
GROUP BY 1
ORDER BY 2 DESC
LIMIT 10;
Python UDF seems to be slow on a large dataset. So, I'm open to suggestions on improving the query.
If you know the prefixes you would like to remove from the domains, then why not just exclude these? The following query simply removes the know www/http/etc prefixes from domain names and counts normalized domain names.
SELECT COUNT(*) from
(select REGEXP_REPLACE(domain, '^(https|http|www|biz)') FROM domains)
GROUP BY regexp_replace;
I have a 130k line text file with patent information and I just want to keep the dates (regex "[0-9]{4}-[0-9]{2}-[0-9]{2} ") for subsequent work in Excel. For this purpose I need to keep the line structure intact (also blank lines). My main problem is that I can't seem to find a way to identify and keep multiple occurrences of date information in the same line while deleting all other information.
Original file structure:
US20110228428A1 | US | | 7 | 2010-03-19 | SEAGATE TECHNOLOGY LLC
US20120026629A1 | US | | 7 | 2010-07-28 | TDK CORP | US20120127612A1 | US | | EXAMINER | 2010-11-24 | | US20120147501A1 | US | | 2 | 2010-12-09 | SAE MAGNETICS HK LTD,HEADWAY TECHNOLOGIES INC
Desired file structure:
2010-03-19
2010-07-28 2010-11-24 2010-12-09
Thank you for your help!
Search for
.*?(?:([0-9]{4}-[0-9]{2}-[0-9]{2})|$)
And replace with
" $1"
Don't put the quotes, just to show there is a space before the $1. This will also put a space before the first match in a row.
This regex will match as less as possible .*? before it finds either the Date or the end of the row (the $). If a date is found it is stored in $1 because of the brackets around. So as replacement just put a space to separate the found dates and then the found date from $1.