Efficient SQL statement to encode text for NLP locutions - regex

Background
A locution is a noun-phrase consisting of at least two words, such as:
black olive
hot pepper sauce
rose finn apple potato
The separate words black and olive are an adjective (black - JJ) and a noun (olive - NN). However, humans know that black olive is a noun (that differentiates it from, say, a green olive).
The problem here is how to most efficiently transform a list of normalized ingredient names (such as the list above) into a specific format for a natural language processor (NLP).
Example Data
The table can be created as follows:
CREATE TABLE ingredient_name (
id bigserial NOT NULL, -- Uniquely identifies the ingredient.
label character varying(30) NOT NULL
);
The following SQL statements show actual database records:
insert into ingredient_name (label) values ('alfalfa sprout');
insert into ingredient_name (label) values ('almond extract');
insert into ingredient_name (label) values ('concentrated apple juice');
insert into ingredient_name (label) values ('black-eyed pea');
insert into ingredient_name (label) values ('rose finn apple potato');
Data Format
The general format is:
lexeme1_lexeme2_<lexemeN> lexeme1_lexeme2_lexemeN NN
Given the list of words above, the NLP expects:
black_<olive> black_olive NN
hot_pepper_<sauce> hot_pepper_sauce NN
rose_finn_apple_<potato> rose_finn_apple_potato NN
The database has a table (recipe.ingredient_name) and a column (label). The labels are normalized (e.g., single space, lower case).
SQL Statement
The code that produces the expected results:
CREATE OR REPLACE VIEW ingredient_locutions_vw AS
SELECT
t.id,
-- Replace spaces with underscores
translate( t.prefix, ' ', '_' )
|| '<' || t.suffix || '>' || ' ' ||
translate( t.label, ' ', '_' )
|| ' NN' AS locution_nlp
FROM (
SELECT
id,
-- Ingredient name
label,
-- All words except the last word
left( label, abs( strpos( reverse( label ), ' ' ) - length( label ) ) + 1 ) AS prefix,
-- Just the last word
substr( label,
length( label ) - strpos( reverse( label ), ' ' ) + 2
) AS suffix
FROM
ingredient_name
WHERE
-- Limit set to ingredient names having at least one space
strpos( label, ' ' ) > 0
) AS t;
Question
What is a more efficient (or elegant) way to split the prefix (all words except the first) and suffix (just the last word) in the above code?
The system is PostgreSQL 9.1.
Thank you!

CREATE OR REPLACE VIEW ingredient_locutions_vw AS
SELECT
t.id,
format('%s_<%s> %s NN',
array_to_string(t.prefix, '_'),
t.suffix,
array_to_string(t.label, '_')
) AS locution_nlp
FROM (
SELECT
id,
-- Ingredient name
label,
-- All words except the last word
label[1:array_length(label, 1) - 1] AS prefix,
-- Just the last word
label[array_length(label, 1)] AS suffix
FROM (
select id, string_to_array(label, ' ') as label
from ingredient_name
) s
WHERE
-- Limit set to ingredient names having at least one space
array_length(label, 1) > 1
) AS t;
select * from ingredient_locutions_vw ;
id | locution_nlp
----+--------------------------------------------------------
1 | alfalfa_<sprout> alfalfa_sprout NN
2 | almond_<extract> almond_extract NN
3 | concentrated_apple_<juice> concentrated_apple_juice NN
4 | black-eyed_<pea> black-eyed_pea NN
5 | rose_finn_apple_<potato> rose_finn_apple_potato NN
(5 rows)

Related

Interactive grid removes spaces

Look at the query and result generated in sqlplus :
SELECT
plan_line_id,
LPAD (a.plan_operation || ' ' || a.plan_options,
LENGTH (a.plan_operation || ' ' || a.plan_options) + a.plan_depth * 3
) OPERATION,
a.plan_cost COST,
output_rows "ROWS"
FROM gV$SQL_PLAN_MONITOR a
WHERE sql_id = '8gan9d0z7bzhm'
Looks pretty cool doesn't it ;)
And the same result in grid report :
Apex 21 removes all spaces generated by lpad function. The same problem is with classic report and classic grid :( any idea why ?
You'll have to use a character which can be correctly interpreted by your browser - instead of a space, use a non-breaking space, the &nbsp character.
Something like this (I'm using the # character in the CTE, which is then replaced by :
WITH
temp
AS
(SELECT plan_line_id,
LPAD (
a.plan_operation || ' ' || a.plan_options,
LENGTH (a.plan_operation || ' ' || a.plan_options)
+ a.plan_depth * 3,
'#') OPERATION, --> here
a.plan_cost COST,
output_rows "ROWS"
FROM a1_test a
WHERE sql_id = '9p4xcx2gd8u49')
SELECT plan_line_id,
REPLACE (operation, '#', ' ') operation, --> here
cost,
"ROWS"
FROM temp
Turn "Operation" column's **Escape special characters" property OFF.
The result is then
Not as pretty as you'd want it to, but it is better than what you currently have.

Oracle PLSQL regexp_substr separate comma separated string with double quotes

I've seen examples of how to separate comma-separated strings into rows like this:
select distinct id, trim(regexp_substr(value,'[^,]+', 1, level) ) value, level
from tbl1
connect by regexp_substr(value, '[^,]+', 1, level) is not null
order by id, level;
but, my question is, how do I do this on double quote and comma delimited strings?
Ex: the above works for strings like "1,2,3,4,5,6,7", but what about "1","2","3","4,5","6,7,8","9" so that the rows end up like:
1
2
3
4,5
6,7,8
9
edit: I'm on Oracle 11.2.0.4, 11gR2.
There is a hack. Replace the pattern "," with # and use it in regular expression.It works like a charm.
Input String : "1","2","3","4,5","6,7,8","9","Ant,B","Gradle","E,F","G"(Can be number/Character doesn't matter)
with temp as (
select replace(replace('"1","2","3","4,5","6,7,8","9","Ant,B","Gradle","E,F","G"','","','#'),'"') Colmn from dual
)
SELECT trim(regexp_substr(str, '[^#]+', 1, level)) str
FROM (SELECT Colmn str FROM temp) t
CONNECT BY instr(str, '#', 1, level - 1) > 0
Output :
STR
1
2
3
4,5
6,7,8
9
Ant,B
Gradle
E,F
G
10 rows
Refer DBFiddle link for demo.
https://dbfiddle.uk/?rdbms=oracle_18&fiddle=d09c326f614d10f5d3c407fdfd3a44c5
here is the another solution.it must be working all string with have number.
i used number values index as base.
with temp as (
select '"1","2","3",x"4,5","6,73,8","9"' Colmn from dual
)
SELECT regexp_substr(Colmn, '\d{1}', REGEXP_INSTR(Colmn, '\d{1}', REGEXP_INSTR(Colmn, '\d{1}') ,level),1 ) from temp
CONNECT BY REGEXP_COUNT (Colmn,'\d{1}')+1> level

Regular expression - Remove special characters except single white space

From stack overflow, I got the standard reg expression
to eliminate -
a) special characters
b) digits
c) more than 2 spaces to single space
to include -
d) - (hyphen)
e) ' (single quote)
SELECT ID, REGEXP_REPLACE(REGEXP_REPLACE(forenames, '[^A-Za-z-]', ' '),'\s{2,}',' ') , REGEXP_REPLACE(REGEXP_REPLACE(surname, '[^A-Za-z-]', ' '),'\s{2,}',' ') , forenames, surname from table1;
Instead of 2 functions how to get the result in single function?
to include '(single quote) \' is not working in regexp_replace.
Thanks.
Oracle Setup:
CREATE TABLE test_data ( id, value ) AS
SELECT 1, '123a45b£$- ''c45d#{e''' FROM DUAL
Query:
SELECT ID,
REGEXP_REPLACE(
value,
'[^a-zA-Z'' -]| +( )',
'\1'
)
FROM test_data
Output:
ID | REGEXP_REPLACE(VALUE,'[^A-ZA-Z''-]|+()','\1')
-: | :--------------------------------------------
1 | ab- 'cde'
db<>fiddle here

PL SQL regular expression substring

I have a long string.
message := 'I loooove my pet animal';
This string in 23 chars long. If message is greater that 15 chars, I need to find the length of message where I can break the string into 2 strings. For example, in this case,
message1 := 'I loove my'
message2 := 'pet animal'
Essentially it should find the position of a whole word at the previous to 15 chars and the break the original string into 2 at that point.
Please give me ideas how I can do this.
Thank you.
Here is a general solution - with possibly more than one input string, and with inputs of any length. The only assumption is that no single word may be more than 15 characters, and that everything between two spaces is considered a word. If a "word" can be more than 15 characters, the solution can be adapted, but the requirement itself would need to state what the desired result is in such a case.
I make up two input strings in a CTE (at the top) - that is not part of the solution, it is just for testing and illustration. I also wrote this in plain SQL - there is no need for PL/SQL code for this type of problem. Set processing (instead of one row at a time) should result in much better execution.
The approach is to identify the location of all spaces (I append and prepend a space to each string, too, so I won't have to deal with exceptions for the first and last substring); then I decide, in a recursive subquery, where each "maximal" substring should begin and where it should end; and then outputting the substrings is trivial. I used a recursive query, that should work in Oracle 11.1 (or 11.2 with the syntax I used, with column names in CTE declarations - it can be changed easily to work in 11.1). In Oracle 12, it would be easier to rewrite the same idea using MATCH_RECOGINZE.
with
inputs ( id, str ) as (
select 101, 'I loooove my pet animal' from dual union all
select 102, '1992 was a great year for - make something up here as needed' from dual
),
positions ( id, pos ) as (
select id, instr(' ' || str || ' ', ' ', 1, level)
from inputs
connect by level <= length(str) - length(replace(str, ' ')) + 2
and prior id = id
and prior sys_guid() is not null
),
r ( id, str, line_number, pos_from, pos_to ) as (
select id, ' ' || str || ' ', 0, null, 1
from inputs
union all
select r.id, r.str, r.line_number + 1, r.pos_to,
( select max(pos)
from positions
where id = r.id and pos - r.pos_to between 1 and 16
)
from r
where pos_to is not null
)
select id, line_number, substr(str, pos_from + 1, pos_to - pos_from - 1) as line_text
from r
where line_number > 0 and pos_to is not null
order by id, line_number
;
Output:
ID LINE_NUMBER LINE_TEXT
---- ----------- ---------------
101 1 I loooove my
101 2 pet animal
102 1 1992 was a
102 2 great year for
102 3 - make
102 4 something up
102 5 here as needed
7 rows selected.
First you reverse string.
SELECT REVERSE(strField) FROM DUAL;
Then you calculate length i = length(strField).
Then find the first space after the middle
j := INSTR( REVERSE(strField), ' ', i / 2, i)`
Finally split by i - j (maybe +/- 1 need to test it)
DEMO
WITH parameter (id, strField) as (
select 101, 'I loooove my pet animal' from dual union all
select 102, '1992 was a great year for - make something up here as needed' from dual union all
select 103, 'You are Supercalifragilisticexpialidocious' from dual
), prepare (id, rev, len, middle) as (
SELECT id, reverse(strField), length(strField), length(strField) / 2
FROM parameter
)
SELECT p.*, l.*,
SUBSTR(strField, 1, len - INSTR(rev, ' ', middle)) as first,
SUBSTR(strField, len - INSTR(rev, ' ', middle) + 2, len) as second
FROM parameter p
JOIN prepare l
ON p.id = l.id
OUTPUT

how to get out string oracle regex

I have the following string my trying get out the 1111111 and 33333333333 with out the |
character
SELECT regexp_substr('7|1111111|2222222|33333333333|0||20140515|||false|0|0|0|0|0|','*[|]*[|][0-9]*')FROM dual
Using REGEXP_REPLACE may be a bit simpler;
SELECT REGEXP_REPLACE('7|1111111|2222222|33333333333|0||20140515|||false|0|0|0|0|0|',
'^([^|]*[|]){1}([^|]*).*$', '\2') FROM dual;
> 1111111
SELECT REGEXP_REPLACE('7|1111111|2222222|33333333333|0||20140515|||false|0|0|0|0|0|',
'^([^|]*[|]){3}([^|]*).*$', '\2') FROM dual;
> 33333333333
You can choose column by choosing how many pipes to skip in the {1} part.
A simple SQLfiddle to test with.
A short explanation of the regexp;
([^|]+[|]){3} -- Matches 3 groups of {optional characters}{pipe}
(\d*) -- Matches the next digit group (the one we want)
.* -- Matches the rest of the expression
What we want is the second paranthesized group, that is, we replace the whole string by the back reference \2.
Because "|" separators always present it's simpler to extract fields with simple substring function rather than using regular expressions.
Just find positions of corresponding separators in source string and extract content between them:
with test_data as (
select
'7|1111111|2222222|33333333333|0||20140515|||false|0|0|0|0|0|ABC' as s,
8 as field_number -- test 1, 3, 8, 10 and 16
from dual
)
select
field_number,
substr(
s,
decode( field_number,
1,1,
instr(s,'|',1,field_number - 1) + 1
),
(
decode( instr(s,'|',1,field_number),
0, length(s)+ 1,
instr(s,'|',1,field_number)
)
-
decode( field_number,
1, 1,
instr(s,'|',1,field_number - 1) + 1
)
)
) as field_value
from
test_data
SQLFiddle
This variant works with empty fields, non-numeric fields and so on.
Possible simplification with appending additional separators to the start and the end of the string:
with test_data as (
select
(
'|' ||
'7|1111111|2222222|33333333333|0||20140515|||false|0|0|0|0|0|ABC' ||
'|'
) as s, -- additional separators appended before and after original string
10 as field_number -- test 1, 3, 8, 10 and 16
from dual
)
select
field_number,
substr(
s,
instr(s, '|', 1, field_number) + 1,
(
instr(s, '|', 1, field_number + 1)
-
(instr(s, '|', 1, field_number) + 1)
)
) as field_value
from
test_data
;
SQLFiddle