Vi: Substitution pattern SQL file. Issue with the Regex - regex
I have to modify a SQL file with vi to delete columns that we do not use. As we have a lot of data, I use the search and replace option with a Regex Pattern.
For instance we have :
(1,2956,2026442,4,NULL,NULL,'ZAC DU BOIS DES COMMUNES','',NULL,NULL,'Rue DU LUXEMBOURG',NULL,
'9999','EVREUX',NULL,1,'27229',NULL,NULL,NULL,NULL,NULL,' Rue DU LUXEMBOURG, 9999 EVREUX',NULL,NULL,NULL,NULL,
NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'2020-07-08 16:34:40',NULL,NULL)
So we have 40 columns and I keep 13 ones. My regex is :
(1),2,(3),4-5,(6-14),15-22,(23),24-39,(40)
:%s/(\(.\{-}\),.\{-},\(.\{-}\),.\{-},.\{-},\(.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-}\),.\{-},
.\{-}, .\{-},.\{-},.\{-},.\{-},.\{-},.\{-},\(.\{-}\),.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},
.\{-},.\{-},.\{-},.\{-},.\{-},\(.\{-}\))/(\1,\2,\3,\4,\5)/g
I enclose in my parenthesis the parts that interest me by putting them in parenthesis (I only get the values in parenthesis on the line above my regex ). Then with the replace I recover these groups.
So normally my result is suppose to be :
(1,2026442,NULL,'ZAC DU BOIS DES COMMUNES','',NULL,NULL,'Rue DU LUXEMBOURG',NULL,
'9999','EVREUX',' Rue DU LUXEMBOURG, 9999 EVREUX',NULL)
But Because in ' Rue DU LUXEMBOURG, 9999 EVREUX' there is a comma (,). My result become :
(1,2026442,NULL,'ZAC DU BOIS DES COMMUNES','',NULL,NULL,'Rue DU LUXEMBOURG',NULL,'9999','EVREUX',' Rue DU LUXEMBOURG',NULL,NULL)
Does Someone who is good in Regex can help me ? thanks in advance. If I wasn't clear tell me too, i will try to explain better next time.
I suggest matching fields that can be strings with a %('[^']*'|\w*) pattern, that is, a non-capturing group that finds either ' + zero or more non-'s and then a ' char, or any zero or more alphanumeric characters.
Also, the use of non-capturing groups (in Vim, it is %(...) in very magic mode, or \%(...\) in a regular mode) and very magic mode can help shorten the pattern.
The whole pattern will look like
:%s/\v\(([^,]*),[^,]*,([^,]*),[^,]*,[^,]*,(%('[^']*'|\w*)%(,%('[^']*'|\w*)){8})%(,%('[^']*'|\w*)){8},('[^']*'|\w*)%(,%('[^']*'|\w*)){16},([^,]*)\)/(\1,\2,\3,\4,\5)/g
See the regex demo converted to a PCRE regex.
Note some fields that are not strings are matched with [^,]* that matches zero or more chars other than a comma. The %(,%('[^']*'|\w*)){8} like patterns match (here) 8 occurrences of a sequence of a , char + '...' substring or zero or more word chars.
Related
Remove spaces (apostrophes) around quotes with regex in ruby
I'm trying to remove all spaces around quotes with one Ruby regex. (not the same question as this) Input: l' avant ou l 'après ou encore ' maintenant' Output: l'avant ou l'après ou encore 'maintenant' What I tried: (/'\s|\s'/, '') It's matching a few cases, but not all. How to perform this ? Thanks.
TLDR:
I assume the spaces were inserted by some automation software and there can only be single spaces around the words.
s = "l' avant ou l 'apres ou encore ' maintenant' ou bien 'ceci ' et ' encore de l ' huile ' d 'accord d' accord d ' accord Je n' en ai pas .... s ' entendre Je m'appelle Victor"
first_rx = /(?<=\b[b-df-hj-np-tv-z]) ' ?(?=\p{L})|(?<=\b[b-df-hj-np-tv-z]) ?' (?=\p{L})/i
# If you find it overmatches, replace [b-df-hj-np-tv-z] with [dlnsmtc],
# i.e. first letters of word that are usually contracted
second_rx = /\b'\b\K|' *((?:\b'\b|[^'])+)(?<=\S) *'/
puts s.gsub(first_rx, "'")
.gsub(second_rx) { $~[1] ? "'#{$~[1]}'" : "" }
Output:
l'avant ou l'apres ou encore 'maintenant' ou bien 'ceci' et 'encore de l'huile' d'accord d'accord d'accord Je n'en ai pas .... s'entendre Je m'appelle Victor
Explanation
The problem is really complex. There are several words that can be abbreviated and used with an apostrophe in French, de, le/la, ne, se, me, te, ce to name a few, but these are all consonants. You may remove all spaces between a single, standalone consonant, apostrophe and the next word using
s.gsub(/(?<=\b[b-df-hj-np-tv-z]) ' ?(?=\p{L})|(?<=\b[b-df-hj-np-tv-z]) ?' (?=\p{L})/i, "'")
If you find it overmatches, replace [b-df-hj-np-tv-z] with [dlnsmtc], i.e. first letters of word that are usually contracted. See the regex demo.
Next step is to remove spaces after initial and before trailing apostrophes. This is tricky:
s.gsub(/\b'\b\K|' *((?:\b'\b|[^'])+)(?<=\S) *'/) { $~[1] ? "'#{$~[1]}'" : "" }
where \b'\b is meant to match all apsotrophes in between word chars, those that we fixed at the previous step. See this regex demo. As there is no (*SKIP)(*F) support in Onigmo regex, the regex is a bit simplified but the replacement is a conditional one: if Group 1 matched, replace with ' + Group 1 value ($1) + ', else, replace with an empty string (since \K reset the match, dropped all text from the match memory buffer).
NOTE: this approach can be extended to handle some specific cases like aujourd'hui, too.
To remove all whitespace around the ', use gsub!, applied in several steps for proper whitespace removal: str = "l' avant ou l 'apres ou encore ' maintenant'" str.gsub!(/\b'\s+\b/, "'").gsub!(/\b\s+'\b/, "'").gsub!(/\b(\s+')\s+\b/, '\1') puts str # l'avant ou l'apres ou encore 'maintenant' Here, \b : word boundary, \s+ : 1 or more whitespace, string.gsub!(regex, replacement_string) : replace in the string argument regex with specified replacement_string (during this, the original string is changed), \1 : in the replacement string, this refers to the first group captured in parenthesis in the regex: (...).
So if you have alot of data like this, all the answers I have seen are wrong, and will not work. No regex can guess wether the preceding word should have a space or not. Unless you came up with a list of words (or patterns) that either do or don't. The problem is, sometimes a space should be left, sometimes not. The only way to script that is to find a pattern which describes when the space should be there, or when not. You must teach your regex French grammar. It may be possible lol. But probably not, or difficult. If this is a one off, my advice is to create regexes for 2 or 3 different situations, and use something like vim, to go through the data, and select manually yes or no to substitute each occurrence. There may be some cases you can run - eg remove all spaces to the right of quotes? - but unfortunately I don't think you can automate this process.
I believe the following should work for you
s.gsub(/'.*?'/){ |e| "'#{e[1...-1].strip}'" }
The regex portion lazy matches all text within single quotes (including quotes). Then, for each match you substitute for the quoted text with leading and trailing whitespace removed, and return this text in quotes.
Split sentence to words containing apostrophe
Supposing I have a group of words as a sentence like this :
Aujourd'hui séparer l'élément en deux
And want the result to be as an individual words (after the split) :
Aujourd'hui | séparer | l' | élément | en | deux
Note : as you can see, « aujourd'hui » is a single word.
What would be the best regex to use here ?
With my current knowledge, all i can achieve is this basic operation :
QString sentence("Aujourd'hui séparer l'élément en deux");
QStringList list = sentence.split(" ");
Output :
Aujourd'hui / Séparer / l'élément / en / deux
Here are the two questions closest to mine : this and this.
Since the contractions that you want to consider as separate words are usually a single letter + an apostrophe in French (like l'huile, n'en, d'accord) you can use a pattern that either matches 1+ whitespace chars, or a location that is immediately preceded with a start of a word, then 1 letter and then an apostrophe.
I also suggest taking into account curly apostrophes. So, use
\s+|(?<=\b\p{L}['’])\b
See the regex demo.
Details
\s+ - 1+ whitespaces
| - or
(?<=\b\p{L}['’])\b - a word boundary (\b) location that is preceded with a start of word (\b), a letter (\p{L}) and a ' or ’.
In Qt, you may use
QStringList result = text.split(
QRegularExpression(R"(\s+|(?<=\b\p{L}['’])\b)",
QRegularExpression::PatternOption::UseUnicodePropertiesOption)
);
The R"(...)" is a raw string literal notation, you may use "\\s+|(?<=\\b\\p{L}['’])\\b" if you are using a C++ environment that does not allow raw string literals.
Not sure if I understood what you are saying but this might help you
QString sentence("Aujourd'hui séparer l'élément en deux");
QStringList list = sentence.split(" '");
I don't know C++ but I guees it supports negative lookbehind.
Have a try with:
(?: |(?<!\w{2})')
This will split on space or apostroph if there are not 2 letters before.
Demo & explanation
Well, you're dealing with a natural language, here, and the first - and toughest - problem to answer is: Can you actually come up with a fixed rule, when splits should happen? In this particular case, there is really no logical reason, why French considers "aujourd'hui" as a single word (when logically, it could be parsed as "au jour de hui").
I'm not familiar with all the possible pitfalls in French, but if you really want to make sure to cover all obscure cases, you'll have to look for a natural language tokenizer.
Anyway, for the example you give, it may be good enough to use a QRegularExpression with negative lookbehind to omit splits when more than one letter precedes the apostrophe:
sentence.split(QRegularExpression("(?<![\\w][\\w])'"));
Need REGEX to find text excluding one character inside
I have a text like this: [mk_dropcaps char=”L”]os especialistas no dejan de insistir en ello: si quieres gozar de una buena salud bucodental. I need a regex that matches [mk_dropcaps char=””], excluding the character between quotes.
Do you want to capture just that within the [ ] square brackets? \[mk_dropcaps char="."\] You may need to replace the quotes with ones you use :P
When creating regexp you can use utilities websites to help you with the task. The answer you seek is : ^.*\[mk_dropcaps char=”.”\].*$ The explaination : EDIT const regex = /^.*(\[mk_dropcaps char=”.”\]).*$/; const str = '[mk_dropcaps char=”L”]os especialistas no dejan de insistir en ello: si quieres gozar de una buena salud bucodental.'; const ret = regex.exec(str); console.log(ret[1]);
For your example data you could use 2 capturing groups to capture [mk_dropcaps char=” and ”] using: (\[mk_dropcaps char=”)[^”]+(”\]) or (\[[^”]+”)[^”]+(”\]) Regex demo 1 Regex demo 2 Then use the capturing groups \1\2 to get [mk_dropcaps char=””]
Regex - Match all Spaces before first "="
I have a large Property file where I need to quickly remove all the SPACE from the keys which got added by mistake. I am using Sublime Text. Where I need to use regex to match the following. What I have: Rio de Janeiro=Рио де Жанейро Rio Grande do Norte=Рио Гранде до Норте Rio Grande do Sul=Рио Гранде направя Sul What I Need: RiodeJaneiro=Рио де Жанейро RioGrandedoNorte=Рио Гранде до Норте RioGrandedoSul=Рио Гранде направя Sul I have gone through the Regex list and tried to come up with my own but failed. I need a quick solution so I am here. Any help is much appreciated. Thanks.
Hit CTRL + H Find What: [ ]+(?=.*?=) Replace With: Live Demo Explanation:
Search for: [ ]+(?=[A-Za-z].*=) And replace with nothing. Explain: [ ]+ : 1 or more spaces (?=[A-Za-z].*=) : a positive lookahead that asserts the space is followed by a letter ([A-Za-z]), then other characters `.*` and a `=` character.
How to use regex correctly with this REGEXEXTRACT function?
I'm dealing with a large spreadsheet of order data which uses a unique, sort of hashed string of syntax to represent order items and attributes.
I currently have this data in Google Sheets and I'm hoping to be able to make use of the REGEXEXTRACT function (https://support.google.com/docs/answer/3098244) to retrieve the pieces of information I need from each row.
Example of function: REGEXEXTRACT("Needle in a haystack", ".e{2}dle")
Order data is huge and I believe I can use this regex function to isolate the piece of information I want.
Examples of the portion of string snippets I need to work with. Keeping in mind the actual order string is much longer than this:
"Location\";s:5:\"value\";s:7:\"Atlanta\"
"Location\";s:2:\"value\";s:8:\"New York\"
"Location\";s:5:\"value\";s:15:\"barrio de boedo\"
So the common string in each row is Location, as value is used multiple times throughout each order.
Let me see if I can articulate this correctly: How would I use regex to specify the value between the 4th and 5th double quotes occurring immediately after the string 'Location' so that in my examples above, the results would be Atlanta, New York, barrio de boedo?
For reference, the barrio de boedo example in its entirety:
\";s:7:\"product\";s:2:\"31\";s:8:\"form_key\";s:16:\"aasdf\";s:7:\"options\";a:2:{i:1;s:1:\"2\";i:2;s:15:\"barrio de boedo\";}s:15:\"super_attribute\";a:2:{i:92;s:1:\"4\";i:132;s:1:\"9\";}s:3:\"qty\";s:1:\"1\";}s:7:\"options\";a:2:{i:0;a:7:{s:5:\"label\";s:15:\"Language-Gender\";s:5:\"value\";s:8:\"spa-male\";s:11:\"print_value\";s:8:\"spa-male\";s:9:\"option_id\";s:1:\"1\";s:11:\"option_type\";s:9:\"drop_down\";s:12:\"option_value\";s:1:\"2\";s:11:\"custom_view\";b:0;}i:1;a:7:{s:5:\"label\";s:8:\"Location\";s:5:\"value\";s:15:\"barrio de boedo\";s:11:\"print_value\";s:15:\"barrio de boedo\";s:9:\"option_id\";s:1:\"2\";s:11:\"option_type\";s:5:\"field\";s:12:\"option_value\";s:15:\"barrio de boedo\";s:11:\"custom_view\";b:0;}}s:15:\"attributes_info\";a:2:{i:0;a:2:{s:5:\"label\";s:5:\"Color\";s:5:\"value\";s:4:\"Grey\";}i:1;a:2:{s:5:\"label\";s:4:\"Size\";s:5:\"value\";s:1:\"L\";}}s:11:\"simple_name\";s:14:\"T-Shirt-Grey-L\";s:10:\"simple_sku\";s:14:\"t-shirt-Grey-L\";s:20:\"product_calculations\";i:1;s:13:\"shipment_type\";i:0;}"
You need to use the following pattern in REGEXEXTRACT:
"Location\\?(?:""[^""]*){3}""([^""]+)\\"""
See the regex demo.
The pattern is Location\\?(?:"[^"]*){3}"([^"]+)\\" and it matches:
Location - a substring Location
\\? - 1 or 0 \ symbols (the ? makes a pattern optional)
(?:"[^"]*){3} - exactly 3 occurrences (due to the limiting quantifier {3}) of a " followed with zero or more (due to the * quantifier) chars other than " (the [^...] is a negated character class that matches any chars but those defined in the class)
" - a single double quote
([^"]+) - capturing group #1 (whose contents will be returned with REGEXEXTRACT): 1 or more (due to + quantifier) chars other than "
\\" - a \" substring.