Remove spaces (apostrophes) around quotes with regex in ruby - regex

I'm trying to remove all spaces around quotes with one Ruby regex. (not the same question as this)
Input: l' avant ou l 'après ou encore ' maintenant'
Output: l'avant ou l'après ou encore 'maintenant'
What I tried:
(/'\s|\s'/, '')
It's matching a few cases, but not all.
How to perform this ? Thanks.

TLDR:
I assume the spaces were inserted by some automation software and there can only be single spaces around the words.
s = "l' avant ou l 'apres ou encore ' maintenant' ou bien 'ceci ' et ' encore de l ' huile ' d 'accord d' accord d ' accord Je n' en ai pas .... s ' entendre Je m'appelle Victor"
first_rx = /(?<=\b[b-df-hj-np-tv-z]) ' ?(?=\p{L})|(?<=\b[b-df-hj-np-tv-z]) ?' (?=\p{L})/i
# If you find it overmatches, replace [b-df-hj-np-tv-z] with [dlnsmtc],
# i.e. first letters of word that are usually contracted
second_rx = /\b'\b\K|' *((?:\b'\b|[^'])+)(?<=\S) *'/
puts s.gsub(first_rx, "'")
.gsub(second_rx) { $~[1] ? "'#{$~[1]}'" : "" }
Output:
l'avant ou l'apres ou encore 'maintenant' ou bien 'ceci' et 'encore de l'huile' d'accord d'accord d'accord Je n'en ai pas .... s'entendre Je m'appelle Victor
Explanation
The problem is really complex. There are several words that can be abbreviated and used with an apostrophe in French, de, le/la, ne, se, me, te, ce to name a few, but these are all consonants. You may remove all spaces between a single, standalone consonant, apostrophe and the next word using
s.gsub(/(?<=\b[b-df-hj-np-tv-z]) ' ?(?=\p{L})|(?<=\b[b-df-hj-np-tv-z]) ?' (?=\p{L})/i, "'")
If you find it overmatches, replace [b-df-hj-np-tv-z] with [dlnsmtc], i.e. first letters of word that are usually contracted. See the regex demo.
Next step is to remove spaces after initial and before trailing apostrophes. This is tricky:
s.gsub(/\b'\b\K|' *((?:\b'\b|[^'])+)(?<=\S) *'/) { $~[1] ? "'#{$~[1]}'" : "" }
where \b'\b is meant to match all apsotrophes in between word chars, those that we fixed at the previous step. See this regex demo. As there is no (*SKIP)(*F) support in Onigmo regex, the regex is a bit simplified but the replacement is a conditional one: if Group 1 matched, replace with ' + Group 1 value ($1) + ', else, replace with an empty string (since \K reset the match, dropped all text from the match memory buffer).
NOTE: this approach can be extended to handle some specific cases like aujourd'hui, too.

To remove all whitespace around the ', use gsub!, applied in several steps for proper whitespace removal:
str = "l' avant ou l 'apres ou encore ' maintenant'"
str.gsub!(/\b'\s+\b/, "'").gsub!(/\b\s+'\b/, "'").gsub!(/\b(\s+')\s+\b/, '\1')
puts str
# l'avant ou l'apres ou encore 'maintenant'
Here,
\b : word boundary,
\s+ : 1 or more whitespace,
string.gsub!(regex, replacement_string) : replace in the string argument regex with specified replacement_string (during this, the original string is changed),
\1 : in the replacement string, this refers to the first group captured in parenthesis in the regex: (...).

So if you have alot of data like this, all the answers I have seen are wrong, and will not work. No regex can guess wether the preceding word should have a space or not. Unless you came up with a list of words (or patterns) that either do or don't.
The problem is, sometimes a space should be left, sometimes not. The only way to script that is to find a pattern which describes when the space should be there, or when not. You must teach your regex French grammar. It may be possible lol. But probably not, or difficult.
If this is a one off, my advice is to create regexes for 2 or 3 different situations, and use something like vim, to go through the data, and select manually yes or no to substitute each occurrence.
There may be some cases you can run - eg remove all spaces to the right of quotes? - but unfortunately I don't think you can automate this process.

I believe the following should work for you
s.gsub(/'.*?'/){ |e| "'#{e[1...-1].strip}'" }
The regex portion lazy matches all text within single quotes (including quotes). Then, for each match you substitute for the quoted text with leading and trailing whitespace removed, and return this text in quotes.

Related

Vi: Substitution pattern SQL file. Issue with the Regex

I have to modify a SQL file with vi to delete columns that we do not use. As we have a lot of data, I use the search and replace option with a Regex Pattern.
For instance we have :
(1,2956,2026442,4,NULL,NULL,'ZAC DU BOIS DES COMMUNES','',NULL,NULL,'Rue DU LUXEMBOURG',NULL,
'9999','EVREUX',NULL,1,'27229',NULL,NULL,NULL,NULL,NULL,' Rue DU LUXEMBOURG, 9999 EVREUX',NULL,NULL,NULL,NULL,
NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'2020-07-08 16:34:40',NULL,NULL)
So we have 40 columns and I keep 13 ones. My regex is :
(1),2,(3),4-5,(6-14),15-22,(23),24-39,(40)
:%s/(\(.\{-}\),.\{-},\(.\{-}\),.\{-},.\{-},\(.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-}\),.\{-},
.\{-}, .\{-},.\{-},.\{-},.\{-},.\{-},.\{-},\(.\{-}\),.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},.\{-},
.\{-},.\{-},.\{-},.\{-},.\{-},\(.\{-}\))/(\1,\2,\3,\4,\5)/g
I enclose in my parenthesis the parts that interest me by putting them in parenthesis (I only get the values in parenthesis on the line above my regex ). Then with the replace I recover these groups.
So normally my result is suppose to be :
(1,2026442,NULL,'ZAC DU BOIS DES COMMUNES','',NULL,NULL,'Rue DU LUXEMBOURG',NULL,
'9999','EVREUX',' Rue DU LUXEMBOURG, 9999 EVREUX',NULL)
But Because in ' Rue DU LUXEMBOURG, 9999 EVREUX' there is a comma (,). My result become :
(1,2026442,NULL,'ZAC DU BOIS DES COMMUNES','',NULL,NULL,'Rue DU LUXEMBOURG',NULL,'9999','EVREUX',' Rue DU LUXEMBOURG',NULL,NULL)
Does Someone who is good in Regex can help me ? thanks in advance. If I wasn't clear tell me too, i will try to explain better next time.
I suggest matching fields that can be strings with a %('[^']*'|\w*) pattern, that is, a non-capturing group that finds either ' + zero or more non-'s and then a ' char, or any zero or more alphanumeric characters.
Also, the use of non-capturing groups (in Vim, it is %(...) in very magic mode, or \%(...\) in a regular mode) and very magic mode can help shorten the pattern.
The whole pattern will look like
:%s/\v\(([^,]*),[^,]*,([^,]*),[^,]*,[^,]*,(%('[^']*'|\w*)%(,%('[^']*'|\w*)){8})%(,%('[^']*'|\w*)){8},('[^']*'|\w*)%(,%('[^']*'|\w*)){16},([^,]*)\)/(\1,\2,\3,\4,\5)/g
See the regex demo converted to a PCRE regex.
Note some fields that are not strings are matched with [^,]* that matches zero or more chars other than a comma. The %(,%('[^']*'|\w*)){8} like patterns match (here) 8 occurrences of a sequence of a , char + '...' substring or zero or more word chars.

Split sentence to words containing apostrophe

Supposing I have a group of words as a sentence like this :
Aujourd'hui séparer l'élément en deux
And want the result to be as an individual words (after the split) :
Aujourd'hui | séparer | l' | élément | en | deux
Note : as you can see, « aujourd'hui » is a single word.
What would be the best regex to use here ?
With my current knowledge, all i can achieve is this basic operation :
QString sentence("Aujourd'hui séparer l'élément en deux");
QStringList list = sentence.split(" ");
Output :
Aujourd'hui / Séparer / l'élément / en / deux
Here are the two questions closest to mine : this and this.
Since the contractions that you want to consider as separate words are usually a single letter + an apostrophe in French (like l'huile, n'en, d'accord) you can use a pattern that either matches 1+ whitespace chars, or a location that is immediately preceded with a start of a word, then 1 letter and then an apostrophe.
I also suggest taking into account curly apostrophes. So, use
\s+|(?<=\b\p{L}['’])\b
See the regex demo.
Details
\s+ - 1+ whitespaces
| - or
(?<=\b\p{L}['’])\b - a word boundary (\b) location that is preceded with a start of word (\b), a letter (\p{L}) and a ' or ’.
In Qt, you may use
QStringList result = text.split(
QRegularExpression(R"(\s+|(?<=\b\p{L}['’])\b)",
QRegularExpression::PatternOption::UseUnicodePropertiesOption)
);
The R"(...)" is a raw string literal notation, you may use "\\s+|(?<=\\b\\p{L}['’])\\b" if you are using a C++ environment that does not allow raw string literals.
Not sure if I understood what you are saying but this might help you
QString sentence("Aujourd'hui séparer l'élément en deux");
QStringList list = sentence.split(" '");
I don't know C++ but I guees it supports negative lookbehind.
Have a try with:
(?: |(?<!\w{2})')
This will split on space or apostroph if there are not 2 letters before.
Demo & explanation
Well, you're dealing with a natural language, here, and the first - and toughest - problem to answer is: Can you actually come up with a fixed rule, when splits should happen? In this particular case, there is really no logical reason, why French considers "aujourd'hui" as a single word (when logically, it could be parsed as "au jour de hui").
I'm not familiar with all the possible pitfalls in French, but if you really want to make sure to cover all obscure cases, you'll have to look for a natural language tokenizer.
Anyway, for the example you give, it may be good enough to use a QRegularExpression with negative lookbehind to omit splits when more than one letter precedes the apostrophe:
sentence.split(QRegularExpression("(?<![\\w][\\w])'"));

python: Removing all kinds of quotation marks

I have the following string:
txt="Daniel's car é à muito esperto"
I am trying to remove all kinds of quotation marks.
I tried:
txt=re.sub(r"\u0022\u201C\u201D\u0027\u2019\u2018\u2019\u0060\u00B4\'\"", ' ', txt)
I expected:
"Daniel s car é à muito esperto"
but actually nothing is happening.
The reason that the regex does not work is that it matches only a single string
r"\u0022\u201C\u201D\u0027\u2019\u2018\u2019\u0060\u00B4\'\""
To fix that one could use either alteration between each character or a character set.
txt=re.sub(r"[\u0022\u201C\u201D\u0027\u2019\u2018\u2019\u0060\u00B4\'\"]", ' ', txt)
One might need to pass the re.UNICODE flag. Untested.

BigQuery REGEXP_MATCH and accents : boundary wildcard fails?

In GAS I can correctly match accents with regular expression having boundary characters, such as \bà\b. The character à is matched only when it is a separate word. This works in GAS:
function test_regExp() {
var str = "la séance est à Paris";
var RegExp = "\\bà\\b";
var PatReg= new RegExp( RegExp);
var found=PatReg.exec(str);
if (found) {
Logger.log( [str.substring(0,found.index),found[0],str.substring(found[0].length+found.index)] );
} else Logger.log("oops! Did not match");
In BigQuery, if boundary characters are next to accents the patterns do not match. \bséance\b matches séance:
SELECT [row],etext,ftext FROM [hcd.hdctextx] WHERE (REGEXP_MATCH(ftext,"\\bséance\\b") ) LIMIT 100;
\bà\b does not match à as a word:
SELECT [row],etext,ftext FROM [hcd.hdctextx] WHERE (REGEXP_MATCH(ftext,"\\bà\\b") ) LIMIT 100;
I'm assuming that BigQuery, unlike GAS, is including accents in the boundary character set. So \bséance\b works because é can function properly as a boundary in that configuration. \bà\b or \bétranger\b or \bmarché\b do not work because accent + \b is interpreted as \b\b, which never matches anything. (Ok, I'm grasping at straws here, because I can't find a better explanation....besides a bug.)
I don't think it is a unicode problem, because it only crops up at boundary positions.
For the moment therefore, no way to use boundary in those particular configurations of accents.
Is there a way to set the Locale in BigQuery or other fix?
Workaround: substitute (?:[^a-zA-Zéàïëâê]) and so on for \b.
Thanks!
BigQuery's behavior is correct with respect to the RE2 syntax documentation. (No surprise, because BigQuery uses RE2 to implement regexps.)
RE2's character classes are:
\b = at word boundary (\w on one side and \W, \A, or \z on the other)
\w = word characters (≡ [0-9A-Za-z_])
\W = not word characters (≡ [^0-9A-Za-z_])
\A = beginning of text
\z = end of text
In other words, you can only use \b to match boundaries of non-accented characters. RE2 has plenty of support for Unicode characters, though, so you can most likely craft an alternative regexp using something like \pL.
I'm not sure why Google Apps Script doesn't follow the RE2 spec here, but I'll follow up with that team to figure out what's going on.
Check this out:
SElect Regexp_extract(StringToParse,r'\b?(à)\b?') as Extract,
Regexp_match(StringToParse,r'\b?(à)\b?') as match,
FROM
(SELECT 'la séance est à Paris' as StringToParse)
Hope this helps
The answer is: in BQ don't use \b with accents; rewrite the regular expresssion:
frenRegExp = frenRegExp.replace(/\\b/g, "(?:[- .,;!?()]|$|^)");
frenRegExp = frenRegExp.replace(/\\w/g, "(?:[A-Za-zÀàÂâÄäÆæÇçÈèÉéÊêËëÎîÏïÔôÙùÛûÜüñ])");
frenRegExp = frenRegExp.replace(/\\W/g, "(?:[^A-Za-zÀàÂâÄäÆæÇçÈèÉéÊêËëÎîÏïÔôÙùÛûÜüñ])");
Also, though the GAS specification has RE2 as its re engine (oops! I really don't know what it uses, since it does not exclude accented characters from \w like BQ), it is only partially implemented. For example \pL does not match a letter.
Here is some test code that works in apps scripts, but not in BQ without a substitution.
////////////////////// TEST ///////////////////
function test_regExp() {
var str = " Voilà la séance générale qui est à Paris";
var RegExpString ="\\bs\\w+an\\w*"
Logger.log(RegExpString);
var RegExpCompiled= new RegExp( RegExpString,"i");
Logger.log(RegExpCompiled.source);
var found=RegExpCompiled.exec(str);
if (found) {
Logger.log("|"+found[0]+"|")
Logger.log( [str.substring(0,found.index),found[0],str.substring(found[0].length+found.index)] );
} else Logger.log("Oops: not found");
}
Output:
[16-02-09 22:15:59:659 PST] \bs\w+anc\w*
[16-02-09 22:15:59:660 PST] \bs\w+an\w*
[16-02-09 22:15:59:660 PST] |séance|
[16-02-09 22:15:59:661 PST] [ Voilà la , séance, générale qui est à Paris]

Regex for quoted string with escaping quotes

How do I get the substring " It's big \"problem " using a regular expression?
s = ' function(){ return " It\'s big \"problem "; }';
/"(?:[^"\\]|\\.)*"/
Works in The Regex Coach and PCRE Workbench.
Example of test in JavaScript:
var s = ' function(){ return " Is big \\"problem\\", \\no? "; }';
var m = s.match(/"(?:[^"\\]|\\.)*"/);
if (m != null)
alert(m);
This one comes from nanorc.sample available in many linux distros. It is used for syntax highlighting of C style strings
\"(\\.|[^\"])*\"
As provided by ePharaoh, the answer is
/"([^"\\]*(\\.[^"\\]*)*)"/
To have the above apply to either single quoted or double quoted strings, use
/"([^"\\]*(\\.[^"\\]*)*)"|\'([^\'\\]*(\\.[^\'\\]*)*)\'/
Most of the solutions provided here use alternative repetition paths i.e. (A|B)*.
You may encounter stack overflows on large inputs since some pattern compiler implements this using recursion.
Java for instance: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6337993
Something like this:
"(?:[^"\\]*(?:\\.)?)*", or the one provided by Guy Bedford will reduce the amount of parsing steps avoiding most stack overflows.
/(["\']).*?(?<!\\)(\\\\)*\1/is
should work with any quoted string
"(?:\\"|.)*?"
Alternating the \" and the . passes over escaped quotes while the lazy quantifier *? ensures that you don't go past the end of the quoted string. Works with .NET Framework RE classes
/"(?:[^"\\]++|\\.)*+"/
Taken straight from man perlre on a Linux system with Perl 5.22.0 installed.
As an optimization, this regex uses the 'posessive' form of both + and * to prevent backtracking, for it is known beforehand that a string without a closing quote wouldn't match in any case.
This one works perfect on PCRE and does not fall with StackOverflow.
"(.*?[^\\])??((\\\\)+)?+"
Explanation:
Every quoted string starts with Char: " ;
It may contain any number of any characters: .*? {Lazy match}; ending with non escape character [^\\];
Statement (2) is Lazy(!) optional because string can be empty(""). So: (.*?[^\\])??
Finally, every quoted string ends with Char("), but it can be preceded with even number of escape sign pairs (\\\\)+; and it is Greedy(!) optional: ((\\\\)+)?+ {Greedy matching}, bacause string can be empty or without ending pairs!
An option that has not been touched on before is:
Reverse the string.
Perform the matching on the reversed string.
Re-reverse the matched strings.
This has the added bonus of being able to correctly match escaped open tags.
Lets say you had the following string; String \"this "should" NOT match\" and "this \"should\" match"
Here, \"this "should" NOT match\" should not be matched and "should" should be.
On top of that this \"should\" match should be matched and \"should\" should not.
First an example.
// The input string.
const myString = 'String \\"this "should" NOT match\\" and "this \\"should\\" match"';
// The RegExp.
const regExp = new RegExp(
// Match close
'([\'"])(?!(?:[\\\\]{2})*[\\\\](?![\\\\]))' +
'((?:' +
// Match escaped close quote
'(?:\\1(?=(?:[\\\\]{2})*[\\\\](?![\\\\])))|' +
// Match everything thats not the close quote
'(?:(?!\\1).)' +
'){0,})' +
// Match open
'(\\1)(?!(?:[\\\\]{2})*[\\\\](?![\\\\]))',
'g'
);
// Reverse the matched strings.
matches = myString
// Reverse the string.
.split('').reverse().join('')
// '"hctam "\dluohs"\ siht" dna "\hctam TON "dluohs" siht"\ gnirtS'
// Match the quoted
.match(regExp)
// ['"hctam "\dluohs"\ siht"', '"dluohs"']
// Reverse the matches
.map(x => x.split('').reverse().join(''))
// ['"this \"should\" match"', '"should"']
// Re order the matches
.reverse();
// ['"should"', '"this \"should\" match"']
Okay, now to explain the RegExp.
This is the regexp can be easily broken into three pieces. As follows:
# Part 1
(['"]) # Match a closing quotation mark " or '
(?! # As long as it's not followed by
(?:[\\]{2})* # A pair of escape characters
[\\] # and a single escape
(?![\\]) # As long as that's not followed by an escape
)
# Part 2
((?: # Match inside the quotes
(?: # Match option 1:
\1 # Match the closing quote
(?= # As long as it's followed by
(?:\\\\)* # A pair of escape characters
\\ #
(?![\\]) # As long as that's not followed by an escape
) # and a single escape
)| # OR
(?: # Match option 2:
(?!\1). # Any character that isn't the closing quote
)
)*) # Match the group 0 or more times
# Part 3
(\1) # Match an open quotation mark that is the same as the closing one
(?! # As long as it's not followed by
(?:[\\]{2})* # A pair of escape characters
[\\] # and a single escape
(?![\\]) # As long as that's not followed by an escape
)
This is probably a lot clearer in image form: generated using Jex's Regulex
Image on github (JavaScript Regular Expression Visualizer.)
Sorry, I don't have a high enough reputation to include images, so, it's just a link for now.
Here is a gist of an example function using this concept that's a little more advanced: https://gist.github.com/scagood/bd99371c072d49a4fee29d193252f5fc#file-matchquotes-js
here is one that work with both " and ' and you easily add others at the start.
("|')(?:\\\1|[^\1])*?\1
it uses the backreference (\1) match exactley what is in the first group (" or ').
http://www.regular-expressions.info/backref.html
One has to remember that regexps aren't a silver bullet for everything string-y. Some stuff are simpler to do with a cursor and linear, manual, seeking. A CFL would do the trick pretty trivially, but there aren't many CFL implementations (afaik).
A more extensive version of https://stackoverflow.com/a/10786066/1794894
/"([^"\\]{50,}(\\.[^"\\]*)*)"|\'[^\'\\]{50,}(\\.[^\'\\]*)*\'|“[^”\\]{50,}(\\.[^“\\]*)*”/
This version also contains
Minimum quote length of 50
Extra type of quotes (open “ and close ”)
If it is searched from the beginning, maybe this can work?
\"((\\\")|[^\\])*\"
I faced a similar problem trying to remove quoted strings that may interfere with parsing of some files.
I ended up with a two-step solution that beats any convoluted regex you can come up with:
line = line.replace("\\\"","\'"); // Replace escaped quotes with something easier to handle
line = line.replaceAll("\"([^\"]*)\"","\"x\""); // Simple is beautiful
Easier to read and probably more efficient.
If your IDE is IntelliJ Idea, you can forget all these headaches and store your regex into a String variable and as you copy-paste it inside the double-quote it will automatically change to a regex acceptable format.
example in Java:
String s = "\"en_usa\":[^\\,\\}]+";
now you can use this variable in your regexp or anywhere.
(?<="|')(?:[^"\\]|\\.)*(?="|')
" It\'s big \"problem "
match result:
It\'s big \"problem
("|')(?:[^"\\]|\\.)*("|')
" It\'s big \"problem "
match result:
" It\'s big \"problem "
Messed around at regexpal and ended up with this regex: (Don't ask me how it works, I barely understand even tho I wrote it lol)
"(([^"\\]?(\\\\)?)|(\\")+)+"