Extracting quoted and unquoted values using regex - c++

I'm trying to to parse a string of type <tag>=<value> using regular expressions but have hit some issues adding support for quoted values. The idea is that any unquoted values should be trimmed of leading / trailing white space so that [ Hello ] becomes [Hello] (Pls ignore the square brackets.)
However, when the value is quoted, I want anything up to and including the double quotes to be removed but no further, so [ " Hello World " ] would become [" Hello World "]
So far, I've come up with the following code with a pattern match for this (note that some of the character have been escaped or doubly escaped to avoid them being interpreted as tri-graphs or other C format characters.)
void getTagVal( const std::string& tagVal )
{
boost::smatch what;
static const boost::regex pp("^\\s*([a-zA-Z0-9_-]+)\\s*=\\s*\"\?\?([%:\\a-zA-Z0-9 /\\._]+?)\"\?\?\\s*$");
if ( boost::regex_match( tagVal, what, pp ) )
{
const string tag = static_cast<const string&>( what[1] );
const string val = static_cast<const string&>( what[2] );
cout << "Tag = [" << tag << "] Val = [" << val << "]" << endl;
}
}
int main( int argc, char* argv[] )
{
getTagVal("Qs1= \" Hello World \" ");
getTagVal("Qs2=\" Hello World \" ");
getTagVal("Qs3= \" Hello World \"");
getTagVal("Qs4=\" Hello World \"");
getTagVal("Qs5=\"Hello World \"");
getTagVal("Qs6=\" Hello World\"");
getTagVal("Qs7=\"Hello World\"");
return 0;
}
Taking out the double escaping, this breaks down as:
^ - Start of line.
\s* - an optional amount of whitespace.
([a-zA-Z0-9_-]+) - One or more alphanumerics or a dash or underscore. This is captured as the tag.
\s* - an optional amount of whitespace.
= - an "equal" symbol.
\s* - an optional amount of whitespace.
"?? - an optional double quote (non-greedy).
([%:\a-zA-Z0-9 /\._]+?) - One or more alphanumerics or a space, underscore, percent, colon, period, forward or back slash. This is captured as the value (non-greedy).
"?? - an optional double quote (non-greedy).
\s* - an optional amount of whitespace.
$ - End of line
For the example calls in main(), I would expect to get:
Tag = [Qs1] Val = [ Hello World ]
Tag = [Qs2] Val = [ Hello World ]
Tag = [Qs3] Val = [ Hello World ]
Tag = [Qs4] Val = [ Hello World ]
Tag = [Qs5] Val = [Hello World ]
Tag = [Qs6] Val = [ Hello World]
Tag = [Qs7] Val = [Hello World]
but what I actually get is:
Tag = [Qs1] Val = [" Hello World ]
Tag = [Qs2] Val = [" Hello World ]
Tag = [Qs3] Val = [" Hello World ]
Tag = [Qs4] Val = [" Hello World ]
Tag = [Qs5] Val = ["Hello World ]
Tag = [Qs6] Val = [" Hello World]
Tag = [Qs7] Val = ["Hello World]
So it's almost correct but for some reason the first quote is hanging around in the output value even though I specifically bracket the value section of the regex with the quote outside it.

I would change the part starting with the first quote to an alternative:
"([^"]+)"|([%:\a-zA-Z0-9 /\._]+)\s*
You would then have to handle the two possibilities of quoted or unquoted text ending up in the second or third capturing parenthesis pair in the host code around the regex.

Figured out what the problem was.
When using \ you have to be careful as this is processed within the C string and so needs to be escaped there, but it will also be processed by the regex engine so if you're not careful \\a becomes \a which is absolutely not what you wanted.
So, to tell it that I want a \ to be in my set of characters in the value (which I do as ironically, they're being used as escape sequences within a format string) then you have to double escape them so
static const boost::regex pp("^\\s*([a-zA-Z0-9_-]+)\\s*=\\s*\"\?\?([%:\\a-zA-Z0-9 /\\._]+?)\"\?\?\\s*$");
becomes:
static const boost::regex pp("^\\s*([a-zA-Z0-9_-]+)\\s*=\\s*\"\?\?([%:\\\\a-zA-Z0-9 /._]+?)\"\?\?\\s*$");
(i.e. you need to make it \\\\)

Related

How to match escaped group signs {&date:dd.\{mm\}.yyyy} but not {&date:dd.{mm}.yyyy} with vba and regex

I'm trying to create a pattern for finding placeholders within a string to be able to replace them with variables later. I'm stuck on a problem to find all these placeholders within a string according to my requirement.
I already found this post, but it only helped a little:
Regex match ; but not \;
Placeholders will look like this
{&var} --> Variable stored in a dictionary --> dict("var")
{$prop} --> Property of a class cls.prop read by CallByName and PropGet
{#const} --> Some constant values by name from a function
Generally I have this pattern and it works well
Dim RegEx As Object
Set RegEx = CreateObject("VBScript.RegExp")
RegEx.pattern = "\{([#\$&])([\w\.]+)\}"
For example I have this string:
"Value of foo is '{&var}' and bar is '{$prop}'"
I get 2 matches as expected
(&)(var)
($)(prop)
I also want to add a formating part like in .Net to this expression.
String.Format("This is a date: {0:dd.mm.yyyy}", DateTime.Now());
// This is a date: 05.07.2019
String.Format("This is a date, too: {0:dd.(mm).yyyy}", DateTime.Now());
// This is a date, too: 05.(07).2019
I extended the RegEx to get that optional formatting string
Dim RegEx As Object
Set RegEx = CreateObject("VBScript.RegExp")
RegEx.pattern = "\{([#\$&])([\w\.]+):{0,1}([^\}]*)\}"
RegEx.Execute("Value of foo is '{&var:DD.MM.YYYY}' and bar is '{$prop}'")
I get 2 matches as expected
(&)(var)(DD.MM.YYYY)
($)(prop)()
At this point I noticed I have to take care for escapet "{" and "}", because maybe I want to have some brackets within the formattet result.
This does not work properly, because my pattern stops after "...{MM"
RegEx.Execute("Value of foo is '{&var:DD.{MM}.YYYY}' and bar is '{$prop}'")
It would be okay to add escape signs to the text before checking the regex:
RegEx.Execute("Value of foo is '{&var:DD.\{MM\}.YYYY}' and bar is '{$prop}'")
But how can I correctly add the negative lookbehind?
And second: How does this also works for variables, that should not be resolved, even if they have the correct syntax bus the outer bracket is escaped?
RegEx.Execute("This should not match '\{&var:DD.\{MM\}.YYYY\}' but this one '{&var:DD.\{MM\}.YYYY}'")
I hope my question is not confusing and someone can help me
Update 05.07.19 at 12:50
After the great help of #wiktor-stribiżew the result is completed.
As requested i provide some example code:
Sub testRegEx()
Debug.Print FillVariablesInText(Nothing, "Date\\\\{$var01:DD.\{MM\}.YYYY}\\\\ Var:\{$nomatch\}{$var02} Double: {#const}{$var01} rest of string")
End Sub
Function FillVariablesInText(ByRef dict As Dictionary, ByVal txt As String) As String
Const c_varPattern As String = "(?:(?:^|[^\\\n])(?:\\{2})*)\{([#&\$])([\w.]+)(?:\:([^}\\]*(?:\\.[^\}\\]*)*))?(?=\})"
Dim part As String
Dim snippets As New Collection
Dim allMatches, m
Dim i As Long, j As Long, x As Long, n As Long
' Create a RegEx object and execute pattern
Dim RegEx As Object
Set RegEx = CreateObject("VBScript.RegExp")
RegEx.pattern = c_varPattern
RegEx.MultiLine = True
RegEx.Global = True
Set allMatches = RegEx.Execute(txt)
' Start at position 1 of txt
j = 1
n = 0
For Each m In allMatches
n = n + 1
Debug.Print "(" & n & "):" & m.value
Debug.Print " [0] = " & m.SubMatches(0) ' Type [&$#]
Debug.Print " [1] = " & m.SubMatches(1) ' Name
Debug.Print " [2] = " & m.SubMatches(2) ' Format
part = "{" & m.SubMatches(0)
' Get offset for pre-match-string
x = 1 ' Index to Postion at least +1
Do While Mid(m.value, x, 2) <> part
x = x + 1
Loop
' Postition in txt
i = m.FirstIndex + x
' Anything to add to result?
If i <> j Then
snippets.Add Mid(txt, j, i - j)
End If
' Next start postition (not Index!) + 1 for lookahead-positive "}"
j = m.FirstIndex + m.Length + 2
' Here comes a function get a actual value
' e.g.: snippets.Add dict(m.SubMatches(1))
' or : snippets.Add Format(dict(m.SubMatches(1)), m.SubMatches(2))
snippets.Add "<<" & m.SubMatches(0) & m.SubMatches(1) & ">>"
Next m
' Any text at the end?
If j < Len(txt) Then
snippets.Add Mid(txt, j)
End If
' Join snippets
For i = 1 To snippets.Count
FillVariablesInText = FillVariablesInText & snippets(i)
Next
End Function
The function testRegEx gives me this result and debug print:
(1):e\\\\{$var01:DD.\{MM\}.YYYY(2):}{$var02
[0] = $
[1] = var02
[2] =
(1):e\\\\{$var01:DD.\{MM\}.YYYY
[0] = $
[1] = var01
[2] = DD.\{MM\}.YYYY
(2):}{$var02
[0] = $
[1] = var02
[2] =
(3): {#const
[0] = #
[1] = const
[2] =
(4):}{$var01
[0] = $
[1] = var01
[2] =
Date\\\\<<$var01>>\\\\ Var:\{$nomatch\}<<$var02>> Double: <<#const>><<$var01>> rest of string
You may use
((?:^|[^\\])(?:\\{2})*)\{([#$&])([\w.]+)(?::([^}\\]*(?:\\.[^}\\]*)*))?}
To make sure the consecutive matches are found, too, turn the last } into a lookahead, and when extracting matches just append it to the result, or if you need the indices increment the match length by 1:
((?:^|[^\\])(?:\\{2})*)\{([#$&])([\w.]+)(?::([^}\\]*(?:\\.[^}\\]*)*))?(?=})
^^^^^
See the regex demo and regex demo #2.
Details
((?:^|[^\\])(?:\\{2})*) - Group 1 (makes sure the { that comes next is not escaped): start of string or any char but \ followed with 0 or more double backslashes
\{ - a { char
([#$&]) - Group 2: any of the three chars
([\w.]+) - Group 3: 1 or more word or dot chars
(?::([^}\\]*(?:\\.[^}\\]*)*))? - an optional sequence of : and then Group 4:
[^}\\]* - 0 or more chars other than } and \
(?:\\.[^}\\]*)* - zero or more reptitions of a \-escaped char and then 0 or more chars other than } and \
} - a } char
Welcome to the site! If you need to only match balanced escapes, you will need something more powerful. If not --- I haven't tested this, but you could try replacing [^\}]* with [^\{\}]|\\\{|\\\}. That is, match non-braces and escaped brace sequences separately. You may need to change this depending on how you want to handle backslashes in your formatting string.

Split with a multicharacter regex pattern and keep delimiters

I have next string and regex for splitting it:
val str = "this is #[loc] sparta"
val regex = "((?<=( #\\[\\w{3,100}\\] ))|(?=( #\\[\\w{3,100}\\] )))"
print(str.split(Regex(regex)))
//print - [this is, #[loc] , sparta]
Works fine. But in develop I did not realize when in #[***] block must be a not only text (\w) - he have and "-" and numbers (UUID), and my correct blocks is -
val str = "this is #[loc_75acca83-a39b-4df1-8c3c-b690df00db62]"
and in this case regex don't work.
How to change this part - "\w{3,100}" for new requirements?
I try change to any - "\.{3,100}" - not work
To fix your issue, you may replace your regex with
val regex = """((?<=( #\[[^\]\[]{3,100}] ))|(?=( #\[[^\]\[]{3,100}] )))"""
The \w can be replaced with [^\]\[] that matches any char but [ and ].
Note the use of a raw string literal, """...""", that allows the use of a single backslash as a regex escape.
See the Kotlin online demo.
Alternatively, you may use the following method to split and keep delimiters:
private fun splitKeepDelims(s: String, rx: Regex, keep_empty: Boolean = true) : MutableList<String> {
var res = mutableListOf<String>() // Declare the mutable list var
var start = 0 // Define var for substring start pos
rx.findAll(s).forEach { // Looking for matches
val substr_before = s.substring(start, it.range.first()) // // Substring before match start
if (substr_before.length > 0 || keep_empty) {
res.add(substr_before) // Adding substring before match start
}
res.add(it.value) // Adding match
start = it.range.last()+1 // Updating start pos of next substring before match
}
if ( start != s.length ) res.add(s.substring(start)) // Adding text after last match if any
return res
}
Then, just use it like
val str = "this is #[loc_75acca83-a39b-4df1-8c3c-b690df00db62] sparta"
val regex = """#\[[\]\[]+]""".toRegex()
print(splitKeepDelims(str, regex))
// => [this is , #[loc_75acca83-a39b-4df1-8c3c-b690df00db62], sparta]
See the Kotlin demo.
The \[[^\]\[]+] pattern matches
\[ - a [ char
[^\]\[]+ - 1+ chars other than [ and ]
] - a ] char.

Exact matching with Question mark in Perl

I want to find string ?Allen in the string array but there is question mark in keyword and it causes some problems.
I write this code to find string in array
#arr = ("My name is ?Allen",
"My name is ?Allens",
"My name is s?Allen",
"My name is s?Allens",
"My name is ?allen");
$keyword = "?Allen";
for (my $i=0; $i <= 4; $i++){
if ($arr[$i] =~ /\b$keyword\b/){
print "str $i = match\n";
}else{
print "str $i = no\n";
}
}
finally I get this result
str 0 = match
str 1 = no
str 2 = match
str 3 = no
str 4 = no
but I want to find only first index array as matching string like this:
str 0 = match
str 1 = no
str 2 = no
str 3 = no
str 4 = no
Note that your regex contains non-word special chars that you need to quote before using them in the actual pattern. Also, the fact that the special chars can appear at the leading/trailing positions means you cannot expect \b to always work the same (since its meaning is context dependent). Thus, you may fix the code with
/(?<!\S)\Q$keyword\E(?!\S)/
where
(?<!\S) - requires a whitespace char or start of string before
\Q$keyword\E - a literal search string (see Quoting Metacharacters)
(?!\S) - that should be followed with a whitespace or end of string.
Another alternative for \Q...\E (mentioned by Dave Cross) is using quotemeta:
This is the internal function implementing the \Q escape in double-quoted strings.

Regex to match tokens in a string using string.gmatch

I need a regex to use in string.gmatch that matches sequences of alphanumeric characters and non alphanumeric characters (quotes, brackets, colons and the like) as separated, single, matches, so basically:
str = [[
function test(arg1, arg2) {
dosomething(0x12f, "String");
}
]]
for token in str:gmatch(regex) do
print(token)
end
Should print:
function
test
(
arg1
,
arg2
)
{
dosomething
(
0x121f
,
"
String
"
)
;
}
How can I achieve this? In standard regex I've found that ([a-zA-Z0-9]+)|([\{\}\(\)\";,]) works for me but I'm not sure on how to translate this to Lua's regex.
local str = [[
function test(arg1, arg2) {
dosomething(0x12f, "String");
}
]]
for p, w in str:gmatch"(%p?)(%w*)" do
if p ~= "" then print(p) end
if w ~= "" then print(w) end
end
You need a workaround involving a temporary char that is not used in your code. E.g., use a § to insert it after the alphanumeric and non-alphanumeric characters:
str = str:gsub("%s*(%w+)%s*", "%1§") -- Trim chunks of 1+ alphanumeric characters and add a temp char after them
str = str:gsub("(%W)%s*", "%1§") -- Right trim the non-alphanumeric char one by one and add the temp char after each
for token in str:gmatch("[^§]+") do -- Match chunks of chars other than the temp char
print(token)
end
See this Lua demo
Note that %w in Lua is an equivalent of JS [a-zA-Z0-9], as it does not match an underscore, _.

Regex that matches specific spaces

I've been trying to do this Regex for a while now. I'd like to create one that matches all the spaces of a text, except those in literal string.
Exemple:
123 Foo "String with spaces"
Space between 123 and Foo would match, as well as the one between Foo and "String with spaces", but only those two.
Thanks
A common, simple strategy for this is to count the number of quotes leading up to your location in the string. If the count is odd, you are inside a quoted string; if the amount is even, you are outside a quoted string. I can't think of a way to do this in regular expressions, but you could use this strategy to filter the results.
You could use re.findall to match either a string or a space and then afterwards inspect the matches:
import re
hits = re.findall("\"(?:\\\\.|[^\\\"])*\"|[ ]", 'foo bar baz "another\\" test\" and done')
for h in hits:
print "found: [%s]" % h
yields:
found: [ ]
found: [ ]
found: [ ]
found: ["another\" test"]
found: [ ]
found: [ ]
A short explanation:
" # match a double quote
(?: # start non-capture group 1
\\\\. # match a backslash followed by any character (except line breaks)
| # OR
[^\\\"] # match any character except a '\' and '"'
)* # end non-capture group 1 and repeat it zero or more times
" # match a double quote
| # OR
[ ] # match a single space
If this ->123 Foo "String with spaces" <- is your structure for a line that is to say text followed by a quoted text you could create 2 groups the quoted and the unquoted text and an tackle them separately.
ex.regex -> (.*)(".*") where $1 should contain ->123 Foo <- and $2 ->"String with spaces"<-
java example.
String aux = "123 Foo \"String with spaces\"";
String regex = "(.*)(\".*\")";
String unquoted = aux.replaceAll(regex, "$1").replace(" ", "");
String quoted = aux.replaceAll(regex, "$2");
System.out.println(unquoted+quoted);
javascript example.
<SCRIPT LANGUAGE="JavaScript">
<!--
str='1 23 Foo \"String with spaces\"';
re = new RegExp('(.*)(".*")') ;
var quoted = str.replace(re, "$1");
var unquoted = str.replace(re, "$2");
document.write (quoted.split(' ').join('')+unquoted);
// -->
</SCRIPT>