I'm trying to write a c++ regex to essentially match a few symbols and identifiers as part of a tokenizer. Currently, I have this:
EDITED
regex tokens("([a-zA-Z_][a-zA-Z0-9_]*)|(\\S?)|(\\S)")
vector<string> identifiers(std::sregex_token_iterator(str.begin(), str.end(),
IDENTIFIER),std::sregex_token_iterator());
https://regex101.com/r/mFTC1Y/2
The problem is, it hangs my program (just takes forever and I never get to the matches). I don't understand how that can be? The regex tester I'm using says it takes a bout 7ms to match...
Please help!
JUST EDITED: so this regex matches what I want, but only via group captures. If it parses:
main()
It will return
main( // full match
main // group 1
( // group 2
new match
) // full match
) // group 3
I just want the group matches without having to explicitly check the respective groups (i.e. I just don't return the full match to me). How can I update my code to do that?
EDIT
So, this is the full, working code. I'd prefer it be more elegant.
regex TOKENS("([a-zA-Z_][a-zA-Z0-9_]*)|(\\S?)|(\\S)")
auto identifier = sregex_iterator(str.cbegin(), str.cend(), TOKENS);
auto it = sregex_iterator();
for_each(identifier, it, [&](smatch const& m){
string group1(m[1].str());
string group2(m[2].str());
string group3(m[3].str());
if(isKeyword(keywords, group1)) cout << "<keyword> " << group1 << " </keyword>" << endl;
else if(group1 != "") cout << "<identifier> " << group1 << " </identifier>" << endl;
if (isSymbol(symbols, group2)) cout << "<symbol> " << group2 << " </symbol>" << endl;
if (isSymbol(symbols, group3)) cout << "<symbol> " << group3 << " </symbol>" << endl;
});
Something more elegant would probably come at the cost of a very complex regex, or else a very clever one, since essentially what I'm trying to do is tokenize code into one of three types: KEYWORD, ID and SYMBOL - all with one regex. Next I'll have to tackle INT/STRING const and comments. What I'm trying to avoid is tokenizing char by char, because then I'll have even more control-flow statements (which I don't want).
I am not sure, if your regex is correct.
Try the below:
#include <iostream>
#include <string>
#include <algorithm>
#include <vector>
#include <regex>
// Our test data (raw string). So, containing also \n and so on
std::string testData(
R"#( :-) IDcorrect1 _wrongID I2DCorrect
3FALSE lowercasecorrect Underscore_not_allowed
i3DCorrect,i4 :-)
}
)#");
std::regex re("(\\b[a-zA-Z][a-zA-Z0-9]*\\b)");
int main(void)
{
// Define the variable id as vector of string and use the range constructor to read the test data and tokenize it
std::vector<std::string> id{ std::sregex_token_iterator(testData.begin(), testData.end(), re, 1), std::sregex_token_iterator() };
// For debug output. Print complete vector to std::cout
std::copy(id.begin(), id.end(), std::ostream_iterator<std::string>(std::cout, "\n"));
return 0;
}
All IDs will be in the vector. Then you can further check.
Related
I'm looking for a regex pattern that returns true if found 7 numbers on given string. There's no order so if a string is set to: "100 my, str1ng y000" it catches that.
RegEx alone won't count exact occurrences for you, it would return true even if there are more than 7 digits in the string because it would try to find out at least 7 digits in the string.
You can use below code to test exact number (7 in your case) of digits in any string:
var temp = "100 my, str1ng y000 3c43fdgd";
var count = (temp.match(/\d/g) || []).length;
alert(count == 7);
I will show you an C++ Example that
Shows a regex for extracting digit groups
Shows a regex for matching at least 7 digits
Shows, if there is a match for the requested predicate
Shows the number of digits in the string (no regex needed)
Shows the group of digits
#include <iostream>
#include <string>
#include <algorithm>
#include <vector>
#include <regex>
// Our test data (raw string). So, containing also \" and so on
std::string testData("100 my, str1ng y000");
std::regex re1(R"#((\d+))#"); // For extracting digit groups
std::regex re2(R"#((\d.*){7,})#"); // For regex match
int main(void)
{
// Define the variable id as vector of string and use the range constructor to read the test data and tokenize it
std::vector<std::string> id{ std::sregex_token_iterator(testData.begin(), testData.end(), re1, 1), std::sregex_token_iterator() };
// Match the regex. Should have at least 7 digits somewhere
std::smatch base_match;
bool containsAtLeast7Digits = std::regex_match(testData, base_match, re2);
// Show result on screen
std::cout << "\nEvaluating string '" << testData <<
"'\n\nThe predicate 'contains-at-leats-7-digits' is " << std::boolalpha << containsAtLeast7Digits <<
"\n\nIt contains overall " <<
std::count_if(
testData.begin(),
testData.end(),
[](const char c) {
return std::isdigit(static_cast<int>(c));
}
) << " digits and " << id.size() << " digit groups. These are:\n\n";
// Print complete vector to std::cout
std::copy(id.begin(), id.end(), std::ostream_iterator<std::string>(std::cout, "\n"));
return 0;
}
Please note: Use std::count for counting. Faster and easier.
Hope this helps . . .
I'm a bit confused about the following C++11 code:
#include <iostream>
#include <string>
#include <regex>
int main()
{
std::string haystack("abcdefabcghiabc");
std::regex needle("abc");
std::smatch matches;
std::regex_search(haystack, matches, needle);
std::cout << matches.size() << std::endl;
}
I'd expect it to print out 3 but instead I get 1. Am I missing something?
You get 1 because regex_search returns only 1 match, and size() will return the number of capture groups + the whole match value.
Your matches is...:
Object of a match_results type (such as cmatch or smatch) that is filled by this function with information about the match results and any submatches found.
If [the regex search is] successful, it is not empty and contains a series of sub_match objects: the first sub_match element corresponds to the entire match, and, if the regex expression contained sub-expressions to be matched (i.e., parentheses-delimited groups), their corresponding sub-matches are stored as successive sub_match elements in the match_results object.
Here is a code that will find multiple matches:
#include <string>
#include <iostream>
#include <regex>
using namespace std;
int main() {
string str("abcdefabcghiabc");
int i = 0;
regex rgx1("abc");
smatch smtch;
while (regex_search(str, smtch, rgx1)) {
std::cout << i << ": " << smtch[0] << std::endl;
i += 1;
str = smtch.suffix().str();
}
return 0;
}
See IDEONE demo returning abc 3 times.
As this method destroys the input string, here is another alternative based on the std::sregex_iterator (std::wsregex_iterator should be used when your subject is an std::wstring object):
int main() {
std::regex r("ab(c)");
std::string s = "abcdefabcghiabc";
for(std::sregex_iterator i = std::sregex_iterator(s.begin(), s.end(), r);
i != std::sregex_iterator();
++i)
{
std::smatch m = *i;
std::cout << "Match value: " << m.str() << " at Position " << m.position() << '\n';
std::cout << " Capture: " << m[1].str() << " at Position " << m.position(1) << '\n';
}
return 0;
}
See IDEONE demo, returning
Match value: abc at Position 0
Capture: c at Position 2
Match value: abc at Position 6
Capture: c at Position 8
Match value: abc at Position 12
Capture: c at Position 14
What you're missing is that matches is populated with one entry for each capture group (including the entire matched substring as the 0th capture).
If you write
std::regex needle("a(b)c");
then you'll get matches.size()==2, with matches[0]=="abc", and matches[1]=="b".
EDIT: Some people have downvoted this answer. That may be for a variety of reasons, but if it is because it does not apply to the answer I criticized (no one left a comment to explain the decision), they should take note that W. Stribizew changed the code two months after I wrote this, and I was unaware of it until today, 2021-01-18. The rest of the answer is unchanged from when I first wrote it.
#stribizhev's solution has quadratic worst case complexity for sane regular expressions. For insane ones (e.g. "y*"), it doesn't terminate. In some applications, these issues could be DoS attacks waiting to happen. Here's a fixed version:
string str("abcdefabcghiabc");
int i = 0;
regex rgx1("abc");
smatch smtch;
auto beg = str.cbegin();
while (regex_search(beg, str.cend(), smtch, rgx1)) {
std::cout << i << ": " << smtch[0] << std::endl;
i += 1;
if ( smtch.length(0) > 0 )
std::advance(beg, smtch.length(0));
else if ( beg != str.cend() )
++beg;
else
break;
}
According to my personal preference, this will find n+1 matches of an empty regex in a string of length n. You might also just exit the loop after an empty match.
If you want to compare the performance for a string with millions of matches, add the following lines after the definition of str (and don't forget to turn on optimizations), once for each version:
for (int j = 0; j < 20; ++j)
str = str + str;
I want to use a regex on the reverse of a string.
I can do the following but all my sub_matches are reversed:
string foo("lorem ipsum");
match_results<string::reverse_iterator> sm;
if (regex_match(foo.rbegin(), foo.rend(), sm, regex("(\\w+)\\s+(\\w+)"))) {
cout << sm[1] << ' ' << sm[2] << endl;
}
else {
cout << "bad\n";
}
[Live example]
What I want is to get out:
ipsum lorem
Is there any provision for getting the sub-matches that are not reversed? That is, any provision beyond reversing the strings after they're matched like this:
string first(sm[1]);
string second(sm[2]);
reverse(first.begin(), first.end());
reverse(second.begin(), second.end());
cout << first << ' ' << second << endl;
EDIT:
It has been suggested that I update the question to clarify what I want:
Running the regex backwards on the string is not about reversing the order that the matches are found in. The regex is far more complex that would be valuable to post here, but running it backwards saves me from needing a look ahead. This question is about the handling of sub-matches obtained from a match_results<string::reverse_iterator>. I need to be able to get them out as they were in the input, here foo. I don't want to have to construct a temporary string and run reverse on it for each sub-match. How can I avoid doing this.
You could just reverse the order in which you use the results:
#include <regex>
#include <string>
#include <iostream>
using namespace std;
int main()
{
string foo("lorem ipsum");
smatch sm;
if (regex_match(foo, sm, regex("(\\w+)\\s+(\\w+)"))) {
cout << sm[2] << ' ' << sm[1] << endl; // use second as first
}
else {
cout << "bad\n";
}
}
Output:
ipsum lorem
This is absolutely possible! The key is in the fact that a sub_match inherits from pair<BidirIt, BidirIt>. Since sub_matches will be obtained from: match_results<string::reverse_iterator> sm, the elements of the pair a sub_match inherits from will be string::reverse_iterators.
So for any given sub_match from sm you can get the forward range from it's second.base() to it's first.base(). You don't have to construct strings to stream ranges but you will need to construct an ostream_iterator:
ostream_iterator<char> output(cout);
copy(sm[1].second.base(), sm[1].first.base(), output);
output = ' ';
copy(sm[2].second.base(), sm[2].first.base(), output);
Take heart though, there is a better solution on the horizon! This answer discusses string_literals as of right now no action has been taken on them, but they have made it into the "Evolution Subgroup".
I have a confusion on how to fetch the result after running the function regex_search in the std::tr1::regex.
Following is a sample code to demonstrate my issue.
string source = "abcd 16000 ";
string exp = "abcd ([^\\s]+)";
std::tr1::cmatch res;
std::tr1::regex rx(exp);
while(std::tr1::regex_search(source.c_str(), res, rx, std::tr1::regex_constants::match_continuous))
{
//HOW TO FETCH THE RESULT???????????
std::cout <<" "<< res.str()<<endl;
source = res.suffix().str();
}
The regular expression mentioned should ideally strip off the "abcd" from the string and return me 16000.
I see that the cmatch res has TWO objects. The second object contains the expected result.(this object has three members (matched, first, second). and the values are {true, "16000", " "}.
My question is what does this size of the object denote? Why is it showing 2 in this specific case( res[0] and res[1]) when I have run regex_search only once? And how do I know which object would have the expected result?
Thanks
Sunil
As stated here:
match[0]: represents the entire match
match[1]: represents the first match
match[2]: represents the second match, and so forth
This means match[0] should - in this case! - hold your full source (abcd 16000) as you match the whole thing, while match[1] contains the content of your capturing group.
If there was, for example, a second capturing group in your regex you'd get a third object in the match-collection and so on.
I'm a guy who understands visualized problems/solutions better, so let's do this:
See the demo#regex101.
See the two colors in the textfield containing the teststring?
The green color is the background for your capturing group while the
blue color represents everything else generally matched by the expression, but not captured by any group.
In other words: blue+green is the equivalent for match[0] and green for match[1] in your case.
This way you can always know which of the objects in match refers to which capturing group:
You initialize a counter in your head, starting at 0. Now go through the regex from the left to the right, add 1 for each ( and subtract 1 for each ) until you reach the opening bracket of the capturing group you want to extract. The number in your head is the array index.
EDIT
Regarding your comment on checking res[0].first:
The member first of the sub_match class is only
denoting the position of the start of the match.
While second denotes the position of the end of the match.
(taken from boost doc)
Both return a char* (VC++10) or an iterator (Boost), thus you get a substring of the sourcestring as the output (which may be the full source in case the match starts at index zero!).
Consider the following program (VC++10):
#include "stdafx.h"
#include <regex>
#include <iostream>
using namespace std;
int _tmain(int argc, _TCHAR* argv[])
{
string source = "abcdababcdefg";
string exp = "ab";
tr1::cmatch res;
tr1::regex rx(exp);
tr1::regex_search(source.c_str(), res, rx);
for (size_t n = 0; n < res.size(); ++n)
{
std::cout << "submatch[" << n << "]: matched == " << std::boolalpha
<< res[n].matched <<
" at position " << res.position(n) << std::endl;
std::cout << " " << res.length(n)
<< " chars, value == " << res[n] << std::endl;
}
std::cout << std::endl;
cout << "res[0].first: " << res[0].first << " - res[0].second: " << res[0].second << std::endl;
cout << "res[0]: " << res[0];
cin.get();
return 0;
}
Execute it and look at the output. The first (and only) match is - obviously - the first to chars ab, so this is actually the whole matched string and the reason why res[0] == "ab".
Now, knowing that .first/.second give us substrings from the start of the match and from the end of the match onwards, the output shouldn't be confusing anymore.
I know there's an "isspace" function that checks for spaces, but that would require me to iterate through every character in the string, which can be bad on performance since this would be called a lot. Is there a fast way to check if a std::string contains only spaces?
ex:
function(" ") // returns true
function(" 4 ") // returns false
One solution I've thought of is to use regex, then i'll know that it only contains whitespace if it's false... but i'm not sure if this would be more efficient than the isspace function.
regex: [\w\W] //checks for any word character(a,b,c..) and non-word character([,],..)
thanks in advance!
With a regular string, the best you can do will be of the form:
return string::find_first_not_of("\t\n ") == string::npos;
This will be O(n) in the worst case, but without knowing else about the string, this will be the best you can do.
Any method would, of necessity, need to look at each character of the string. A loop that calls isspace() on each character is pretty efficient. If isspace() is inlined by the compiler, then this would be darn near optimal.
The loop should, of course, abort as soon as a non-space character is seen.
You are making the assumption regex doesnt iterate over the string. Regex is probably much heavier than a linear search since it might build a FSM and traverse based on that.
The only way you could speed it up further and make it a near-constant time operation is to amortize the cost by iterating on every update to the string and caching a bool/bit that tracks if there is a space-like character, returning that value if no changes have been made since, and updating that bit whenever you do a write operation to that string. However, this sacrifices/slows that speed of modifying operations in order to increase the speed of your custom has_space().
For what it's worth, a locale has a function (scan_is) to do things like this:
#include <locale>
#include <iostream>
#include <iomanip>
int main() {
std::string inputs[] = {
"all lower",
"including a space"
};
std::locale loc(std::locale::classic());
std::ctype_base::mask m = std::ctype_base::space;
for (int i=0; i<2; i++) {
char const *pos;
char const *b = &*inputs[i].begin();
char const *e = &*inputs[i].end();
std::cout << "Input: " << std::setw(20) << inputs[i] << ":\t";
if ((pos=std::use_facet<std::ctype<char> >(loc).scan_is(m, b, e)) == e)
std::cout << "No space character\n";
else
std::cout << "First space character at position " << pos - b << "\n";
}
return 0;
}
It's probably open to (a lot of) question whether this gives much (if any) real advantage over using isspace in a loop (or using std::find_if).
You can also use find_first_not_of if you all the characters to be in a given list.
Then you can avoid loops.
Here is an example
#include <string>
#include <algorithm>
using namespace std;
int main()
{
string str1=" ";
string str2=" u ";
bool ContainsNotBlank1=(str1.find_first_not_of("\t\n ")==string::npos);
bool ContainsNotBlank2=(str2.find_first_not_of("\t\n ")==string::npos);
bool ContainsNotBlank3=(str2.find_first_not_of("\t\n u")==string::npos);
cout << ContainsNotBlank1 <<endl;
cout << ContainsNotBlank2 <<endl;
cout << ContainsNotBlank3 <<endl;
return 0;
}
Output:
1: because only blanks and a tab
0: because u is not into the list "\t\n "
1: because str2 contains blanks, tabs and a u.
Hope it helps
Tell me if you have any questions