Rcpp - Capture result of sregex_token_iterator to vector - c++

Question

I'm an R user and am learning c++ to leverage in Rcpp. Recently, I wrote an alternative to R's strsplit in Rcpp using string.h but it isn't regex based (afaik). I've been reading about Boost and found sregex_token_iterator.
The website below has an example:
std::string input("This is his face");
sregex re = sregex::compile(" "); // find white space
// iterate over all non-white space in the input. Note the -1 below:
sregex_token_iterator begin( input.begin(), input.end(), re, -1 ), end;
// write all the words to std::cout
std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
std::copy( begin, end, out_iter );
My rcpp function runs just fine:
#include <Rcpp.h>
#include <boost/xpressive/xpressive.hpp>
using namespace Rcpp;
// [[Rcpp::export]]
StringVector testMe(std::string input,std::string uregex) {
boost::xpressive::sregex re = boost::xpressive::sregex::compile(uregex); // find a date
// iterate over the days, months and years in the input
boost::xpressive::sregex_token_iterator begin( input.begin(), input.end(), re ,-1), end;
// write all the words to std::cout
std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
std::copy( begin, end, out_iter );
return("Done");
}
/*** R
testMe("This is a funny sentence"," ")
*/
But all it does is print out the tokens. I am very new to C++ but I understand the idea of making a vector in rcpp with StringVector res(10); (make a vector named res of length 10) which I can then index res[1] = "blah".
My question is - how do I take the output of boost::xpressive::sregex_token_iterator begin( input.begin(), input.end(), re ,-1), end; and store it in a vector so I can return it?
http://www.boost.org/doc/libs/1_54_0/doc/html/xpressive/user_s_guide.html#boost_xpressive.user_s_guide.string_splitting_and_tokenization
Final working Rcpp solution
Including this because my need was Rcpp specific and I had to make some minor changes to the solution provided.
#include <Rcpp.h>
#include <boost/xpressive/xpressive.hpp>
typedef std::vector<std::string> StringVector;
using boost::xpressive::sregex;
using boost::xpressive::sregex_token_iterator;
using Rcpp::List;
void tokenWorker(/*in*/ const std::string& input,
/*in*/ const sregex re,
/*inout*/ StringVector& v)
{
sregex_token_iterator begin( input.begin(), input.end(), re ,-1), end;
// write all the words to v
std::copy(begin, end, std::back_inserter(v));
}
//[[Rcpp::export]]
List tokenize(StringVector t, std::string tok = " "){
List final_res(t.size());
sregex re = sregex::compile(tok);
for(int z=0;z<t.size();z++){
std::string x = "";
for(int y=0;y<t[z].size();y++){
x += t[z][y];
}
StringVector v;
tokenWorker(x, re, v);
final_res[z] = v;
}
return(final_res);
}
/*** R
tokenize("Please tokenize this sentence")
*/

Answer 1

My question is - how do I take the output of
boost::xpressive::sregex_token_iterator begin( input.begin(),
input.end(), re ,-1), end; and store it in a vector so I can return
it?
You're already halfway there.
The missing link is just std::back_inserter
#include <iostream>
#include <algorithm>
#include <vector>
#include <string>
#include <boost/xpressive/xpressive.hpp>
typedef std::vector<std::string> StringVector;
using boost::xpressive::sregex;
using boost::xpressive::sregex_token_iterator;
void testMe(/*in*/ const std::string& input,
/*in*/ const std::string& uregex,
/*inout*/ StringVector& v)
{
sregex re = sregex::compile(uregex);
sregex_token_iterator begin( input.begin(), input.end(), re ,-1), end;
// write all the words to v
std::copy(begin, end, std::back_inserter(v));
}
int main()
{
std::string input("This is his face");
std::string blank(" ");
StringVector v;
// find white space
testMe(input, blank, v);
std::copy(v.begin(), v.end(),
std::ostream_iterator<std::string>(std::cout, "|"));
std::cout << std::endl;
return 0;
}
output:
This|is|his|face|
I used legacy C++ because you used a regex lib from boost instead of std <regex>; maybe you better consider C++14 right from the start when you learn c++ right now; C++14 would have shortened even this small snippet and made it more expressive.

Answer 2

And here's the C++11 version.
Aside from the benefits of using a standardized <regex>, the <regex>-using version compiles roughly twice as fast as the boost::xpressive version with gcc-4.9 and clang-3.5 (-g -O0 -std=c++11) on a QuadCore-Box running with Debian x86_64 Jessie.
#include <iostream>
#include <algorithm>
#include <vector>
#include <string>
//////////////////////////////////////////////////////////////////////////////
// A minimal adaption layer atop boost::xpressive and c++11 std's <regex> //
//--------------------------------------------------------------------------//
// remove the comment sign from the #define if your compiler suite's //
// <regex> implementation is not complete //
//#define USE_REGEX_FALLBACK_33509467 1 //
//////////////////////////////////////////////////////////////////////////////
#if defined(USE_REGEX_FALLBACK_33509467)
#include <boost/xpressive/xpressive.hpp>
using regex = boost::xpressive::sregex;
using sregex_iterator = boost::xpressive::sregex_token_iterator;
auto compile = [] (const std::string& s) {
return boost::xpressive::sregex::compile(s);
};
auto make_sregex_iterator = [] (const std::string& s, const regex& re) {
return sregex_iterator(s.begin(), s.end(), re ,-1);
};
#else // #if !defined(USE_REGEX_FALLBACK_33509467)
#include <regex>
using regex = std::regex;
using sregex_iterator = std::sregex_token_iterator;
auto compile = [] (const std::string& s) {
return regex(s);
};
auto make_sregex_iterator = [] (const std::string& s, const regex& re) {
return std::sregex_token_iterator(s.begin(), s.end(), re, -1);
};
#endif // #if defined(USE_REGEX_FALLBACK_33509467)
//////////////////////////////////////////////////////////////////////////////
typedef std::vector<std::string> StringVector;
StringVector testMe(/*in*/const std::string& input,
/*in*/const std::string& uregex)
{
regex re = compile(uregex);
sregex_iterator begin = make_sregex_iterator(input, re),
end;
return StringVector(begin, end); // doesn't steal the strings
// but try (and succeed) to move the vector
}
int main() {
std::string input("This is his face");
std::string blank(" ");
// tokenize by white space
StringVector v = testMe(input, blank);
std::copy(v.begin(), v.end(),
std::ostream_iterator<std::string>(std::cout, "|"));
std::cout << std::endl;
return EXIT_SUCCESS;
}