How can I trim empty/whitespace lines? - c++

I have to process badly mismanaged text with creative indentation. I want to remove the empty (or whitespace) lines at the beginning and end of my text without touching anything else; meaning that if the first or last actual lines respectively begin or end with whitespace, these will stay.
For example, this:
<lines, empty or with whitespaces ...>
<text, maybe preceded by whitespace>
<lines with or without text...>
<text, maybe followed by whitespace>
<lines, empty or with whitespaces ...>
turns to
<text, maybe preceded by whitespace>
<lines with or without text...>
<text, maybe followed by whitespace>
preserving the spaces at the beginning and the end of the actual text lines (the text might also be entirely whitespace)
A regex replacing (\A\s*(\r\n|\Z)|\r\n\s*\Z) by emptiness does exactly what I want, but regex is kind of overkill, and I fear it might cost me some time when processing texts with a lot of lines but not much to trim.
On the other hand, an explicit algorithm is easy to make (just read until a non-whitespace/the end while remembering the last line feed, then truncate, and do the same backwards) but it feels like I'm missing something obvious.
How can I do this?

As you can see from this discussion, trimming whitespace requires a lot of work in C++. This should definitely be included in the standard library.
Anyway, I've checked how to do it as simply as possible, but nothing comes near the compactness of RegEx. For speed, it's a different story.
In the following you can find three versions of a program which does the required task. With regex, with std functions and with just a couple of indexes. The last one can be also made faster because you can avoid copying altogether, but I left it for fair comparison:
#include <string>
#include <sstream>
#include <chrono>
#include <iostream>
#include <regex>
#include <exception>
struct perf {
std::chrono::steady_clock::time_point start_;
perf() : start_(std::chrono::steady_clock::now()) {}
double elapsed() const {
auto stop = std::chrono::steady_clock::now();
std::chrono::duration<double> elapsed_seconds = stop - start_;
return elapsed_seconds.count();
}
};
std::string Generate(size_t line_len, size_t empty, size_t nonempty) {
std::string es(line_len, ' ');
es += '\n';
for (size_t i = 0; i < empty; ++i) {
es += es;
}
std::string nes(line_len - 1, ' ');
es += "a\n";
for (size_t i = 0; i < nonempty; ++i) {
nes += nes;
}
return es + nes + es;
}
int main()
{
std::string test;
//test = " \n\t\n \n \tTEST\n\tTEST\n\t\t\n TEST\t\n \t\n \n ";
std::cout << "Generating...";
std::cout.flush();
test = Generate(1000, 8, 10);
std::cout << " done." << std::endl;
std::cout << "Test 1...";
std::cout.flush();
perf p1;
std::string out1;
std::regex re(R"(^\s*\n|\n\s*$)");
try {
out1 = std::regex_replace(test, re, "");
}
catch (std::exception& e) {
std::cout << e.what() << std::endl;
}
std::cout << " done. Elapsed time: " << p1.elapsed() << "s" << std::endl;
std::cout << "Test 2...";
std::cout.flush();
perf p2;
std::stringstream is(test);
std::string line;
while (std::getline(is, line) && line.find_first_not_of(" \t\n\v\f\r") == std::string::npos);
std::string out2 = line;
size_t end = out2.size();
while (std::getline(is, line)) {
out2 += '\n';
out2 += line;
if (line.find_first_not_of(" \t\n\v\f\r") != std::string::npos) {
end = out2.size();
}
}
out2.resize(end);
std::cout << " done. Elapsed time: " << p2.elapsed() << "s" << std::endl;
if (out1 == out2) {
std::cout << "out1 == out2\n";
}
else {
std::cout << "out1 != out2\n";
}
std::cout << "Test 3...";
std::cout.flush();
perf p3;
static bool whitespace_table[] = {
1,1,1,1,1,1,1,1,1,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
};
size_t sfl = 0; // Start of first line
for (size_t i = 0, end = test.size(); i < end; ++i) {
if (test[i] == '\n') {
sfl = i + 1;
}
else if (whitespace_table[(unsigned char)test[i]]) {
break;
}
}
size_t ell = test.size(); // End of last line
for (size_t i = test.size(); i-- > 0;) {
if (test[i] == '\n') {
ell = i;
}
else if (whitespace_table[(unsigned char)test[i]]) {
break;
}
}
std::string out3 = test.substr(sfl, ell - sfl);
std::cout << " done. Elapsed time: " << p3.elapsed() << "s" << std::endl;
if (out1 == out3) {
std::cout << "out1 == out3\n";
}
else {
std::cout << "out1 != out3\n";
}
return 0;
}
Running it on C++ Shell you get these timings:
Generating... done.
Test 1... done. Elapsed time: 4.2288s
Test 2... done. Elapsed time: 0.0077323s
out1 == out2
Test 3... done. Elapsed time: 0.000695783s
out1 == out3
If performance is important, it's better to really test it with the real files.
As a side note, this regex doesn't work on MSVC, because I couldn't find a way of avoiding ^ and $ to match the start and end of lines, that is disable the multiline mode of operation. If you run this, it throws an exception saying regex_error(error_complexity): The complexity of an attempted match against a regular expression exceeded a pre-set level.
I think I'll ask how to cope with this!

If whitespace in front of the first line or after the last non-whitespace-only line can be removed then this answer https://stackoverflow.com/a/217605/14258355 will suffice.
However, due to this constraint and if you do not want to use regex, I would propose to convert the string into lines and then build the string back up again from the first to the last non-whitespace-only line.
Here is a working example: https://godbolt.org/z/rozxj6saj
Convert the string to lines:
std::vector<std::string> StringToLines(const std::string &s) {
// Create vector with lines (not using input stream to keep line break
// characters)
std::vector<std::string> result;
std::string line;
for (auto c : s) {
line.push_back(c);
// Check for line break
if (c == '\n' || c == '\r') {
result.push_back(line);
line.clear();
}
}
// add last bit
result.push_back(line);
return result;
}
Build the string from the first to the last non-whitespace-only line:
bool IsNonWhiteSpaceString(const std::string &s) {
return s.end() != std::find_if(s.begin(), s.end(), [](unsigned char uc) {
return !std::isspace(uc);
});
}
std::string TrimVectorEmptyEndsIntoString(const std::vector<std::string> &v) {
std::string result;
// Find first non-whitespace line
auto it_begin = std::find_if(v.begin(), v.end(), [](const std::string &s) {
return IsNonWhiteSpaceString(s);
});
// Find last non-whitespace line
auto it_end = std::find_if(v.rbegin(), v.rend(), [](const std::string &s) {
return IsNonWhiteSpaceString(s);
});
// Build the string
for (auto it = it_begin; it != it_end.base(); std::advance(it, 1)) {
result.append(*it);
}
return result;
}
Usage example:
// Create a test string
std::string test_string(
" \n\t\n \n TEST\n\tTEST\n\t\tTEST\n TEST\t\n \t");
// Output result
std::cout << TrimVectorEmptyEndsIntoString(StringToLines(test_string));
Output showing whitespace:

Related

How can i split adjacent numbers and letters in c++?

I've got a large text document that including adjacent numbers and letters.
Just like that,
JACK1940383DAVID30284HAROLD68372TROY4392 etc.
How can i split this like below in C++
List: Jack / 1940383 , David/30284, ...
You can use std::string::find_first_of() and std::string::find_first_not_of() in a loop, using std::string::substr() to extract each piece, eg:
std::string s = "JACK1940383DAVID30284HAROLD68372TROY4392";
std::string::size_type start = 0, end;
while ((end = s.find_first_of("0123456789", start)) != std::string::npos) {
std::string name = s.substr(start, end-start);
start = end;
int number;
if ((end = s.find_first_not_of("0123456789", start)) != std::string::npos) {
number = std::stoi(s.substr(start, end-start));
}
else {
number = std::stoi(s.substr(start));
}
start = end;
// use name and number as needed...
}
Online Demo
You can use regex like this:
#include <iostream>
#include <string>
#include <regex>
#include <vector>
// create a struct to group your data
// this makes it easy to store it in a vector.
struct person_t
{
std::string name;
std::string number;
};
// overloaded output operator for printing one person's details
std::ostream& operator<<(std::ostream& os, const person_t& person)
{
std::cout << person.name << ": " << person.number << std::endl;
return os;
}
// get a vector of person_t based on the input
auto get_persons(const std::string& input)
{
// make a regex in this case a regex that will match one or more capital letters
// and groups them using the ()
// then match one or more digits and group them too.
static const std::regex rx{ "([A-Z]+)([0-9]+)" };
std::smatch match;
// a vector to hold all the persons
std::vector<person_t> persons;
// start at begin of string and look for first part of the string
// that matches the regex.
auto cbegin = input.cbegin();
while (std::regex_search(cbegin, input.cend(), match, rx))
{
// match[0] will contain the whole match,
// match[1]-match[n] will contain the groups from the regular expressions
// match[1] will contain the match with characters and thus the name
// match[2] will contain the match with the numbers and thus the number.
// create a person_t struct with this info
person_t person{ match[1], match[2] };
// and add it to the vector
persons.push_back(person);
cbegin = match.suffix().first;
}
return persons;
}
int main()
{
// parse and split the string
auto persons = get_persons("JACK1940383DAVID30284HAROLD68372TROY4392");
// show the output
for (const auto& person : persons)
{
std::cout << person;
}
}
As pointed in other good answers you can use
find_first_of(), find_first_not_of() and substr() from std::string in a loop
regex
But it may be too much. I will add 3 more examples that you may find
simpler.
The first 2 programs expects the file name on the command line for (my) convenience here, and the test file is in.txt. Contents are the same as posted
JACK1940383DAVID30284HAROLD68372TROY4392
The last example just parses the string data declared as a char[]
1. Using fscanf()
Since the target is to consume formatted data, fscanf() is an option. As the data structure is very simple, the program is just a one line loop:
char mask[] = "%50[^0-9]%50[0-9]";
while ( 2 == fscanf(F, mask, tk_key, tk_value))
std::cout << tk_key << "/" << tk_value << "\n";
program output
output is the same for all examples
JACK/1940383
DAVID/30284
HAROLD/68372
TROY/4392
code for ex. 1
#include <errno.h>
#include <iostream>
int main(int argc,char** argv)
{
if (argc < 2)
{ std::cerr << "Use: pgm FileName\n";
return -1;
}
FILE* F = fopen(argv[1], "r");
if (F == NULL)
{
perror("Could not open file");
return -1;
}
std::cerr << "File: \"" << argv[1] << "\"\n";
char tk_key[50], tk_value[50];
char mask[] = "%50[^0-9]%50[0-9]";
while ( 2 == fscanf(F, mask, tk_key, tk_value))
std::cout << tk_key << "/" << tk_value << "\n";
fclose(F);
return 0;
}
using a state machine
There are just 2 states so it is not a fancy FSA ;) State machines are good for representing this kind of stuff, albeit here this seems to be overkill.
#define S_LETTER 0
#define S_DIGIT 1
#include <algorithm>
#include <iostream>
#include <fstream>
using iich = std::istream_iterator<char>;
int main(int argc,char** argv)
{
std::ifstream in_file{argv[1]};
if ( not in_file.good()) return -1;
iich p {in_file}, eofile{};
std::string token{}; // string to build values
char st = S_LETTER; // state value for FSA
std::for_each(p, eofile,
[&token,&st](char ch)
{
char temp = 0;
switch (st)
{
case S_LETTER:
if ((ch >= '0') && (ch <= '9'))
{
std::cout << token << "/";
token = ch;
st = S_DIGIT; // now in number
}
else token += ch; // concat in string
break;
case S_DIGIT:
default:
if ((ch < '0') || (ch > '9'))
{ // is a letter
std::cout << token << "\n";
token = ch;
st = S_LETTER; // now in name
}
else token += ch; // concat in string
break;
}; // switch()
});
std::cout << token << "\n"; // print last token
}
Here we have no loop. for_each gets the data from an iterator and passes it to a function that builds the name and the value as strings and couts them
Output is the same
3. a simple FSA to consume the data
#define S_LETTER 0
#define S_DIGIT 1
#include <iostream>
int main(void)
{
char one[] = "JACK1940383DAVID30284HAROLD68372TROY4392";
char* p = (char*)&one;
char* token = p;
char st = S_LETTER;
char temp = 0;
while (*p != 0)
{
switch (st)
{
case S_LETTER:
if ((*p >= '0') && (*p <= '9'))
{
temp = *p;
*p = 0;
std::cout << token << "/";
*p = temp;
token = p;
st = S_DIGIT; // now in number
}
break;
case S_DIGIT:
default:
if ( (*p < '0') || (*p > '9'))
{ // letter
temp = *p;
*p = 0;
std::cout << token << "\n";
*p = temp;
token = p;
st = S_LETTER; // now in name
}
break;
}; // switch()
p += 1; // next symbol
}; // while()
std::cout << token << "\n"; // print last token
}
This code just uses a C-style loop to parse the input data

i.m trying to split string by whitespace using c++, where the data from database [duplicate]

What would be easiest method to split a string using c++11?
I've seen the method used by this post, but I feel that there ought to be a less verbose way of doing it using the new standard.
Edit: I would like to have a vector<string> as a result and be able to delimitate on a single character.
std::regex_token_iterator performs generic tokenization based on a regex. It may or may not be overkill for doing simple splitting on a single character, but it works and is not too verbose:
std::vector<std::string> split(const string& input, const string& regex) {
// passing -1 as the submatch index parameter performs splitting
std::regex re(regex);
std::sregex_token_iterator
first{input.begin(), input.end(), re, -1},
last;
return {first, last};
}
Here is a (maybe less verbose) way to split string (based on the post you mentioned).
#include <string>
#include <sstream>
#include <vector>
std::vector<std::string> split(const std::string &s, char delim) {
std::stringstream ss(s);
std::string item;
std::vector<std::string> elems;
while (std::getline(ss, item, delim)) {
elems.push_back(item);
// elems.push_back(std::move(item)); // if C++11 (based on comment from #mchiasson)
}
return elems;
}
Here's an example of splitting a string and populating a vector with the extracted elements using boost.
#include <boost/algorithm/string.hpp>
std::string my_input("A,B,EE");
std::vector<std::string> results;
boost::algorithm::split(results, my_input, boost::is_any_of(","));
assert(results[0] == "A");
assert(results[1] == "B");
assert(results[2] == "EE");
Another regex solution inspired by other answers but hopefully shorter and easier to read:
std::string s{"String to split here, and here, and here,..."};
std::regex regex{R"([\s,]+)"}; // split on space and comma
std::sregex_token_iterator it{s.begin(), s.end(), regex, -1};
std::vector<std::string> words{it, {}};
I don't know if this is less verbose, but it might be easier to grok for those more seasoned in dynamic languages such as javascript. The only C++11 features it uses is auto and range-based for loop.
#include <string>
#include <cctype>
#include <iostream>
#include <vector>
using namespace std;
int main()
{
string s = "hello how are you won't you tell me your name";
vector<string> tokens;
string token;
for (const auto& c: s) {
if (!isspace(c))
token += c;
else {
if (token.length()) tokens.push_back(token);
token.clear();
}
}
if (token.length()) tokens.push_back(token);
return 0;
}
#include <iostream>
#include <algorithm>
#include <vector>
#include <string>
using namespace std;
vector<string> split(const string& str, int delimiter(int) = ::isspace){
vector<string> result;
auto e=str.end();
auto i=str.begin();
while(i!=e){
i=find_if_not(i,e, delimiter);
if(i==e) break;
auto j=find_if(i,e, delimiter);
result.push_back(string(i,j));
i=j;
}
return result;
}
int main(){
string line;
getline(cin,line);
vector<string> result = split(line);
for(auto s: result){
cout<<s<<endl;
}
}
My choice is boost::tokenizer but I didn't have any heavy tasks and test with huge data.
Example from boost doc with lambda modification:
#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>
#include <vector>
int main()
{
using namespace std;
using namespace boost;
string s = "This is, a test";
vector<string> v;
tokenizer<> tok(s);
for_each (tok.begin(), tok.end(), [&v](const string & s) { v.push_back(s); } );
// result 4 items: 1)This 2)is 3)a 4)test
return 0;
}
This is my answer. Verbose, readable and efficient.
std::vector<std::string> tokenize(const std::string& s, char c) {
auto end = s.cend();
auto start = end;
std::vector<std::string> v;
for( auto it = s.cbegin(); it != end; ++it ) {
if( *it != c ) {
if( start == end )
start = it;
continue;
}
if( start != end ) {
v.emplace_back(start, it);
start = end;
}
}
if( start != end )
v.emplace_back(start, end);
return v;
}
#include <string>
#include <vector>
#include <sstream>
inline vector<string> split(const string& s) {
vector<string> result;
istringstream iss(s);
for (string w; iss >> w; )
result.push_back(w);
return result;
}
Here is a C++11 solution that uses only std::string::find(). The delimiter can be any number of characters long. Parsed tokens are output via an output iterator, which is typically a std::back_inserter in my code.
I have not tested this with UTF-8, but I expect it should work as long as the input and delimiter are both valid UTF-8 strings.
#include <string>
template<class Iter>
Iter splitStrings(const std::string &s, const std::string &delim, Iter out)
{
if (delim.empty()) {
*out++ = s;
return out;
}
size_t a = 0, b = s.find(delim);
for ( ; b != std::string::npos;
a = b + delim.length(), b = s.find(delim, a))
{
*out++ = std::move(s.substr(a, b - a));
}
*out++ = std::move(s.substr(a, s.length() - a));
return out;
}
Some test cases:
void test()
{
std::vector<std::string> out;
size_t counter;
std::cout << "Empty input:" << std::endl;
out.clear();
splitStrings("", ",", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, empty delimiter:" << std::endl;
out.clear();
splitStrings("Hello, world!", "", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", no delimiter in string:" << std::endl;
out.clear();
splitStrings("abxycdxyxydefxya", "xyz", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", delimiter exists string:" << std::endl;
out.clear();
splitStrings("abxycdxy!!xydefxya", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", delimiter exists string"
", input contains blank token:" << std::endl;
out.clear();
splitStrings("abxycdxyxydefxya", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", delimiter exists string"
", nothing after last delimiter:" << std::endl;
out.clear();
splitStrings("abxycdxyxydefxy", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", only delimiter exists string:" << std::endl;
out.clear();
splitStrings("xy", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
}
Expected output:
Empty input:
0:
Non-empty input, empty delimiter:
0: Hello, world!
Non-empty input, non-empty delimiter, no delimiter in string:
0: abxycdxyxydefxya
Non-empty input, non-empty delimiter, delimiter exists string:
0: ab
1: cd
2: !!
3: def
4: a
Non-empty input, non-empty delimiter, delimiter exists string, input contains blank token:
0: ab
1: cd
2:
3: def
4: a
Non-empty input, non-empty delimiter, delimiter exists string, nothing after last delimiter:
0: ab
1: cd
2:
3: def
4:
Non-empty input, non-empty delimiter, only delimiter exists string:
0:
1:
One possible way of doing this is finding all occurrences of the split string and storing locations to a list. Then count input string characters and when you get to a position where there is a 'search hit' in the position list then you jump forward by 'length of the split string'. This approach takes a split string of any length. Here is my tested and working solution.
#include <iostream>
#include <string>
#include <list>
#include <vector>
using namespace std;
vector<string> Split(string input_string, string search_string)
{
list<int> search_hit_list;
vector<string> word_list;
size_t search_position, search_start = 0;
// Find start positions of every substring occurence and store positions to a hit list.
while ( (search_position = input_string.find(search_string, search_start) ) != string::npos) {
search_hit_list.push_back(search_position);
search_start = search_position + search_string.size();
}
// Iterate through hit list and reconstruct substring start and length positions
int character_counter = 0;
int start, length;
for (auto hit_position : search_hit_list) {
// Skip over substrings we are splitting with. This also skips over repeating substrings.
if (character_counter == hit_position) {
character_counter = character_counter + search_string.size();
continue;
}
start = character_counter;
character_counter = hit_position;
length = character_counter - start;
word_list.push_back(input_string.substr(start, length));
character_counter = character_counter + search_string.size();
}
// If the search string is not found in the input string, then return the whole input_string.
if (word_list.size() == 0) {
word_list.push_back(input_string);
return word_list;
}
// The last substring might be still be unprocessed, get it.
if (character_counter < input_string.size()) {
word_list.push_back(input_string.substr(character_counter, input_string.size() - character_counter));
}
return word_list;
}
int main() {
vector<string> word_list;
string search_string = " ";
// search_string = "the";
string text = "thetheThis is some text to test with the split-thethe function.";
word_list = Split(text, search_string);
for (auto item : word_list) {
cout << "'" << item << "'" << endl;
}
cout << endl;
}

C++ alternative of Java's split(str, -1) [duplicate]

What would be easiest method to split a string using c++11?
I've seen the method used by this post, but I feel that there ought to be a less verbose way of doing it using the new standard.
Edit: I would like to have a vector<string> as a result and be able to delimitate on a single character.
std::regex_token_iterator performs generic tokenization based on a regex. It may or may not be overkill for doing simple splitting on a single character, but it works and is not too verbose:
std::vector<std::string> split(const string& input, const string& regex) {
// passing -1 as the submatch index parameter performs splitting
std::regex re(regex);
std::sregex_token_iterator
first{input.begin(), input.end(), re, -1},
last;
return {first, last};
}
Here is a (maybe less verbose) way to split string (based on the post you mentioned).
#include <string>
#include <sstream>
#include <vector>
std::vector<std::string> split(const std::string &s, char delim) {
std::stringstream ss(s);
std::string item;
std::vector<std::string> elems;
while (std::getline(ss, item, delim)) {
elems.push_back(item);
// elems.push_back(std::move(item)); // if C++11 (based on comment from #mchiasson)
}
return elems;
}
Here's an example of splitting a string and populating a vector with the extracted elements using boost.
#include <boost/algorithm/string.hpp>
std::string my_input("A,B,EE");
std::vector<std::string> results;
boost::algorithm::split(results, my_input, boost::is_any_of(","));
assert(results[0] == "A");
assert(results[1] == "B");
assert(results[2] == "EE");
Another regex solution inspired by other answers but hopefully shorter and easier to read:
std::string s{"String to split here, and here, and here,..."};
std::regex regex{R"([\s,]+)"}; // split on space and comma
std::sregex_token_iterator it{s.begin(), s.end(), regex, -1};
std::vector<std::string> words{it, {}};
I don't know if this is less verbose, but it might be easier to grok for those more seasoned in dynamic languages such as javascript. The only C++11 features it uses is auto and range-based for loop.
#include <string>
#include <cctype>
#include <iostream>
#include <vector>
using namespace std;
int main()
{
string s = "hello how are you won't you tell me your name";
vector<string> tokens;
string token;
for (const auto& c: s) {
if (!isspace(c))
token += c;
else {
if (token.length()) tokens.push_back(token);
token.clear();
}
}
if (token.length()) tokens.push_back(token);
return 0;
}
#include <iostream>
#include <algorithm>
#include <vector>
#include <string>
using namespace std;
vector<string> split(const string& str, int delimiter(int) = ::isspace){
vector<string> result;
auto e=str.end();
auto i=str.begin();
while(i!=e){
i=find_if_not(i,e, delimiter);
if(i==e) break;
auto j=find_if(i,e, delimiter);
result.push_back(string(i,j));
i=j;
}
return result;
}
int main(){
string line;
getline(cin,line);
vector<string> result = split(line);
for(auto s: result){
cout<<s<<endl;
}
}
My choice is boost::tokenizer but I didn't have any heavy tasks and test with huge data.
Example from boost doc with lambda modification:
#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>
#include <vector>
int main()
{
using namespace std;
using namespace boost;
string s = "This is, a test";
vector<string> v;
tokenizer<> tok(s);
for_each (tok.begin(), tok.end(), [&v](const string & s) { v.push_back(s); } );
// result 4 items: 1)This 2)is 3)a 4)test
return 0;
}
This is my answer. Verbose, readable and efficient.
std::vector<std::string> tokenize(const std::string& s, char c) {
auto end = s.cend();
auto start = end;
std::vector<std::string> v;
for( auto it = s.cbegin(); it != end; ++it ) {
if( *it != c ) {
if( start == end )
start = it;
continue;
}
if( start != end ) {
v.emplace_back(start, it);
start = end;
}
}
if( start != end )
v.emplace_back(start, end);
return v;
}
#include <string>
#include <vector>
#include <sstream>
inline vector<string> split(const string& s) {
vector<string> result;
istringstream iss(s);
for (string w; iss >> w; )
result.push_back(w);
return result;
}
Here is a C++11 solution that uses only std::string::find(). The delimiter can be any number of characters long. Parsed tokens are output via an output iterator, which is typically a std::back_inserter in my code.
I have not tested this with UTF-8, but I expect it should work as long as the input and delimiter are both valid UTF-8 strings.
#include <string>
template<class Iter>
Iter splitStrings(const std::string &s, const std::string &delim, Iter out)
{
if (delim.empty()) {
*out++ = s;
return out;
}
size_t a = 0, b = s.find(delim);
for ( ; b != std::string::npos;
a = b + delim.length(), b = s.find(delim, a))
{
*out++ = std::move(s.substr(a, b - a));
}
*out++ = std::move(s.substr(a, s.length() - a));
return out;
}
Some test cases:
void test()
{
std::vector<std::string> out;
size_t counter;
std::cout << "Empty input:" << std::endl;
out.clear();
splitStrings("", ",", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, empty delimiter:" << std::endl;
out.clear();
splitStrings("Hello, world!", "", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", no delimiter in string:" << std::endl;
out.clear();
splitStrings("abxycdxyxydefxya", "xyz", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", delimiter exists string:" << std::endl;
out.clear();
splitStrings("abxycdxy!!xydefxya", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", delimiter exists string"
", input contains blank token:" << std::endl;
out.clear();
splitStrings("abxycdxyxydefxya", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", delimiter exists string"
", nothing after last delimiter:" << std::endl;
out.clear();
splitStrings("abxycdxyxydefxy", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
std::cout << "Non-empty input, non-empty delimiter"
", only delimiter exists string:" << std::endl;
out.clear();
splitStrings("xy", "xy", std::back_inserter(out));
counter = 0;
for (auto i = out.begin(); i != out.end(); ++i, ++counter) {
std::cout << counter << ": " << *i << std::endl;
}
}
Expected output:
Empty input:
0:
Non-empty input, empty delimiter:
0: Hello, world!
Non-empty input, non-empty delimiter, no delimiter in string:
0: abxycdxyxydefxya
Non-empty input, non-empty delimiter, delimiter exists string:
0: ab
1: cd
2: !!
3: def
4: a
Non-empty input, non-empty delimiter, delimiter exists string, input contains blank token:
0: ab
1: cd
2:
3: def
4: a
Non-empty input, non-empty delimiter, delimiter exists string, nothing after last delimiter:
0: ab
1: cd
2:
3: def
4:
Non-empty input, non-empty delimiter, only delimiter exists string:
0:
1:
One possible way of doing this is finding all occurrences of the split string and storing locations to a list. Then count input string characters and when you get to a position where there is a 'search hit' in the position list then you jump forward by 'length of the split string'. This approach takes a split string of any length. Here is my tested and working solution.
#include <iostream>
#include <string>
#include <list>
#include <vector>
using namespace std;
vector<string> Split(string input_string, string search_string)
{
list<int> search_hit_list;
vector<string> word_list;
size_t search_position, search_start = 0;
// Find start positions of every substring occurence and store positions to a hit list.
while ( (search_position = input_string.find(search_string, search_start) ) != string::npos) {
search_hit_list.push_back(search_position);
search_start = search_position + search_string.size();
}
// Iterate through hit list and reconstruct substring start and length positions
int character_counter = 0;
int start, length;
for (auto hit_position : search_hit_list) {
// Skip over substrings we are splitting with. This also skips over repeating substrings.
if (character_counter == hit_position) {
character_counter = character_counter + search_string.size();
continue;
}
start = character_counter;
character_counter = hit_position;
length = character_counter - start;
word_list.push_back(input_string.substr(start, length));
character_counter = character_counter + search_string.size();
}
// If the search string is not found in the input string, then return the whole input_string.
if (word_list.size() == 0) {
word_list.push_back(input_string);
return word_list;
}
// The last substring might be still be unprocessed, get it.
if (character_counter < input_string.size()) {
word_list.push_back(input_string.substr(character_counter, input_string.size() - character_counter));
}
return word_list;
}
int main() {
vector<string> word_list;
string search_string = " ";
// search_string = "the";
string text = "thetheThis is some text to test with the split-thethe function.";
word_list = Split(text, search_string);
for (auto item : word_list) {
cout << "'" << item << "'" << endl;
}
cout << endl;
}

Difficulties with string declaration/reference parameters (c++)

Last week I got an homework to write a function: the function gets a string and a char value and should divide the string in two parts, before and after the first occurrence of the existing char.
The code worked but my teacher told me to do it again, because it is not well written code. But I don't understand how to make it better. I understand so far that defining two strings with white spaces is not good, but i get out of bounds exceptions otherwise. Since the string input changes, the string size changes everytime.
#include <iostream>
#include <string>
using namespace std;
void divide(char search, string text, string& first_part, string& sec_part)
{
bool firstc = true;
int counter = 0;
for (int i = 0; i < text.size(); i++) {
if (text.at(i) != search && firstc) {
first_part.at(i) = text.at(i);
}
else if (text.at(i) == search&& firstc == true) {
firstc = false;
sec_part.at(counter) = text.at(i);
}
else {
sec_part.at(counter) = text.at(i);
counter++;
}
}
}
int main() {
string text;
string part1=" ";
string part2=" ";
char search_char;
cout << "Please enter text? ";
getline(cin, text);
cout << "Please enter a char: ? ";
cin >> search_char;
divide(search_char,text,aprt1,part2);
cout << "First string: " << part1 <<endl;
cout << "Second string: " << part2 << endl;
system("PAUSE");
return 0;
}
I would suggest you, learn to use c++ standard functions. there are plenty utility function that can help you in programming.
void divide(const std::string& text, char search, std::string& first_part, std::string& sec_part)
{
std::string::const_iterator pos = std::find(text.begin(), text.end(), search);
first_part.append(text, 0, pos - text.begin());
sec_part.append(text, pos - text.begin());
}
int main()
{
std::string text = "thisisfirst";
char search = 'f';
std::string first;
std::string second;
divide(text, search, first, second);
}
Here I used std::find that you can read about it from here and also Iterators.
You have some other mistakes. you are passing your text by value that will do a copy every time you call your function. pass it by reference but qualify it with const that will indicate it is an input parameter not an output.
Why is your teacher right ?
The fact that you need to initialize your destination strings with empty space is terrible:
If the input string is longer, you'll get out of bound errors.
If it's shorter, you got wrong answer, because in IT and programming, "It works " is not the same as "It works".
In addition, your code does not fit the specifications. It should work all the time, independently of the current value which is stored in your output strings.
Alternative 1: your code but working
Just clear the destination strings at the beginning. Then iterate as you did, but use += or push_back() to add chars at the end of the string.
void divide(char search, string text, string& first_part, string& sec_part)
{
bool firstc = true;
first_part.clear(); // make destinations strings empty
sec_part.clear();
for (int i = 0; i < text.size(); i++) {
char c = text.at(i);
if (firstc && c != search) {
first_part += c;
}
else if (firstc && c == search) {
firstc = false;
sec_part += c;
}
else {
sec_part += c;
}
}
}
I used a temporary c instead of text.at(i) or text\[i\], in order to avoid multiple indexing But this is not really required: nowadays, optimizing compilers should produce equivalent code, whatever variant you use here.
Alternative 2: use string member functions
This alternative uses the find() function, and then constructs a string from the start until that position, and another from that position. There is a special case when the character was not found.
void divide(char search, string text, string& first_part, string& sec_part)
{
auto pos = text.find(search);
first_part = string(text, 0, pos);
if (pos== string::npos)
sec_part.clear();
else sec_part = string(text, pos, string::npos);
}
As you understand yourself these declarations
string part1=" ";
string part2=" ";
do not make sense because the entered string in the object text can essentially exceed the both initialized strings. In this case using the string method at can result in throwing an exception or the strings will have trailing spaces.
From the description of the assignment it is not clear whether the searched character should be included in one of the strings. You suppose that the character should be included in the second string.
Take into account that the parameter text should be declared as a constant reference.
Also instead of using loops it is better to use methods of the class std::string such as for example find.
The function can look the following way
#include <iostream>
#include <string>
void divide(const std::string &text, char search, std::string &first_part, std::string &sec_part)
{
std::string::size_type pos = text.find(search);
first_part = text.substr(0, pos);
if (pos == std::string::npos)
{
sec_part.clear();
}
else
{
sec_part = text.substr(pos);
}
}
int main()
{
std::string text("Hello World");
std::string first_part;
std::string sec_part;
divide(text, ' ', first_part, sec_part);
std::cout << "\"" << text << "\"\n";
std::cout << "\"" << first_part << "\"\n";
std::cout << "\"" << sec_part << "\"\n";
}
The program output is
"Hello World"
"Hello"
" World"
As you can see the separating character is included in the second string though I think that maybe it would be better to exclude it from the both strings.
An alternative and in my opinion more clear approach can look the following way
#include <iostream>
#include <string>
#include <utility>
std::pair<std::string, std::string> divide(const std::string &s, char c)
{
std::string::size_type pos = s.find(c);
return { s.substr(0, pos), pos == std::string::npos ? "" : s.substr(pos) };
}
int main()
{
std::string text("Hello World");
auto p = divide(text, ' ');
std::cout << "\"" << text << "\"\n";
std::cout << "\"" << p.first << "\"\n";
std::cout << "\"" << p.second << "\"\n";
}
Your code will only work as long the character is found within part1.length(). You need something similar to this:
void string_split_once(const char s, const string & text, string & first, string & second) {
first.clear();
second.clear();
std::size_t pos = str.find(s);
if (pos != string::npos) {
first = text.substr(0, pos);
second = text.substr(pos);
}
}
The biggest problem I see is that you are using at where you should be using push_back. See std::basic_string::push_back. at is designed to access an existing character to read or modify it. push_back appends a new character to the string.
divide could look like this :
void divide(char search, string text, string& first_part,
string& sec_part)
{
bool firstc = true;
for (int i = 0; i < text.size(); i++) {
if (text.at(i) != search && firstc) {
first_part.push_back(text.at(i));
}
else if (text.at(i) == search&& firstc == true) {
firstc = false;
sec_part.push_back(text.at(i));
}
else {
sec_part.push_back(text.at(i));
}
}
}
Since you aren't handling exceptions, consider using text[i] rather than text.at(i).

Different behavior in C regex VS C++11 regex

I need a code that splits math-notation permutations into its elements, lets suppose this permutation:
The permutation string will be:
"(1,2,5)(3,4)" or "(3,4)(1,2,5)" or "(3,4)(5,1,2)"
The patterns i've tried are this:
([0-9]+[ ]*,[ ]*)*[0-9]+ for each permutation cycle. This would split the "(1,2,5)(3,4)" string in two strings "1,2,5" and "3,4".
([0-9]+) for each element in cycle. This would split each cycle in individual numbers.
When i've tried this patterns in this page they work well. And also, i've used them with the C++11 regex library with good results:
#include <iostream>
#include <string>
#include <regex>
void elements(const std::string &input)
{
const std::regex ElementRegEx("[0-9]+");
for (std::sregex_iterator Element(input.begin(), input.end(), ElementRegEx); Element != std::sregex_iterator(); ++Element)
{
const std::string CurrentElement(*Element->begin());
std::cout << '\t' << CurrentElement << '\n';
}
}
void cycles(const std::string &input)
{
const std::regex CycleRegEx("([0-9]+[ ]*,[ ]*)*[0-9]+");
for (std::sregex_iterator Cycle(input.begin(), input.end(), CycleRegEx); Cycle != std::sregex_iterator(); ++Cycle)
{
const std::string CurrentCycle(*Cycle->begin());
std::cout << CurrentCycle << '\n';
elements(CurrentCycle);
}
}
int main(int argc, char **argv)
{
std::string input("(1,2,5)(3,4)");
std::cout << "input: " << input << "\n\n";
cycles(input);
return 0;
}
The Output compiling with Visual Studio 2010 (10.0):
input: (1,2,5)(3,4)
1,2,5
1
2
5
3,4
3
4
But unfortunately, i cannot use the C++11 tools on my project, the project will run under a Linux plataform and it must be compiled with gcc 4.2.3; so i'm forced to use the C regex library in the regex.h header. So, using the same patterns but with different library i'm getting different results:
Here is the test code:
void elements(const std::string &input)
{
regex_t ElementRegEx;
regcomp(&ElementRegEx, "([0-9]+)", REG_EXTENDED);
regmatch_t ElementMatches[MAX_MATCHES];
if (!regexec(&ElementRegEx, input.c_str(), MAX_MATCHES, ElementMatches, 0))
{
int Element = 0;
while ((ElementMatches[Element].rm_so != -1) && (ElementMatches[Element].rm_eo != -1))
{
regmatch_t &ElementMatch = ElementMatches[Element];
std::stringstream CurrentElement(input.substr(ElementMatch.rm_so, ElementMatch.rm_eo - ElementMatch.rm_so));
std::cout << '\t' << CurrentElement << '\n';
++Element;
}
}
regfree(&ElementRegEx);
}
void cycles(const std::string &input)
{
regex_t CycleRegEx;
regcomp(&CycleRegEx, "([0-9]+[ ]*,[ ]*)*[0-9]+", REG_EXTENDED);
regmatch_t CycleMatches[MAX_MATCHES];
if (!regexec(&CycleRegEx, input.c_str(), MAX_MATCHES, CycleMatches, 0))
{
int Cycle = 0;
while ((CycleMatches[Cycle].rm_so != -1) && (CycleMatches[Cycle].rm_eo != -1))
{
regmatch_t &CycleMatch = CycleMatches[Cycle];
const std::string CurrentCycle(input.substr(CycleMatch.rm_so, CycleMatch.rm_eo - CycleMatch.rm_so));
std::cout << CurrentCycle << '\n';
elements(CurrentCycle);
++Cycle;
}
}
regfree(&CycleRegEx);
}
int main(int argc, char **argv)
{
cycles("(1,2,5)(3,4)")
return 0;
}
The expected output is the same as using C++11 regex, but the real ouput was:
input: (1,2,5)(3,4)
1,2,5
1
1
2,
2
2
Finally, the questions are:
Could someone give me a hint about where i'm misunderstanding the C regex engine?
Why the behavior is different in the C regex vs the C++ regex?
You're misunderstanding the output of regexec. The pmatch buffer (after pmatch[0]) is filled with sub-matches of the regex, not with consecutive matches in the string.
For example, if your regex is [a-z]([+ ])([0-9]) matched against x+5, then pmatch[0] will reference x+5 (the whole match), and pmatch[1] and pmatch[2] will reference + and 5 respectively.
You need to repeat the regexec in a loop, starting from the end of the previous match:
int start = 0;
while (!regexec(&ElementRegEx, input.c_str() + start, MAX_MATCHES, ElementMatches, 0))
{
regmatch_t &ElementMatch = ElementMatches[0];
std::string CurrentElement(input.substr(start + ElementMatch.rm_so, ElementMatch.rm_eo - ElementMatch.rm_so));
std::cout << '\t' << CurrentElement << '\n';
start += ElementMatch.rm_eo;
}