I'm currently using the following code to right-trim all the std::strings in my programs:
std::string s;
s.erase(s.find_last_not_of(" \n\r\t")+1);
It works fine, but I wonder if there are some end-cases where it might fail?
Of course, answers with elegant alternatives and also left-trim solution are welcome.
EDIT Since c++17, some parts of the standard library were removed. Fortunately, starting with c++11, we have lambdas which are a superior solution.
#include <algorithm>
#include <cctype>
#include <locale>
// trim from start (in place)
static inline void ltrim(std::string &s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](unsigned char ch) {
return !std::isspace(ch);
}));
}
// trim from end (in place)
static inline void rtrim(std::string &s) {
s.erase(std::find_if(s.rbegin(), s.rend(), [](unsigned char ch) {
return !std::isspace(ch);
}).base(), s.end());
}
// trim from both ends (in place)
static inline void trim(std::string &s) {
rtrim(s);
ltrim(s);
}
// trim from start (copying)
static inline std::string ltrim_copy(std::string s) {
ltrim(s);
return s;
}
// trim from end (copying)
static inline std::string rtrim_copy(std::string s) {
rtrim(s);
return s;
}
// trim from both ends (copying)
static inline std::string trim_copy(std::string s) {
trim(s);
return s;
}
Thanks to https://stackoverflow.com/a/44973498/524503 for bringing up the modern solution.
Original answer:
I tend to use one of these 3 for my trimming needs:
#include <algorithm>
#include <functional>
#include <cctype>
#include <locale>
// trim from start
static inline std::string <rim(std::string &s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(),
std::not1(std::ptr_fun<int, int>(std::isspace))));
return s;
}
// trim from end
static inline std::string &rtrim(std::string &s) {
s.erase(std::find_if(s.rbegin(), s.rend(),
std::not1(std::ptr_fun<int, int>(std::isspace))).base(), s.end());
return s;
}
// trim from both ends
static inline std::string &trim(std::string &s) {
return ltrim(rtrim(s));
}
They are fairly self-explanatory and work very well.
EDIT: BTW, I have std::ptr_fun in there to help disambiguate std::isspace because there is actually a second definition which supports locales. This could have been a cast just the same, but I tend to like this better.
EDIT: To address some comments about accepting a parameter by reference, modifying and returning it. I Agree. An implementation that I would likely prefer would be two sets of functions, one for in place and one which makes a copy. A better set of examples would be:
#include <algorithm>
#include <functional>
#include <cctype>
#include <locale>
// trim from start (in place)
static inline void ltrim(std::string &s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(),
std::not1(std::ptr_fun<int, int>(std::isspace))));
}
// trim from end (in place)
static inline void rtrim(std::string &s) {
s.erase(std::find_if(s.rbegin(), s.rend(),
std::not1(std::ptr_fun<int, int>(std::isspace))).base(), s.end());
}
// trim from both ends (in place)
static inline void trim(std::string &s) {
rtrim(s);
ltrim(s);
}
// trim from start (copying)
static inline std::string ltrim_copy(std::string s) {
ltrim(s);
return s;
}
// trim from end (copying)
static inline std::string rtrim_copy(std::string s) {
rtrim(s);
return s;
}
// trim from both ends (copying)
static inline std::string trim_copy(std::string s) {
trim(s);
return s;
}
I am keeping the original answer above though for context and in the interest of keeping the high voted answer still available.
Using Boost's string algorithms would be easiest:
#include <boost/algorithm/string.hpp>
std::string str("hello world! ");
boost::trim_right(str);
str is now "hello world!". There's also trim_left and trim, which trims both sides.
If you add _copy suffix to any of above function names e.g. trim_copy, the function will return a trimmed copy of the string instead of modifying it through a reference.
If you add _if suffix to any of above function names e.g. trim_copy_if, you can trim all characters satisfying your custom predicate, as opposed to just whitespaces.
What you are doing is fine and robust. I have used the same method for a long time and I have yet to find a faster method:
const char* ws = " \t\n\r\f\v";
// trim from end of string (right)
inline std::string& rtrim(std::string& s, const char* t = ws)
{
s.erase(s.find_last_not_of(t) + 1);
return s;
}
// trim from beginning of string (left)
inline std::string& ltrim(std::string& s, const char* t = ws)
{
s.erase(0, s.find_first_not_of(t));
return s;
}
// trim from both ends of string (right then left)
inline std::string& trim(std::string& s, const char* t = ws)
{
return ltrim(rtrim(s, t), t);
}
By supplying the characters to be trimmed you have the flexibility to trim non-whitespace characters and the efficiency to trim only the characters you want trimmed.
Use the following code to right trim (trailing) spaces and tab characters from std::strings (ideone):
// trim trailing spaces
size_t endpos = str.find_last_not_of(" \t");
size_t startpos = str.find_first_not_of(" \t");
if( std::string::npos != endpos )
{
str = str.substr( 0, endpos+1 );
str = str.substr( startpos );
}
else {
str.erase(std::remove(std::begin(str), std::end(str), ' '), std::end(str));
}
And just to balance things out, I'll include the left trim code too (ideone):
// trim leading spaces
size_t startpos = str.find_first_not_of(" \t");
if( string::npos != startpos )
{
str = str.substr( startpos );
}
Try this, it works for me.
inline std::string trim(std::string& str)
{
str.erase(str.find_last_not_of(' ')+1); //suffixing spaces
str.erase(0, str.find_first_not_of(' ')); //prefixing spaces
return str;
}
Bit late to the party, but never mind. Now C++11 is here, we have lambdas and auto variables. So my version, which also handles all-whitespace and empty strings, is:
#include <cctype>
#include <string>
#include <algorithm>
inline std::string trim(const std::string &s)
{
auto wsfront=std::find_if_not(s.begin(),s.end(),[](int c){return std::isspace(c);});
auto wsback=std::find_if_not(s.rbegin(),s.rend(),[](int c){return std::isspace(c);}).base();
return (wsback<=wsfront ? std::string() : std::string(wsfront,wsback));
}
We could make a reverse iterator from wsfront and use that as the termination condition in the second find_if_not but that's only useful in the case of an all-whitespace string, and gcc 4.8 at least isn't smart enough to infer the type of the reverse iterator (std::string::const_reverse_iterator) with auto. I don't know how expensive constructing a reverse iterator is, so YMMV here. With this alteration, the code looks like this:
inline std::string trim(const std::string &s)
{
auto wsfront=std::find_if_not(s.begin(),s.end(),[](int c){return std::isspace(c);});
return std::string(wsfront,std::find_if_not(s.rbegin(),std::string::const_reverse_iterator(wsfront),[](int c){return std::isspace(c);}).base());
}
http://ideone.com/nFVtEo
std::string trim(const std::string &s)
{
std::string::const_iterator it = s.begin();
while (it != s.end() && isspace(*it))
it++;
std::string::const_reverse_iterator rit = s.rbegin();
while (rit.base() != it && isspace(*rit))
rit++;
return std::string(it, rit.base());
}
I like tzaman's solution, the only problem with it is that it doesn't trim a string containing only spaces.
To correct that 1 flaw, add a str.clear() in between the 2 trimmer lines
std::stringstream trimmer;
trimmer << str;
str.clear();
trimmer >> str;
With C++17 you can use basic_string_view::remove_prefix and basic_string_view::remove_suffix:
std::string_view trim(std::string_view s)
{
s.remove_prefix(std::min(s.find_first_not_of(" \t\r\v\n"), s.size()));
s.remove_suffix(std::min(s.size() - s.find_last_not_of(" \t\r\v\n") - 1, s.size()));
return s;
}
A nice alternative:
std::string_view ltrim(std::string_view s)
{
s.remove_prefix(std::distance(s.cbegin(), std::find_if(s.cbegin(), s.cend(),
[](int c) {return !std::isspace(c);})));
return s;
}
std::string_view rtrim(std::string_view s)
{
s.remove_suffix(std::distance(s.crbegin(), std::find_if(s.crbegin(), s.crend(),
[](int c) {return !std::isspace(c);})));
return s;
}
std::string_view trim(std::string_view s)
{
return ltrim(rtrim(s));
}
In the case of an empty string, your code assumes that adding 1 to string::npos gives 0. string::npos is of type string::size_type, which is unsigned. Thus, you are relying on the overflow behaviour of addition.
s.erase(0, s.find_first_not_of(" \n\r\t"));
s.erase(s.find_last_not_of(" \n\r\t")+1);
Hacked off of Cplusplus.com
std::string choppa(const std::string &t, const std::string &ws)
{
std::string str = t;
size_t found;
found = str.find_last_not_of(ws);
if (found != std::string::npos)
str.erase(found+1);
else
str.clear(); // str is all whitespace
return str;
}
This works for the null case as well. :-)
My solution based on the answer by #Bill the Lizard.
Note that these functions will return the empty string if the input string contains nothing but whitespace.
const std::string StringUtils::WHITESPACE = " \n\r\t";
std::string StringUtils::Trim(const std::string& s)
{
return TrimRight(TrimLeft(s));
}
std::string StringUtils::TrimLeft(const std::string& s)
{
size_t startpos = s.find_first_not_of(StringUtils::WHITESPACE);
return (startpos == std::string::npos) ? "" : s.substr(startpos);
}
std::string StringUtils::TrimRight(const std::string& s)
{
size_t endpos = s.find_last_not_of(StringUtils::WHITESPACE);
return (endpos == std::string::npos) ? "" : s.substr(0, endpos+1);
}
str.erase(0, str.find_first_not_of("\t\n\v\f\r ")); // left trim
str.erase(str.find_last_not_of("\t\n\v\f\r ") + 1); // right trim
Try it online!
With C++11 also came a regular expression module, which of course can be used to trim leading or trailing spaces.
Maybe something like this:
std::string ltrim(const std::string& s)
{
static const std::regex lws{"^[[:space:]]*", std::regex_constants::extended};
return std::regex_replace(s, lws, "");
}
std::string rtrim(const std::string& s)
{
static const std::regex tws{"[[:space:]]*$", std::regex_constants::extended};
return std::regex_replace(s, tws, "");
}
std::string trim(const std::string& s)
{
return ltrim(rtrim(s));
}
My answer is an improvement upon the top answer for this post that trims control characters as well as spaces (0-32 and 127 on the ASCII table).
std::isgraph determines if a character has a graphical representation, so you can use this to alter Evan's answer to remove any character that doesn't have a graphical representation from either side of a string. The result is a much more elegant solution:
#include <algorithm>
#include <functional>
#include <string>
/**
* #brief Left Trim
*
* Trims whitespace from the left end of the provided std::string
*
* #param[out] s The std::string to trim
*
* #return The modified std::string&
*/
std::string& ltrim(std::string& s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(),
std::ptr_fun<int, int>(std::isgraph)));
return s;
}
/**
* #brief Right Trim
*
* Trims whitespace from the right end of the provided std::string
*
* #param[out] s The std::string to trim
*
* #return The modified std::string&
*/
std::string& rtrim(std::string& s) {
s.erase(std::find_if(s.rbegin(), s.rend(),
std::ptr_fun<int, int>(std::isgraph)).base(), s.end());
return s;
}
/**
* #brief Trim
*
* Trims whitespace from both ends of the provided std::string
*
* #param[out] s The std::string to trim
*
* #return The modified std::string&
*/
std::string& trim(std::string& s) {
return ltrim(rtrim(s));
}
Note: Alternatively you should be able to use std::iswgraph if you need support for wide characters, but you will also have to edit this code to enable std::wstring manipulation, which is something that I haven't tested (see the reference page for std::basic_string to explore this option).
This is what I use. Just keep removing space from the front, and then, if there's anything left, do the same from the back.
void trim(string& s) {
while(s.compare(0,1," ")==0)
s.erase(s.begin()); // remove leading whitespaces
while(s.size()>0 && s.compare(s.size()-1,1," ")==0)
s.erase(s.end()-1); // remove trailing whitespaces
}
An elegant way of doing it can be like
std::string & trim(std::string & str)
{
return ltrim(rtrim(str));
}
And the supportive functions are implemented as:
std::string & ltrim(std::string & str)
{
auto it = std::find_if( str.begin() , str.end() , [](char ch){ return !std::isspace<char>(ch , std::locale::classic() ) ; } );
str.erase( str.begin() , it);
return str;
}
std::string & rtrim(std::string & str)
{
auto it = std::find_if( str.rbegin() , str.rend() , [](char ch){ return !std::isspace<char>(ch , std::locale::classic() ) ; } );
str.erase( it.base() , str.end() );
return str;
}
And once you've all these in place, you can write this as well:
std::string trim_copy(std::string const & str)
{
auto s = str;
return ltrim(rtrim(s));
}
Here is a solution for trim with regex
#include <string>
#include <regex>
string trim(string str){
return regex_replace(str, regex("(^[ ]+)|([ ]+$)"),"");
}
I guess if you start asking for the "best way" to trim a string, I'd say a good implementation would be one that:
Doesn't allocate temporary strings
Has overloads for in-place trim and copy trim
Can be easily customized to accept different validation sequences / logic
Obviously there are too many different ways to approach this and it definitely depends on what you actually need. However, the C standard library still has some very useful functions in <string.h>, like memchr. There's a reason why C is still regarded as the best language for IO - its stdlib is pure efficiency.
inline const char* trim_start(const char* str)
{
while (memchr(" \t\n\r", *str, 4)) ++str;
return str;
}
inline const char* trim_end(const char* end)
{
while (memchr(" \t\n\r", end[-1], 4)) --end;
return end;
}
inline std::string trim(const char* buffer, int len) // trim a buffer (input?)
{
return std::string(trim_start(buffer), trim_end(buffer + len));
}
inline void trim_inplace(std::string& str)
{
str.assign(trim_start(str.c_str()),
trim_end(str.c_str() + str.length()));
}
int main()
{
char str [] = "\t \nhello\r \t \n";
string trimmed = trim(str, strlen(str));
cout << "'" << trimmed << "'" << endl;
system("pause");
return 0;
}
For what it's worth, here is a trim implementation with an eye towards performance. It's much quicker than many other trim routines I've seen around. Instead of using iterators and std::finds, it uses raw c strings and indices. It optimizes the following special cases: size 0 string (do nothing), string with no whitespace to trim (do nothing), string with only trailing whitespace to trim (just resize the string), string that's entirely whitespace (just clear the string). And finally, in the worst case (string with leading whitespace), it does its best to perform an efficient copy construction, performing only 1 copy and then moving that copy in place of the original string.
void TrimString(std::string & str)
{
if(str.empty())
return;
const auto pStr = str.c_str();
size_t front = 0;
while(front < str.length() && std::isspace(int(pStr[front]))) {++front;}
size_t back = str.length();
while(back > front && std::isspace(int(pStr[back-1]))) {--back;}
if(0 == front)
{
if(back < str.length())
{
str.resize(back - front);
}
}
else if(back <= front)
{
str.clear();
}
else
{
str = std::move(std::string(str.begin()+front, str.begin()+back));
}
}
Trim C++11 implementation:
static void trim(std::string &s) {
s.erase(s.begin(), std::find_if_not(s.begin(), s.end(), [](char c){ return std::isspace(c); }));
s.erase(std::find_if_not(s.rbegin(), s.rend(), [](char c){ return std::isspace(c); }).base(), s.end());
}
Contributing my solution to the noise. trim defaults to creating a new string and returning the modified one while trim_in_place modifies the string passed to it. The trim function supports c++11 move semantics.
#include <string>
// modifies input string, returns input
std::string& trim_left_in_place(std::string& str) {
size_t i = 0;
while(i < str.size() && isspace(str[i])) { ++i; };
return str.erase(0, i);
}
std::string& trim_right_in_place(std::string& str) {
size_t i = str.size();
while(i > 0 && isspace(str[i - 1])) { --i; };
return str.erase(i, str.size());
}
std::string& trim_in_place(std::string& str) {
return trim_left_in_place(trim_right_in_place(str));
}
// returns newly created strings
std::string trim_right(std::string str) {
return trim_right_in_place(str);
}
std::string trim_left(std::string str) {
return trim_left_in_place(str);
}
std::string trim(std::string str) {
return trim_left_in_place(trim_right_in_place(str));
}
#include <cassert>
int main() {
std::string s1(" \t\r\n ");
std::string s2(" \r\nc");
std::string s3("c \t");
std::string s4(" \rc ");
assert(trim(s1) == "");
assert(trim(s2) == "c");
assert(trim(s3) == "c");
assert(trim(s4) == "c");
assert(s1 == " \t\r\n ");
assert(s2 == " \r\nc");
assert(s3 == "c \t");
assert(s4 == " \rc ");
assert(trim_in_place(s1) == "");
assert(trim_in_place(s2) == "c");
assert(trim_in_place(s3) == "c");
assert(trim_in_place(s4) == "c");
assert(s1 == "");
assert(s2 == "c");
assert(s3 == "c");
assert(s4 == "c");
}
This can be done more simply in C++11 due to the addition of back() and pop_back().
while ( !s.empty() && isspace(s.back()) ) s.pop_back();
Here's what I came up with:
std::stringstream trimmer;
trimmer << str;
trimmer >> str;
Stream extraction eliminates whitespace automatically, so this works like a charm.
Pretty clean and elegant too, if I do say so myself. ;)
I'm not sure if your environment is the same, but in mine, the empty string case will cause the program to abort. I would either wrap that erase call with an if(!s.empty()) or use Boost as already mentioned.
Here is my version:
size_t beg = s.find_first_not_of(" \r\n");
return (beg == string::npos) ? "" : in.substr(beg, s.find_last_not_of(" \r\n") - beg);
Here's a solution easy to understand for beginners not used to write std:: everywhere and not yet familiar with const-correctness, iterators, STL algorithms, etc...
#include <string>
#include <cctype> // for isspace
using namespace std;
// Left trim the given string (" hello! " --> "hello! ")
string left_trim(string str) {
int numStartSpaces = 0;
for (int i = 0; i < str.length(); i++) {
if (!isspace(str[i])) break;
numStartSpaces++;
}
return str.substr(numStartSpaces);
}
// Right trim the given string (" hello! " --> " hello!")
string right_trim(string str) {
int numEndSpaces = 0;
for (int i = str.length() - 1; i >= 0; i--) {
if (!isspace(str[i])) break;
numEndSpaces++;
}
return str.substr(0, str.length() - numEndSpaces);
}
// Left and right trim the given string (" hello! " --> "hello!")
string trim(string str) {
return right_trim(left_trim(str));
}
Hope it helps...
The above methods are great, but sometimes you want to use a combination of functions for what your routine considers to be whitespace. In this case, using functors to combine operations can get messy so I prefer a simple loop I can modify for the trim. Here is a slightly modified trim function copied from the C version here on SO. In this example, I am trimming non alphanumeric characters.
string trim(char const *str)
{
// Trim leading non-letters
while(!isalnum(*str)) str++;
// Trim trailing non-letters
end = str + strlen(str) - 1;
while(end > str && !isalnum(*end)) end--;
return string(str, end+1);
}
What about this...?
#include <iostream>
#include <string>
#include <regex>
std::string ltrim( std::string str ) {
return std::regex_replace( str, std::regex("^\\s+"), std::string("") );
}
std::string rtrim( std::string str ) {
return std::regex_replace( str, std::regex("\\s+$"), std::string("") );
}
std::string trim( std::string str ) {
return ltrim( rtrim( str ) );
}
int main() {
std::string str = " \t this is a test string \n ";
std::cout << "-" << trim( str ) << "-\n";
return 0;
}
Note: I'm still relatively new to C++, so please forgive me if I'm off base here.
Related
I write this split function, can't find easy way to split by string_view(several chars).
My function:
size_t split(std::vector<std::string_view>& result, std::string_view in, char sep) {
result.reserve(std::count(in.begin(), in.end(), in.find(sep) != std::string::npos) + 1);
for (auto pfirst = in.begin();; ++pfirst) {
auto pbefore = pfirst;
pfirst = std::find(pfirst, in.end(), sep);
result.emplace_back(q, pfirst-pbefore);
if (pfirst == in.end())
return result.size();
}
}
I want to call this split function with string_view separator. For example:
str = "apple, phone, bread\n keyboard, computer"
split(result, str, "\n,")
Result:['apple', 'phone', 'bread', 'keyboard', 'computer']
My question is, how can i implement this function as fast as possible?
First, you are using std::count() incorrectly.
Second, std::string_view has its own find_first_of() and substr() methods, which you can use in this situation, instead of using iterators. find_first_of() allows you to specify multiple characters to search for.
Try something more like this:
size_t split(std::vector<std::string_view>& result, std::string_view in, std::string_view seps) {
result.reserve(std::count_if(in.begin(), in.end(), [&](char ch){ return seps.find(ch) != std::string_view::npos; }) + 1);
std::string_view::size_type start = 0, end;
while ((end = in.find_first_of(seps, start)) != std::string_view::npos) {
result.push_back(in.substr(start, end-start));
start = in.find_first_not_of(' ', end+1);
}
if (start != std::string_view::npos)
result.push_back(in.substr(start));
return result.size();
}
Online Demo
This is my take on splitting a string view, just loops once over all the characters in the string view and returns a vector of string_views (so no copying of data)
The calling code can still use words.size() to get the size if needed.
(I use C++20 std::set contains function)
Live demo here : https://onlinegdb.com/tHfPIeo1iM
#include <iostream>
#include <set>
#include <string_view>
#include <vector>
auto split(const std::string_view& string, const std::set<char>& separators)
{
std::vector<std::string_view> words;
auto word_begin{ string.data() };
std::size_t word_len{ 0ul };
for (const auto& c : string)
{
if (!separators.contains(c))
{
word_len++;
}
else
{
// we found a word and not a seperator repeat
if (word_len > 0)
{
words.emplace_back(word_begin, word_len);
word_begin += word_len;
word_len = 0;
}
word_begin++;
}
}
// string_view doesn't have a trailing zero so
// also no trailing separator so if there is still
// a word in the "pipeline" add it too
if (word_len > 0)
{
words.emplace_back(word_begin, word_len);
}
return words;
}
int main()
{
std::set<char> seperators{ ' ', ',', '.', '!', '\n' };
auto words = split("apple, phone, bread\n keyboard, computer", seperators);
bool comma = false;
std::cout << "[";
for (const auto& word : words)
{
if (comma) std::cout << ", ";
std::cout << word;
comma = true;
}
std::cout << "]\n";
return 0;
}
I do not know about performance, but this code seems a lot simpler
std::vector<std::string> ParseDelimited(
const std::string &l, char delim )
{
std::vector<std::string> token;
std::stringstream sst(l);
std::string a;
while (getline(sst, a, delim))
token.push_back(a);
return token;
}
I am processing CSV and using the following code to process a single line.
play with code
std::vector<std::string> string_to_vector(const std::string& s, const char delimiter, const char escape) {
std::stringstream sstr{s};
std::vector<std::string> result;
while (sstr.good()) {
std::string substr;
getline(sstr, substr, delimiter);
while (substr.back() == escape) {
std::string tmp;
getline(sstr, tmp, delimiter);
substr += "," + tmp;
}
result.emplace_back(substr);
}
return result;
}
What it does: Function breaks up string s based on delimiter. If the delimiter is escaped with escape the delimiter will be ignored.
This code works but is super slow. How can I speed it up?
Do you know any existing csv processing implementation that does exactly this and which I could use?
The fastest way to do something is to not do it at all.
If you can ensure that your source string s will outlive the use of the returned vector, you could replace your std::vector<std::string> with std::vector<char*> which would point to the beginning of each substring. You then replace your identified delimiters with zeroes.
[EDIT] I have not moved up to C++17, so no string_view for me :)
NOTE: typical CSV is different from what you imply; it doesn't use escape for the comma, but surrounds entries with comma in it with double quotes. But I assume you know your data.
Implementation:
#include <iostream>
#include <vector>
#include <string>
std::vector<char*> string_to_vector(std::string& s,
const char delimiter, const char escape)
{
size_t prev(0), pos(0), from(0);
std::vector<char*> v;
while ((pos = s.find(delimiter, from)) != s.npos)
{
if (pos == 0 || s[pos - 1] != escape)
{
s[pos] = 0;
v.push_back(&s[prev]);
prev = pos + 1;
}
from = pos + 1;
}
v.push_back(&s[prev]);
return v;
}
int main() {
std::string test("this,is,a\\,test");
std::vector<char*> v = string_to_vector(test, ',', '\\');
for (auto& s : v)
std::cout << s << " ";
}
Java has a convenient split method:
String str = "The quick brown fox";
String[] results = str.split(" ");
Is there an easy way to do this in C++?
The Boost tokenizer class can make this sort of thing quite simple:
#include <iostream>
#include <string>
#include <boost/foreach.hpp>
#include <boost/tokenizer.hpp>
using namespace std;
using namespace boost;
int main(int, char**)
{
string text = "token, test string";
char_separator<char> sep(", ");
tokenizer< char_separator<char> > tokens(text, sep);
BOOST_FOREACH (const string& t, tokens) {
cout << t << "." << endl;
}
}
Updated for C++11:
#include <iostream>
#include <string>
#include <boost/tokenizer.hpp>
using namespace std;
using namespace boost;
int main(int, char**)
{
string text = "token, test string";
char_separator<char> sep(", ");
tokenizer<char_separator<char>> tokens(text, sep);
for (const auto& t : tokens) {
cout << t << "." << endl;
}
}
Here's a real simple one:
#include <vector>
#include <string>
using namespace std;
vector<string> split(const char *str, char c = ' ')
{
vector<string> result;
do
{
const char *begin = str;
while(*str != c && *str)
str++;
result.push_back(string(begin, str));
} while (0 != *str++);
return result;
}
C++ standard library algorithms are pretty universally based around iterators rather than concrete containers. Unfortunately this makes it hard to provide a Java-like split function in the C++ standard library, even though nobody argues that this would be convenient. But what would its return type be? std::vector<std::basic_string<…>>? Maybe, but then we’re forced to perform (potentially redundant and costly) allocations.
Instead, C++ offers a plethora of ways to split strings based on arbitrarily complex delimiters, but none of them is encapsulated as nicely as in other languages. The numerous ways fill whole blog posts.
At its simplest, you could iterate using std::string::find until you hit std::string::npos, and extract the contents using std::string::substr.
A more fluid (and idiomatic, but basic) version for splitting on whitespace would use a std::istringstream:
auto iss = std::istringstream{"The quick brown fox"};
auto str = std::string{};
while (iss >> str) {
process(str);
}
Using std::istream_iterators, the contents of the string stream could also be copied into a vector using its iterator range constructor.
Multiple libraries (such as Boost.Tokenizer) offer specific tokenisers.
More advanced splitting require regular expressions. C++ provides the std::regex_token_iterator for this purpose in particular:
auto const str = "The quick brown fox"s;
auto const re = std::regex{R"(\s+)"};
auto const vec = std::vector<std::string>(
std::sregex_token_iterator{begin(str), end(str), re, -1},
std::sregex_token_iterator{}
);
Another quick way is to use getline. Something like:
stringstream ss("bla bla");
string s;
while (getline(ss, s, ' ')) {
cout << s << endl;
}
If you want, you can make a simple split() method returning a vector<string>, which is
really useful.
Use strtok. In my opinion, there isn't a need to build a class around tokenizing unless strtok doesn't provide you with what you need. It might not, but in 15+ years of writing various parsing code in C and C++, I've always used strtok. Here is an example
char myString[] = "The quick brown fox";
char *p = strtok(myString, " ");
while (p) {
printf ("Token: %s\n", p);
p = strtok(NULL, " ");
}
A few caveats (which might not suit your needs). The string is "destroyed" in the process, meaning that EOS characters are placed inline in the delimter spots. Correct usage might require you to make a non-const version of the string. You can also change the list of delimiters mid parse.
In my own opinion, the above code is far simpler and easier to use than writing a separate class for it. To me, this is one of those functions that the language provides and it does it well and cleanly. It's simply a "C based" solution. It's appropriate, it's easy, and you don't have to write a lot of extra code :-)
You can use streams, iterators, and the copy algorithm to do this fairly directly.
#include <string>
#include <vector>
#include <iostream>
#include <istream>
#include <ostream>
#include <iterator>
#include <sstream>
#include <algorithm>
int main()
{
std::string str = "The quick brown fox";
// construct a stream from the string
std::stringstream strstr(str);
// use stream iterators to copy the stream to the vector as whitespace separated strings
std::istream_iterator<std::string> it(strstr);
std::istream_iterator<std::string> end;
std::vector<std::string> results(it, end);
// send the vector to stdout.
std::ostream_iterator<std::string> oit(std::cout);
std::copy(results.begin(), results.end(), oit);
}
A solution using regex_token_iterators:
#include <iostream>
#include <regex>
#include <string>
using namespace std;
int main()
{
string str("The quick brown fox");
regex reg("\\s+");
sregex_token_iterator iter(str.begin(), str.end(), reg, -1);
sregex_token_iterator end;
vector<string> vec(iter, end);
for (auto a : vec)
{
cout << a << endl;
}
}
No offense folks, but for such a simple problem, you are making things way too complicated. There are a lot of reasons to use Boost. But for something this simple, it's like hitting a fly with a 20# sledge.
void
split( vector<string> & theStringVector, /* Altered/returned value */
const string & theString,
const string & theDelimiter)
{
UASSERT( theDelimiter.size(), >, 0); // My own ASSERT macro.
size_t start = 0, end = 0;
while ( end != string::npos)
{
end = theString.find( theDelimiter, start);
// If at end, use length=maxLength. Else use length=end-start.
theStringVector.push_back( theString.substr( start,
(end == string::npos) ? string::npos : end - start));
// If at end, use start=maxSize. Else use start=end+delimiter.
start = ( ( end > (string::npos - theDelimiter.size()) )
? string::npos : end + theDelimiter.size());
}
}
For example (for Doug's case),
#define SHOW(I,X) cout << "[" << (I) << "]\t " # X " = \"" << (X) << "\"" << endl
int
main()
{
vector<string> v;
split( v, "A:PEP:909:Inventory Item", ":" );
for (unsigned int i = 0; i < v.size(); i++)
SHOW( i, v[i] );
}
And yes, we could have split() return a new vector rather than passing one in. It's trivial to wrap and overload. But depending on what I'm doing, I often find it better to re-use pre-existing objects rather than always creating new ones. (Just as long as I don't forget to empty the vector in between!)
Reference: http://www.cplusplus.com/reference/string/string/.
(I was originally writing a response to Doug's question: C++ Strings Modifying and Extracting based on Separators (closed). But since Martin York closed that question with a pointer over here... I'll just generalize my code.)
Boost has a strong split function: boost::algorithm::split.
Sample program:
#include <vector>
#include <boost/algorithm/string.hpp>
int main() {
auto s = "a,b, c ,,e,f,";
std::vector<std::string> fields;
boost::split(fields, s, boost::is_any_of(","));
for (const auto& field : fields)
std::cout << "\"" << field << "\"\n";
return 0;
}
Output:
"a"
"b"
" c "
""
"e"
"f"
""
This is a simple STL-only solution (~5 lines!) using std::find and std::find_first_not_of that handles repetitions of the delimiter (like spaces or periods for instance), as well leading and trailing delimiters:
#include <string>
#include <vector>
void tokenize(std::string str, std::vector<string> &token_v){
size_t start = str.find_first_not_of(DELIMITER), end=start;
while (start != std::string::npos){
// Find next occurence of delimiter
end = str.find(DELIMITER, start);
// Push back the token found into vector
token_v.push_back(str.substr(start, end-start));
// Skip all occurences of the delimiter to find new start
start = str.find_first_not_of(DELIMITER, end);
}
}
Try it out live!
I know you asked for a C++ solution, but you might consider this helpful:
Qt
#include <QString>
...
QString str = "The quick brown fox";
QStringList results = str.split(" ");
The advantage over Boost in this example is that it's a direct one to one mapping to your post's code.
See more at Qt documentation
Here is a sample tokenizer class that might do what you want
//Header file
class Tokenizer
{
public:
static const std::string DELIMITERS;
Tokenizer(const std::string& str);
Tokenizer(const std::string& str, const std::string& delimiters);
bool NextToken();
bool NextToken(const std::string& delimiters);
const std::string GetToken() const;
void Reset();
protected:
size_t m_offset;
const std::string m_string;
std::string m_token;
std::string m_delimiters;
};
//CPP file
const std::string Tokenizer::DELIMITERS(" \t\n\r");
Tokenizer::Tokenizer(const std::string& s) :
m_string(s),
m_offset(0),
m_delimiters(DELIMITERS) {}
Tokenizer::Tokenizer(const std::string& s, const std::string& delimiters) :
m_string(s),
m_offset(0),
m_delimiters(delimiters) {}
bool Tokenizer::NextToken()
{
return NextToken(m_delimiters);
}
bool Tokenizer::NextToken(const std::string& delimiters)
{
size_t i = m_string.find_first_not_of(delimiters, m_offset);
if (std::string::npos == i)
{
m_offset = m_string.length();
return false;
}
size_t j = m_string.find_first_of(delimiters, i);
if (std::string::npos == j)
{
m_token = m_string.substr(i);
m_offset = m_string.length();
return true;
}
m_token = m_string.substr(i, j - i);
m_offset = j;
return true;
}
Example:
std::vector <std::string> v;
Tokenizer s("split this string", " ");
while (s.NextToken())
{
v.push_back(s.GetToken());
}
pystring is a small library which implements a bunch of Python's string functions, including the split method:
#include <string>
#include <vector>
#include "pystring.h"
std::vector<std::string> chunks;
pystring::split("this string", chunks);
// also can specify a separator
pystring::split("this-string", chunks, "-");
I posted this answer for similar question.
Don't reinvent the wheel. I've used a number of libraries and the fastest and most flexible I have come across is: C++ String Toolkit Library.
Here is an example of how to use it that I've posted else where on the stackoverflow.
#include <iostream>
#include <vector>
#include <string>
#include <strtk.hpp>
const char *whitespace = " \t\r\n\f";
const char *whitespace_and_punctuation = " \t\r\n\f;,=";
int main()
{
{ // normal parsing of a string into a vector of strings
std::string s("Somewhere down the road");
std::vector<std::string> result;
if( strtk::parse( s, whitespace, result ) )
{
for(size_t i = 0; i < result.size(); ++i )
std::cout << result[i] << std::endl;
}
}
{ // parsing a string into a vector of floats with other separators
// besides spaces
std::string s("3.0, 3.14; 4.0");
std::vector<float> values;
if( strtk::parse( s, whitespace_and_punctuation, values ) )
{
for(size_t i = 0; i < values.size(); ++i )
std::cout << values[i] << std::endl;
}
}
{ // parsing a string into specific variables
std::string s("angle = 45; radius = 9.9");
std::string w1, w2;
float v1, v2;
if( strtk::parse( s, whitespace_and_punctuation, w1, v1, w2, v2) )
{
std::cout << "word " << w1 << ", value " << v1 << std::endl;
std::cout << "word " << w2 << ", value " << v2 << std::endl;
}
}
return 0;
}
Adam Pierce's answer provides an hand-spun tokenizer taking in a const char*. It's a bit more problematic to do with iterators because incrementing a string's end iterator is undefined. That said, given string str{ "The quick brown fox" } we can certainly accomplish this:
auto start = find(cbegin(str), cend(str), ' ');
vector<string> tokens{ string(cbegin(str), start) };
while (start != cend(str)) {
const auto finish = find(++start, cend(str), ' ');
tokens.push_back(string(start, finish));
start = finish;
}
Live Example
If you're looking to abstract complexity by using standard functionality, as On Freund suggests strtok is a simple option:
vector<string> tokens;
for (auto i = strtok(data(str), " "); i != nullptr; i = strtok(nullptr, " ")) tokens.push_back(i);
If you don't have access to C++17 you'll need to substitute data(str) as in this example: http://ideone.com/8kAGoa
Though not demonstrated in the example, strtok need not use the same delimiter for each token. Along with this advantage though, there are several drawbacks:
strtok cannot be used on multiple strings at the same time: Either a nullptr must be passed to continue tokenizing the current string or a new char* to tokenize must be passed (there are some non-standard implementations which do support this however, such as: strtok_s)
For the same reason strtok cannot be used on multiple threads simultaneously (this may however be implementation defined, for example: Visual Studio's implementation is thread safe)
Calling strtok modifies the string it is operating on, so it cannot be used on const strings, const char*s, or literal strings, to tokenize any of these with strtok or to operate on a string who's contents need to be preserved, str would have to be copied, then the copy could be operated on
c++20 provides us with split_view to tokenize strings, in a non-destructive manner: https://topanswers.xyz/cplusplus?q=749#a874
The previous methods cannot generate a tokenized vector in-place, meaning without abstracting them into a helper function they cannot initialize const vector<string> tokens. That functionality and the ability to accept any white-space delimiter can be harnessed using an istream_iterator. For example given: const string str{ "The quick \tbrown \nfox" } we can do this:
istringstream is{ str };
const vector<string> tokens{ istream_iterator<string>(is), istream_iterator<string>() };
Live Example
The required construction of an istringstream for this option has far greater cost than the previous 2 options, however this cost is typically hidden in the expense of string allocation.
If none of the above options are flexable enough for your tokenization needs, the most flexible option is using a regex_token_iterator of course with this flexibility comes greater expense, but again this is likely hidden in the string allocation cost. Say for example we want to tokenize based on non-escaped commas, also eating white-space, given the following input: const string str{ "The ,qu\\,ick ,\tbrown, fox" } we can do this:
const regex re{ "\\s*((?:[^\\\\,]|\\\\.)*?)\\s*(?:,|$)" };
const vector<string> tokens{ sregex_token_iterator(cbegin(str), cend(str), re, 1), sregex_token_iterator() };
Live Example
Check this example. It might help you..
#include <iostream>
#include <sstream>
using namespace std;
int main ()
{
string tmps;
istringstream is ("the dellimiter is the space");
while (is.good ()) {
is >> tmps;
cout << tmps << "\n";
}
return 0;
}
If you're using C++ ranges - the full ranges-v3 library, not the limited functionality accepted into C++20 - you could do it this way:
auto results = str | ranges::views::tokenize(" ",1);
... and this is lazily-evaluated. You can alternatively set a vector to this range:
auto results = str | ranges::views::tokenize(" ",1) | ranges::to<std::vector>();
this will take O(m) space and O(n) time if str has n characters making up m words.
See also the library's own tokenization example, here.
MFC/ATL has a very nice tokenizer. From MSDN:
CAtlString str( "%First Second#Third" );
CAtlString resToken;
int curPos= 0;
resToken= str.Tokenize("% #",curPos);
while (resToken != "")
{
printf("Resulting token: %s\n", resToken);
resToken= str.Tokenize("% #",curPos);
};
Output
Resulting Token: First
Resulting Token: Second
Resulting Token: Third
If you're willing to use C, you can use the strtok function. You should pay attention to multi-threading issues when using it.
For simple stuff I just use the following:
unsigned TokenizeString(const std::string& i_source,
const std::string& i_seperators,
bool i_discard_empty_tokens,
std::vector<std::string>& o_tokens)
{
unsigned prev_pos = 0;
unsigned pos = 0;
unsigned number_of_tokens = 0;
o_tokens.clear();
pos = i_source.find_first_of(i_seperators, pos);
while (pos != std::string::npos)
{
std::string token = i_source.substr(prev_pos, pos - prev_pos);
if (!i_discard_empty_tokens || token != "")
{
o_tokens.push_back(i_source.substr(prev_pos, pos - prev_pos));
number_of_tokens++;
}
pos++;
prev_pos = pos;
pos = i_source.find_first_of(i_seperators, pos);
}
if (prev_pos < i_source.length())
{
o_tokens.push_back(i_source.substr(prev_pos));
number_of_tokens++;
}
return number_of_tokens;
}
Cowardly disclaimer: I write real-time data processing software where the data comes in through binary files, sockets, or some API call (I/O cards, camera's). I never use this function for something more complicated or time-critical than reading external configuration files on startup.
You can simply use a regular expression library and solve that using regular expressions.
Use expression (\w+) and the variable in \1 (or $1 depending on the library implementation of regular expressions).
Many overly complicated suggestions here. Try this simple std::string solution:
using namespace std;
string someText = ...
string::size_type tokenOff = 0, sepOff = tokenOff;
while (sepOff != string::npos)
{
sepOff = someText.find(' ', sepOff);
string::size_type tokenLen = (sepOff == string::npos) ? sepOff : sepOff++ - tokenOff;
string token = someText.substr(tokenOff, tokenLen);
if (!token.empty())
/* do something with token */;
tokenOff = sepOff;
}
I thought that was what the >> operator on string streams was for:
string word; sin >> word;
Here's an approach that allows you control over whether empty tokens are included (like strsep) or excluded (like strtok).
#include <string.h> // for strchr and strlen
/*
* want_empty_tokens==true : include empty tokens, like strsep()
* want_empty_tokens==false : exclude empty tokens, like strtok()
*/
std::vector<std::string> tokenize(const char* src,
char delim,
bool want_empty_tokens)
{
std::vector<std::string> tokens;
if (src and *src != '\0') // defensive
while( true ) {
const char* d = strchr(src, delim);
size_t len = (d)? d-src : strlen(src);
if (len or want_empty_tokens)
tokens.push_back( std::string(src, len) ); // capture token
if (d) src += len+1; else break;
}
return tokens;
}
Seems odd to me that with all us speed conscious nerds here on SO no one has presented a version that uses a compile time generated look up table for the delimiter (example implementation further down). Using a look up table and iterators should beat std::regex in efficiency, if you don't need to beat regex, just use it, its standard as of C++11 and super flexible.
Some have suggested regex already but for the noobs here is a packaged example that should do exactly what the OP expects:
std::vector<std::string> split(std::string::const_iterator it, std::string::const_iterator end, std::regex e = std::regex{"\\w+"}){
std::smatch m{};
std::vector<std::string> ret{};
while (std::regex_search (it,end,m,e)) {
ret.emplace_back(m.str());
std::advance(it, m.position() + m.length()); //next start position = match position + match length
}
return ret;
}
std::vector<std::string> split(const std::string &s, std::regex e = std::regex{"\\w+"}){ //comfort version calls flexible version
return split(s.cbegin(), s.cend(), std::move(e));
}
int main ()
{
std::string str {"Some people, excluding those present, have been compile time constants - since puberty."};
auto v = split(str);
for(const auto&s:v){
std::cout << s << std::endl;
}
std::cout << "crazy version:" << std::endl;
v = split(str, std::regex{"[^e]+"}); //using e as delim shows flexibility
for(const auto&s:v){
std::cout << s << std::endl;
}
return 0;
}
If we need to be faster and accept the constraint that all chars must be 8 bits we can make a look up table at compile time using metaprogramming:
template<bool...> struct BoolSequence{}; //just here to hold bools
template<char...> struct CharSequence{}; //just here to hold chars
template<typename T, char C> struct Contains; //generic
template<char First, char... Cs, char Match> //not first specialization
struct Contains<CharSequence<First, Cs...>,Match> :
Contains<CharSequence<Cs...>, Match>{}; //strip first and increase index
template<char First, char... Cs> //is first specialization
struct Contains<CharSequence<First, Cs...>,First>: std::true_type {};
template<char Match> //not found specialization
struct Contains<CharSequence<>,Match>: std::false_type{};
template<int I, typename T, typename U>
struct MakeSequence; //generic
template<int I, bool... Bs, typename U>
struct MakeSequence<I,BoolSequence<Bs...>, U>: //not last
MakeSequence<I-1, BoolSequence<Contains<U,I-1>::value,Bs...>, U>{};
template<bool... Bs, typename U>
struct MakeSequence<0,BoolSequence<Bs...>,U>{ //last
using Type = BoolSequence<Bs...>;
};
template<typename T> struct BoolASCIITable;
template<bool... Bs> struct BoolASCIITable<BoolSequence<Bs...>>{
/* could be made constexpr but not yet supported by MSVC */
static bool isDelim(const char c){
static const bool table[256] = {Bs...};
return table[static_cast<int>(c)];
}
};
using Delims = CharSequence<'.',',',' ',':','\n'>; //list your custom delimiters here
using Table = BoolASCIITable<typename MakeSequence<256,BoolSequence<>,Delims>::Type>;
With that in place making a getNextToken function is easy:
template<typename T_It>
std::pair<T_It,T_It> getNextToken(T_It begin,T_It end){
begin = std::find_if(begin,end,std::not1(Table{})); //find first non delim or end
auto second = std::find_if(begin,end,Table{}); //find first delim or end
return std::make_pair(begin,second);
}
Using it is also easy:
int main() {
std::string s{"Some people, excluding those present, have been compile time constants - since puberty."};
auto it = std::begin(s);
auto end = std::end(s);
while(it != std::end(s)){
auto token = getNextToken(it,end);
std::cout << std::string(token.first,token.second) << std::endl;
it = token.second;
}
return 0;
}
Here is a live example: http://ideone.com/GKtkLQ
I know this question is already answered but I want to contribute. Maybe my solution is a bit simple but this is what I came up with:
vector<string> get_words(string const& text, string const& separator)
{
vector<string> result;
string tmp = text;
size_t first_pos = 0;
size_t second_pos = tmp.find(separator);
while (second_pos != string::npos)
{
if (first_pos != second_pos)
{
string word = tmp.substr(first_pos, second_pos - first_pos);
result.push_back(word);
}
tmp = tmp.substr(second_pos + separator.length());
second_pos = tmp.find(separator);
}
result.push_back(tmp);
return result;
}
Please comment if there is a better approach to something in my code or if something is wrong.
UPDATE: added generic separator
you can take advantage of boost::make_find_iterator. Something similar to this:
template<typename CH>
inline vector< basic_string<CH> > tokenize(
const basic_string<CH> &Input,
const basic_string<CH> &Delimiter,
bool remove_empty_token
) {
typedef typename basic_string<CH>::const_iterator string_iterator_t;
typedef boost::find_iterator< string_iterator_t > string_find_iterator_t;
vector< basic_string<CH> > Result;
string_iterator_t it = Input.begin();
string_iterator_t it_end = Input.end();
for(string_find_iterator_t i = boost::make_find_iterator(Input, boost::first_finder(Delimiter, boost::is_equal()));
i != string_find_iterator_t();
++i) {
if(remove_empty_token){
if(it != i->begin())
Result.push_back(basic_string<CH>(it,i->begin()));
}
else
Result.push_back(basic_string<CH>(it,i->begin()));
it = i->end();
}
if(it != it_end)
Result.push_back(basic_string<CH>(it,it_end));
return Result;
}
Here's my Swiss® Army Knife of string-tokenizers for splitting up strings by whitespace, accounting for single and double-quote wrapped strings as well as stripping those characters from the results. I used RegexBuddy 4.x to generate most of the code-snippet, but I added custom handling for stripping quotes and a few other things.
#include <string>
#include <locale>
#include <regex>
std::vector<std::wstring> tokenize_string(std::wstring string_to_tokenize) {
std::vector<std::wstring> tokens;
std::wregex re(LR"(("[^"]*"|'[^']*'|[^"' ]+))", std::regex_constants::collate);
std::wsregex_iterator next( string_to_tokenize.begin(),
string_to_tokenize.end(),
re,
std::regex_constants::match_not_null );
std::wsregex_iterator end;
const wchar_t single_quote = L'\'';
const wchar_t double_quote = L'\"';
while ( next != end ) {
std::wsmatch match = *next;
const std::wstring token = match.str( 0 );
next++;
if (token.length() > 2 && (token.front() == double_quote || token.front() == single_quote))
tokens.emplace_back( std::wstring(token.begin()+1, token.begin()+token.length()-1) );
else
tokens.emplace_back(token);
}
return tokens;
}
I wrote a simplified version (and maybe a little bit efficient) of https://stackoverflow.com/a/50247503/3976739 for my own use. I hope it would help.
void StrTokenizer(string& source, const char* delimiter, vector<string>& Tokens)
{
size_t new_index = 0;
size_t old_index = 0;
while (new_index != std::string::npos)
{
new_index = source.find(delimiter, old_index);
Tokens.emplace_back(source.substr(old_index, new_index-old_index));
if (new_index != std::string::npos)
old_index = ++new_index;
}
}
If the maximum length of the input string to be tokenized is known, one can exploit this and implement a very fast version. I am sketching the basic idea below, which was inspired by both strtok() and the "suffix array"-data structure described Jon Bentley's "Programming Perls" 2nd edition, chapter 15. The C++ class in this case only gives some organization and convenience of use. The implementation shown can be easily extended for removing leading and trailing whitespace characters in the tokens.
Basically one can replace the separator characters with string-terminating '\0'-characters and set pointers to the tokens withing the modified string. In the extreme case when the string consists only of separators, one gets string-length plus 1 resulting empty tokens. It is practical to duplicate the string to be modified.
Header file:
class TextLineSplitter
{
public:
TextLineSplitter( const size_t max_line_len );
~TextLineSplitter();
void SplitLine( const char *line,
const char sep_char = ',',
);
inline size_t NumTokens( void ) const
{
return mNumTokens;
}
const char * GetToken( const size_t token_idx ) const
{
assert( token_idx < mNumTokens );
return mTokens[ token_idx ];
}
private:
const size_t mStorageSize;
char *mBuff;
char **mTokens;
size_t mNumTokens;
inline void ResetContent( void )
{
memset( mBuff, 0, mStorageSize );
// mark all items as empty:
memset( mTokens, 0, mStorageSize * sizeof( char* ) );
// reset counter for found items:
mNumTokens = 0L;
}
};
Implementattion file:
TextLineSplitter::TextLineSplitter( const size_t max_line_len ):
mStorageSize ( max_line_len + 1L )
{
// allocate memory
mBuff = new char [ mStorageSize ];
mTokens = new char* [ mStorageSize ];
ResetContent();
}
TextLineSplitter::~TextLineSplitter()
{
delete [] mBuff;
delete [] mTokens;
}
void TextLineSplitter::SplitLine( const char *line,
const char sep_char /* = ',' */,
)
{
assert( sep_char != '\0' );
ResetContent();
strncpy( mBuff, line, mMaxLineLen );
size_t idx = 0L; // running index for characters
do
{
assert( idx < mStorageSize );
const char chr = line[ idx ]; // retrieve current character
if( mTokens[ mNumTokens ] == NULL )
{
mTokens[ mNumTokens ] = &mBuff[ idx ];
} // if
if( chr == sep_char || chr == '\0' )
{ // item or line finished
// overwrite separator with a 0-terminating character:
mBuff[ idx ] = '\0';
// count-up items:
mNumTokens ++;
} // if
} while( line[ idx++ ] );
}
A scenario of usage would be:
// create an instance capable of splitting strings up to 1000 chars long:
TextLineSplitter spl( 1000 );
spl.SplitLine( "Item1,,Item2,Item3" );
for( size_t i = 0; i < spl.NumTokens(); i++ )
{
printf( "%s\n", spl.GetToken( i ) );
}
output:
Item1
Item2
Item3
I'm currently using the following code to right-trim all the std::strings in my programs:
std::string s;
s.erase(s.find_last_not_of(" \n\r\t")+1);
It works fine, but I wonder if there are some end-cases where it might fail?
Of course, answers with elegant alternatives and also left-trim solution are welcome.
EDIT Since c++17, some parts of the standard library were removed. Fortunately, starting with c++11, we have lambdas which are a superior solution.
#include <algorithm>
#include <cctype>
#include <locale>
// trim from start (in place)
static inline void ltrim(std::string &s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](unsigned char ch) {
return !std::isspace(ch);
}));
}
// trim from end (in place)
static inline void rtrim(std::string &s) {
s.erase(std::find_if(s.rbegin(), s.rend(), [](unsigned char ch) {
return !std::isspace(ch);
}).base(), s.end());
}
// trim from both ends (in place)
static inline void trim(std::string &s) {
rtrim(s);
ltrim(s);
}
// trim from start (copying)
static inline std::string ltrim_copy(std::string s) {
ltrim(s);
return s;
}
// trim from end (copying)
static inline std::string rtrim_copy(std::string s) {
rtrim(s);
return s;
}
// trim from both ends (copying)
static inline std::string trim_copy(std::string s) {
trim(s);
return s;
}
Thanks to https://stackoverflow.com/a/44973498/524503 for bringing up the modern solution.
Original answer:
I tend to use one of these 3 for my trimming needs:
#include <algorithm>
#include <functional>
#include <cctype>
#include <locale>
// trim from start
static inline std::string <rim(std::string &s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(),
std::not1(std::ptr_fun<int, int>(std::isspace))));
return s;
}
// trim from end
static inline std::string &rtrim(std::string &s) {
s.erase(std::find_if(s.rbegin(), s.rend(),
std::not1(std::ptr_fun<int, int>(std::isspace))).base(), s.end());
return s;
}
// trim from both ends
static inline std::string &trim(std::string &s) {
return ltrim(rtrim(s));
}
They are fairly self-explanatory and work very well.
EDIT: BTW, I have std::ptr_fun in there to help disambiguate std::isspace because there is actually a second definition which supports locales. This could have been a cast just the same, but I tend to like this better.
EDIT: To address some comments about accepting a parameter by reference, modifying and returning it. I Agree. An implementation that I would likely prefer would be two sets of functions, one for in place and one which makes a copy. A better set of examples would be:
#include <algorithm>
#include <functional>
#include <cctype>
#include <locale>
// trim from start (in place)
static inline void ltrim(std::string &s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(),
std::not1(std::ptr_fun<int, int>(std::isspace))));
}
// trim from end (in place)
static inline void rtrim(std::string &s) {
s.erase(std::find_if(s.rbegin(), s.rend(),
std::not1(std::ptr_fun<int, int>(std::isspace))).base(), s.end());
}
// trim from both ends (in place)
static inline void trim(std::string &s) {
rtrim(s);
ltrim(s);
}
// trim from start (copying)
static inline std::string ltrim_copy(std::string s) {
ltrim(s);
return s;
}
// trim from end (copying)
static inline std::string rtrim_copy(std::string s) {
rtrim(s);
return s;
}
// trim from both ends (copying)
static inline std::string trim_copy(std::string s) {
trim(s);
return s;
}
I am keeping the original answer above though for context and in the interest of keeping the high voted answer still available.
Using Boost's string algorithms would be easiest:
#include <boost/algorithm/string.hpp>
std::string str("hello world! ");
boost::trim_right(str);
str is now "hello world!". There's also trim_left and trim, which trims both sides.
If you add _copy suffix to any of above function names e.g. trim_copy, the function will return a trimmed copy of the string instead of modifying it through a reference.
If you add _if suffix to any of above function names e.g. trim_copy_if, you can trim all characters satisfying your custom predicate, as opposed to just whitespaces.
What you are doing is fine and robust. I have used the same method for a long time and I have yet to find a faster method:
const char* ws = " \t\n\r\f\v";
// trim from end of string (right)
inline std::string& rtrim(std::string& s, const char* t = ws)
{
s.erase(s.find_last_not_of(t) + 1);
return s;
}
// trim from beginning of string (left)
inline std::string& ltrim(std::string& s, const char* t = ws)
{
s.erase(0, s.find_first_not_of(t));
return s;
}
// trim from both ends of string (right then left)
inline std::string& trim(std::string& s, const char* t = ws)
{
return ltrim(rtrim(s, t), t);
}
By supplying the characters to be trimmed you have the flexibility to trim non-whitespace characters and the efficiency to trim only the characters you want trimmed.
Use the following code to right trim (trailing) spaces and tab characters from std::strings (ideone):
// trim trailing spaces
size_t endpos = str.find_last_not_of(" \t");
size_t startpos = str.find_first_not_of(" \t");
if( std::string::npos != endpos )
{
str = str.substr( 0, endpos+1 );
str = str.substr( startpos );
}
else {
str.erase(std::remove(std::begin(str), std::end(str), ' '), std::end(str));
}
And just to balance things out, I'll include the left trim code too (ideone):
// trim leading spaces
size_t startpos = str.find_first_not_of(" \t");
if( string::npos != startpos )
{
str = str.substr( startpos );
}
Try this, it works for me.
inline std::string trim(std::string& str)
{
str.erase(str.find_last_not_of(' ')+1); //suffixing spaces
str.erase(0, str.find_first_not_of(' ')); //prefixing spaces
return str;
}
Bit late to the party, but never mind. Now C++11 is here, we have lambdas and auto variables. So my version, which also handles all-whitespace and empty strings, is:
#include <cctype>
#include <string>
#include <algorithm>
inline std::string trim(const std::string &s)
{
auto wsfront=std::find_if_not(s.begin(),s.end(),[](int c){return std::isspace(c);});
auto wsback=std::find_if_not(s.rbegin(),s.rend(),[](int c){return std::isspace(c);}).base();
return (wsback<=wsfront ? std::string() : std::string(wsfront,wsback));
}
We could make a reverse iterator from wsfront and use that as the termination condition in the second find_if_not but that's only useful in the case of an all-whitespace string, and gcc 4.8 at least isn't smart enough to infer the type of the reverse iterator (std::string::const_reverse_iterator) with auto. I don't know how expensive constructing a reverse iterator is, so YMMV here. With this alteration, the code looks like this:
inline std::string trim(const std::string &s)
{
auto wsfront=std::find_if_not(s.begin(),s.end(),[](int c){return std::isspace(c);});
return std::string(wsfront,std::find_if_not(s.rbegin(),std::string::const_reverse_iterator(wsfront),[](int c){return std::isspace(c);}).base());
}
http://ideone.com/nFVtEo
std::string trim(const std::string &s)
{
std::string::const_iterator it = s.begin();
while (it != s.end() && isspace(*it))
it++;
std::string::const_reverse_iterator rit = s.rbegin();
while (rit.base() != it && isspace(*rit))
rit++;
return std::string(it, rit.base());
}
I like tzaman's solution, the only problem with it is that it doesn't trim a string containing only spaces.
To correct that 1 flaw, add a str.clear() in between the 2 trimmer lines
std::stringstream trimmer;
trimmer << str;
str.clear();
trimmer >> str;
With C++17 you can use basic_string_view::remove_prefix and basic_string_view::remove_suffix:
std::string_view trim(std::string_view s)
{
s.remove_prefix(std::min(s.find_first_not_of(" \t\r\v\n"), s.size()));
s.remove_suffix(std::min(s.size() - s.find_last_not_of(" \t\r\v\n") - 1, s.size()));
return s;
}
A nice alternative:
std::string_view ltrim(std::string_view s)
{
s.remove_prefix(std::distance(s.cbegin(), std::find_if(s.cbegin(), s.cend(),
[](int c) {return !std::isspace(c);})));
return s;
}
std::string_view rtrim(std::string_view s)
{
s.remove_suffix(std::distance(s.crbegin(), std::find_if(s.crbegin(), s.crend(),
[](int c) {return !std::isspace(c);})));
return s;
}
std::string_view trim(std::string_view s)
{
return ltrim(rtrim(s));
}
In the case of an empty string, your code assumes that adding 1 to string::npos gives 0. string::npos is of type string::size_type, which is unsigned. Thus, you are relying on the overflow behaviour of addition.
s.erase(0, s.find_first_not_of(" \n\r\t"));
s.erase(s.find_last_not_of(" \n\r\t")+1);
Hacked off of Cplusplus.com
std::string choppa(const std::string &t, const std::string &ws)
{
std::string str = t;
size_t found;
found = str.find_last_not_of(ws);
if (found != std::string::npos)
str.erase(found+1);
else
str.clear(); // str is all whitespace
return str;
}
This works for the null case as well. :-)
My solution based on the answer by #Bill the Lizard.
Note that these functions will return the empty string if the input string contains nothing but whitespace.
const std::string StringUtils::WHITESPACE = " \n\r\t";
std::string StringUtils::Trim(const std::string& s)
{
return TrimRight(TrimLeft(s));
}
std::string StringUtils::TrimLeft(const std::string& s)
{
size_t startpos = s.find_first_not_of(StringUtils::WHITESPACE);
return (startpos == std::string::npos) ? "" : s.substr(startpos);
}
std::string StringUtils::TrimRight(const std::string& s)
{
size_t endpos = s.find_last_not_of(StringUtils::WHITESPACE);
return (endpos == std::string::npos) ? "" : s.substr(0, endpos+1);
}
str.erase(0, str.find_first_not_of("\t\n\v\f\r ")); // left trim
str.erase(str.find_last_not_of("\t\n\v\f\r ") + 1); // right trim
Try it online!
With C++11 also came a regular expression module, which of course can be used to trim leading or trailing spaces.
Maybe something like this:
std::string ltrim(const std::string& s)
{
static const std::regex lws{"^[[:space:]]*", std::regex_constants::extended};
return std::regex_replace(s, lws, "");
}
std::string rtrim(const std::string& s)
{
static const std::regex tws{"[[:space:]]*$", std::regex_constants::extended};
return std::regex_replace(s, tws, "");
}
std::string trim(const std::string& s)
{
return ltrim(rtrim(s));
}
My answer is an improvement upon the top answer for this post that trims control characters as well as spaces (0-32 and 127 on the ASCII table).
std::isgraph determines if a character has a graphical representation, so you can use this to alter Evan's answer to remove any character that doesn't have a graphical representation from either side of a string. The result is a much more elegant solution:
#include <algorithm>
#include <functional>
#include <string>
/**
* #brief Left Trim
*
* Trims whitespace from the left end of the provided std::string
*
* #param[out] s The std::string to trim
*
* #return The modified std::string&
*/
std::string& ltrim(std::string& s) {
s.erase(s.begin(), std::find_if(s.begin(), s.end(),
std::ptr_fun<int, int>(std::isgraph)));
return s;
}
/**
* #brief Right Trim
*
* Trims whitespace from the right end of the provided std::string
*
* #param[out] s The std::string to trim
*
* #return The modified std::string&
*/
std::string& rtrim(std::string& s) {
s.erase(std::find_if(s.rbegin(), s.rend(),
std::ptr_fun<int, int>(std::isgraph)).base(), s.end());
return s;
}
/**
* #brief Trim
*
* Trims whitespace from both ends of the provided std::string
*
* #param[out] s The std::string to trim
*
* #return The modified std::string&
*/
std::string& trim(std::string& s) {
return ltrim(rtrim(s));
}
Note: Alternatively you should be able to use std::iswgraph if you need support for wide characters, but you will also have to edit this code to enable std::wstring manipulation, which is something that I haven't tested (see the reference page for std::basic_string to explore this option).
This is what I use. Just keep removing space from the front, and then, if there's anything left, do the same from the back.
void trim(string& s) {
while(s.compare(0,1," ")==0)
s.erase(s.begin()); // remove leading whitespaces
while(s.size()>0 && s.compare(s.size()-1,1," ")==0)
s.erase(s.end()-1); // remove trailing whitespaces
}
An elegant way of doing it can be like
std::string & trim(std::string & str)
{
return ltrim(rtrim(str));
}
And the supportive functions are implemented as:
std::string & ltrim(std::string & str)
{
auto it = std::find_if( str.begin() , str.end() , [](char ch){ return !std::isspace<char>(ch , std::locale::classic() ) ; } );
str.erase( str.begin() , it);
return str;
}
std::string & rtrim(std::string & str)
{
auto it = std::find_if( str.rbegin() , str.rend() , [](char ch){ return !std::isspace<char>(ch , std::locale::classic() ) ; } );
str.erase( it.base() , str.end() );
return str;
}
And once you've all these in place, you can write this as well:
std::string trim_copy(std::string const & str)
{
auto s = str;
return ltrim(rtrim(s));
}
Here is a solution for trim with regex
#include <string>
#include <regex>
string trim(string str){
return regex_replace(str, regex("(^[ ]+)|([ ]+$)"),"");
}
I guess if you start asking for the "best way" to trim a string, I'd say a good implementation would be one that:
Doesn't allocate temporary strings
Has overloads for in-place trim and copy trim
Can be easily customized to accept different validation sequences / logic
Obviously there are too many different ways to approach this and it definitely depends on what you actually need. However, the C standard library still has some very useful functions in <string.h>, like memchr. There's a reason why C is still regarded as the best language for IO - its stdlib is pure efficiency.
inline const char* trim_start(const char* str)
{
while (memchr(" \t\n\r", *str, 4)) ++str;
return str;
}
inline const char* trim_end(const char* end)
{
while (memchr(" \t\n\r", end[-1], 4)) --end;
return end;
}
inline std::string trim(const char* buffer, int len) // trim a buffer (input?)
{
return std::string(trim_start(buffer), trim_end(buffer + len));
}
inline void trim_inplace(std::string& str)
{
str.assign(trim_start(str.c_str()),
trim_end(str.c_str() + str.length()));
}
int main()
{
char str [] = "\t \nhello\r \t \n";
string trimmed = trim(str, strlen(str));
cout << "'" << trimmed << "'" << endl;
system("pause");
return 0;
}
For what it's worth, here is a trim implementation with an eye towards performance. It's much quicker than many other trim routines I've seen around. Instead of using iterators and std::finds, it uses raw c strings and indices. It optimizes the following special cases: size 0 string (do nothing), string with no whitespace to trim (do nothing), string with only trailing whitespace to trim (just resize the string), string that's entirely whitespace (just clear the string). And finally, in the worst case (string with leading whitespace), it does its best to perform an efficient copy construction, performing only 1 copy and then moving that copy in place of the original string.
void TrimString(std::string & str)
{
if(str.empty())
return;
const auto pStr = str.c_str();
size_t front = 0;
while(front < str.length() && std::isspace(int(pStr[front]))) {++front;}
size_t back = str.length();
while(back > front && std::isspace(int(pStr[back-1]))) {--back;}
if(0 == front)
{
if(back < str.length())
{
str.resize(back - front);
}
}
else if(back <= front)
{
str.clear();
}
else
{
str = std::move(std::string(str.begin()+front, str.begin()+back));
}
}
Trim C++11 implementation:
static void trim(std::string &s) {
s.erase(s.begin(), std::find_if_not(s.begin(), s.end(), [](char c){ return std::isspace(c); }));
s.erase(std::find_if_not(s.rbegin(), s.rend(), [](char c){ return std::isspace(c); }).base(), s.end());
}
Contributing my solution to the noise. trim defaults to creating a new string and returning the modified one while trim_in_place modifies the string passed to it. The trim function supports c++11 move semantics.
#include <string>
// modifies input string, returns input
std::string& trim_left_in_place(std::string& str) {
size_t i = 0;
while(i < str.size() && isspace(str[i])) { ++i; };
return str.erase(0, i);
}
std::string& trim_right_in_place(std::string& str) {
size_t i = str.size();
while(i > 0 && isspace(str[i - 1])) { --i; };
return str.erase(i, str.size());
}
std::string& trim_in_place(std::string& str) {
return trim_left_in_place(trim_right_in_place(str));
}
// returns newly created strings
std::string trim_right(std::string str) {
return trim_right_in_place(str);
}
std::string trim_left(std::string str) {
return trim_left_in_place(str);
}
std::string trim(std::string str) {
return trim_left_in_place(trim_right_in_place(str));
}
#include <cassert>
int main() {
std::string s1(" \t\r\n ");
std::string s2(" \r\nc");
std::string s3("c \t");
std::string s4(" \rc ");
assert(trim(s1) == "");
assert(trim(s2) == "c");
assert(trim(s3) == "c");
assert(trim(s4) == "c");
assert(s1 == " \t\r\n ");
assert(s2 == " \r\nc");
assert(s3 == "c \t");
assert(s4 == " \rc ");
assert(trim_in_place(s1) == "");
assert(trim_in_place(s2) == "c");
assert(trim_in_place(s3) == "c");
assert(trim_in_place(s4) == "c");
assert(s1 == "");
assert(s2 == "c");
assert(s3 == "c");
assert(s4 == "c");
}
This can be done more simply in C++11 due to the addition of back() and pop_back().
while ( !s.empty() && isspace(s.back()) ) s.pop_back();
Here's what I came up with:
std::stringstream trimmer;
trimmer << str;
trimmer >> str;
Stream extraction eliminates whitespace automatically, so this works like a charm.
Pretty clean and elegant too, if I do say so myself. ;)
I'm not sure if your environment is the same, but in mine, the empty string case will cause the program to abort. I would either wrap that erase call with an if(!s.empty()) or use Boost as already mentioned.
Here is my version:
size_t beg = s.find_first_not_of(" \r\n");
return (beg == string::npos) ? "" : in.substr(beg, s.find_last_not_of(" \r\n") - beg);
Here's a solution easy to understand for beginners not used to write std:: everywhere and not yet familiar with const-correctness, iterators, STL algorithms, etc...
#include <string>
#include <cctype> // for isspace
using namespace std;
// Left trim the given string (" hello! " --> "hello! ")
string left_trim(string str) {
int numStartSpaces = 0;
for (int i = 0; i < str.length(); i++) {
if (!isspace(str[i])) break;
numStartSpaces++;
}
return str.substr(numStartSpaces);
}
// Right trim the given string (" hello! " --> " hello!")
string right_trim(string str) {
int numEndSpaces = 0;
for (int i = str.length() - 1; i >= 0; i--) {
if (!isspace(str[i])) break;
numEndSpaces++;
}
return str.substr(0, str.length() - numEndSpaces);
}
// Left and right trim the given string (" hello! " --> "hello!")
string trim(string str) {
return right_trim(left_trim(str));
}
Hope it helps...
The above methods are great, but sometimes you want to use a combination of functions for what your routine considers to be whitespace. In this case, using functors to combine operations can get messy so I prefer a simple loop I can modify for the trim. Here is a slightly modified trim function copied from the C version here on SO. In this example, I am trimming non alphanumeric characters.
string trim(char const *str)
{
// Trim leading non-letters
while(!isalnum(*str)) str++;
// Trim trailing non-letters
end = str + strlen(str) - 1;
while(end > str && !isalnum(*end)) end--;
return string(str, end+1);
}
What about this...?
#include <iostream>
#include <string>
#include <regex>
std::string ltrim( std::string str ) {
return std::regex_replace( str, std::regex("^\\s+"), std::string("") );
}
std::string rtrim( std::string str ) {
return std::regex_replace( str, std::regex("\\s+$"), std::string("") );
}
std::string trim( std::string str ) {
return ltrim( rtrim( str ) );
}
int main() {
std::string str = " \t this is a test string \n ";
std::cout << "-" << trim( str ) << "-\n";
return 0;
}
Note: I'm still relatively new to C++, so please forgive me if I'm off base here.
Java has a convenient split method:
String str = "The quick brown fox";
String[] results = str.split(" ");
Is there an easy way to do this in C++?
The Boost tokenizer class can make this sort of thing quite simple:
#include <iostream>
#include <string>
#include <boost/foreach.hpp>
#include <boost/tokenizer.hpp>
using namespace std;
using namespace boost;
int main(int, char**)
{
string text = "token, test string";
char_separator<char> sep(", ");
tokenizer< char_separator<char> > tokens(text, sep);
BOOST_FOREACH (const string& t, tokens) {
cout << t << "." << endl;
}
}
Updated for C++11:
#include <iostream>
#include <string>
#include <boost/tokenizer.hpp>
using namespace std;
using namespace boost;
int main(int, char**)
{
string text = "token, test string";
char_separator<char> sep(", ");
tokenizer<char_separator<char>> tokens(text, sep);
for (const auto& t : tokens) {
cout << t << "." << endl;
}
}
Here's a real simple one:
#include <vector>
#include <string>
using namespace std;
vector<string> split(const char *str, char c = ' ')
{
vector<string> result;
do
{
const char *begin = str;
while(*str != c && *str)
str++;
result.push_back(string(begin, str));
} while (0 != *str++);
return result;
}
C++ standard library algorithms are pretty universally based around iterators rather than concrete containers. Unfortunately this makes it hard to provide a Java-like split function in the C++ standard library, even though nobody argues that this would be convenient. But what would its return type be? std::vector<std::basic_string<…>>? Maybe, but then we’re forced to perform (potentially redundant and costly) allocations.
Instead, C++ offers a plethora of ways to split strings based on arbitrarily complex delimiters, but none of them is encapsulated as nicely as in other languages. The numerous ways fill whole blog posts.
At its simplest, you could iterate using std::string::find until you hit std::string::npos, and extract the contents using std::string::substr.
A more fluid (and idiomatic, but basic) version for splitting on whitespace would use a std::istringstream:
auto iss = std::istringstream{"The quick brown fox"};
auto str = std::string{};
while (iss >> str) {
process(str);
}
Using std::istream_iterators, the contents of the string stream could also be copied into a vector using its iterator range constructor.
Multiple libraries (such as Boost.Tokenizer) offer specific tokenisers.
More advanced splitting require regular expressions. C++ provides the std::regex_token_iterator for this purpose in particular:
auto const str = "The quick brown fox"s;
auto const re = std::regex{R"(\s+)"};
auto const vec = std::vector<std::string>(
std::sregex_token_iterator{begin(str), end(str), re, -1},
std::sregex_token_iterator{}
);
Another quick way is to use getline. Something like:
stringstream ss("bla bla");
string s;
while (getline(ss, s, ' ')) {
cout << s << endl;
}
If you want, you can make a simple split() method returning a vector<string>, which is
really useful.
Use strtok. In my opinion, there isn't a need to build a class around tokenizing unless strtok doesn't provide you with what you need. It might not, but in 15+ years of writing various parsing code in C and C++, I've always used strtok. Here is an example
char myString[] = "The quick brown fox";
char *p = strtok(myString, " ");
while (p) {
printf ("Token: %s\n", p);
p = strtok(NULL, " ");
}
A few caveats (which might not suit your needs). The string is "destroyed" in the process, meaning that EOS characters are placed inline in the delimter spots. Correct usage might require you to make a non-const version of the string. You can also change the list of delimiters mid parse.
In my own opinion, the above code is far simpler and easier to use than writing a separate class for it. To me, this is one of those functions that the language provides and it does it well and cleanly. It's simply a "C based" solution. It's appropriate, it's easy, and you don't have to write a lot of extra code :-)
You can use streams, iterators, and the copy algorithm to do this fairly directly.
#include <string>
#include <vector>
#include <iostream>
#include <istream>
#include <ostream>
#include <iterator>
#include <sstream>
#include <algorithm>
int main()
{
std::string str = "The quick brown fox";
// construct a stream from the string
std::stringstream strstr(str);
// use stream iterators to copy the stream to the vector as whitespace separated strings
std::istream_iterator<std::string> it(strstr);
std::istream_iterator<std::string> end;
std::vector<std::string> results(it, end);
// send the vector to stdout.
std::ostream_iterator<std::string> oit(std::cout);
std::copy(results.begin(), results.end(), oit);
}
A solution using regex_token_iterators:
#include <iostream>
#include <regex>
#include <string>
using namespace std;
int main()
{
string str("The quick brown fox");
regex reg("\\s+");
sregex_token_iterator iter(str.begin(), str.end(), reg, -1);
sregex_token_iterator end;
vector<string> vec(iter, end);
for (auto a : vec)
{
cout << a << endl;
}
}
No offense folks, but for such a simple problem, you are making things way too complicated. There are a lot of reasons to use Boost. But for something this simple, it's like hitting a fly with a 20# sledge.
void
split( vector<string> & theStringVector, /* Altered/returned value */
const string & theString,
const string & theDelimiter)
{
UASSERT( theDelimiter.size(), >, 0); // My own ASSERT macro.
size_t start = 0, end = 0;
while ( end != string::npos)
{
end = theString.find( theDelimiter, start);
// If at end, use length=maxLength. Else use length=end-start.
theStringVector.push_back( theString.substr( start,
(end == string::npos) ? string::npos : end - start));
// If at end, use start=maxSize. Else use start=end+delimiter.
start = ( ( end > (string::npos - theDelimiter.size()) )
? string::npos : end + theDelimiter.size());
}
}
For example (for Doug's case),
#define SHOW(I,X) cout << "[" << (I) << "]\t " # X " = \"" << (X) << "\"" << endl
int
main()
{
vector<string> v;
split( v, "A:PEP:909:Inventory Item", ":" );
for (unsigned int i = 0; i < v.size(); i++)
SHOW( i, v[i] );
}
And yes, we could have split() return a new vector rather than passing one in. It's trivial to wrap and overload. But depending on what I'm doing, I often find it better to re-use pre-existing objects rather than always creating new ones. (Just as long as I don't forget to empty the vector in between!)
Reference: http://www.cplusplus.com/reference/string/string/.
(I was originally writing a response to Doug's question: C++ Strings Modifying and Extracting based on Separators (closed). But since Martin York closed that question with a pointer over here... I'll just generalize my code.)
Boost has a strong split function: boost::algorithm::split.
Sample program:
#include <vector>
#include <boost/algorithm/string.hpp>
int main() {
auto s = "a,b, c ,,e,f,";
std::vector<std::string> fields;
boost::split(fields, s, boost::is_any_of(","));
for (const auto& field : fields)
std::cout << "\"" << field << "\"\n";
return 0;
}
Output:
"a"
"b"
" c "
""
"e"
"f"
""
This is a simple STL-only solution (~5 lines!) using std::find and std::find_first_not_of that handles repetitions of the delimiter (like spaces or periods for instance), as well leading and trailing delimiters:
#include <string>
#include <vector>
void tokenize(std::string str, std::vector<string> &token_v){
size_t start = str.find_first_not_of(DELIMITER), end=start;
while (start != std::string::npos){
// Find next occurence of delimiter
end = str.find(DELIMITER, start);
// Push back the token found into vector
token_v.push_back(str.substr(start, end-start));
// Skip all occurences of the delimiter to find new start
start = str.find_first_not_of(DELIMITER, end);
}
}
Try it out live!
I know you asked for a C++ solution, but you might consider this helpful:
Qt
#include <QString>
...
QString str = "The quick brown fox";
QStringList results = str.split(" ");
The advantage over Boost in this example is that it's a direct one to one mapping to your post's code.
See more at Qt documentation
Here is a sample tokenizer class that might do what you want
//Header file
class Tokenizer
{
public:
static const std::string DELIMITERS;
Tokenizer(const std::string& str);
Tokenizer(const std::string& str, const std::string& delimiters);
bool NextToken();
bool NextToken(const std::string& delimiters);
const std::string GetToken() const;
void Reset();
protected:
size_t m_offset;
const std::string m_string;
std::string m_token;
std::string m_delimiters;
};
//CPP file
const std::string Tokenizer::DELIMITERS(" \t\n\r");
Tokenizer::Tokenizer(const std::string& s) :
m_string(s),
m_offset(0),
m_delimiters(DELIMITERS) {}
Tokenizer::Tokenizer(const std::string& s, const std::string& delimiters) :
m_string(s),
m_offset(0),
m_delimiters(delimiters) {}
bool Tokenizer::NextToken()
{
return NextToken(m_delimiters);
}
bool Tokenizer::NextToken(const std::string& delimiters)
{
size_t i = m_string.find_first_not_of(delimiters, m_offset);
if (std::string::npos == i)
{
m_offset = m_string.length();
return false;
}
size_t j = m_string.find_first_of(delimiters, i);
if (std::string::npos == j)
{
m_token = m_string.substr(i);
m_offset = m_string.length();
return true;
}
m_token = m_string.substr(i, j - i);
m_offset = j;
return true;
}
Example:
std::vector <std::string> v;
Tokenizer s("split this string", " ");
while (s.NextToken())
{
v.push_back(s.GetToken());
}
pystring is a small library which implements a bunch of Python's string functions, including the split method:
#include <string>
#include <vector>
#include "pystring.h"
std::vector<std::string> chunks;
pystring::split("this string", chunks);
// also can specify a separator
pystring::split("this-string", chunks, "-");
I posted this answer for similar question.
Don't reinvent the wheel. I've used a number of libraries and the fastest and most flexible I have come across is: C++ String Toolkit Library.
Here is an example of how to use it that I've posted else where on the stackoverflow.
#include <iostream>
#include <vector>
#include <string>
#include <strtk.hpp>
const char *whitespace = " \t\r\n\f";
const char *whitespace_and_punctuation = " \t\r\n\f;,=";
int main()
{
{ // normal parsing of a string into a vector of strings
std::string s("Somewhere down the road");
std::vector<std::string> result;
if( strtk::parse( s, whitespace, result ) )
{
for(size_t i = 0; i < result.size(); ++i )
std::cout << result[i] << std::endl;
}
}
{ // parsing a string into a vector of floats with other separators
// besides spaces
std::string s("3.0, 3.14; 4.0");
std::vector<float> values;
if( strtk::parse( s, whitespace_and_punctuation, values ) )
{
for(size_t i = 0; i < values.size(); ++i )
std::cout << values[i] << std::endl;
}
}
{ // parsing a string into specific variables
std::string s("angle = 45; radius = 9.9");
std::string w1, w2;
float v1, v2;
if( strtk::parse( s, whitespace_and_punctuation, w1, v1, w2, v2) )
{
std::cout << "word " << w1 << ", value " << v1 << std::endl;
std::cout << "word " << w2 << ", value " << v2 << std::endl;
}
}
return 0;
}
Adam Pierce's answer provides an hand-spun tokenizer taking in a const char*. It's a bit more problematic to do with iterators because incrementing a string's end iterator is undefined. That said, given string str{ "The quick brown fox" } we can certainly accomplish this:
auto start = find(cbegin(str), cend(str), ' ');
vector<string> tokens{ string(cbegin(str), start) };
while (start != cend(str)) {
const auto finish = find(++start, cend(str), ' ');
tokens.push_back(string(start, finish));
start = finish;
}
Live Example
If you're looking to abstract complexity by using standard functionality, as On Freund suggests strtok is a simple option:
vector<string> tokens;
for (auto i = strtok(data(str), " "); i != nullptr; i = strtok(nullptr, " ")) tokens.push_back(i);
If you don't have access to C++17 you'll need to substitute data(str) as in this example: http://ideone.com/8kAGoa
Though not demonstrated in the example, strtok need not use the same delimiter for each token. Along with this advantage though, there are several drawbacks:
strtok cannot be used on multiple strings at the same time: Either a nullptr must be passed to continue tokenizing the current string or a new char* to tokenize must be passed (there are some non-standard implementations which do support this however, such as: strtok_s)
For the same reason strtok cannot be used on multiple threads simultaneously (this may however be implementation defined, for example: Visual Studio's implementation is thread safe)
Calling strtok modifies the string it is operating on, so it cannot be used on const strings, const char*s, or literal strings, to tokenize any of these with strtok or to operate on a string who's contents need to be preserved, str would have to be copied, then the copy could be operated on
c++20 provides us with split_view to tokenize strings, in a non-destructive manner: https://topanswers.xyz/cplusplus?q=749#a874
The previous methods cannot generate a tokenized vector in-place, meaning without abstracting them into a helper function they cannot initialize const vector<string> tokens. That functionality and the ability to accept any white-space delimiter can be harnessed using an istream_iterator. For example given: const string str{ "The quick \tbrown \nfox" } we can do this:
istringstream is{ str };
const vector<string> tokens{ istream_iterator<string>(is), istream_iterator<string>() };
Live Example
The required construction of an istringstream for this option has far greater cost than the previous 2 options, however this cost is typically hidden in the expense of string allocation.
If none of the above options are flexable enough for your tokenization needs, the most flexible option is using a regex_token_iterator of course with this flexibility comes greater expense, but again this is likely hidden in the string allocation cost. Say for example we want to tokenize based on non-escaped commas, also eating white-space, given the following input: const string str{ "The ,qu\\,ick ,\tbrown, fox" } we can do this:
const regex re{ "\\s*((?:[^\\\\,]|\\\\.)*?)\\s*(?:,|$)" };
const vector<string> tokens{ sregex_token_iterator(cbegin(str), cend(str), re, 1), sregex_token_iterator() };
Live Example
Check this example. It might help you..
#include <iostream>
#include <sstream>
using namespace std;
int main ()
{
string tmps;
istringstream is ("the dellimiter is the space");
while (is.good ()) {
is >> tmps;
cout << tmps << "\n";
}
return 0;
}
If you're using C++ ranges - the full ranges-v3 library, not the limited functionality accepted into C++20 - you could do it this way:
auto results = str | ranges::views::tokenize(" ",1);
... and this is lazily-evaluated. You can alternatively set a vector to this range:
auto results = str | ranges::views::tokenize(" ",1) | ranges::to<std::vector>();
this will take O(m) space and O(n) time if str has n characters making up m words.
See also the library's own tokenization example, here.
MFC/ATL has a very nice tokenizer. From MSDN:
CAtlString str( "%First Second#Third" );
CAtlString resToken;
int curPos= 0;
resToken= str.Tokenize("% #",curPos);
while (resToken != "")
{
printf("Resulting token: %s\n", resToken);
resToken= str.Tokenize("% #",curPos);
};
Output
Resulting Token: First
Resulting Token: Second
Resulting Token: Third
If you're willing to use C, you can use the strtok function. You should pay attention to multi-threading issues when using it.
For simple stuff I just use the following:
unsigned TokenizeString(const std::string& i_source,
const std::string& i_seperators,
bool i_discard_empty_tokens,
std::vector<std::string>& o_tokens)
{
unsigned prev_pos = 0;
unsigned pos = 0;
unsigned number_of_tokens = 0;
o_tokens.clear();
pos = i_source.find_first_of(i_seperators, pos);
while (pos != std::string::npos)
{
std::string token = i_source.substr(prev_pos, pos - prev_pos);
if (!i_discard_empty_tokens || token != "")
{
o_tokens.push_back(i_source.substr(prev_pos, pos - prev_pos));
number_of_tokens++;
}
pos++;
prev_pos = pos;
pos = i_source.find_first_of(i_seperators, pos);
}
if (prev_pos < i_source.length())
{
o_tokens.push_back(i_source.substr(prev_pos));
number_of_tokens++;
}
return number_of_tokens;
}
Cowardly disclaimer: I write real-time data processing software where the data comes in through binary files, sockets, or some API call (I/O cards, camera's). I never use this function for something more complicated or time-critical than reading external configuration files on startup.
You can simply use a regular expression library and solve that using regular expressions.
Use expression (\w+) and the variable in \1 (or $1 depending on the library implementation of regular expressions).
Many overly complicated suggestions here. Try this simple std::string solution:
using namespace std;
string someText = ...
string::size_type tokenOff = 0, sepOff = tokenOff;
while (sepOff != string::npos)
{
sepOff = someText.find(' ', sepOff);
string::size_type tokenLen = (sepOff == string::npos) ? sepOff : sepOff++ - tokenOff;
string token = someText.substr(tokenOff, tokenLen);
if (!token.empty())
/* do something with token */;
tokenOff = sepOff;
}
I thought that was what the >> operator on string streams was for:
string word; sin >> word;
Here's an approach that allows you control over whether empty tokens are included (like strsep) or excluded (like strtok).
#include <string.h> // for strchr and strlen
/*
* want_empty_tokens==true : include empty tokens, like strsep()
* want_empty_tokens==false : exclude empty tokens, like strtok()
*/
std::vector<std::string> tokenize(const char* src,
char delim,
bool want_empty_tokens)
{
std::vector<std::string> tokens;
if (src and *src != '\0') // defensive
while( true ) {
const char* d = strchr(src, delim);
size_t len = (d)? d-src : strlen(src);
if (len or want_empty_tokens)
tokens.push_back( std::string(src, len) ); // capture token
if (d) src += len+1; else break;
}
return tokens;
}
Seems odd to me that with all us speed conscious nerds here on SO no one has presented a version that uses a compile time generated look up table for the delimiter (example implementation further down). Using a look up table and iterators should beat std::regex in efficiency, if you don't need to beat regex, just use it, its standard as of C++11 and super flexible.
Some have suggested regex already but for the noobs here is a packaged example that should do exactly what the OP expects:
std::vector<std::string> split(std::string::const_iterator it, std::string::const_iterator end, std::regex e = std::regex{"\\w+"}){
std::smatch m{};
std::vector<std::string> ret{};
while (std::regex_search (it,end,m,e)) {
ret.emplace_back(m.str());
std::advance(it, m.position() + m.length()); //next start position = match position + match length
}
return ret;
}
std::vector<std::string> split(const std::string &s, std::regex e = std::regex{"\\w+"}){ //comfort version calls flexible version
return split(s.cbegin(), s.cend(), std::move(e));
}
int main ()
{
std::string str {"Some people, excluding those present, have been compile time constants - since puberty."};
auto v = split(str);
for(const auto&s:v){
std::cout << s << std::endl;
}
std::cout << "crazy version:" << std::endl;
v = split(str, std::regex{"[^e]+"}); //using e as delim shows flexibility
for(const auto&s:v){
std::cout << s << std::endl;
}
return 0;
}
If we need to be faster and accept the constraint that all chars must be 8 bits we can make a look up table at compile time using metaprogramming:
template<bool...> struct BoolSequence{}; //just here to hold bools
template<char...> struct CharSequence{}; //just here to hold chars
template<typename T, char C> struct Contains; //generic
template<char First, char... Cs, char Match> //not first specialization
struct Contains<CharSequence<First, Cs...>,Match> :
Contains<CharSequence<Cs...>, Match>{}; //strip first and increase index
template<char First, char... Cs> //is first specialization
struct Contains<CharSequence<First, Cs...>,First>: std::true_type {};
template<char Match> //not found specialization
struct Contains<CharSequence<>,Match>: std::false_type{};
template<int I, typename T, typename U>
struct MakeSequence; //generic
template<int I, bool... Bs, typename U>
struct MakeSequence<I,BoolSequence<Bs...>, U>: //not last
MakeSequence<I-1, BoolSequence<Contains<U,I-1>::value,Bs...>, U>{};
template<bool... Bs, typename U>
struct MakeSequence<0,BoolSequence<Bs...>,U>{ //last
using Type = BoolSequence<Bs...>;
};
template<typename T> struct BoolASCIITable;
template<bool... Bs> struct BoolASCIITable<BoolSequence<Bs...>>{
/* could be made constexpr but not yet supported by MSVC */
static bool isDelim(const char c){
static const bool table[256] = {Bs...};
return table[static_cast<int>(c)];
}
};
using Delims = CharSequence<'.',',',' ',':','\n'>; //list your custom delimiters here
using Table = BoolASCIITable<typename MakeSequence<256,BoolSequence<>,Delims>::Type>;
With that in place making a getNextToken function is easy:
template<typename T_It>
std::pair<T_It,T_It> getNextToken(T_It begin,T_It end){
begin = std::find_if(begin,end,std::not1(Table{})); //find first non delim or end
auto second = std::find_if(begin,end,Table{}); //find first delim or end
return std::make_pair(begin,second);
}
Using it is also easy:
int main() {
std::string s{"Some people, excluding those present, have been compile time constants - since puberty."};
auto it = std::begin(s);
auto end = std::end(s);
while(it != std::end(s)){
auto token = getNextToken(it,end);
std::cout << std::string(token.first,token.second) << std::endl;
it = token.second;
}
return 0;
}
Here is a live example: http://ideone.com/GKtkLQ
I know this question is already answered but I want to contribute. Maybe my solution is a bit simple but this is what I came up with:
vector<string> get_words(string const& text, string const& separator)
{
vector<string> result;
string tmp = text;
size_t first_pos = 0;
size_t second_pos = tmp.find(separator);
while (second_pos != string::npos)
{
if (first_pos != second_pos)
{
string word = tmp.substr(first_pos, second_pos - first_pos);
result.push_back(word);
}
tmp = tmp.substr(second_pos + separator.length());
second_pos = tmp.find(separator);
}
result.push_back(tmp);
return result;
}
Please comment if there is a better approach to something in my code or if something is wrong.
UPDATE: added generic separator
you can take advantage of boost::make_find_iterator. Something similar to this:
template<typename CH>
inline vector< basic_string<CH> > tokenize(
const basic_string<CH> &Input,
const basic_string<CH> &Delimiter,
bool remove_empty_token
) {
typedef typename basic_string<CH>::const_iterator string_iterator_t;
typedef boost::find_iterator< string_iterator_t > string_find_iterator_t;
vector< basic_string<CH> > Result;
string_iterator_t it = Input.begin();
string_iterator_t it_end = Input.end();
for(string_find_iterator_t i = boost::make_find_iterator(Input, boost::first_finder(Delimiter, boost::is_equal()));
i != string_find_iterator_t();
++i) {
if(remove_empty_token){
if(it != i->begin())
Result.push_back(basic_string<CH>(it,i->begin()));
}
else
Result.push_back(basic_string<CH>(it,i->begin()));
it = i->end();
}
if(it != it_end)
Result.push_back(basic_string<CH>(it,it_end));
return Result;
}
Here's my Swiss® Army Knife of string-tokenizers for splitting up strings by whitespace, accounting for single and double-quote wrapped strings as well as stripping those characters from the results. I used RegexBuddy 4.x to generate most of the code-snippet, but I added custom handling for stripping quotes and a few other things.
#include <string>
#include <locale>
#include <regex>
std::vector<std::wstring> tokenize_string(std::wstring string_to_tokenize) {
std::vector<std::wstring> tokens;
std::wregex re(LR"(("[^"]*"|'[^']*'|[^"' ]+))", std::regex_constants::collate);
std::wsregex_iterator next( string_to_tokenize.begin(),
string_to_tokenize.end(),
re,
std::regex_constants::match_not_null );
std::wsregex_iterator end;
const wchar_t single_quote = L'\'';
const wchar_t double_quote = L'\"';
while ( next != end ) {
std::wsmatch match = *next;
const std::wstring token = match.str( 0 );
next++;
if (token.length() > 2 && (token.front() == double_quote || token.front() == single_quote))
tokens.emplace_back( std::wstring(token.begin()+1, token.begin()+token.length()-1) );
else
tokens.emplace_back(token);
}
return tokens;
}
I wrote a simplified version (and maybe a little bit efficient) of https://stackoverflow.com/a/50247503/3976739 for my own use. I hope it would help.
void StrTokenizer(string& source, const char* delimiter, vector<string>& Tokens)
{
size_t new_index = 0;
size_t old_index = 0;
while (new_index != std::string::npos)
{
new_index = source.find(delimiter, old_index);
Tokens.emplace_back(source.substr(old_index, new_index-old_index));
if (new_index != std::string::npos)
old_index = ++new_index;
}
}
If the maximum length of the input string to be tokenized is known, one can exploit this and implement a very fast version. I am sketching the basic idea below, which was inspired by both strtok() and the "suffix array"-data structure described Jon Bentley's "Programming Perls" 2nd edition, chapter 15. The C++ class in this case only gives some organization and convenience of use. The implementation shown can be easily extended for removing leading and trailing whitespace characters in the tokens.
Basically one can replace the separator characters with string-terminating '\0'-characters and set pointers to the tokens withing the modified string. In the extreme case when the string consists only of separators, one gets string-length plus 1 resulting empty tokens. It is practical to duplicate the string to be modified.
Header file:
class TextLineSplitter
{
public:
TextLineSplitter( const size_t max_line_len );
~TextLineSplitter();
void SplitLine( const char *line,
const char sep_char = ',',
);
inline size_t NumTokens( void ) const
{
return mNumTokens;
}
const char * GetToken( const size_t token_idx ) const
{
assert( token_idx < mNumTokens );
return mTokens[ token_idx ];
}
private:
const size_t mStorageSize;
char *mBuff;
char **mTokens;
size_t mNumTokens;
inline void ResetContent( void )
{
memset( mBuff, 0, mStorageSize );
// mark all items as empty:
memset( mTokens, 0, mStorageSize * sizeof( char* ) );
// reset counter for found items:
mNumTokens = 0L;
}
};
Implementattion file:
TextLineSplitter::TextLineSplitter( const size_t max_line_len ):
mStorageSize ( max_line_len + 1L )
{
// allocate memory
mBuff = new char [ mStorageSize ];
mTokens = new char* [ mStorageSize ];
ResetContent();
}
TextLineSplitter::~TextLineSplitter()
{
delete [] mBuff;
delete [] mTokens;
}
void TextLineSplitter::SplitLine( const char *line,
const char sep_char /* = ',' */,
)
{
assert( sep_char != '\0' );
ResetContent();
strncpy( mBuff, line, mMaxLineLen );
size_t idx = 0L; // running index for characters
do
{
assert( idx < mStorageSize );
const char chr = line[ idx ]; // retrieve current character
if( mTokens[ mNumTokens ] == NULL )
{
mTokens[ mNumTokens ] = &mBuff[ idx ];
} // if
if( chr == sep_char || chr == '\0' )
{ // item or line finished
// overwrite separator with a 0-terminating character:
mBuff[ idx ] = '\0';
// count-up items:
mNumTokens ++;
} // if
} while( line[ idx++ ] );
}
A scenario of usage would be:
// create an instance capable of splitting strings up to 1000 chars long:
TextLineSplitter spl( 1000 );
spl.SplitLine( "Item1,,Item2,Item3" );
for( size_t i = 0; i < spl.NumTokens(); i++ )
{
printf( "%s\n", spl.GetToken( i ) );
}
output:
Item1
Item2
Item3