C++-Tokenizer extremely slow - c++

I just don't get what i do wrong. The unicode tokenizer function shown below is very very slow. Maybe somebody can give me a hint how to speed it up?
Thanks for any help. By the way, ustring is Glib::ustring.
sep1 are separators that should not show up in the results
sep2 are separators that should be single tokens in the result
void tokenize(const ustring & u, const ustring & sep1,
const ustring & sep2, vector<ustring> & tokens) {
ustring s;
s.reserve(100);
ostringstream os;
gunichar c;
for (int i = 0; i < u.length(); i++) {
c = u[i];
if (sep1.find(c) != ustring::npos) {
tokens.push_back(s);
s = "";
}
else if (sep2.find(c) != ustring::npos) {
tokens.push_back(s);
s = "";
s.append(1, c);
tokens.push_back(s);
s = "";
}
else {
s.append(1, c);
}
}
if (s!="")
tokens.push_back(s);
}
I now changed it to (now between 1 and 2 seconds) :
ustring s;
s.reserve(100);
ostringstream os;
gunichar c;
set<gunichar> set_sep1;
int i=0;
for (i=0;i<sep1.size();i++)
{
set_sep1.insert(sep1[i]);
}
set<gunichar> set_sep2;
for (i=0;i<sep2.size();i++)
{
set_sep2.insert(sep2[i]);
}
int start_index=-1;
int ulen=u.length();
i=0;
for (ustring::const_iterator it=u.begin();it!=u.end();++it)
{
c=*it;
if (set_sep1.find(c)!=set_sep1.end())
{
if (start_index!=-1 && start_index<i)
tokens.push_back(u.substr(start_index,i-start_index));
start_index=i+1;
s="";
}
else if (set_sep2.find(c)!=set_sep2.end())
{
tokens.push_back(s);
s="";
tokens.push_back(s);
start_index=i+1;
s="";
}
i++;
}
if (start_index!=-1 && start_index<ulen)
tokens.push_back(u.substr(start_index,ulen-start_index));

The to things that may be "very very slow" here are:
ustring::length()
ustring::append()
Random access to ustring via operator[] : e.g. c=u[i];
Try the following:
Instead of calling u.length() in a loop, store the length in a variable and in the loop compare to that variable
Append the characters of the current token to an ostringstream or wostringstream instead of a ustring
Iterate through ustring using iterators instead of indices involving random access.
Example:
for(ustring::const_iterator it = u.cbegin(); it != u.cend(); it++)
{
c = *it;
//implementation follows
}

I think the following will speed up your code significantly, but it's a little work to find out. Currently you are:
iterating over each character of u.
doing a find in sep1 to see if that character is among the separators.
appending one character at a time, as needed.
Assuming your list of separators is smaller than the strings you are parsing, you are probably better off doing the following:
For each seperator, do a find to see if the separator is in the string
If found, append the entire substring in one go, and do a find on the remaing substring.
A second optimization is to order your separators in terms of most likely to succeed. If for e.g. "," is the most commonly use separator, make sure the find runs on that first. If one separator is much hotter than others, this will make a big difference.

Related

string::replace not working correctly 100% of the time?

I'm trying to replace every space character with '%20' in a string, and I'm thinking of using the built in replace function for the string class.
Currently, I have:
void replaceSpace(string& s)
{
int len = s.length();
string str = "%20";
for(int i = 0; i < len; i++) {
if(s[i] == ' ') {
s.replace(i, 1, str);
}
}
}
When I pass in the string "_a_b_c_e_f_g__", where the underscores represent space, my output is "%20a%20b%20c%20e_f_g__". Again, underscores represent space.
Why is that the spaces near the beginning of the string are replaced, but the spaces towards the end aren't?
You are making s longer with each replacement, but you are not updating len which is used in the loop condition.
Modifying the string that you are just scanning is like cutting the branch under your feet. It may work if you are careful, but in this case you aren't.
Namely, you take the string len at the beginning but with each replacement your string gets longer and you are pushing the replacement places further away (so you never reach all of them).
The correct way to cut this branch is from its end (tip) towards the trunk - this way you always have a safe footing:
void replaceSpace(string& s)
{
int len = s.length();
string str = "%20";
for(int i = len - 1; i >= 0; i--) {
if(s[i] == ' ') {
s.replace(i, 1, str);
}
}
}
You're growing the string but only looping to its initial size.
Looping over a collection while modifying it is very prone to error.
Here's a solution that doesn't:
void replace(string& s)
{
string s1;
std::for_each(s.begin(),
s.end(),
[&](char c) {
if (c == ' ') s1 += "%20";
else s1 += c;
});
s.swap(s1);
}
As others have already mentioned, the problem is you're using the initial string length in your loop, but the string gets bigger along the way. Your loop never reaches the end of the string.
You have a number of ways to fix this. You can correct your solution and make sure you go to the end of the string as it is now, not as it was before you started looping.
Or you can use #molbdnilo 's way, which creates a copy of the string along the way.
Or you can use something like this:
std::string input = " a b c e f g ";
std::string::size_type pos = 0;
while ((pos = input.find(' ', pos)) != std::string::npos)
{
input.replace(pos, 1, "%20");
}
Here's a function that can make it easier for you:
string replace_char_str(string str, string find_str, string replace_str)
{
size_t pos = 0;
for ( pos = str.find(find_str); pos != std::string::npos; pos = str.find(find_str,pos) )
{
str.replace(pos ,1, replace_str);
}
return str;
}
So if when you want to replace the spaces, try it like this:
string new_str = replace_char_str(yourstring, " ", "%20");
Hope this helps you ! :)

std::string returning inappropriate value

I wrote a program which perform string compression using counts of repeated characters. The program in C++ is :
#include<iostream>
#include<cstring>
std::string compressBad(std::string str)
{
std::string mystr = "";
int count = 1;
char last = str[0];
for (int i = 0; i < str.length();++i)
{
if(str[i] == last)
count++;
else
{
std::string lastS = last+"";
std::string countS = std::to_string(count);
mystr.append(lastS);
mystr.append(countS);
//mystr = mystr + last + count;
count = 1;
last = str[i];
}
}
std::string lastS = last+"";
std::string countS = std::to_string(count);
mystr.append(lastS);
mystr.append(countS);
return mystr;
//return mystr+last+count;
}
int main()
{
std::string str;
std::getline(std::cin, str);
std::string str2 = compressBad(str);
std::cout<<str2;
/*if (str.length() < str2.length())
std::cout<<str;
else
std::cout<<str2;*/
std::cout<<std::endl;
return 0;
}
Few example on running this are :
Input : sssaaddddd
Output : ùÿÿ*425
Output it should print : s3a2d5
Second example:
Input : sssaaddd
Output: ùÿÿ*423
Output it should print : s3a2d3
I also implemented the same concept in Java and there it is working fine. The java implementation is here
Why is this problem happening with above code.
There may be other issues in your code, but I think that this line might be to blame:
std::string lastS = last+"";
Here, you're trying to convert the character last to a string by concatenating the empty string to the end. Unfortunately, in C++ this is interpreted to mean "take the numeric value of the character last, then add that to a pointer that points to the empty string, producing a new pointer to a character." This pointer points into random memory, hence the garbage you're seeing. (Notice that this is quite different from how Java works!)
Try changing this line to read
std::string lastS(1, last);
This will initialize lastS to be a string consisting of just the character stored in last.
Another option would be to use an ostringstream:
std::ostringstream myStr;
myStr << last << count;
// ...
return myStr.str();
This eliminates all the calls to .append() and std::to_string and is probably a lot easier to read.
last + "" doesn't do what you think.
just do
mystr.append(1, last);

How to validate that there are only digits in a string?

I'm new to C++. I'm working on a project where I need to read mostly integers from the user through the console. In order to avoid someone entering non-digit characters I thought about reading the input as a string, checking there are only digits in it, and then converting it to an integer. I created a function since I need to check for integers several times:
bool isanInt(int *y){
string z;
int x;
getline(cin,z);
for (int n=0; n < z.length(); n++) {
if(!((z[n] >= '0' && z[n] <= '9') || z[n] == ' ') ){
cout << "That is not a valid input!" << endl;
return false;
}
}
istringstream convert(z); //converting the string to integer
convert >> x;
*y = x;
return true;
}
When I need the user to input an integer I'll call this function. But for some reason when I make a call tho this function the program doesn't wait for an input, it jumps immediately to the for-loop processing an empty string. Any thoughts? Thanks for your help.
There are many ways to test a string for only numeric characters. One is
bool is_digits(const std::string &str) {
return str.find_first_not_of("0123456789") == std::string::npos;
}
This would work:
#include <algorithm> // for std::all_of
#include <cctype> // for std::isdigit
bool all_digits(const std::string& s)
{
return std::all_of(s.begin(),
s.end(),
[](char c) { return std::isdigit(c); });
}
You can cast the string in a try/catch block so that if the cast fails you it would raise an exception and you can write whatever you want in the console.
For example:
try
{
int myNum = strtoint(myString);
}
catch (std::bad_cast& bc)
{
std::cerr << "Please insert only numbers "<< '\n';
}
Character-classification is a job typically delegated to the ctype facets of a locale. You're going to need a function that takes into account all 9 digits including the thousands separator and the radix point:
bool is_numeric_string(const std::string& str, std::locale loc = std::locale())
{
using ctype = std::ctype<char>;
using numpunct = std::numpunct<char>;
using traits_type = std::string::traits_type;
auto& ct_f = std::use_facet<ctype>(loc);
auto& np_f = std::use_facet<numpunct>(loc);
return std::all_of(str.begin(), str.end(), [&str, &ct_f, &np_f] (char c)
{
return ct_f.is(std::ctype_base::digit, c) || traits_type::eq(c, np_f.thousands_sep())
|| traits_type::eq(c, np_f.decimal_point());
});
}
Note that extra effort can go into making sure the thousands separator is not the first character.
try another way like cin.getline(str,sizeof(str)), and str here is char*. I think ur problem may be cause by other functions before calling this function. Maybe u can examine other parts of ur codes carefully. Breakpoints setting is recommended too.
Always use off-the-shelf functions. Never write alone.
I recommend
std::regex
Enjoy.

How to replace all occurrences of a character in string?

What is the effective way to replace all occurrences of a character with another character in std::string?
std::string doesn't contain such function but you could use stand-alone replace function from algorithm header.
#include <algorithm>
#include <string>
void some_func() {
std::string s = "example string";
std::replace( s.begin(), s.end(), 'x', 'y'); // replace all 'x' to 'y'
}
The question is centered on character replacement, but, as I found this page very useful (especially Konrad's remark), I'd like to share this more generalized implementation, which allows to deal with substrings as well:
std::string ReplaceAll(std::string str, const std::string& from, const std::string& to) {
size_t start_pos = 0;
while((start_pos = str.find(from, start_pos)) != std::string::npos) {
str.replace(start_pos, from.length(), to);
start_pos += to.length(); // Handles case where 'to' is a substring of 'from'
}
return str;
}
Usage:
std::cout << ReplaceAll(string("Number Of Beans"), std::string(" "), std::string("_")) << std::endl;
std::cout << ReplaceAll(string("ghghjghugtghty"), std::string("gh"), std::string("X")) << std::endl;
std::cout << ReplaceAll(string("ghghjghugtghty"), std::string("gh"), std::string("h")) << std::endl;
Outputs:
Number_Of_Beans
XXjXugtXty
hhjhugthty
EDIT:
The above can be implemented in a more suitable way, in case performance is of your concern, by returning nothing (void) and performing the changes "in-place"; that is, by directly modifying the string argument str, passed by reference instead of by value. This would avoid an extra costly copy of the original string by overwriting it.
Code :
static inline void ReplaceAll2(std::string &str, const std::string& from, const std::string& to)
{
// Same inner code...
// No return statement
}
Hope this will be helpful for some others...
I thought I'd toss in the boost solution as well:
#include <boost/algorithm/string/replace.hpp>
// in place
std::string in_place = "blah#blah";
boost::replace_all(in_place, "#", "#");
// copy
const std::string input = "blah#blah";
std::string output = boost::replace_all_copy(input, "#", "#");
Imagine a large binary blob where all 0x00 bytes shall be replaced by "\1\x30" and all 0x01 bytes by "\1\x31" because the transport protocol allows no \0-bytes.
In cases where:
the replacing and the to-replaced string have different lengths,
there are many occurences of the to-replaced string within the source string and
the source string is large,
the provided solutions cannot be applied (because they replace only single characters) or have a performance problem, because they would call string::replace several times which generates copies of the size of the blob over and over.
(I do not know the boost solution, maybe it is OK from that perspective)
This one walks along all occurrences in the source string and builds the new string piece by piece once:
void replaceAll(std::string& source, const std::string& from, const std::string& to)
{
std::string newString;
newString.reserve(source.length()); // avoids a few memory allocations
std::string::size_type lastPos = 0;
std::string::size_type findPos;
while(std::string::npos != (findPos = source.find(from, lastPos)))
{
newString.append(source, lastPos, findPos - lastPos);
newString += to;
lastPos = findPos + from.length();
}
// Care for the rest after last occurrence
newString += source.substr(lastPos);
source.swap(newString);
}
A simple find and replace for a single character would go something like:
s.replace(s.find("x"), 1, "y")
To do this for the whole string, the easy thing to do would be to loop until your s.find starts returning npos. I suppose you could also catch range_error to exit the loop, but that's kinda ugly.
For completeness, here's how to do it with std::regex.
#include <regex>
#include <string>
int main()
{
const std::string s = "example string";
const std::string r = std::regex_replace(s, std::regex("x"), "y");
}
If you're looking to replace more than a single character, and are dealing only with std::string, then this snippet would work, replacing sNeedle in sHaystack with sReplace, and sNeedle and sReplace do not need to be the same size. This routine uses the while loop to replace all occurrences, rather than just the first one found from left to right.
while(sHaystack.find(sNeedle) != std::string::npos) {
sHaystack.replace(sHaystack.find(sNeedle),sNeedle.size(),sReplace);
}
As Kirill suggested, either use the replace method or iterate along the string replacing each char independently.
Alternatively you can use the find method or find_first_of depending on what you need to do. None of these solutions will do the job in one go, but with a few extra lines of code you ought to make them work for you. :-)
What about Abseil StrReplaceAll? From the header file:
// This file defines `absl::StrReplaceAll()`, a general-purpose string
// replacement function designed for large, arbitrary text substitutions,
// especially on strings which you are receiving from some other system for
// further processing (e.g. processing regular expressions, escaping HTML
// entities, etc.). `StrReplaceAll` is designed to be efficient even when only
// one substitution is being performed, or when substitution is rare.
//
// If the string being modified is known at compile-time, and the substitutions
// vary, `absl::Substitute()` may be a better choice.
//
// Example:
//
// std::string html_escaped = absl::StrReplaceAll(user_input, {
// {"&", "&"},
// {"<", "<"},
// {">", ">"},
// {"\"", """},
// {"'", "'"}});
#include <iostream>
#include <string>
using namespace std;
// Replace function..
string replace(string word, string target, string replacement){
int len, loop=0;
string nword="", let;
len=word.length();
len--;
while(loop<=len){
let=word.substr(loop, 1);
if(let==target){
nword=nword+replacement;
}else{
nword=nword+let;
}
loop++;
}
return nword;
}
//Main..
int main() {
string word;
cout<<"Enter Word: ";
cin>>word;
cout<<replace(word, "x", "y")<<endl;
return 0;
}
Old School :-)
std::string str = "H:/recursos/audio/youtube/libre/falta/";
for (int i = 0; i < str.size(); i++) {
if (str[i] == '/') {
str[i] = '\\';
}
}
std::cout << str;
Result:
H:\recursos\audio\youtube\libre\falta\
For simple situations this works pretty well without using any other library then std::string (which is already in use).
Replace all occurences of character a with character b in some_string:
for (size_t i = 0; i < some_string.size(); ++i) {
if (some_string[i] == 'a') {
some_string.replace(i, 1, "b");
}
}
If the string is large or multiple calls to replace is an issue, you can apply the technique mentioned in this answer: https://stackoverflow.com/a/29752943/3622300
here's a solution i rolled, in a maximal DRI spirit.
it will search sNeedle in sHaystack and replace it by sReplace,
nTimes if non 0, else all the sNeedle occurences.
it will not search again in the replaced text.
std::string str_replace(
std::string sHaystack, std::string sNeedle, std::string sReplace,
size_t nTimes=0)
{
size_t found = 0, pos = 0, c = 0;
size_t len = sNeedle.size();
size_t replen = sReplace.size();
std::string input(sHaystack);
do {
found = input.find(sNeedle, pos);
if (found == std::string::npos) {
break;
}
input.replace(found, len, sReplace);
pos = found + replen;
++c;
} while(!nTimes || c < nTimes);
return input;
}
I think I'd use std::replace_if()
A simple character-replacer (requested by OP) can be written by using standard library functions.
For an in-place version:
#include <string>
#include <algorithm>
void replace_char(std::string& in,
std::string::value_type srch,
std::string::value_type repl)
{
std::replace_if(std::begin(in), std::end(in),
[&srch](std::string::value_type v) { return v==srch; },
repl);
return;
}
and an overload that returns a copy if the input is a const string:
std::string replace_char(std::string const& in,
std::string::value_type srch,
std::string::value_type repl)
{
std::string result{ in };
replace_char(result, srch, repl);
return result;
}
This works! I used something similar to this for a bookstore app, where the inventory was stored in a CSV (like a .dat file). But in the case of a single char, meaning the replacer is only a single char, e.g.'|', it must be in double quotes "|" in order not to throw an invalid conversion const char.
#include <iostream>
#include <string>
using namespace std;
int main()
{
int count = 0; // for the number of occurences.
// final hold variable of corrected word up to the npos=j
string holdWord = "";
// a temp var in order to replace 0 to new npos
string holdTemp = "";
// a csv for a an entry in a book store
string holdLetter = "Big Java 7th Ed,Horstman,978-1118431115,99.85";
// j = npos
for (int j = 0; j < holdLetter.length(); j++) {
if (holdLetter[j] == ',') {
if ( count == 0 )
{
holdWord = holdLetter.replace(j, 1, " | ");
}
else {
string holdTemp1 = holdLetter.replace(j, 1, " | ");
// since replacement is three positions in length,
// must replace new replacement's 0 to npos-3, with
// the 0 to npos - 3 of the old replacement
holdTemp = holdTemp1.replace(0, j-3, holdWord, 0, j-3);
holdWord = "";
holdWord = holdTemp;
}
holdTemp = "";
count++;
}
}
cout << holdWord << endl;
return 0;
}
// result:
Big Java 7th Ed | Horstman | 978-1118431115 | 99.85
Uncustomarily I am using CentOS currently, so my compiler version is below . The C++ version (g++), C++98 default:
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
This is not the only method missing from the standard library, it was intended be low level.
This use case and many other are covered by general libraries such as:
POCO
Abseil
Boost
QtCore
QtCore & QString has my preference: it supports UTF8 and uses less templates, which means understandable errors and faster compilation. It uses the "q" prefix which makes namespaces unnecessary and simplifies headers.
Boost often generates hideous error messages and slow compile time.
POCO seems to be a reasonable compromise.
How about replace any character string with any character string using only good-old C string functions?
char original[256]="First Line\nNext Line\n", dest[256]="";
char* replace_this = "\n"; // this is now a single character but could be any string
char* with_this = "\r\n"; // this is 2 characters but could be of any length
/* get the first token */
char* token = strtok(original, replace_this);
/* walk through other tokens */
while (token != NULL) {
strcat(dest, token);
strcat(dest, with_this);
token = strtok(NULL, replace_this);
}
dest should now have what we are looking for.

C++ Remove new line from multiline string

Whats the most efficient way of removing a 'newline' from a std::string?
#include <algorithm>
#include <string>
std::string str;
str.erase(std::remove(str.begin(), str.end(), '\n'), str.cend());
The behavior of std::remove may not quite be what you'd expect.
A call to remove is typically followed by a call to a container's erase method, which erases the unspecified values and reduces the physical size of the container to match its new logical size.
See an explanation of it here.
If the newline is expected to be at the end of the string, then:
if (!s.empty() && s[s.length()-1] == '\n') {
s.erase(s.length()-1);
}
If the string can contain many newlines anywhere in the string:
std::string::size_type i = 0;
while (i < s.length()) {
i = s.find('\n', i);
if (i == std::string:npos) {
break;
}
s.erase(i);
}
You should use the erase-remove idiom, looking for '\n'. This will work for any standard sequence container; not just string.
Here is one for DOS or Unix new line:
void chomp( string &s)
{
int pos;
if((pos=s.find('\n')) != string::npos)
s.erase(pos);
}
Slight modification on edW's solution to remove all exisiting endline chars
void chomp(string &s){
size_t pos;
while (((pos=s.find('\n')) != string::npos))
s.erase(pos,1);
}
Note that size_t is typed for pos, it is because npos is defined differently for different types, for example, -1 (unsigned int) and -1 (unsigned float) are not the same, due to the fact the max size of each type are different. Therefore, comparing int to size_t might return false even if their values are both -1.
s.erase(std::remove(s.begin(), s.end(), '\n'), s.end());
The code removes all newlines from the string str.
O(N) implementation best served without comments on SO and with comments in production.
unsigned shift=0;
for (unsigned i=0; i<length(str); ++i){
if (str[i] == '\n') {
++shift;
}else{
str[i-shift] = str[i];
}
}
str.resize(str.length() - shift);
std::string some_str = SOME_VAL;
if ( some_str.size() > 0 && some_str[some_str.length()-1] == '\n' )
some_str.resize( some_str.length()-1 );
or (removes several newlines at the end)
some_str.resize( some_str.find_last_not_of(L"\n")+1 );
Another way to do it in the for loop
void rm_nl(string &s) {
for (int p = s.find("\n"); p != (int) string::npos; p = s.find("\n"))
s.erase(p,1);
}
Usage:
string data = "\naaa\nbbb\nccc\nddd\n";
rm_nl(data);
cout << data; // data = aaabbbcccddd
All these answers seem a bit heavy to me.
If you just flat out remove the '\n' and move everything else back a spot, you are liable to have some characters slammed together in a weird-looking way. So why not just do the simple (and most efficient) thing: Replace all '\n's with spaces?
for (int i = 0; i < str.length();i++) {
if (str[i] == '\n') {
str[i] = ' ';
}
}
There may be ways to improve the speed of this at the edges, but it will be way quicker than moving whole chunks of the string around in memory.
If its anywhere in the string than you can't do better than O(n).
And the only way is to search for '\n' in the string and erase it.
for(int i=0;i<s.length();i++) if(s[i]=='\n') s.erase(s.begin()+i);
For more newlines than:
int n=0;
for(int i=0;i<s.length();i++){
if(s[i]=='\n'){
n++;//we increase the number of newlines we have found so far
}else{
s[i-n]=s[i];
}
}
s.resize(s.length()-n);//to delete only once the last n elements witch are now newlines
It erases all the newlines once.
About answer 3 removing only the last \n off string code :
if (!s.empty() && s[s.length()-1] == '\n') {
s.erase(s.length()-1);
}
Will the if condition not fail if the string is really empty ?
Is it not better to do :
if (!s.empty())
{
if (s[s.length()-1] == '\n')
s.erase(s.length()-1);
}
To extend #Greg Hewgill's answer for C++11:
If you just need to delete a newline at the very end of the string:
This in C++98:
if (!s.empty() && s[s.length()-1] == '\n') {
s.erase(s.length()-1);
}
...can now be done like this in C++11:
if (!s.empty() && s.back() == '\n') {
s.pop_back();
}
Optionally, wrap it up in a function. Note that I pass it by ptr here simply so that when you take its address as you pass it to the function, it reminds you that the string will be modified in place inside the function.
void remove_trailing_newline(std::string* str)
{
if (str->empty())
{
return;
}
if (str->back() == '\n')
{
str->pop_back();
}
}
// usage
std::string str = "some string\n";
remove_trailing_newline(&str);
Whats the most efficient way of removing a 'newline' from a std::string?
As far as the most efficient way goes--that I'd have to speed test/profile and see. I'll see if I can get back to you on that and run some speed tests between the top two answers here, and a C-style way like I did here: Removing elements from array in C. I'll use my nanos() timestamp function for speed testing.
Other References:
See these "new" C++11 functions in this reference wiki here: https://en.cppreference.com/w/cpp/string/basic_string
https://en.cppreference.com/w/cpp/string/basic_string/empty
https://en.cppreference.com/w/cpp/string/basic_string/back
https://en.cppreference.com/w/cpp/string/basic_string/pop_back