This question already has answers here:
Most optimized way of concatenation in strings
(9 answers)
Closed 9 years ago.
Consider first that the amount of total data that will be stored in the output string will almost certainly be small and so I doubt any of these have a noticeable affect on performance. My primary goal is to find a way to concatenate a series of const char*'s of unknown size that doesn't look terrible while also keeping efficiency in mind. Below are the results of my search:
Method 1:
std::string str = std::string(array1) + array2 + array3;
Method 2:
std::string str(array1);
str += array2;
str += array3;
I decided on the first method as it is short and concise. If I'm not mistaken, both methods will invoke the same series of operations. the unoptimized compiler would first create a temporary string and internally allocate some amount of space for its buffer >= sizeof(array1). If that buffer is sufficiently large, the additional + operations will not require any new allocations. Finally, if move semantics are supported, then the buffers of the temporary and named str are swapped.
Are there any other ways to perform such an operation that also look nice and don't incur terrible overhead?
Remember, that, in case of arrays, sizeof(array) returns actual size (aka length) of it's parameter, if it has been declared as an array of explicit size (and you wrote 'series of const char*'s of unknown size'). So, assuming you want to create universal solution, strlen() should come under consideration instead.
I don't think you can avoid all additional operations. In case of many concatenations, the best solution would be to allocate buffer, that is large enough to store all concatenated strings.
We can easily deduce, that the most optimal version of append() in this case is:
string& append (const char* s, size_t n);
Why? Because reference says: 'If s does not point to an array long enough (...), it causes undefined behavior'. So we can assume, that internally no additional checks take place (especially additional strlen() calls). Which is good, since you are completely sure, that values passed to append() are correct and you can avoid unnecesary overhead.
Now, the actual concatenation can be done like this:
len_1 = strlen(array_1);
len_2 = strlen(array_2);
len_3 = strlen(array_3);
//Preallocate enough space for all arrays. Only one reallocation takes place.
target_string.reserve(len_1 + len_2 + len_3);
target_string.append(array_1, len_1);
target_string.append(array_2, len_2);
target_string.append(array_3, len_3);
I do not know if this solution 'looks good' in your opinion, but it's definitely clear and is optimized for this use case.
Related
Is there a STL/library method to reduce the string size (trim it) in constant time.
In C this this can be done in constant time by just adding the '\0' past the last index.
C++ resize compexity is undefined and mostly likely be O(N)
http://www.cplusplus.com/reference/string/string/resize/
#SamVarshavchik is being coy in the comments, but it's worth spelling out: in many implementations, including libstdc++, std::string::resize() will reduce the size of a string in constant time, by reducing the length of the string and not reallocating/copying the data:
https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/include/bits/basic_string.tcc . The estimate of O(n) is if you increase the size, forcing the string to be reallocated and copied over.
Alternatively, in C++17, std::string_view is rough immutable counterpart to "trimming" a C string by slapping a null byte. It takes a slice of a string (pointer + size) without copying its content, and you can then pass it around to various STL functions or print it from a stream.
std::string hello_world = "hello world";
auto hello = std::string_view(hello_world.data(), 5);
std::cout << hello; // prints hello
The caveat is that the string_view doesn't own the data, so you can't use it once the original string goes out of scope, and you don't want to modify the original string in a way that might cause it to reallocate.
The C++17 way, we can achieve the substr operation in O(1).
https://www.modernescpp.com/index.php/c-17-avoid-copying-with-std-string-view
std::string_view do not allocate the memory on heap for large string as well.
std::string allocate memory on heap but exception is for std::string size is 15 for MSVC and GCC and 23 for Clang. std::string below above mentioned size are not allocated memory on heap.
I created a mathematical vector class, and because it could be called 60 times a second in real time, I'm concerned that this algorithm is too slow. It basically grabs (x,y) in vector format and returns the string. Is there a faster way to do the same thing?
I researched that placing 'static' before the buffer array is one way. I also don't like the binary copy of the returned string (or assumedly the copy constructor). It does need to return std::string.
std::string Vector2D::toString() const
{
char buffer[20];
snprintf(buffer, 20, "(%.02f, %.02f)", m_x, m_y);
return std::string(buffer, strlen(buffer));
}
I agree with the commenters that performance is unlikely to be a measurable problem for you -- at least not on any modern computer -- but if you want to reduce overhead anyway, just an exercise, you can change the method to write into a caller-supplied char-array rather than returning a std::string:
void Vector2D::writeToCharBuffer(char * buffer) const
{
snprintf(buffer, 20, "(%.02f, %.02f)", m_x, m_y);
}
This avoids the call to strlen(), the need for copying the chars from buffer into a std:string object's internal array, and perhaps the need for the std:string to allocate (and later free) a buffer from the heap, thus increasing your program's efficiency by some negligible amount.
The downside is that it is harder for the caller to use correctly, e.g. instead of
std::string s = vec.toString();
he now has to do something like:
char tempBuf[20];
vec.writeToCharBuffer(tempBuf);
... and God help him if he forgets that the function needs a buffer of at least 20 bytes:
char tempBuf[10]; // oopsie
vec.writeToCharBuffer(tempBuf); // buffer overflow -> undefined behavior, chaos and despair!
You can save the strlen call, by using the return value from snprintf.
You can potentially save the copy into the string buffer, by preallocating the string to the buffer size, calling snprintf directly into the string's storage, and then resizing to the return value of snprintf.
You can do slightly better than that by calling some dedicated number formatting functions (along the lines of itoa, but not exactly itoa, because that doesn't do floating-point or accept a precision) to fill the buffer, instead of snprintf which has to process a control string.
None of this will matter if you call this function only a few thousand times. You need to be calling it tens of millions of times in order for it to be a noticeable difference.
This really is a question just for my own interest I haven't been able to determine through the documentation.
I see on http://www.cplusplus.com/reference/string/string/ that append has complexity:
"Unspecified, but generally up to linear in the new string length."
while push_back() has complexity:
"Unspecified; Generally amortized constant, but up to linear in the new string length."
As a toy example, suppose I wanted to append the characters "foo" to a string. Would
myString.push_back('f');
myString.push_back('o');
myString.push_back('o');
and
myString.append("foo");
amount to exactly the same thing? Or is there any difference? You might figure that append would be more efficient because the compiler would know how much memory is required to extend the string the specified number of characters, while push_back may need to secure memory each call?
In C++03 (for which most of "cplusplus.com"'s documentation is written), the complexities were unspecified because library implementers were allowed to do Copy-On-Write or "rope-style" internal representations for strings. For instance, a COW implementation might require copying the entire string if a character is modified and there is sharing going on.
In C++11, COW and rope implementations are banned. You should expect constant amortized time per character added or linear amortized time in the number of characters added for appending to a string at the end. Implementers may still do relatively crazy things with strings (in comparison to, say std::vector), but most implementations are going to be limited to things like the "small string optimization".
In comparing push_back and append, push_back deprives the underlying implementation of potentially useful length information which it might use to preallocate space. On the other hand, append requires that an implementation walk over the input twice in order to find that length, so the performance gain or loss is going to depend on a number of unknowable factors such as the length of the string before you attempt the append. That said, the difference is probably extremely Extremely EXTREMELY small. Go with append for this -- it is far more readable.
I had the same doubt, so I made a small test to check this (g++ 4.8.5 with C++11 profile on Linux, Intel, 64 bit under VmWare Fusion).
And the result is interesting:
push :19
append :21
++++ :34
Could be possible this is because of the string length (big), but the operator + is very expensive compared with the push_back and the append.
Also it is interesting that when the operator only receives a character (not a string), it behaves very similar to the push_back.
For not to depend on pre-allocated variables, each cycle is defined in a different scope.
Note : the vCounter simply uses gettimeofday to compare the differences.
TimeCounter vCounter;
{
string vTest;
vCounter.start();
for (int vIdx=0;vIdx<1000000;vIdx++) {
vTest.push_back('a');
vTest.push_back('b');
vTest.push_back('c');
}
vCounter.stop();
cout << "push :" << vCounter.elapsed() << endl;
}
{
string vTest;
vCounter.start();
for (int vIdx=0;vIdx<1000000;vIdx++) {
vTest.append("abc");
}
vCounter.stop();
cout << "append :" << vCounter.elapsed() << endl;
}
{
string vTest;
vCounter.start();
for (int vIdx=0;vIdx<1000000;vIdx++) {
vTest += 'a';
vTest += 'b';
vTest += 'c';
}
vCounter.stop();
cout << "++++ :" << vCounter.elapsed() << endl;
}
Add one more opinion here.
I personally consider it better to use push_back() when adding characters one by one from another string. For instance:
string FilterAlpha(const string& s) {
string new_s;
for (auto& it: s) {
if (isalpha(it)) new_s.push_back(it);
}
return new_s;
}
If using append()here, I would replace push_back(it) with append(1,it), which is not that readable to me.
Yes, I would also expect append() to perform better for the reasons you gave, and in a situation where you need to append a string, using append() (or operator+=) is certainly preferable (not least also because the code is much more readable).
But what the Standard specifies is the complexity of the operation. And that is generally linear even for append(), because ultimately each character of the string being appended (and possible all characters, if reallocation occurs) needs to be copied (this is true even if memcpy or similar are used).
I'm finding standard string addition to be very slow so I'm looking for some tips/hacks that can speed up some code I have.
My code is basically structured as follows:
inline void add_to_string(string data, string &added_data) {
if(added_data.length()<1) added_data = added_data + "{";
added_data = added_data+data;
}
int main()
{
int some_int = 100;
float some_float = 100.0;
string some_string = "test";
string added_data;
added_data.reserve(1000*64);
for(int ii=0;ii<1000;ii++)
{
//variables manipulated here
some_int = ii;
some_float += ii;
some_string.assign(ii%20,'A');
//then we concatenate the strings!
stringstream fragment;
fragment<<some_int <<","<<some_float<<","<<some_string;
add_to_string(fragment.str(),added_data);
}
return;
}
Doing some basic profiling, I'm finding that a ton of time is being used in the for loop. Are there some things I can do that will significantly speed this up? Will it help to use c strings instead of c++ strings?
String addition is not the problem you are facing. std::stringstream is known to be slow due to it's design. On every iteration of your for-loop the stringstream is responsible for at least 2 allocations and 2 deletions. The cost of each of these 4 operations is likely more than that of the string addition.
Profile the following and measure the difference:
std::string stringBuffer;
for(int ii=0;ii<1000;ii++)
{
//variables manipulated here
some_int = ii;
some_float += ii;
some_string.assign(ii%20,'A');
//then we concatenate the strings!
char buffer[128];
sprintf(buffer, "%i,%f,%s",some_int,some_float,some_string.c_str());
stringBuffer = buffer;
add_to_string(stringBuffer ,added_data);
}
Ideally, replace sprintf with _snprintf or the equivalent supported by your compiler.
As a rule of thumb, use stringstream for formatting by default and switch to the faster and less safe functions like sprintf, itoa, etc. whenever performance matters.
Edit: that, and what didierc said: added_data += data;
You can save lots of string operations if you do not call add_to_string in your loop.
I believe this does the same (although I am not a C++ expert and do not know exactly what stringstream does):
stringstream fragment;
for(int ii=0;ii<1000;ii++)
{
//variables manipulated here
some_int = ii;
some_float += ii;
some_string.assign(ii%20,'A');
//then we concatenate the strings!
fragment<<some_int<<","<<some_float<<","<<some_string;
}
// inlined add_to_string call without the if-statement ;)
added_data = "{" + fragment.str();
I see you used the reserve method on added_data, which should help by avoiding multiple reallocations of the string as it grows.
You should also use the += string operator where possible:
added_data += data;
I think that the above should save up some significant time by avoiding unecessary copies back and forth of added_data in a temporary string when doing the catenation.
This += operator is a simpler version of the string::append method, it just copies data directly at the end of added_data. Since you made the reserve, that operation alone should be very fast (almost equivalent to a strcpy).
But why going through all this, when you are already using a stringstream to handle input? Keep it all in there to begin with!
The stringstream class is indeed not very efficient.
You may have a look at the stringstream class for more information on how to use it, if necessary, but your solution of using a string as a buffer seems to avoid that class speed issue.
At any rate, stay away from any attempt at reimplementing the speed critical code in pure C unless you really know what you are doing. Some other SO posts support the idea of doing it,, but I think it's best (read safer) to rely as much as possible on the standard library, which will be enhanced over time, and take care of many corner cases you (or I) wouldn't think of. If your input data format is set in stone, then you might start thinking about taking that road, but otherwise it's premature optimization.
If you start added_data with a "{", you would be able to remove the if from your add_to_string method: the if gets executed exactly once, when the string is empty, so you might as well make it non-empty right away.
In addition, your add_to_string makes a copy of the data; this is not necessary, because it does not get modified. Accepting the data by const reference should speed things up for you.
Finally, changing your added_data from string to sstream should let you append to it in a loop, without the sstream intermediary that gets created, copied, and thrown away on each iteration of the loop.
Please have a look at Twine used in LLVM.
A Twine is a kind of rope, it represents a concatenated string using a
binary-tree, where the string is the preorder of the nodes. Since the
Twine can be efficiently rendered into a buffer when its result is used,
it avoids the cost of generating temporary values for intermediate string
results -- particularly in cases when the Twine result is never
required. By explicitly tracking the type of leaf nodes, we can also avoid
the creation of temporary strings for conversions operations (such as
appending an integer to a string).
It may helpful in solving your problem.
How about this approach?
This is a DevPartner for MSVC 2010 report.
string newstring = stringA & stringB;
i dont think strings are slow, its the conversions that can make it slow
and maybe your compiler that might check variable types for mismatches.
Is it possible to somehow adapt a c-style string/buffer (char* or wchar_t*) to work with the Boost String Algorithms Library?
That is, for example, it's trimalgorithm has the following declaration:
template<typename SequenceT>
void trim(SequenceT &, const std::locale & = std::locale());
and the implementation (look for trim_left_if) requires that the sequence type has a member function erase.
How could I use that with a raw character pointer / c string buffer?
char* pStr = getSomeCString(); // example, could also be something like wchar_t buf[256];
...
boost::trim(pStr); // HOW?
Ideally, the algorithms would work directly on the supplied buffer. (As far as possible. it obviously can't work if an algorithm needs to allocate additional space in the "string".)
#Vitaly asks: why can't you create a std::string from char buffer and then use it in algorithms?
The reason I have char* at all is that I'd like to use a few algorthims on our existing codebase. Refactoring all the char buffers to string would be more work than it's worth, and when changing or adapting something it would be nice to just be able to apply a given algorithm to any c-style string that happens to live in the current code.
Using a string would mean to (a) copy char* to string, (b) apply algorithm to string and (c) copy string back into char buffer.
For the SequenceT-type operations, you probably have to use std::string. If you wanted to implement that by yourself, you'd have to fulfill many more requirements for creation, destruction, value semantics etc. You'd basically end up with your implementation of std::string.
The RangeT-type operations might be, however, usable on char*s using the iterator_range from Boost.Range library. I didn't try it, though.
There exist some code which implements a std::string like string with a fixed buffer. With some tinkering you can modify this code to create a string type which uses an external buffer:
char buffer[100];
strcpy(buffer, " HELLO ");
xstr::xstring<xstr::fixed_char_buf<char> >
str(buffer, strlen(buffer), sizeof(buffer));
boost::algorithm::trim(str);
buffer[str.size()] = 0;
std::cout << buffer << std::endl; // prints "HELLO"
For this I added an constructor to xstr::xstring and xstr::fixed_char_buf to take the buffer, the size of the buffer which is in use and the maximum size of the buffer. Further I replaced the SIZE template argument with a member variable and changed the internal char array into a char pointer.
The xstr code is a bit old and will not compile without trouble on newer compilers but it needs some minor changes. Further I only added the things needed in this case. If you want to use this for real, you need to make some more changes to make sure it can not use uninitialized memory.
Anyway, it might be a good start for writing you own string adapter.
I don't know what platform you're targeting, but on most modern computers (including mobile ones like ARM) memory copy is so fast you shouldn't even waste your time optimizing memory copies. I say - wrap char* in std::string and check whether the performance suits your needs. Don't waste time on premature optimization.