I'm working in a codebase with a mixture of CString, const char* and std::string (non-unicode), where all new code uses std::string exclusively. I've now had to do the following:
{
CString tempstring;
load_cstring_legacy_method(tempstring);
stdstring = tempstring;
}
and worry about performance. The strings are DNA sequences so we can easily have 100+ of them with each of them ~3M characters. Note that adjusting load_cstring_legacy_method is not an option. I did a quick test:
// 3M
const int stringsize = 3000000;
const int repeat = 1000;
std::chrono::steady_clock::time_point startTime = std::chrono::steady_clock::now();
for ( int i = 0; i < repeat; ++i ){
CString cstring('A', stringsize);
std::string stdstring(cstring); // Comment out
cstring.Empty();
}
std::cout << std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - startTime).count() << " ms" << std::endl;
and commenting out the std::string gives 850 ms, with the assignment its 3600 ms. The magnitude of the difference is suprising so I guess the benchmark might not be doing what I expect. Assuming there is a penalty, is there a way I can avoid it?
So your question is to make the std::string construction faster?
On my machine, comparing this
std::string stdstring(cstring); // 4741 ms
I get better performance this way:
std::string stdstring(cstring, stringsize); // 3419 ms
or if the std::string already exists like the first part of your question suggests:
stdstring.assign(cstring, stringsize); // 3408 ms
Use a more efficient memory allocator. Something like a memory arena/region would substantially help with allocation costs.
If you're really, really desperate, you could theoretically combine ReleaseBuffer with some hideous allocator hacks to avoid the copy altogether. This would involve a lot of pain, though.
In addition, if you have a serious problem, you could consider changing your string implementation. The std::string that ships with Visual Studio employs SSO, or Small String Optimization. This does exactly what it sounds like- it optimizes very small strings, which are quite common all around but not necessarily good for this use case. Another implementation like COW could be more appropriate (be super careful if doing so in a multi-threaded environment).
Finally, if you're using an old version of VS, you should also consider upgrading. Move semantics are a huge instawin as far as performance goes.
CString is probably the Unicode version, which explains the slowness. The generic conversion routine cannot know assume that the characters used are limited to "ACGT".
You can, however, and shamelessly take advantage of that.
{
CString tempstring;
load_cstring_legacy_method(tempstring);
int len = tempstring.GetLength();
stdstring.reserve(len);
for(int i = 0; i != len; ++i)
{
stdstring.push_back(static_cast<char>(tempstring[i]));
}
}
Portable? Only so far as CString is, so Windows variants.
Related
I want to create a function that will take a string and an integer as parameters and return a string that contains the string parameter repeated the given number of times.
For example:
std::string MakeDuplicate( const std::string& str, int x )
{
...
}
Calling MakeDuplicate( "abc", 3 ); would return "abcabcabc".
I know I can do this just by looping x number of times but I'm sure there must be a better way.
I don't see a problem with looping, just make sure you do a reserve first:
std::string MakeDuplicate( const std::string& str, int x )
{
std::string newstr;
newstr.reserve(str.length()*x); // prevents multiple reallocations
// loop...
return newstr;
}
At some point it will have to be a loop. You may be able to hide the looping in some fancy language idiom, but ultimately you're going to have to loop.
For small 'x' simple loop is your friend. For large 'x and relatively short 'str' we can think of a "smarter" solution by reusing already concatenated string.
std::string MakeDuplicate( const std::string& str, unsigned int x ) {
std::string newstr;
if (x>0) {
unsigned int y = 2;
newstr.reserve(str.length()*x);
newstr.append(str);
while (y<x) {
newstr.append(newstr);
y*=2;
}
newstr.append(newstr.c_str(), (x-y/2)*str.length());
}
return newstr;
}
Or something like that :o) (I think it can be written in a nicer way but idea is there).
EDIT: I was intersted myself and did some tests comparing three solutions on my notebook with visual studio (reuse version, simple loop with preallocation, simple copy&loop-1 without preallocation). Results as expected: for small x(<10) preallocation version is generally fastest, no preallocation was tiny bit slower, for larger x speedup of 'reuse' version is really significant (log n vs n complexity). Nice, I just can't think of any real problem that could use it :o)
There is an alternative to a loop, its called recursion, and of recursion tail-recursion is the nicest variety since you can theoretically do it till the end of time -- just like a loop :D
p.s., tail-recursion is often syntactic sugar for a loop -- however in the case of procedural languages (C++), the compiler is generally at loss, so the tail-recursion is not optimised and you might run out of memory (but if you wrote a recursion that runs out of memory than you have bigger problems) :D
more downvotes please !!
recursion is obviously not a construct used in computer science for the same job as looping
For example:
char s[]="abcde";
for(int i=0;i<strlen(s);i++)
cout<<s+i<<endl;
Output:
abcde
bcde
cde
de
e
And this doesn't ruin the char array.
What I want to ask for help is, is there another way we can do that with the std::string class?
Cheap and nasty way, closest to what you already have: use std::string::c_str()
std::string const s{"abcde"};
char const *c = s.c_str();
for (std::size_t i=0; i<s.size(); ++i)
std::cout << c+i << '\n';
The more string-like way is to use std::string::substr:
std::string const s{"abcde"};
for (std::size_t i=0; i<s.size(); ++i)
std::cout << s.substr(i) << '\n';
Note that the second approach does involve allocating memory for the new string every time, but I think here the benefit of clarity (using a function specifically named and designed for what you are trying to do), and avoiding raw pointers (often a cause of confusion for beginners) both outweigh the performance (non-)issue here.
A more modern approach that combines the clarity of using specific C++ components designed for what you want, with avoiding extra allocations, is to use std::experimental::string_view, which is designed to wrap an existing string of characters:
std::string const s{"abcde"};
std::experimental::string_view sv{s};
for (std::size_t i=0; i<s.size(); ++i) {
std::cout << sv << '\n';
sv.remove_prefix(1);
}
std::experimental::string_view::remove_prefix does what you might expect: shrinks the view of the string it is wrapping at the front, by the number specified.
string_view is not technically part of The Standard yet, so I don't really recommend it for production use, since you won't get good cross-platform support. But it's quite fun to learn about. You might consider the very similar (not coincidentally) component, boost::string_ref to be production quality.
Also, please reconsider your use of what are often considered bad practices: using namespace std; and endl (those are links to explanations).
I am working on a embedded SW project. A lot of strings are stored inside flash memory. I would use these strings (usually const char* or const wchar*) as std::string's data. That means I want to avoid creating a copy of the original data because of memory restrictions.
An extended use might be to read the flash data via stringstream directly out of the flash memory.
Example which unfortunately is not working in place:
const char* flash_adr = 0x00300000;
size_t length = 3000;
std::string str(flash_adr, length);
Any ideas will be appreciated!
If you are willing to go with compiler and library specific implementations, here is an example that works in MSVC 2013.
#include <iostream>
#include <string>
int main() {
std::string str("A std::string with a larger length than yours");
char *flash_adr = "Your source from the flash";
char *reset_adr = str._Bx._Ptr; // Keep the old address around
// Change the inner buffer
(char*)str._Bx._Ptr = flash_adr;
std::cout << str << std::endl;
// Reset the pointer or the program will crash
(char*)str._Bx._Ptr = reset_adr;
return 0;
}
It will print Your source from the flash.
The idea is to reserve a std::string capable of fitting the strings in your flash and keep on changing its inner buffer pointer.
You need to customize this for your compiler and as always, you need to be very very careful.
I have now used string_span described in CPP Core Guidelines (https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md). GSL provides a complete implementation (GSL: Guidelines Support Library https://github.com/Microsoft/GSL).
If you know the address of your string inside flash memory you can just use the address directly with the following constructor to create a string_span.
constexpr basic_string_span(pointer ptr, size_type length) noexcept
: span_(ptr, length)
{}
std::string_view might have done the same job as Captain Obvlious (https://stackoverflow.com/users/845568/captain-obvlious) commented as my favourite comment.
I am quite happy with the solution. It works good from performance side including providing a good readability.
I have a simple scenario. I need to join two C-strings together into a std::string. I have decided to do this in one of two ways:
Solution 1
void ProcessEvent(char const* pName) {
std::string fullName;
fullName.reserve(50); // Ensure minimal reallocations for small event names (50 is an arbitrary limit).
fullName += "com.domain.events.";
fullName += pName;
// Use fullName as needed
}
Solution 2
void ProcessEvent(char const* pName) {
std::ostringstream ss;
ss << "com.domain.events." << pName;
std::string fullName{ss.str()};
// Use fullName as needed
}
I like solution 2 better because the code is more natural. Solution 1 seems like a response to a measurable bottleneck from performance testing. However, Solution 1 exists for 2 reasons:
It's a light optimization to reduce allocations. Event management in this application is used quite frequently so there might be benefits (but no measurements have been taken).
I've heard criticism regarding STL streams WRT performance. Some have recommended to only use stringstream when doing heavy string building, especially those involving number conversions and/or usage of manipulators.
Is it a premature pessimization to prefer solution 2 for its simplicity? Or is it a premature optimization to choose solution 1? I'm wondering if I'm too overly concerned about STL streams.
Let's measure it
A quick test with the following functions:
void func1(const char* text) {
std::string s;
s.reserve(50);
s += "com.domain.event.";
s += text;
}
void func2(const char* text) {
std::ostringstream oss;
oss << "com.domain.event." << text;
std::string s = oss.str();
}
Running each 100 000 times in a loop gives the following results on average on my computer (using gcc-4.9.1):
func1 : 37 milliseconds
func2 : 87 milliseconds
That is, func1 is more than twice as fast.
That being said, I would recommend using the clearest most readable syntax until you really need the performance. Implement a testable program first, then optimize if its too slow.
Edit:
As suggested by #Ken P:
void func3(const char* text) {
std::string s = "com.domain.event" + std::string{text};
}
func3 : 27 milliseconds
The simplest solution is often the fastest.
You didn't mention the 3rd alternative of not pre-allocating anything at all in the string and just let the optimizer do what it's best at.
Given these two functions, func1 and func3:
void func1(const char* text) {
std::string s;
s.reserve(50);
s += "com.domain.event.";
s += text;
std::cout << s;
}
void func3(const char* text) {
std::string s;
s += "com.domain.event.";
s += text;
std::cout << s;
}
It can be seen in the example at http://goo.gl/m8h2Ks that the gcc assembly for func1 just for reserving space for 50 characters will add an additional 3 instructions compared to when no pre-allocation is done in func3. One of the calls is a string append call, which in turn will give some overhead:
leaq 16(%rsp), %rdi
movl $50, %esi
call std::basic_string<char>::append(char const*, unsigned long)
Looking at the code alone doesn't guarantee that func3 is faster than func1 though, just because it has fewer instructions. Cache and other things also contributes to the actual performance, which can only be properly assessed by measuring, as others pointed out.
I'm finding standard string addition to be very slow so I'm looking for some tips/hacks that can speed up some code I have.
My code is basically structured as follows:
inline void add_to_string(string data, string &added_data) {
if(added_data.length()<1) added_data = added_data + "{";
added_data = added_data+data;
}
int main()
{
int some_int = 100;
float some_float = 100.0;
string some_string = "test";
string added_data;
added_data.reserve(1000*64);
for(int ii=0;ii<1000;ii++)
{
//variables manipulated here
some_int = ii;
some_float += ii;
some_string.assign(ii%20,'A');
//then we concatenate the strings!
stringstream fragment;
fragment<<some_int <<","<<some_float<<","<<some_string;
add_to_string(fragment.str(),added_data);
}
return;
}
Doing some basic profiling, I'm finding that a ton of time is being used in the for loop. Are there some things I can do that will significantly speed this up? Will it help to use c strings instead of c++ strings?
String addition is not the problem you are facing. std::stringstream is known to be slow due to it's design. On every iteration of your for-loop the stringstream is responsible for at least 2 allocations and 2 deletions. The cost of each of these 4 operations is likely more than that of the string addition.
Profile the following and measure the difference:
std::string stringBuffer;
for(int ii=0;ii<1000;ii++)
{
//variables manipulated here
some_int = ii;
some_float += ii;
some_string.assign(ii%20,'A');
//then we concatenate the strings!
char buffer[128];
sprintf(buffer, "%i,%f,%s",some_int,some_float,some_string.c_str());
stringBuffer = buffer;
add_to_string(stringBuffer ,added_data);
}
Ideally, replace sprintf with _snprintf or the equivalent supported by your compiler.
As a rule of thumb, use stringstream for formatting by default and switch to the faster and less safe functions like sprintf, itoa, etc. whenever performance matters.
Edit: that, and what didierc said: added_data += data;
You can save lots of string operations if you do not call add_to_string in your loop.
I believe this does the same (although I am not a C++ expert and do not know exactly what stringstream does):
stringstream fragment;
for(int ii=0;ii<1000;ii++)
{
//variables manipulated here
some_int = ii;
some_float += ii;
some_string.assign(ii%20,'A');
//then we concatenate the strings!
fragment<<some_int<<","<<some_float<<","<<some_string;
}
// inlined add_to_string call without the if-statement ;)
added_data = "{" + fragment.str();
I see you used the reserve method on added_data, which should help by avoiding multiple reallocations of the string as it grows.
You should also use the += string operator where possible:
added_data += data;
I think that the above should save up some significant time by avoiding unecessary copies back and forth of added_data in a temporary string when doing the catenation.
This += operator is a simpler version of the string::append method, it just copies data directly at the end of added_data. Since you made the reserve, that operation alone should be very fast (almost equivalent to a strcpy).
But why going through all this, when you are already using a stringstream to handle input? Keep it all in there to begin with!
The stringstream class is indeed not very efficient.
You may have a look at the stringstream class for more information on how to use it, if necessary, but your solution of using a string as a buffer seems to avoid that class speed issue.
At any rate, stay away from any attempt at reimplementing the speed critical code in pure C unless you really know what you are doing. Some other SO posts support the idea of doing it,, but I think it's best (read safer) to rely as much as possible on the standard library, which will be enhanced over time, and take care of many corner cases you (or I) wouldn't think of. If your input data format is set in stone, then you might start thinking about taking that road, but otherwise it's premature optimization.
If you start added_data with a "{", you would be able to remove the if from your add_to_string method: the if gets executed exactly once, when the string is empty, so you might as well make it non-empty right away.
In addition, your add_to_string makes a copy of the data; this is not necessary, because it does not get modified. Accepting the data by const reference should speed things up for you.
Finally, changing your added_data from string to sstream should let you append to it in a loop, without the sstream intermediary that gets created, copied, and thrown away on each iteration of the loop.
Please have a look at Twine used in LLVM.
A Twine is a kind of rope, it represents a concatenated string using a
binary-tree, where the string is the preorder of the nodes. Since the
Twine can be efficiently rendered into a buffer when its result is used,
it avoids the cost of generating temporary values for intermediate string
results -- particularly in cases when the Twine result is never
required. By explicitly tracking the type of leaf nodes, we can also avoid
the creation of temporary strings for conversions operations (such as
appending an integer to a string).
It may helpful in solving your problem.
How about this approach?
This is a DevPartner for MSVC 2010 report.
string newstring = stringA & stringB;
i dont think strings are slow, its the conversions that can make it slow
and maybe your compiler that might check variable types for mismatches.