I generate a set of data files. As the files are supposed to be readable, they text files (opposed to binary files).
To output information to my files, I used very comfortable std::ofstream object.
In the beginning, when the data to be exported was smaller, the time needed to write to the files was not noticeable. However, as the information to be exported has accumulated, it takes now around 5 minutes to generate them.
As I started being bothered by waiting, my question is obvious: Is there any faster alternative to std::ofstream, please? In case there is faster alternative, will it be worth of rewritting my application? In other words, could the time saved be +50%? Thank you.
Update:
I was asked to show you my code that generates the above files, so here you are - the most time consuming loop:
ofstream fout;
fout.open(strngCollectiveSourceFileName,ios::out);
fout << "#include \"StdAfx.h\"" << endl;
fout << "#include \"Debug.h\"" << endl;
fout << "#include \"glm.hpp\"" << endl;
fout << "#include \"" << strngCollectiveHeaderFileName.substr( strngCollectiveHeaderFileName.rfind(TEXT("\\")) + 1) << "\"" << endl << endl;
fout << "using namespace glm;" << endl << endl << endl;
for (unsigned int nSprite = 0; nSprite < vpTilesetSprites.size(); nSprite++ )
{
for(unsigned int nFrameSet = 0; nFrameSet < vpTilesetSprites[nSprite]->vpFrameSets.size(); nFrameSet++)
{
// display index definition
fout << "// Index Definition: " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetLongDescription() << "\n";
string strngIndexSignature = strngIndexDefinitionSignature;
strngIndexSignature.replace(strngIndexSignature.find(TEXT("#aIndexArrayName#")), strlen(TEXT("#aIndexArrayName#")), TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("IndexData") );
strngIndexSignature.replace(strngIndexSignature.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName );
fout << strngIndexSignature << "[4] = {0, 1, 2, 3};\t\t" << "// " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << ": Index Definition\n\n";
// display vertex definition
fout << "// Vertex Definition: " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetLongDescription() << "\n";
string strngVertexSignature = strngVertexDefinitionSignature;
strngVertexSignature.replace(strngVertexSignature.find(TEXT("#aVertexArrayName#")), strlen(TEXT("#aVertexArrayName#")), TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("VertexData") );
strngVertexSignature.replace(strngVertexSignature.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName );
fout << strngVertexSignature << "[" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFramesCount() << "] =\n";
fout << "{\n";
for (int nFrameNo = 0; nFrameNo < vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFramesCount(); nFrameNo++)
{
fout << "\t" << "{{ vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vTextureUV.fv << "f) }, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 1: vec4(x, y, z, w), vec2(u, v) \n";
fout << "\t" << " { vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vTextureUV.fv << "f) }, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 2: vec4(x, y, z, w), vec2(u, v) \n";
fout << "\t" << " { vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vTextureUV.fv << "f) }, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 3: vec4(x, y, z, w), vec2(u, v) \n";
fout << "\t" << " { vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vTextureUV.fv << "f) }}, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 4: vec4(x, y, z, w), vec2(u, v) \n\n";
}
fout << "};\n\n\n\n";
}
}
fout.close();
If you don't want to use C file I/O then you can give a try to; FastFormat. Look at the comparison for more info.
How are vpTilesetSprites and vpTilesetSprites[nSprite] stored? Are they implemented with lists or arrays? There is a lot of indexed access to them, and if they are list-like structures, you'll spend a lot of extra time following needless pointers. Ed S.'s comment is right: giving the long indexed temporary variables and linebreaks could make it easier to read, and maybe faster, too:
fout << "// Index Definition: " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetLongDescription() << "\n";
string strngIndexSignature = strngIndexDefinitionSignature;
strngIndexSignature.replace(strngIndexSignature.find(TEXT("#aIndexArrayName#")), strlen(TEXT("#aIndexArrayName#")), TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("IndexData") );
strngIndexSignature.replace(strngIndexSignature.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName );
vs
string idxsig = strngIndexDefinitionSignature;
sprite sp = vpTilesetSprites[nSprite];
frameset fs = sp->vpFrameSets[nFrameSet];
fout << "// Index Definition: " << fs->GetLongDescription() << "\n";
idxsig.replace(idxsig.find(TEXT("#aIndexArrayName#")), strlen(TEXT("#aIndexArrayName#")),
TEXT("a") + sp->GetObjectName() + fs->getFrameSetName() + TEXT("IndexData"));
idxsig.replace(idxsig.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")),
strngCollectiveArrayClassName);
But, the much bigger problem is how you're using strings as templates; you're looking for a given text string (and computing the length of your needle string every single time you need it!) over and over again.
Consider this: You're performing the find and replace operations nSprite * nFrameSet times. Each time through, this loop:
makes a copy of strngIndexDefinitionSignature
creates four temporary string objects when concatenating static and dynamic strings
compute strlen(TEXT("#ClassName#"))
compute strlen(TEXT("#aIndexArrayName#"))
find start point of both
replace both texts with new texts
And that's just the first four lines of your loop.
Can you replace your strngIndexDefinitionSignature with a format string? I assume it currently looks like this:
"flubber #aIndexArrayName# { blubber } #ClassName# blorp"
If you re-write it like this:
"flubber a %s%sIndexData { blubber } %s blorp"
Then your two find and replace lines can be replaced with:
sprintf(out, index_def_sig, sp->GetObjectName(), fs->getFrameSetName(),
strngCollectiveArrayClassName);
This would remove two find() operations, two replace() operations, creating and destroying four temporary string objects, a string duplicate that was promptly over-written with two replace() calls, and two strlen() operations that return the same result every time (but aren't actually needed anyway).
You can then output your string with << as usual. Or, you can change sprintf(3) to fprintf(3), and avoid even the temporary C string.
Assuming you do it in large enough chunks, calling write() directly might be faster; that said, it's more likely that your biggest bottleneck doesn't have anything directly to do with std::ofstream. The most obvious thing is to make sure you aren't using std::endl (because flushing the stream frequently will kill performance). Beyond that, I would suggest profiling your app to see where it's actually spending the time.
The performance of ostream is probably not your actual issue; I suggest using a profiler to determine where your real bottlenecks are. If ostream turns out to be your actual problem, you can drop down to <cstdio> and use fprintf(FILE*, const char*, ...) for formatted output to a file handle.
The best answer will depend on what sort of text you are generating, and how you are generating it. C++ streams can be slow, but that mostly is because they can also do a lot more for you, such as locale-dependent formatting, and so on.
You may find speed ups with streams by bypassing some of the formatting (eg. ostream::write), or by writing characters directly to a streambuf instead (streambuf::sputn). Sometimes increasing the buffer size on the relevant streambuf helps (via streambuf::pubsetbuf).
If this isn't good enough, you might want to try C-style stdio files, eg fopen, fprintf, etc. It takes a little while to get used to the way the text is formatted if you're not used to that method but the performance is usually pretty good.
For the absolute top performance you usually have to go to OS-specific routines. Sometimes the direct low-level file routines are significantly better than the C stdio, but sometimes not - for example, I've seen some people say WriteFile on Win32 is the fastest method on Windows, whereas some Google hits report it as being slower than stdio. Another approach might be a memory-mapped file, eg. mmap + msync - this essentially uses your system memory as the disk and writes the actual data to disk in large blocks, which is likely to be near optimal. However you run the risk of losing all the data if you incur a crash half way for some reason, which may or may not be a problem for you.
Related
Consider this scenario.
I'm creating a struct(flatbuffers::Table) using CreateXXX generated code. This creates the struct on the FlatBuffer buffer and gives me the offset.
Then I can get the memory block with GetBufferPointer() and transfer it.
Reversely, if I get a memory block, I can use GetXXX to get my struct(flatbuffers::Table) from that.
But after I get this struct, if I need to serialize it again, how can I do it? After the serialization, I should be able to transfer that data and do GetXXX on that data like before.
flatbuffers::Parser parser;
parser.Parse(schema.c_str());
parser.SetRootType("license");
parser.Parse(j.c_str());
auto* buf = parser.builder_.GetBufferPointer();
auto li = flatbuffers::GetRoot<license>(buf);
std::cout << "ID: " << li->id()->c_str() << " Rand: " << li->rand()->c_str() << " Secret: " << li->secret()->c_str() << std::endl;
uint8_t* buf2 = ????????????
auto li2 = flatbuffers::GetRoot<license>(buf2);
std::cout << "ID: " << li2->id()->c_str() << " Rand: " << li2->rand()->c_str() << " Secret: " << li2->secret()->c_str() << std::endl;
The obvious answer is that you keep the original buffer pointer (and size) around. Then you can "re-serialize" it by just writing out the existing buffer.
There is a function GetBufferStartFromRootPointer if you really must use just the root (li in your example).
I am using Qt, and I have an unsigned char *bytePointer and want to print out a number-value of the current byte. Below is my code, which is meant to give the int-value and the hex-value of the continuous bytes that I receive from a machine attached to the computer:
int byteHex=0;
byteHex = (int)*bytePointer;
qDebug << "\n int: " //this is the main issue here.
<< *bytePointer;
std::cout << " (hex: "
<< std::hex
<< byteHex
<< ")\n";
}
This gives perfect results, and I get actual numbers, however this code is going into an API and I don't want to use Qt-only functions, such as qDebug. So when I try this:
int byteHex=0;
byteHex = (int)*bytePointer;
std::cout << "\n int: " //I changed qDebug to std::cout
<< *bytePointer;
std::cout << " (hex: "
<< std::hex
<< byteHex
<< ")\n";
}
The output does give the hex-values perfectly, however the int-values return symbols (like ☺, └, §, to list a few).
My question is: How do I get std::cout to give the same output as qDebug?
EDIT: for some reason the symbols only occur with a certain Qt setting. I have no idea why it happened but it's fixed now.
As others pointed out in comment, you change the outputting to hex, but you do not actually set it back here:
std::cout << " (hex: "
<< std::hex
<< byteHex
<< ")\n";
You will need to apply this afterwards:
std::cout << std::dec;
Standard output streams will output any character type as a character, not a numeric value. To output the numeric value, convert to a non-character integer type:
std::cout << int(*bytePointer);
I am having problems with getting this table to line up correctly, this is a cout statement from my table. My problem is how can I format different things in one line without it messing up the next entry in the line. For example, when I use setprecision(2) for my goldweight, the goldvalue gets messed up and gives me a weird number like 5656e+02
cout << " Value Analysis" <<endl;
cout << "Gold: " << setw(6) << "" <<goldweight<< " Oz # "<<costgold<<"
("<<setw(1)<< ""<<carats<<" Carats) $"<<goldvalue<<endl;
Yes, setw(), setprecision(), etc will mess up stuff.
What you can do is to use temporary std::ostringstream.
#include <sstream>
...
std::ostringstream oss_goldweight ;
std::ostringstream oss_goldvalue ;
std::ostringstream oss_goldcarats ;
oss_goldweight << setw(6) << goldweight ;
oss_goldvalue << setprecision(2) << goldvalue ;
oss_goldcarats << setw(1) << carats ;
and then use oss_goldxyz variables instead of raw values.
you need to reset it for the next input, like if you already set a precision, then you set it back (unset)
std::cout.unsetf ( std::ios::floatfield );
My code:
std::vector<double> thePoint(4);
thePoint[0] = 86;
thePoint[1] = -334.8836574;
thePoint[2] = 24.283;
thePoint[3] = 345.67675;
ofstream file1(tempFileName, ios::trunc);
file1 << std::setprecision(16) << thePoint[0] << " ";
file1 << std::fixed << std::setprecision(2) << thePoint[1] << " ";
file1 << std::setprecision(16) << thePoint[2] << " ";
file1 << std::setprecision(16) << thePoint[3];
I get:
86 -334.88 24.28300000000000 345.6767500000000
I want:
86 -334.88 24.283 345.67675
The odd formatting is needed for an interface with other picky code.
You should do this:
file1 << std::fixed << std::setprecision(2) << thePoint[1] << " ";
file1.unsetf(ios_base::fixed);
file1 << std::setprecision(16) << thePoint[2];
The floatfield format flag can take any of its two possible values (using the manipulators fixed and scientific), or none of them (using ios_base::unsetf).
You can do it by forcing the floatfield to an empty value:
file1.setf( std::ios_base::fmtflags(), std::floatfield );
In practice, it's rare to want to, however. The usual protocol is to
save the format flags, and restore them when you're through:
std::ios_base::fmtflags originalFlags = file1.flags();
// ...
file1.flags( originalFlags );
Of course, you'd normally use RAII to do this in a real program. You
should have a IOSave class in your toolbox which will save the flags,
the precision and the fill character in its constructor, and restore
them in the destructor.
It's also not very good practice to use std::setprection etc.
directly. A better solution would be to define your own manipulators,
with names like pression or volume, and use those. This is logical
markup, and means that you control the format for e.g. pression from one
central location, rather than having it spread throughout the program.
And if you write your own manipulators, it's relatively easy to have
them restore the original formatting parameters at the end of the full
expression. (The manipulator objects will be temporaries, destructed at
the end of the full expression.)
I am trying to print the value of a const but it is not working. I am making a return to C++ after years so I know casting is a possible solution but I can't get that working either.
The code is as follows:
//the number of blanks surrounding the greeting
const int pad = 0;
//the number of rows and columns to write
const int rows = pad * 2 + 3;
const string::size_type cols = greeting.size() + pad * 2 + 2;
cout << endl << "Rows : " + rows;
I am trying to print the value of 'rows' without success.
You want:
cout << endl << "Rows : " << rows;
Note this has nothing to do with const - C++ does not allow you to concatenate strings and numbers with the + operator. What you were actually doing was that mysterious thing called pointer arithmetic.
You're almost there:
cout << endl << "Rows : " << rows;
The error is because "Rows : " is a string literal, thus is a constant, and generally speaking is not modified as you may think.
Going slightly further, you likely used + (colloquially used as a concatenation operation) assuming you needed to build a string to give to the output stream. Instead operator << returns the output stream when it is done, allowing chaining.
// It is almost as if you did:
(((cout << endl) << "Rows : ") << rows)
I think you want:
std::cout << std::endl << "Rows : " << rows << std::endl;
I make this mistake all the time as I also work with java a lot.
As others have pointed out, you need
std::cout << std::endl << "Rows : " << rows << std::endl;
The reason (or one of the reasons) is that "Rows : " is a char* and the + operator for char*s doesn't concatenate strings, like the one for std::string and strings in languages like Java and Python.