How to enhance the speed of my C++ program in reading delimited text files? - c++

Question

I show you C# and C++ code that execute the same job: to read the same text file delimited by “|” and save with “#” delimited text.
When I execute C++ program, the time elapsed is 169 seconds.
UPDATE 1: Thanks to Seth (compilation with: cl /EHsc /Ox /Ob2 /Oi) and GWW for changing the positions of string s outside the loops, the elapsed time was reduced to 53 seconds. I updated the code also.
UPDATE 2: Do you have any other suggestion to enhace the C++ code?
When I execute the C# program, the elapsed time is 34 seconds!
The question is, how can I enhance the speed of C++ comparing with the C# one?
C++ Program:
int main ()
{
Timer t;
cout << t.ShowStart() << endl;
ifstream input("in.txt");
ofstream output("out.txt", ios::out);
char const row_delim = '\n';
char const field_delim = '|';
string s1, s2;
while (input)
{
if (!getline( input, s1, row_delim ))
break;
istringstream iss(s1);
while (iss)
{
if (!getline(iss, s2, field_delim ))
break;
output << s2 << "#";
}
output << "\n";
}
t.Stop();
cout << t.ShowEnd() << endl;
cout << "Executed in: " << t.ElapsedSeconds() << " seconds." << endl;
return 0;
}
C# program:
static void Main(string[] args)
{
long i;
Stopwatch sw = new Stopwatch();
Console.WriteLine(DateTime.Now);
sw.Start();
StreamReader sr = new StreamReader("in.txt", Encoding.Default);
StreamWriter wr = new StreamWriter("out.txt", false, Encoding.Default);
object[] cols = new object[0]; // allocates more elements automatically when filling
string line;
while (!string.Equals(line = sr.ReadLine(), null)) // Fastest way
{
cols = line.Split('|'); // Faster than using a List<>
foreach (object col in cols)
wr.Write(col + "#");
wr.WriteLine();
}
sw.Stop();
Console.WriteLine("Conteo tomó {0} secs", sw.Elapsed);
Console.WriteLine(DateTime.Now);
}
UPDATE 3:
Well, I must say I am very happy for the help received and because the answer to my question has been satisfied.
I changed the text of the question a little to be more specific, and I tested the solutions that kindly raised Molbdlino and Bo Persson.
Keeping Seth indications for the compile command (i.e. cl /EHsc /Ox /Ob2 /Oi pgm.cpp):
Bo Persson's solution took 18 seconds on average to complete the execution, really a good one taking in account that the code is near to what I like).
Molbdlino solution took 6 seconds on average, really amazing!! (thanks to Constantine also).
Never too late to learn, and I learned valuable things with my question.
My best regards.

Answer 1

As Constantine suggests, read large chunks at a time using read.
I cut the time from ~25s to ~3s on a 129M file with 5M "entries" (26 bytes each) in 100,000 lines.
#include <iostream>
#include <fstream>
#include <sstream>
#include <algorithm>
using namespace std;
int main ()
{
ifstream input("in.txt");
ofstream output("out.txt", ios::out);
const size_t size = 512 * 1024;
char buffer[size];
while (input) {
input.read(buffer, size);
size_t readBytes = input.gcount();
replace(buffer, buffer+readBytes, '|', '#');
output.write(buffer, readBytes);
}
input.close();
output.close();
return 0;
}

Answer 2

How about this for the central loop
while (getline( input, s1, row_delim ))
{
for (string::iterator c = s1.begin(); c != s1.end(); ++c)
if (*c == field_delim)
*c = '#';
output << s1 << '\n';
}

Answer 3

It seems to me that Your slow part is within getline. I don't have precise documentation which would support my idea, but it's how it feels for me. You should try using read instead. Because getline has the delimiter, so it need to check every symbol whether it has found the delimiter symbol, so that looks like multiple in operations, so Your program accesses a symbol in a file, then write it to the memory of your program, in other words, the time consumed on disk head movement. But if You use read function, You will copy the block of symbols and then work with them within program's memory, that may reduce time consuming.
PS again, I don't have documentation about getline and how it works, but I'm sure about read, hope it is helpful.

Answer 4

If you know the max line length you can your stdio+fgets and null terminated strings, it will rock.
For c# if it will fit in memory (probably not if it takes 34 sec) I'd be curious to see how IO.File.WriteAllText("out.txt",IO.File.ReadAllText("in.txt").Replace("|","#")); performs!

Answer 5

I'd be really surprised if this beat #molbdnilo's version, but it's probably the second fastest, and (I would posit) the simplest and cleanest:
#include <fstream>
#include <string>
#include <sstream>
#include <algorithm>
int main() {
std::ifstream in("in.txt");
std::ostringstream buffer;
buffer << in.rdbuf();
std::string s(buffer.str());
std::replace(s.begin(), s.end(), '|', '#');
std::ofstream out("out.txt");
out << s;
return 0;
}
Based on past experience with this method, I'd expect it to be no worse than half the speed of what #molbdnilo posted -- which should still be around triple the speed of your C# version, and over ten times as fast as your original version in C++. [Edit: I just wrote a file generator, and on a file a little over 100 megabytes, it's even closer than I expected -- I'm getting 4.4 seconds, versus 3.5 for #molbdnilo's code.] The combination of reasonable speed with really short, simple code is often quite a decent trade-off. Of course, that's all predicated on your having enough physical RAM to hold the entire file content in memory, but that's generally a fairly safe assumption these days.