Strange performance issues reading from stdout - c++

Question

I'm working on some code that will be used to test other executables. For convenience I'll refer to my code as the tester and the code being tested as the client. The tester will spawn the client and send commands to the client's stdin and receive results from the client's stdout.
I wanted to do some performance testing first so I wrote a very simple example tester and client. The tester waits for the client to write "READY" to its stdout and in response it sends "GO" to the client's stdin. The client then writes some number of bytes to stdout, configured via a command line flag, and then writes "\nREADY\n" at which point the tester will again write "GO". This repeats 10,000 times after which I calculate the time it took to complete the test and the "throughput", the 10,000 divided by the time to complete.
I ran the above test having the client send 0, 10, 100, 1000, 10000, and 100000 bytes of data before it sends "READY". For each byte size I repeated the test 10 times and took the average. When run on my laptop in an Ubuntu VMWare instance I got a throughput of about 100k GO/READY pairs per second. The performance was fairly stable and had virtually no dependence on the number of binary bytes the client sends to the tester. I then repeated the test on a very fast, 24 core server running CentOS. With a 0 byte payload I observed only about 55k GO/READY pairs per second and the performance degraded noticably as the number of bytes the client sent increased. When the client sends 100k bytes between "GO" and "READY" the throughput was only about 6k operations per second.
So I have three questions
Why would the same code run much more slowly on a faster machine
Why would the performance in the virtual machine be independent of payload size but the performance on the fast server be heavily dependent on payload size?
Is there anything I can do to make things faster on the server
One possible explanation is that I recompiled the code on the fast server and it is using a different version of the C++ libraries. The VMWare machine is running Ubuntu 11.10 and the fast sever is running CentOS 6. Both are 64 bit machines.
The relevant tester code is as follows:
ios_base::sync_with_stdio(false);
const int BUFFER_SIZE = 2 << 20;
char buffer[BUFFER_SIZE];
process_stdout->rdbuf()->pubsetbuf(buffer, BUFFER_SIZE);
Timer timer;
// Wait until the process is ready
string line;
line.reserve(2 << 20);
getline(*process_stdout, line);
CHECK(line == "READY");
timer.Start();
for (int i = 0; i < num_trials; ++i) {
*process_stdin << "GO\n";
process_stdin->flush();
line = "";
while (line != "READY") {
getline(*process_stdout, line);
}
}
double elapsed = timer.Elapsed();
cout << "Done. Did " << num_trials << " iterations in "
<< elapsed << " seconds. Throughput: "
<< double(num_trials) / elapsed << " per second." << endl;
I also tried versions using read() calls (from unistd.h) into a 1MB buffer and calls to memchr to find the "\n" characters and look for READY but got the same performance results.
The relevant client code is as follows:
// Create a vector of binary data. Some portion of the data will be sent
// to stdout each time a "GO" is received before sending "READY"
vector<char> byte_source;
const int MAX_BYTES = 1 << 20;
for (int i = 0; i < MAX_BYTES; ++i) {
byte_source.push_back(i % 256);
}
cout << "READY" << endl;
while (cin.good()) {
string line;
getline(cin, line);
if (line == "GO") {
// The value of response_bytes comes from a command line flag
OutputData(response_bytes, byte_source);
cout << "READY" << endl;
}
}
// write bytes worth of data from byte_source to stdout
void OutputData(unsigned int bytes,
const vector<char>& byte_source) {
if (bytes == 0) {
return;
}
cout.write(&byte_source[0], bytes);
cout << "\n";
}
Any help would be greatly appreciated!

Answer 1

The fact that the speed in VM is independent of the payload size indicates that you're doing something wrong. These are not complete programs so it's hard to pinpoint what. Use strace to see what's going on, i.e., whether the client actually does send all data you believe it is (and also check that the tester is receiving all data it should be).
100k READY/GO pairs is way too much; it's basically near the upper limit of the number of context switches per second, without doing anything else.