Python parsing stdin much faster than C++ - c++

Question

I have a python function which I was hoping to translate into C++ to try and gain some extra speed (as it will be used to parse >100GB files). I am very inexperienced with C++ and was horrified to find my C++ function running much slower after my basic translation. Any pointers as to why this is, or what I can do to improve my C++ code would be much appreciated.
Script overview: the function reads stdin from another program, checks each line for any substring matches, and prints each line to stdout
Python function:
def find_tagPy(conditions):
# conditions e.g. ['TTAT', 'TAT'] etc
for line in stdin:
# Check conditionss against this line
l = line.split("\t")
if l[0][0] == "#":
stdout.write(line)
continue
FLAG = int(l[1])
if 1 & FLAG: # Read has a pair
for bases in conditions:
if bases in l[9]:
ADD_MATE = 1
stdout.write(line)
break # stop looking
C++ function:
void find_tagCpp (vector<string> conditions) {
cin.sync_with_stdio(false);
cin.tie(NULL);
string line;
while (getline(cin, line)) {
vector<string> l;
boost::split(l, line, boost::is_any_of("\t"), boost::token_compress_on);
if (l[0][0] == '#') {
cout << line << "\n";
continue;
}
int FLAG = stoi(l[1]);
int pair_FLAG = 1;
if (pair_FLAG & FLAG) { // Read has a pair
for (int i=0; i < conditions.size(); i++) { // If bases in SEQ
if (l[9].find(conditions[i]) != string::npos) {
cout << line << "\n";
break; // Stop looking
}
}
}
}
}
An example of a stdin line is:
FCC2CCMACXX:4:1105:10758:14389# 81 chrM 1 32 10S90M = 16151 16062 CATCACGATGGATCACAGGTCTATCACCCTATTAACCACTCACGGGAGCTTTCCATGCATTTGGTATTTTCGTCTGGGGGGTGTGCACGCGATAGCATTG bbb^Wcbbbbccbbbcbccbba]WQG^bbcdcb_^_c_^`ccdddeeeeeffggggiiiiihiiiiihiiihihiiiihghhiihgfgfgeeeeebbb NM:i:1 AS:i:85 XS:i:65 RG:Z:1_DB31
On my machine the python function takes 1.97 s and the C++ function takes 11.05 s (file size around 25 mb, but this includes processing with upstream and downstream tools)
EDIT:
I've found a bottleneck in boost::split which is a bit suprising:
Python:
for i in range(100000):
l = line.split("\t")
C++:
for (int i=0; i < 100000; i++) {
vector<string> l;
boost::split(l, line, boost::is_any_of("\t"), boost::token_compress_on);
}
Python = 0.0325 s
C++ = 1.245 s
However my file is only 156,980 lines, so this cant be the whole problem.

Answer 1

The split copies the pieces into new strings. This is slow and you do not need them. Instead search the line for the start of the piece you want ( 10th ) and then call find starting from there.

Answer 2

I realised my original code is not amenable to testing, so I thought I would refactor it here, and discuss what I have found.
I turned on compiler optimisations using the -Ofast (fastest, aggressive optimisations, Apple LLVM 6.1) as suggested, and for comparison Python is 2.7.10.
Python function
import time
def fun(line):
l = line.split(" ", 10)
if 'TTAGGG' in l[9]:
pass
line = "FCC2CCMACXX:4:1105:10758:14389# 81 chrM 1 32 10S90M = 16151 16062 CATCACGATGGATCACAGGTCTATCACCCTATTAACCACTCACGGGAGCTTTCCATGCATTTGGTATTTTCGTCTGGGGGGTGTGCACGCTTAGGGGATAGCATTG bbb^Wcbbbbccbbbcbccbba]WQG^bbcdcb_^_c_^`ccdddeeeeeffggggiiiiihiiiiihiiihihiiiihghhiihgfgfgeeeeebbb NM:i:1 AS:i:85 XS:i:65 RG:Z:1_DB31"
time0 = time.time()
for i in range(100000):
fun(line)
print time.time() - time0
C++ function
void fun(string* line, string* substring) {
vector<string> l;
boost::split(l, *line, boost::is_any_of(" "));
if (l[9].find(*substring) != string::npos) {
// Do nothing
}
}
int main(int argc, const char * argv[]) {
string line = "FCC2CCMACXX:4:1105:10758:14389# 81 chrM 1 32 10S90M = 16151 16062 CATCACGATGGATCACAGGTCTATCACCCTATTAACCACTCACGGGAGCTTTCCATGCATTTGGTATTTTCGTCTGGGGGGTGTGCACGCTTAGGGGATAGCATTG bbb^Wcbbbbccbbbcbccbba]WQG^bbcdcb_^_c_^`ccdddeeeeeffggggiiiiihiiiiihiiihihiiiihghhiihgfgfgeeeeebbb NM:i:1 AS:i:85 XS:i:65 RG:Z:1_DB31";
string substring = "TTAGGG";
boost::timer t;
for (int i=0; i<100000; i++) {
fun(&line, &substring);
}
cout << t.elapsed() << endl;
return 0;
}
On my machine I now time the c++ function at 205 ms and the python function at 66 ms. Interestingly, now almost the entire runtime is taken up by the boost::split function.
If I get rid of this function and use string.find to search the whole line (not quite what I wanted though):
if ((*line).find(*substring) != string::npos) {
// Do nothing
}
The c++ runtime is reduced to about <1 ms! So it appears boost::split was the problem all alone. Thanks for the suggestions.

Answer 3

Try this code with some optimization
C++ function:
void find_tagCpp (vector<string> conditions) {
cin.sync_with_stdio(false);
cin.tie(NULL);
string line;
vector<string> l;
while (getline(cin, line)) {
l.clear();
boost::split(l, line, boost::is_any_of("\t"), boost::token_compress_on);
if (l[0][0] == '#') {
cout << line << "\n";
continue;
}
int FLAG = stoi(l[1]);
int pair_FLAG = 1;
if (pair_FLAG & FLAG) { // Read has a pair
for (int i=0; i < conditions.size(); i++) { // If bases in SEQ
if (l[9].find(conditions[i]) != string::npos) {
printf("%s\n", line.c_str());
break; // Stop looking
}
}
}
}
}