Finding most common k-mers and their number of apperance in C++ - c++

Question

Regarding kmer https://en.wikipedia.org/wiki/K-mer
I am trying to find most frequent k-mers in a large fastq file. My plan was to use misra-gries algorithm to find most frequent k-mers, then searching each frequent k-mer's count in file with a second pass. Yet I don't think my algorithm is efficient enough. Here is my first draft below. I try to be memory efficient as possible.(program must not run out of memory)
I also found this DSK algorithm, yet this one is too hard for me to understand without seeing a simple implementation. http://minia.genouest.org/dsk/
Note: Also ID of each counter will be integers not strings, I am going to change it later in my final draft.
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
using namespace std;
struct node {
string id;
int count;
};
void searchCount(vector <node>&memory, string line,int k) {
int count = 0;
string kMer;
for (int i = 0; i < memory.size(); i++) {
if (memory[i].id != "") {
for (int j = 0; j < line.length() - k + 1; j++) {
kMer = line.substr(j, k);
if (kMer == memory[i].id) {
count++;
}
}
}
memory[i].count = count;
count = 0;
}
}
int doWeHaveSpace(vector <node> memory) {
for (int i = 0; i < memory.size(); i++) {
if (memory[i].id == "") {
return i;
}
}
return -1;
}
void MisraGries(string element, vector <node> &memory) {
bool isHere = false;
int index;
for (int i = 0; i < memory.size(); i++) {
if (memory[i].id == element) {
isHere = true;
index = i;
}
}
if (isHere) {
memory[index].count++;
}
else {
int freeIndex = doWeHaveSpace(memory);
if (freeIndex+1) {
memory[freeIndex].count++;
memory[freeIndex].id = element;
}
else {
for (int i = 0; i < memory.size(); i++) {
if (memory[i].count != 0) {
memory[i].count--;
if (memory[i].count == 0) {
memory[i].id = "";
}
}
}
}
}
}
void filecheck(ifstream & input, string prompt) // this function checks and opens input files
{
string filename;
cout << "Please enter file directory and name for " << prompt << ": ";
do
{
getline(cin, filename);
input.open(filename.c_str());
if (input.fail())
cout << " wrong file directory. Please enter real directory. ";
} while (input.fail());
}
int main() {
int line = 1;
string filename;
ifstream input;
ofstream output;
string search;
vector <node> frequent(1000);
for (int i = 0; i < frequent.size(); i++) {
frequent[i].id = "";
frequent[i].count = 0;
}
int k = 30;
string kMer;
filecheck(input, "input file");
while (!input.eof())
{
getline(input, search); // it gets infos line by line to count lines
line++;
if (line == 3) {
for (int i = 0; i < search.length() - k + 1; i++) {
kMer = search.substr(i, k);
MisraGries(kMer, frequent);
}
line = -1;
}
}
return 0;
}

Answer 1

You can speed up your code by storing the most frequent k-mers in a hash table instead of an array. This way, you'll be able to process one k-mer in O(1) time (assuming that the length is constant) if it's already in the cache (if it's not, it would still require a linear pass, but it might give a big improvement on average).
You could also make it even faster if there're a lot of misses by keeping additional information in some kind of auxiliary data structure (like a priority queue) so that you can find the element with count = 0 and remove them without checking all other elements.
Taking into account that k is pretty small in your example, you could increase the size of your in-memory cache (a typical computer should easily keep a few millions of such strings in memory) so that there're less misses.
You could store even more data during the first pass by hashing k-mers (this way, you'll just need to store integers in memory instead of strings).
To sum it up, I'll recommend to make the cache larger (as long as it fits into memory) and use a more suitable data structure that supports fast lookups, like a hash table (std::unordered_map in C++).