Sorting referenced substrings using C++'s sort? - c++

Question

I have two long strings (a million or so characters) that I want to generate suffixes from and sort in order to find the longest shared substrings as this will be much faster than brute-forcing all possible substrings. I'm most familiar with Python, but my quick calculation estimated 40 Tb of suffixes so I'm hoping it's possible to use C++ (suggested) to sort references to the substrings in each main, unchanging string.
I'll need to retain the index of each substring to find the value as well as the origin string later, so any advice on the type of data structure I could use that would 1) allow sorting of reference strings and 2) keep track of the original index would be super helpful!
Current pseudocode:
//Function to make vector? of structures that contain the reference to the string and the original index
int main() {
//Declare strings
string str1="This is a very long string with some repeats of strings."
string str2="This is another string with some repeats that is very long."
//Call function to make array
//Pass vector to sort(v.begin(), v.end), somehow telling it to deference?
//Process the output in multilayer loop to find the longest exact match
// "string with some repeats"
return 0;}

Answer 1

First of all, you should use a suffix tree for this. But I'll answer your original question.
C++17 :
NOTE: uses experimental features
You may use std::string_view to reference the strings without copying. Here is an example code:
//Declare string
char* str1 = "This is a very long string with some repeats of strings."
int main() {
//Call function to make array
vector<string_view> substrings;
//example of adding substring [5,19) into vector
substrings.push_back(string_view(str1 + 5, 19 - 5));
//Pass vector to sort(v.begin(), v.end)
sort(substrings.begin(), substrings.end());
return 0;
}
Everything before C++17:
You could use a custom predicate with the sort function. Instead of making your vector store the actual strings, make it store pair which contains the index.
Here is an example of code needed to make it work:
//Declare string
string str1="This is a very long string with some repeats of strings."
bool pred(pair<int,int> a, pair<int,int> b){
int substring1start=a.first,
substring1end=a.second;
int substring2start=b.first,
substring2end=b.second;
//use a for loop to manually compare substring1 and substring 2
...
//return true if substring1 should go before substring2 in vector
//otherwise return false
}
int main() {
//Call function to make array
vector<pair<int,int>> substrings;
//example of adding substring [1,19) into vector
substrings.push_back({1,19});
//Pass vector to sort(v.begin(), v.end), passing custom predicate
sort(substrings.begin(), substrings.end(), pred);
return 0;
}
Even if you reduce your memory usage, your program will still take 40T iterations to run anyways (since you need to compare the strings). Unless you use some sort of hashing string comparison algorithm.

Answer 2

You could use a combination of std::string_view, std::hash and std::set.
#include <iostream>
#include <set>
#include <string>
#include <string_view>
#include <vector>
std::string str1="This is a very long string with some repeats of strings.";
std::string str2="This is another string with some repeats that is very long.";
std::set<std::size_t> substringhashes;
std::vector<std::string_view> matches;
bool makeSubHashes(std::string& str, std::size_t lenght) {
for (int pos=0; pos+lenght <= str.size(); ++pos) {
std::string_view sv(str.data()+pos, lenght);
auto hash = std::hash<std::string_view>()(sv);
if (!substringhashes.insert(hash).second) {
matches.push_back(sv);
if (matches.size() > 99) // Optional break after finding the 100 longest matches
return true;
}
}
return false;
}
int main() {
for (int lenght=std::min(str1.size(), str2.size()); lenght>0; --lenght) {
if (makeSubHashes(str1, lenght) || makeSubHashes(str2, lenght))
break;
}
for (auto& sv : matches) {
std::cout << sv << std::endl;
}
return 0;
}
If the amount of suffixes are extremely high, there is a chance for false positives with the std::set. It has std::size_ts max value number of different hashes, which is normally a uint64.
It also starts searching for matches at the maximum lenght of the strings, maybe a more reasonable approach is to set some sort of maximum lenght for the suffixes.

Answer 3

std::sort sorts data in main memory.
If you can fit the data in main memory, then you can sort it with std::sort.
Otherwise not.