Merging files with mergesort algorithm in c++ - c++

Question

I wrote a program to do an external mergesort on a file of 100,000 doubles. I couldn't quickly find and external storage libraries for c++ because googling it just leads to a bunch of pages about the extern keyword, so I decided to just write my own, and I think that's where the problem is.
The program actually works, except for a couples details. The output fill will have all of the doubles in sorted order, but at the end of the file are 30 lines of
-9.2559631349317831e+061
which is not in the input file. I also have 21 more values in the output file and the input file, not counting the 30 lines of the single number I just mentioned.
How the program runs is it reads the 100,000 doubles ~4000 lines at a time and sorts them, then stores them in to 26 text files, then those 26 files are merged into 13 files, and those 13 into 7, etc... until there is only one file.
I'm sorry if the code is really ugly, I figured out all of the external storage stuff on my own by pencil, paper, trial, and error. The program is not going to be used for anything. I haven't cleaned it up yet. The driver doesn't do much other than call these methods.
//reads an ifstream file and stores the data in a deque. returns a bool indicating if the file has not reached EOF
bool readFile(ifstream &file, deque<DEQUE_TYPE> &data){
double d;
for(int i = 0; i < DEQUE_SIZE && file.good(); i++){
file >> d;
data.push_back(d);
}
return file.good();
}
//opens a file with the specified filename and prints the contents of the deque to it. if append is true, the data will be appended to the file, else it will be overwritten
void printFile(string fileName, deque<DEQUE_TYPE> &data, bool append){
ofstream outputFile;
if(append)
outputFile.open(fileName, ios::app);
else
outputFile.open(fileName);
outputFile.precision(23);
while(data.size() > 0){
outputFile << data.front() << endl;
data.pop_front();
}
}
//merges the sortfiles until there is one file left
void mergeFiles(){
ifstream inFile1, inFile2;
ofstream outFile;
string fileName1, fileName2;
int i, k, max;
deque<DEQUE_TYPE> data1;
deque<DEQUE_TYPE> data2;
bool fileGood1, fileGood2;
i = 0;
k = 0;
max = 25;
while(max > 1){
fileName1 = ""; fileName1 += "sortfile_"; fileName1 += to_string(i); fileName1 += ".txt";
fileName2 = ""; fileName2 += "sortfile_"; fileName2 += to_string(i+1); fileName2 += ".txt";
try{
inFile1.open(fileName1);
inFile2.open(fileName2);
} catch(int e){
cout << "Could not open the open the files!\nError " << e;
}
fileGood1 = true;
fileGood2 = true;
while(fileGood1 || fileGood2){
fileGood1 = readFile(inFile1, data1);
fileGood2 = readFile(inFile2, data2);
data1 = merge(data1, data2);
printFile("temp", data1, true);
data1.clear();
}
inFile1.close();
inFile2.close();
remove(fileName1.c_str());
remove(fileName2.c_str());
fileName1 = ""; fileName1 += "sortfile_"; fileName1 += to_string(k); fileName1 += ".txt";
rename("temp", fileName1.c_str());
i = i + 2;
k++;
if(i >= max){
max = max / 2 + max % 2;
i = 0;
k = 0;
}
}
}
//merge function
deque<double> merge(deque<double> &left, deque<double> &right){
deque<double> result;
while(left.size() > 0 || right.size() > 0){
if (left.size() > 0 && right.size() > 0){
if (left.front() <= right.front()){
result.push_back(left.front());
left.pop_front();
}
else{
result.push_back(right.front());
right.pop_front();
}
}
else if(left.size() > 0){
result.push_back(left.front());
left.pop_front();
}
else if(right.size() > 0){
result.push_back(right.front());
right.pop_front();
}
}
return result;
}
I sorted a file of 26 numbers (0 - 25), as ThePosey suggested, and here are the results:
-9.2559631349317831e+061 (47 lines of this)
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
25
25
25
25
25
So I'm pretty sure the last number of the file is being duplicated, but I'm still not sure what the 47 occurrences of the random large number is caused by. I checked and the last number of the 100,000 number word is only in the output file twice, not 22 times, So I think I have 11 separate last number being duplicated.

Answer 1

I don't know if this is the whole problem or not, but you have a classic error in your input loop. file.good() doesn't guarantee that the next read will succeed, it only tells you that the previous one did. Try restructuring it like this:
for(int i = 0; i < DEQUE_SIZE && (file >> d); i++){
data.push_back(d);
}
The expression file >> d returns a reference to file, which calls good when you try to evaluate it as a boolean.

Answer 2

Is there a reason why you can't use a few megs of memory to read the entire list in at once into RAM and sort it all at once? It would simplify your program a lot. If you are trying to do this as a challenge I would start by shrinking the problem to say like 1 file of 100 doubles, split that into 4, 25 double reads, and then it should be very easy to trace through and see where the additional lines are coming from.

Answer 3

Assuming your files are in text format, you can use std::merge to do an external merge just as well as an internal one, by using std::istream_iterators.
std::ifstream in1("temp1.txt");
std::ifstream in2("temp2.txt");
std::ofstream out("output.txt");
std::merge(std::istream_iterator<double>(in1),
std::istream_iterator<double>(),
std::istream_iterator<double>(in2),
std::istream_iteraror<double>(),
std::ostream_iterator<double>(out, "\n"));