Removing duplicates and counting duplicates in a text file with C++ - c++

Question

I am a beginner at C++. I created a text file with two columns in it. However, there are around 1 million rows and there are many rows that repeat each other. I want to delete the duplicates and count how many duplicates there were making it into a third row. This is what it would look like before and after:
Before:
10 8
11 7
10 8
10 8
15 12
11 7
After:
10 8 3
11 7 2
15 12 1
I don't really know where to start can someone point me in the right direction of what I should be looking up in order to do this?

Answer 1

You can create std::map<std::pair<int, int>, int>, and after each insertion check if the given pair is contained in the map. If pair is contained just increment number of duplicates, otherwise emplace it in the map.
Something like this:
#include <iostream>
#include <map>
int main(int argc, char* argv[]) {
std::map<std::pair<int, int>, int> rows;
int num1;
int num2;
while (std::cin >> num1 >> num2) {
auto pair = std::make_pair(num1, num2);
if (rows.find(pair) != rows.end())
++rows[pair];
else
rows.emplace(pair, 1);
}
}

Answer 2

#include <string>
#include <fstream>
#include <unordered_map>
using namespace std;
int main()
{
string line;
unordered_map<string, int> count_map;
ifstream src("input.txt");
if (!src.is_open())
{
return -1;
}
while (getline(src, line))
{
if (line.empty())
continue;
count_map[line]++;
}
src.close();
ofstream dst("output.txt");
if (!dst.is_open())
{
return -2;
}
for (auto & iter : count_map)
{
dst << iter.first << " " << iter.second << endl;
}
dst.close();
return 0;
}

Answer 3

#include <iostream>
#include <fstream>
#include <string>
#include <map>
#include <set>
using namespace std;
int main() {
ifstream src("input.txt");
if (!src.is_open()) {
return -1;
}
// store each line, filter out all of duplicated strings
set<string> container;
// key is to maintain the order of lines, value is a pair<K, V>
// K is the itor pointed to the string in the container
// V is the counts of the string
map<int, std::pair<set<string>::iterator, int>> mp;
// key is the pointer which points to the string in the container
// value is the index of string in the file
map<const string *, int> index;
string line;
int idx = 0; // index of the string in the file
while (getline(src, line)) {
if (line.empty()) {
continue;
}
auto res = container.insert(line);
if (res.second) {
index[&(*res.first)] = idx;
mp[idx] = {res.first, 1};
idx++;
} else {
mp[index[&(*res.first)]].second += 1;
}
}
src.close();
ofstream dst("output.txt");
if (!dst.is_open()) {
return -2;
}
for (const auto & iter : mp) {
dst << *iter.second.first << " " << iter.second.second << endl;
}
dst.close();
return 0;
}
BTW, Redis can solve this problem easily if you are allowed to use it.

Answer 4

This can be done with a std::priority_queue, which automatically sorts the entries. With the data sorted like this, one only has to count the number of subsequent identical entries:
#include <queue>
#include <iostream>
#include <vector>
#include <utility> // for std::pair
int main() {
std::priority_queue<std::pair<int,int>> mydat;
mydat.push(std::make_pair(10,8));
mydat.push(std::make_pair(11,7));
mydat.push(std::make_pair(10,8));
mydat.push(std::make_pair(10,8));
mydat.push(std::make_pair(15,12));
mydat.push(std::make_pair(11,7));
std::vector<std::vector<int>> out;
std::pair<int,int> previous;
int counter;
while(!mydat.empty()) {
counter = 1;
previous = mydat.top();
mydat.pop(); // move on to next entry
while(previous == mydat.top() && !mydat.empty()) {
previous = mydat.top();
mydat.pop();
counter++;
}
out.push_back({previous.first, previous.second, counter});
}
for(int i = 0; i < out.size(); ++i) {
std::cout << out[i][0] << " " << out[i][1] << " " << out[i][2] << std::endl;
}
}
godbolt demo
Output:
15 12 1
11 7 2
10 8 3