Calculating a file's mean value of data bytes - c++

Question

Just for fun, I am trying to calculate a file's mean value of data bytes, essentially replicating a feature available in an already existing tool (ent). Basically, it is simply the result of summing all the bytes of a file and dividing by the file length. If the data are close to random, this should be about 127.5. I am testing 2 methods of computing the mean value, one is a simple for loop which works on an unordered_map and the other is using std::accumulate directly on a string object.
Benchmarking both methods show that it is much slower to use std::accumulate than a simple for loop. Also, mesured on my system, on average, clang++ is about 4 times faster for the accumulate method than g++.
So here are my questions:
Why is the for loop method producing bad output at around 2.5GB input for g++ but not with clang++. My guess is I am doing things wrong (UB probably), but they happen to work with clang++. (solved and code modified accordingly)
Why is the std::accumulate method so much slower on g++ with the same optimization settings?
Thanks!
Compiler info (target is x86_64-pc-linux-gnu):
clang version 11.1.0
gcc version 11.1.0 (GCC)
Build info:
g++ -Wall -Wextra -pedantic -O3 -DNDEBUG -std=gnu++2a main.cpp -o main-g
clang++ -Wall -Wextra -pedantic -O3 -DNDEBUG -std=gnu++20 main.cpp -o main-clang
Sample file (using random data):
dd if=/dev/urandom iflag=fullblock bs=1G count=8 of=test-8g.bin (example for 8GB random data file)
Code:
#include <chrono>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <numeric>
#include <stdexcept>
#include <string>
#include <unordered_map>
auto main(int argc, char** argv) -> int {
using std::cout;
std::filesystem::path file_path{};
if (argc == 2) {
file_path = std::filesystem::path(argv[1]);
} else {
return 1;
}
std::string input{};
std::unordered_map<char, int> char_map{};
std::ifstream istrm(file_path, std::ios::binary);
if (!istrm.is_open()) {
throw std::runtime_error("Could not open file");
}
const auto file_size = std::filesystem::file_size(file_path);
input.resize(file_size);
istrm.read(input.data(), static_cast<std::streamsize>(file_size));
istrm.close();
// store frequency of individual chars in unordered_map
for (const auto& c : input) {
if (!char_map.contains(c)) {
char_map.insert(std::pair<char, int>(c, 1));
} else {
char_map[c]++;
}
}
double sum_for_loop = 0.0;
cout << "using for loop\n";
// start stopwatch
auto start_timer = std::chrono::steady_clock::now();
// for loop method
for (const auto& item : char_map) {
sum_for_loop += static_cast<unsigned char>(item.first) * static_cast<double>(item.second);
}
// stop stopwatch
cout << std::chrono::duration<double>(std::chrono::steady_clock::now() - start_timer).count() << " s\n";
auto mean_for_loop = static_cast<double>(sum_for_loop) / static_cast<double>(input.size());
cout << std::fixed << "sum_for_loop: " << sum_for_loop << " size: " << input.size() << '\n';
cout << "mean value of data bytes: " << mean_for_loop << '\n';
cout << "using accumulate()\n";
// start stopwatch
start_timer = std::chrono::steady_clock::now();
// accumulate method, but is slow (much slower in g++)
auto sum_accum =
std::accumulate(input.begin(), input.end(), 0.0, [](auto current_val, auto each_char) { return current_val + static_cast<unsigned char>(each_char); });
// stop stopwatch
cout << std::chrono::duration<double>(std::chrono::steady_clock::now() - start_timer).count() << " s\n";
auto mean_accum = sum_accum / static_cast<double>(input.size());
cout << std::fixed << "sum_for_loop: " << sum_accum << " size: " << input.size() << '\n';
cout << "mean value of data bytes: " << mean_accum << '\n';
}
Sample output from 2GB file (clang++):
using for loop
2.024e-05 s
sum_for_loop: 273805913805 size: 2147483648
mean value of data bytes: 127.500814
using accumulate()
1.317576 s
sum_for_loop: 273805913805.000000 size: 2147483648
mean value of data bytes: 127.500814
Sample output from 2GB file (g++):
using for loop
2.41e-05 s
sum_for_loop: 273805913805 size: 2147483648
mean value of data bytes: 127.500814
using accumulate()
5.269024 s
sum_for_loop: 273805913805.000000 size: 2147483648
mean value of data bytes: 127.500814
Sample output from 8GB file (clang++):
using for loop
1.853e-05 s
sum_for_loop: 1095220441576 size: 8589934592
mean value of data bytes: 127.500440
using accumulate()
5.247585 s
sum_for_loop: 1095220441576.000000 size: 8589934592
mean value of data bytes: 127.500440
Sample output from 8GB file (g++):
using for loop
7.5e-07 s
sum_for_loop: 1095220441576.000000 size: 8589934592
mean value of data bytes: 127.500440
using accumulate()
21.484348 s
sum_for_loop: 1095220441576.000000 size: 8589934592
mean value of data bytes: 127.500440

Answer 1

There are numerous issues with the code. The first - and the one that is causing your display problem - is that sum_for_loop should be a double, not an unsigned long. The sum is overflowing what can be stored in an unsigned long, resulting in your incorrect result when that happens.
The timers should be started after the cout, otherwise you're including the output time in the compute time. In addition, the "for loop" elapsed time excludes the time taken to construct char_map.
When building char_map, you don't need the if. If an entry is not found in the map it will be zero initialized. A better approach (since you only have 256 unique values) is to use an indexed vector (remembering to cast the char to unsigned char).