Why slower function run faster if surrounded with another functions? - c++

Question

Just a little c++ code, confirmed behavior in java.
This is example code what reproduce this behavior compiled with Visual Studio 2019 Release x64. I got:
611ms for just increment element.
631ms for increment element with cache, so additional 20ms for overhead.
But when i add heavy op for before each increment(i choised random number generation) and got:
2073ms for just increment element.
1432ms for increment element using cache.
I have intel cpu 10700K, and 3200RAM if it matter.
#include <iostream>
#include <random>
#include <chrono>
#include <cstdlib>
#define ARR_SIZE 256 * 256 * 256
#define ACCESS_SIZE 256 * 256
#define CACHE_SIZE 1024
#define ITERATIONS 1000
using namespace std;
using chrono::high_resolution_clock;
using chrono::duration_cast;
using chrono::milliseconds;
int* arr;
int* cache;
int counter = 0;
void flushCache() {
for (int j = 0; j < CACHE_SIZE; ++j)
{
++arr[cache[j]];
}
counter = 0;
}
void incWithCache(int i) {
cache[counter] = i;
++counter;
if (counter == CACHE_SIZE) {
flushCache();
}
}
void incWithoutCache(int i) {
++arr[i];
}
int heavyOp() {
return rand() % 107;
}
void main()
{
arr = new int[ARR_SIZE];
cache = new int[CACHE_SIZE];
int* access = new int[ACCESS_SIZE];
random_device rd;
mt19937 gen(rd());
for (int i = 0; i < ACCESS_SIZE; ++i) {
access[i] = gen() % (ARR_SIZE);
}
for (int i = 0; i < ARR_SIZE; ++i) {
arr[i] = 0;
}
auto t1 = high_resolution_clock::now();
for (int iter = 0; iter < ITERATIONS; ++iter) {
for (int i = 0; i < ACCESS_SIZE; ++i) {
incWithoutCache(access[i]);
}
}
auto t2 = high_resolution_clock::now();
auto ms_int = duration_cast<milliseconds>(t2 - t1);
cout << "Time without cache " << ms_int.count() << "ms\n";
t1 = high_resolution_clock::now();
for (int iter = 0; iter < ITERATIONS; ++iter) {
for (int i = 0; i < ACCESS_SIZE; ++i) {
incWithCache(access[i]);
}
flushCache();
}
t2 = high_resolution_clock::now();
ms_int = duration_cast<milliseconds>(t2 - t1);
cout << "Time with cache " << ms_int.count() << "ms\n";
t1 = high_resolution_clock::now();
for (int iter = 0; iter < ITERATIONS; ++iter) {
for (int i = 0; i < ACCESS_SIZE; ++i) {
heavyOp();
incWithoutCache(access[i]);
}
}
t2 = high_resolution_clock::now();
ms_int = duration_cast<milliseconds>(t2 - t1);
cout << "Time without cache and time between " << ms_int.count() << "ms\n";
t1 = high_resolution_clock::now();
for (int iter = 0; iter < ITERATIONS; ++iter) {
for (int i = 0; i < ACCESS_SIZE; ++i) {
heavyOp();
incWithCache(access[i]);
}
flushCache();
}
t2 = high_resolution_clock::now();
ms_int = duration_cast<milliseconds>(t2 - t1);
cout << "Time with cache and time between " << ms_int.count() << "ms\n";
}

Answer 1

I think these kind of questions are extremely hard to answer - optimizing compilers, instruction reordering, and caching all make this hard to analyze but I do have a hypothesis.
First, the difference between incWithoutCache and incWithCache without heavyOp seems reasonable - the second one is simply doing more work.
When you introduce heavyOp is where it gets interesting.
heavyOp + incWithoutCache: incWithoutCache requires a fetch from memory to output to arr. When that memory fetch is complete, it can do the addition. The processor may begin the next heavyOp operation before the increment is complete because of pipelining.
heavyOp + incWithCache: incWithCache does not require a fetch from memory in every iteration as it only has to write out a value. The processor can queue that write to a memory controller and continue. It does do ++counter, but in that case you are always accessing the same value and so I assume this could be cached better then ++arr[i] could from incWithoutCache. When the cache is flushed, the flushing loop can be heavily pipelined - each iteration of the flushing loop is independent and so many iterations will be operating at a time.
So I think the big difference here is that the actual writes to arr cannot be as efficiently pipelined without the cache because heavyOp is trashing your pipeline and potentially your cache. Your heavyOp is taking the same amount of time in either case, but in heavyOp + incWithoutCache the amortized cost of a write to arr is higher because it is not overlapped with other writes to arr such as what can occur with heavyOp + incWithCache.
I think vectorization could theoretically be used for the flushing operation but I did not see that on Compiler Explorer so that may not be a cause of the discrepancy. If vectorization was being used that could explain this speed difference.
I will say I am not an expert in this and could easily be completely wrong about all of this... but it makes sense to me.