How should I improve the performance of this C++ code? - c++

Question

The following code operates on two std::vectors v1 and v2, each containing multiple 128-element vectors. Loops through the outer vectors (using i1 and i2) contain an inner loop, designed to limit the combinations of i1 and i2 for which further complex processing is performed. Around 99.9% of the combinations are filtered out.
Unfortunately the filtering loop is a major bottleneck in my program - profiling shows that 26% of the entire run time is spent on the line if(a[k] + b[k] > LIMIT).
const vector<vector<uint16_t>> & v1 = ...
const vector<vector<uint16_t>> & v2 = ...
for(size_t i1 = 0; i1 < v1.size(); ++i1) { //v1.size() and v2.size() about 20000
for(size_t i2 = 0; i2 < v2.size(); ++i2) {
const vector<uint16_t> & a = v1[i1];
const vector<uint16_t> & b = v2[i2];
bool good = true;
for(std::size_t k = 0; k < 128; ++k) {
if(a[k] + b[k] > LIMIT) { //LIMIT is a const uint16_t: approx 16000
good = false;
break;
}
}
if(!good) continue;
// Further processing involving i1 and i2
}
}
I think the performance of this code could be improved by increasing memory locality, plus perhaps vectorizing. Any suggestions on how to do this, or on other improvements that could be made?

Answer 1

You could apply SIMD to the inner loop:
bool good = true;
for(std::size_t k = 0; k < 128; ++k) {
if(a[k] + b[k] > LIMIT) { //LIMIT is a const uint16_t: approx 16000
good = false;
break;
}
as follows:
#include <emmintrin.h> // SSE2 intrinsics
#include <limits.h> // SHRT_MIN
// ...
// some useful constants - declare these somewhere before the outermost loop
const __m128i vLIMIT = _mm_set1_epi16(LIMIT + SHRT_MIN); // signed version of LIMIT
const __m128i vOFFSET = _mm_set1_epi16(SHRT_MIN); // offset for uint16_t -> int16_t conversion
// ...
bool good = true;
for(std::size_t k = 0; k < 128; k += 8) {
__m128i v, va, vb; // iterate through a, b, 8 elements at a time
int mask;
va = _mm_loadu_si128(&a[k]); // get 8 elements from a[k], b[k]
vb = _mm_loadu_si128(&b[k]);
v = _mm_add_epi16(va, vb); // add a and b vectors
v = _mm_add_epi16(v, vOFFSET); // subtract 32768 to make signed
v = _mm_cmpgt_epi16(v, vLIMIT); // compare against LIMIT
mask = _mm_maskmove_epi8(v); // get comparison results as 16 bit mask
if (mask != 0) { // if any value exceeded limit
good = false; // clear good flag and exit loop
break;
}
Warning: untested code - may need debugging, but the general approach should be sound.

Answer 2

You've got the most efficient access pattern for v1, but you are sequentially scanning through all of v2 for each iteration of the outer loop. This is very inefficient, because v2 access will continually cause (L2 and probably also L3) cache misses.
A better access pattern is to increase the loop nesting, so that outer loops stride through v1 and v2, and inner loops process elements within a subsegment of both v1 and v2 that's small enough to fit in cache.
Basically, instead of
for(size_t i1 = 0; i1 < v1.size(); ++i1) { //v1.size() and v2.size() about 20000
for(size_t i2 = 0; i2 < v2.size(); ++i2) {
Do
for(size_t i2a = 0; i2a < v2.size(); i2a += 32) {
for(size_t i1 = 0; i1 < v1.size(); ++i1) {
for(size_t i2 = i2a; i2 < v2.size() && i2 < i2a + 32; ++i2) {
Or
size_t i2a = 0;
// handle complete blocks
for(; i2a < v2.size() - 31; i2a += 32) {
for(size_t i1 = 0; i1 < v1.size(); ++i1) {
for(size_t i2 = i2a; i2 < i2a + 32; ++i2) {
}
}
}
// handle leftover partial block
for(size_t i1 = 0; i1 < v1.size(); ++i1) {
for(size_t i2 = i2a; i2 < v2.size(); ++i2) {
}
}
This way, a chunk of 32 * 128 * sizeof (uint16_t) bytes, or 8kB, will be loaded from v2 into cache, and then reused 20,000 times.
This improvement is orthogonal to SIMD (SSE) vectorization. It will interact with thread-based parallelism, but probably in a good way.

Answer 3

First, One simple optimization can be this, but the compiler could do this by itself so i'm not sure how much it could improve:
for(std::size_t k = 0; k < 128 && good; ++k)
{
good = a[k] + b[k] <= LIMIT;
}
Second, i think it could better to keep the good result in a second vector because any
processing involved with i1 and i2 could break the CPU cache.
Third, and this could be major optimization, i think you can rewrite the second for loop as this:
for(size_t i2 = i1; i2 < v2.size(); ++i2) since you are using + operations for a and b vectors which is commutative so the result of i1 and i2 will be the same as i2 and i1.
For this you need to have the v1 and v2 the same size. If the size is different you need to write the iteration in a different way.
Fort, As far as i can see you are processing two matrices, it would be better to keep a vector of elements rather than a vector of vectors.
Hope this helps.
Razvan.

Answer 4

A few suggestions:
As suggested in the comments, replace the inner 128 element vector with an array for better memory locality.
This code looks highly parallelizable, have you tried that? You could split the combinations for filtering across all available cores and then rebalance the collected work and split the processing across all cores.
I implemented a version using arrays for the inner 128 elements, PPL for parallelization (requires VS 2012 or higher) and a bit of SSE code for the filtering and got a pretty significant speedup. Depending on what exactly the 'further processing' involves there may be benefits to structuring things slightly differently (in this example I don't rebalance the work after filtering for example).
Update: I implemented the cache blocking suggested by Ben Voigt and got a bit more of a speed up.
#include <vector>
#include <array>
#include <random>
#include <limits>
#include <cstdint>
#include <iostream>
#include <numeric>
#include <chrono>
#include <iterator>
#include <ppl.h>
#include <immintrin.h>
using namespace std;
using namespace concurrency;
namespace {
const int outerVecSize = 20000;
const int innerVecSize = 128;
const int LIMIT = 16000;
auto engine = default_random_engine();
};
typedef vector<uint16_t> InnerVec;
typedef array<uint16_t, innerVecSize> InnerArr;
template <typename Cont> void randomFill(Cont& c) {
// We want approx 0.1% to pass filter, the mean and standard deviation are chosen to get close to that
static auto dist = normal_distribution<>(LIMIT / 4.0, LIMIT / 4.6);
generate(begin(c), end(c), [] {
auto clamp = [](double x, double minimum, double maximum) { return min(max(minimum, x), maximum); };
return static_cast<uint16_t>(clamp(dist(engine), 0.0, numeric_limits<uint16_t>::max()));
});
}
void resizeInner(InnerVec& v) { v.resize(innerVecSize); }
void resizeInner(InnerArr& a) {}
template <typename Inner> Inner generateRandomInner() {
auto inner = Inner();
resizeInner(inner);
randomFill(inner);
return inner;
}
template <typename Inner> vector<Inner> generateRandomInput() {
auto outer = vector<Inner>(outerVecSize);
generate(begin(outer), end(outer), generateRandomInner<Inner>);
return outer;
}
void Report(const chrono::high_resolution_clock::duration elapsed, size_t in1Size, size_t in2Size,
const int passedFilter, const uint32_t specialValue) {
cout << passedFilter << "/" << in1Size* in2Size << " ("
<< 100.0 * (double(passedFilter) / double(in1Size * in2Size)) << "%) passed filter\n";
cout << specialValue << "\n";
cout << "Elapsed time = " << chrono::duration_cast<chrono::milliseconds>(elapsed).count() << "ms" << endl;
}
void TestOriginalVersion() {
cout << __FUNCTION__ << endl;
engine.seed();
const auto v1 = generateRandomInput<InnerVec>();
const auto v2 = generateRandomInput<InnerVec>();
int passedFilter = 0;
uint32_t specialValue = 0;
auto startTime = chrono::high_resolution_clock::now();
for (size_t i1 = 0; i1 < v1.size(); ++i1) { // v1.size() and v2.size() about 20000
for (size_t i2 = 0; i2 < v2.size(); ++i2) {
const vector<uint16_t>& a = v1[i1];
const vector<uint16_t>& b = v2[i2];
bool good = true;
for (std::size_t k = 0; k < 128; ++k) {
if (static_cast<int>(a[k]) + static_cast<int>(b[k])
> LIMIT) { // LIMIT is a const uint16_t: approx 16000
good = false;
break;
}
}
if (!good) continue;
// Further processing involving i1 and i2
++passedFilter;
specialValue += inner_product(begin(a), end(a), begin(b), 0);
}
}
auto endTime = chrono::high_resolution_clock::now();
Report(endTime - startTime, v1.size(), v2.size(), passedFilter, specialValue);
}
bool needsProcessing(const InnerArr& a, const InnerArr& b) {
static_assert(sizeof(a) == sizeof(b) && (sizeof(a) % 16) == 0, "Array size must be multiple of 16 bytes.");
static const __m128i mmLimit = _mm_set1_epi16(LIMIT);
static const __m128i mmLimitPlus1 = _mm_set1_epi16(LIMIT + 1);
static const __m128i mmOnes = _mm_set1_epi16(-1);
auto to_m128i = [](const uint16_t* p) { return reinterpret_cast<const __m128i*>(p); };
return equal(to_m128i(a.data()), to_m128i(a.data() + a.size()), to_m128i(b.data()), [&](const __m128i& a, const __m128i& b) {
// avoid overflow due to signed compare by clamping sum to LIMIT + 1
const __m128i clampSum = _mm_min_epu16(_mm_adds_epu16(a, b), mmLimitPlus1);
return _mm_test_all_zeros(_mm_cmpgt_epi16(clampSum, mmLimit), mmOnes);
});
}
void TestArrayParallelVersion() {
cout << __FUNCTION__ << endl;
engine.seed();
const auto v1 = generateRandomInput<InnerArr>();
const auto v2 = generateRandomInput<InnerArr>();
combinable<int> passedFilterCombinable;
combinable<uint32_t> specialValueCombinable;
auto startTime = chrono::high_resolution_clock::now();
const size_t blockSize = 64;
parallel_for(0u, v1.size(), blockSize, [&](size_t i) {
for (const auto& b : v2) {
const auto blockBegin = begin(v1) + i;
const auto blockEnd = begin(v1) + min(v1.size(), i + blockSize);
for (auto it = blockBegin; it != blockEnd; ++it) {
const InnerArr& a = *it;
if (!needsProcessing(a, b))
continue;
// Further processing involving a and b
++passedFilterCombinable.local();
specialValueCombinable.local() += inner_product(begin(a), end(a), begin(b), 0);
}
}
});
auto passedFilter = passedFilterCombinable.combine(plus<int>());
auto specialValue = specialValueCombinable.combine(plus<uint32_t>());
auto endTime = chrono::high_resolution_clock::now();
Report(endTime - startTime, v1.size(), v2.size(), passedFilter, specialValue);
}
int main() {
TestOriginalVersion();
TestArrayParallelVersion();
}
On my 8 core system I see a pretty good speedup, your results will vary depending on how many cores you have etc.
TestOriginalVersion
441579/400000000 (0.110395%) passed filter
2447300015
Elapsed time = 12525ms
TestArrayParallelVersion
441579/400000000 (0.110395%) passed filter
2447300015
Elapsed time = 657ms