C++ pool allocator vs static allocation , cache performance - c++

Question

Given that I have two parallel and identically sized arrays of the following structs:
struct Matrix
{
float data[16];
};
struct Vec4
{
float data[4];
}
//Matrix arrM[256]; //for illustration
//Vec4 arrV[256];
Lets say I wish to iterate over the two arrays sequentially as fast as possible. Lets say the function is something like:
for (int i=0; i < 256; ++i)
{
readonlyfunc(arrMPtr[i].data);
readonlyfunc(arrVPtr[i].data
}
Assuming that my allocations are aligned for each array, both in the case of statically allocated or heap memory. Assuming that my cache line size 64 bytes.
Would I achieve the same cache locality and performance if I were to store my data as:
A)
//aligned
static Matrix arrM[256];
static Vec4 arrV[256];
Matrix* arrMPtr = arrM[0];
Vec4* arrVPtr = arrV[0];
vs
B)
//aligned
char* ptr = (char*) malloc(256*sizeof(Matrix)+256*sizeof(Vec4));
Matrix* arrMPtr = (Matrix*) ptr;
Vec4* arrVPtr = (Vec4*) ptr+256*sizeof(Matrix);

Answer 1

How the memory is allocated (heap or statically allocated) makes no difference to the memory's ability to be cached. Since both of these data structures are fairly large (1024 and 4096 bytes, respectively), the exact alignment of the first and last elements probably doesn't matter either (but it does matter if you are using SSE instructions to access the content!).
Whether the memory is close together or not won't make a huge difference, as long as the allocation is small enough to easily fit in the cache, but big enough to take up multiple cache-lines.
You may find that using a structure with 20 float values works out better, if you are working sequentially through both arrays. But that only works if you don't ever need to do other things with the data where having a single array makes more sense.
There may be a difference in the compiler's ability to translate the code to avoid an extra memory access. This will clearly depend on the actual code (e.g. will the compiler inline function containing the for-loop, does it inline the readonlyfunc code, etc, etc. If so, the static allocation can be translated from the pointer variant (which loads the address of the pointer to get the address of the data) into a constant address calculation. It probably doesn't make a huge difference in such a large loop as this.
Always, when it comes to performance, sometimes small things can make big differences, so if this is really important, do some experiments, using YOUR compiler, YOUR actual code. We can only give relatively speculative advice, based on our experience. Different compilers do different things with the same code, different processors do different things with the same machine code (both different actual architectures (whether it's instruction set architecture ARM vs X86, or implementation of the architecture such as AMD Opteron vs Intel Atom or ARM Cortex A15 vs Cortex M3). Memory configurations on your particular system will also affect things, how big caches are, etc, etc.

Answer 2

It's impossible to say without knowning more about what your doing and testing. It might be more efficient to use a struct of arrays.
struct MatrixVec
{
float m[16];
float v[4];
};
One important point is that malloc allocates memory from the heap whereas static arrays are allocated on the stack. The stack is already likely in the L1 cache whereas the memory from the heap will have to be read in. Instead you can try a less known function for dynamic memory allocation called alloca which allocates memory on the stack. In your case you could try
char* ptr = (char*) alloca(256*sizeof(Matrix)+256*sizeof(Vec4))
See Agner Fog's Optimizing software in C++. See the section "9.6 Dynamic memory allocation". Here are the advantages he lists for alloca compared to malloc
There is very little overhead to the allocation process because the microprocessor
has hardware support for the stack.
The memory space never becomes fragmented thanks to the first-in-last-out nature
of the stack.
Deallocation has no cost because it goes automatically when the function returns.
There is no need for garbage collection.
The allocated memory is contiguous with other objects on the stack, which makes
data caching very efficient.