Best way to cache meta-data from a kv storage - c++

Question

Currently, we have a system where metadata will be stored in a kv storage cluster. We store it simply by serializing the application metadata with protobuf and then send to kv cluster. As the system gets bigger and bigger, fetching metadata itself becomes expensive. Therefore, we develop an in-memory meta-cache component, simply an LRU cache, and the cached items are the protobuf objects. Recently, we have faced some challenges:
Concurrently read-write seem a big issue: when we add new data to our system, we need to update the cache as well, so we will need to partially lock part of the cache to ensure repeatable read, this brings very high lock contention on the cache.
The cache gets bigger and bigger over time.
I'm thinking that maybe our cache design is not good enough and considering using 3rd party lib like Facebook Cachelib (our system is written in C++). Can anyone who has experience with the matter give me some advice? Should we use 3rd party lib, or should we improve our own? If we improve our own, what can we do?
Thanks so much :).

Answer 1

If you are in-memory caching on only a single server node and if you can hash all of keys (for metadata indexing) into unique positive integers and
if ~75M lookups per second on an old CPU is ok for you, there is a multi-threaded multi-level read-write cache implementation:
https://github.com/tugrul512bit/LruClockCache/blob/main/MultiLevelCache.h
It works like this:
int main()
{
std::vector<int> data(1024*1024); // simulating a backing-store
MultiLevelCache<int,int> cache(
64*1024 /* direct-mapped L1 cache elements */,
256,1024 /* n-way set-associative (LRU approximation) L2 cache elements */,
[&](int key){ return data[key]; } /* cache-miss function to get data from backingstore into cache */,
[&](int key, int value){ data[key]=value; } /* cache-miss function to set data on backging-store during eviction */
);
cache.set(5,10); // this is single-thread example, sets value 10 at key position of 5
cache.flush(); // writes all latest bits of data to backing-store
std::cout<<data[5]<<std::endl;
auto val = cache.getThreadSafe(5); // this is thread-safe from any number of threads
cache.setThreadSafe(10,val); // thread-safe, any number of threads
return 0;
}
It consists of two caches, L1 is fast&sharded but weak on cache-hits, L2 is slower&sharded but is better on cache-hits although not as good as perfect-LRU.
If there is a read-only part (like just distributing values to threads from a read-only backing-store) in the application, there is another way to improve performance up to 2.5 billion lookups per second:
// shared last level cache (LRU approximation)
auto LLC = std::make_shared<LruClockCache<int,int>>(LLCsize,
[ & ](int key) {
return backingStore[key];
},
[ & ](int key, int value) {
backingStore[key] = value;
});
#pragma omp parallel num_threads(8)
{
// each thread builds its own lockless L1&L2 caches, binds to LLC (LLC has thread-safe bindings to L2)
// building overhead is linearly dependent on cache sizes
CacheThreader<LruClockCache,int,int> cache(LLC,L1size,L2size);
for(int i=0;i<many_times;i++)
cache.get(i); // 300M-400M per second per thread
}
Performance depends on accessing pattern prediction. For user-input dependent access order, it has lower performance due to inefficient pipelining. For compile-time-known indexing, it has better performance. For true random access, it has lower performance due to cache-misses & lack of vectorization. For sequential access, it has better performance.
If you need to use your main thread asynchronously during get/set calls and still require a (weak) cache coherence between gets and sets, there is another implementation which has a performance that relies on number of keys requested at a time from a thread:
// backing-store
std::vector<int> data(1000000);
// L1 direct mapped 128 tags
// L2 n-way set-associative 128 sets + 1024 tags per set
AsyncCache<int,int> cache(128,128,1024,[&](int key){ return data[key]; },[&](int key, int value){ data[key]=value; });
// a variable to use as output for the get() call
int val;
// thread-safe, returns immediately
// each set/get call should be given a slot id per thread
// or it returns a slot id instead
int slot = cache.setAsync(5,100);
// returns immediately
cache.getAsync(5,&val,slot);
// garbage garbage
std::cout<<data[5]<<" "<<val<<std::endl;
// waits for operations made in a slot
cache.barrier(slot);
// garbage 100
std::cout<<data[5]<<" "<<val<<std::endl;
// writes latest bits to backing-store
cache.flush();
// 100 100
std::cout<<data[5]<<" "<<val<<std::endl;
this is not fully-coherent and threads are responsible to do right ordering of gets/sets between each other.
If your metadata indexing is two/three-dimensional, there are 2D/3D direct-mapped multi-thread caches:
https://github.com/tugrul512bit/LruClockCache/blob/main/integer_key_specialization/DirectMapped2DMultiThreadCache.h
https://github.com/tugrul512bit/LruClockCache/blob/main/integer_key_specialization/DirectMapped3DMultiThreadCache.h
int backingStore[10][10];
DirectMapped2DMultiThreadCache<int,int> cache(4,4,
[&](int x, int y){ return backingStore[x][y]; },
[&](int x, int y, int value){ backingStore[x][y]=value; });
for(int i=0;i<10;i++)
for(int j=0;j<10;j++)
cache.set(i,j,0); // row-major
cache.flush();
for(int i=0;i<10;i++)for(int j=0;j<10;j++)
std::cout<<backingStore[i][j]<<std::endl;
it has better cache-hit ratio than normal direct-mapped cache if you do tiled-processing.