How to use multiple cache layers in c++ (RAM, HDD, cold) - c++

Question

I have a simple POD data class like
struct hash{
char buffer[16];
};
I need to have a vector of many instances of it it will shorelly not fit into ram (20 PB). It is conceptually grouped into a vector (tree). I want to have a way to have a pointer like thing that would hide RAM, filesystem, cold storage, and have a simple array\pointer like interface (makeing fs, operations invisible after initialisation yet allowing to give it multiple places to put data in - RAM, Fast SSD, SSD, HDD, Tape, Cloud drive locations)
How to do such thing in C++?

Answer 1

There is no support for this at the language level.
One solution would be use a memory mapped file, for example see:
Creating a File Mapping Using Large Pages
If you need a more platform independant solution then it is possible you could use boost that has some support for memory mapped files as well in the boost-filesystem library.
Besides that you, you can always make a pointer like object facade to manage the underlying logics (ala. smart pointers).
template<class T>
struct MyMappedPointerType {
T& operator* MyPointerType();//derefence - may throw..
//implement rest of semantics
};

Answer 2

I think the usual would be to use some handle. Then when you want to access the object, you would pass the handle to a function which will load the memory and give you the address, and then you would close the handle. In C++ you would use RAII.
#include <string>
#include <cstdio>
template <class T>
class Access
{
private:
FILE* f= nullptr;
public:
Access(const std::string& filename)
{
f= fopen(filename.data(), "rw");
}
~Access()
{
fclose(f);
}
class WriteAccess
{
T buffer{};
bool dirty= false;
FILE* f;
int64_t elementNumber;
public:
WriteAccess(FILE* f, int64_t elementNumber)
: f(f)
, elementNumber(elementNumber)
{
if (f) {
fseek(f, elementNumber*sizeof(buffer), SEEK_SET);
fread(&buffer, sizeof(buffer), 1, f);
}
}
T& get() { dirty= true; return buffer; }
const T& get() const { return buffer; }
~WriteAccess()
{
if (dirty && f) {
fseek(f, elementNumber*sizeof(buffer), SEEK_SET);
fwrite(&buffer, sizeof(buffer), 1, f);
}
}
};
WriteAccess operator[] (int64_t elementNumber)
{
return WriteAccess(f, elementNumber);
}
};
struct SomeData
{
int a= 0;
int b= 0;
int c= 0;
};
int main()
{
Access<SomeData> myfile("thedata.bin");
myfile[0].get().a= 1;
auto pos1= myfile[1];
pos1.get().a= 10;
pos1.get().b= 10;
}
Of course, you would provide read acccess and write access, probably not using fopen but new c++ files, you should check for errors, and maybe you could get rid of get() function in form of a conversion operator to T.
You should also note that you could use some ref counting, in my simple example Access class should outlive WriteAccess class.
Also, you should lock if this is going to get used by more than one thread, and I assumed that you would not access the same element twice.
Or you could also use memory mapped file access like they've told you.