How to reduce latency when access new page in linux - c++

Question

The page size is 4KB in linux, but I access a new page, the latency is about twice large than inside a page. How can I reduce this latency?
Here is my test code. I use clock_gettime to measure the time cost in the main function.
#define MEM_SIZE 4096 * 12
long long GetRT() {
struct timespec tp;
clock_gettime(CLOCK_REALTIME, &tp);
return (long long) tp.tv_sec * 1000000000 + tp.tv_nsec;
}
void *InitSharedMemory() {
int fd = shm_open("/test-steve", O_CREAT | O_RDWR, ACCESSPERMS);
if (fd < 0) {
perror("shm_open");
}
if (ftruncate(fd, MEM_SIZE) < 0) {
perror("ftruncate");
}
void *mmap_ptr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_POPULATE, fd, 0);
if (mmap_ptr == (caddr_t)-1) {
perror("mmap");
}
return mmap_ptr;
}
int main()
{
constexpr auto STEP = 512;
char arr[STEP];
for (int i = 0; i < STEP; ++i) {
arr[i] = i;
}
int ret = 0;
void *buffer = InitSharedMemory();
for (int i = 0; i < 40; ++i) {
if ((i % 0x8) == 0) {
// *(char *) buffer = 'a';
}
auto t1 = GetRT();
*(char *) buffer = 'a';
auto t2 = GetRT();
ret += t2 - t1;
printf("cost: %lld ns\n", t2 - t1);
buffer = (char *) buffer + STEP;
}
return ret;
}
The result is some like follows, is the extra latency TLB missing? how to avoid it?
cost: 272 ns
cost: 73 ns
cost: 76 ns
cost: 74 ns
cost: 74 ns
cost: 75 ns
cost: 73 ns
cost: 76 ns
cost: 281 ns
cost: 74 ns
cost: 73 ns
cost: 76 ns
cost: 76 ns
cost: 74 ns
cost: 73 ns
cost: 76 ns
cost: 334 ns
cost: 76 ns
cost: 76 ns
cost: 76 ns
cost: 76 ns
cost: 73 ns
cost: 76 ns
cost: 76 ns
cost: 267 ns
cost: 74 ns
cost: 74 ns
cost: 75 ns
cost: 73 ns
cost: 76 ns
cost: 73 ns
cost: 76 ns
cost: 260 ns

Answer 1

I am afraid that you cannot avoid the cost of the first hit to a page (as illustrated by your example).
May be this cost could be amortised when using huge-pages (2MB) with MAP_HUGETLB in the flags of mmap().
Depending on the applicative context, touching the first byte (or any) of each page once for all, right after the allocation, will force the commit (may be useful on numa hardware).
This will have a large initial cost but could lead to more predictable timings for the subsequent accesses.