Revision history [back]

Thanks to @sgermanserrano for pointing out to his workaround on how to decrease memory usage this gave me a first clue on what's going on.

What happens in lock_and_prefault_dynamic function?

On every iteration 262144 bytes (64 * page_size) allocated on the heap memory by new char array. Also there is a std::vector which holds pointers to allocated arrays. So on every iteration this vector increased by 8 bytes (size of 64 bit pointer). As soon as vector needs to be resized, it reallocated 2x new heap memory, copies data there and deallocates old memory (see details here).

This means that heap memory fragmented with chunks of free memory from vector reallocations. And as soon as there will be available free chunk of size 262144 bytes it will be used for new char array and no minor page fault will happen, which will meet while loop condition.

Why 8GB?

In order to reach size of 262144 bytes vector needs 262144 / 8 = 32768 elements, which is 32768 iterations, which is 32768 of char arrays of size 64 * 4 * 1024 totals into 8589934592 bytes (8GB). Plus memory from reallocated vector data, which is not so significant (actually two char arrays - 512KB).

Code

I did a small patch with debug logging, which shows the following

DEBUG: Memory chunk required 262144
DEBUG: Heap memory chunk of size 8 will be available between 0x55599f4fe0 - 0x55599f4fe0.
DEBUG: Heap memory chunk of size 16 will be available between 0x5559a7b220 - 0x5559a7b228.
DEBUG: Heap memory chunk of size 32 will be available between 0x5559abb250 - 0x5559abb268.
DEBUG: Heap memory chunk of size 64 will be available between 0x5559b3b2a0 - 0x5559b3b2d8.
DEBUG: Heap memory chunk of size 128 will be available between 0x5559c3b330 - 0x5559c3b3a8.
DEBUG: Heap memory chunk of size 256 will be available between 0x5559e3b440 - 0x5559e3b538.
DEBUG: Heap memory chunk of size 512 will be available between 0x555a23b650 - 0x555a23b848.
DEBUG: Heap memory chunk of size 1024 will be available between 0x555aa3ba60 - 0x555aa3be58.
DEBUG: Heap memory chunk of size 2048 will be available between 0x555ba3c270 - 0x555ba3ca68.
DEBUG: Heap memory chunk of size 4096 will be available between 0x555da3d280 - 0x555da3e278.
DEBUG: Heap memory chunk of size 8192 will be available between 0x5561a3f290 - 0x5561a41288.
DEBUG: Heap memory chunk of size 16384 will be available between 0x5569a432a0 - 0x5569a47298.
DEBUG: Heap memory chunk of size 32768 will be available between 0x5579a4b2b0 - 0x5579a532a8.
DEBUG: Heap memory chunk of size 65536 will be available between 0x5599a5b2c0 - 0x5599a6b2b8.
DEBUG: Heap memory chunk of size 131072 will be available between 0x55d9a7b2d0 - 0x55d9a9b2c8.
DEBUG: Heap memory chunk of size 262144 will be available between 0x5659abb2e0 - 0x5659afb2d8.
DEBUG: New char array address 0x5659abb2e0 is lower the old one 0x5759b7b2f0.
DEBUG: This means it was created in already allocated memory.

Patch

diff --git a/rttest/src/rttest.cpp b/rttest/src/rttest.cpp
index cb21d13..6d78bfb 100644
--- a/rttest/src/rttest.cpp
+++ b/rttest/src/rttest.cpp
@@ -676,6 +676,8 @@ int Rttest::lock_and_prefault_dynamic()
   size_t prev_majflts = usage.ru_majflt;
   size_t encountered_minflts = 1;
   size_t encountered_majflts = 1;
+
+  printf("DEBUG: Memory chunk required %zu\n", (64 * page_size));
   // prefault until you see no more pagefaults
   while (encountered_minflts > 0 || encountered_majflts > 0) {
     char * ptr;
@@ -694,6 +696,22 @@ int Rttest::lock_and_prefault_dynamic()
       munlockall();
       return -1;
     }
+
+    if (&prefaulter.front() != nullptr)
+    {
+      if (prefaulter.size() == prefaulter.capacity())
+      {
+        long int v_size = ((&prefaulter.back() - &prefaulter.front()) + 1) * sizeof(void*);
+        printf("DEBUG: Heap memory chunk of size %ld will be available between %p - %p.\n", v_size, &prefaulter.front(), &prefaulter.back());
+      }
+
+      if ((ptr - prefaulter.back()) < 0)
+      {
+        printf("DEBUG: New char array address %p is lower the old one %p.\n", ptr, &prefaulter.back());
+        printf("DEBUG: This means it was created in already allocated memory.\n");
+      }
+    }
+
     prefaulter.push_back(ptr);
     getrusage(RUSAGE_SELF, &usage);
     size_t current_minflt = usage.ru_minflt;