title: Copying Memory Pages in SLES Linux is much faster then in Ubuntu
body: <p>I have two virtual machines running on the exact same Hardware. One of them is SLES12SP5 (Kernel 4.12) (I have similar results with SLES15SP2, but got no at hand at the moment) and one of them is Ubuntu 20.04 (Kernel 5.4). I have a really simple C-Program which measures writing into allocated memory pages and copying them. This program is WAY faster on SLES then on Ubuntu and I can't figure out why. Unfortunately I can't activate the performance counters on the esx Host because of the cluster configuration, maybe that would help to find out whats happening here.</p>
  <p>Okay, so here is the Programm:</p>
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;string.h&gt;
  #ifdef __linux__
  #include &lt;sys/time.h&gt;
  #elif _WIN32
  #include &lt;sys/timeb.h&gt;
  #endif
  
  #ifdef __linux__
  double get_time_insec() {
      struct timeval tstruct;
  
      gettimeofday(&amp;tstruct, NULL);
      long long milliseconds = tstruct.tv_sec*1000LL + tstruct.tv_usec/1000;
      return (double)milliseconds/1000;
  }
  #elif _WIN32
  double get_time_insec() {
      struct timeb tstruct;
  
      ftime(&amp;tstruct);
      return (double)tstruct.time + ((double)tstruct.millitm / (double)1000);
  }
  #endif
  
  #define PAGESIZE 8 * 1024
  
  int main(int argc, char *argv[]) {
      int     i = 0;
      int     x = 0;
      double  startval = 0;
      int     pagenum = 0;
      int     offset = 0;
      int     numOfPages = 0;
      int     numOfTimes = 0;
      size_t  allocBytes = 0;
  
      if (argc != 3) {
           printf(&quot;Usage %s numOfPages numOfTimes\n&quot;, argv[0]);
           return -1;
      }
  
      numOfPages = atoi(argv[1]);
      numOfTimes = atoi(argv[2]);
      allocBytes = numOfPages*PAGESIZE*2;
  
      // Allocate memory Pages
      printf(&quot;Allocating %ld Bytes\n&quot;, allocBytes);
      char *mymem = malloc(allocBytes);
  
      if (mymem == NULL) {
          printf(&quot;Allocation failed!\n&quot;);
          return -1;
      }
  
      // Fill the first half of Pages with text
      printf(&quot;Filling %d Pages %d times...&quot;, numOfPages, numOfTimes);
      fflush(stdout);
      startval = get_time_insec();
      for (x = 0; x &lt; numOfTimes; x++) {
          for (pagenum = 0; pagenum &lt; numOfPages; pagenum++) {
              for (i = 0; i &lt; PAGESIZE; i += 32) {
                  offset = (pagenum * PAGESIZE) + i;
                  memcpy(mymem + offset, &amp;&quot;ABCDEFGHIJKLMNOPQRSTUVWXYZ123456&quot;, 32);
              }
          }
      }
      printf(&quot;Time taken: %.6f sec\n&quot;, get_time_insec() - startval);
  
      // And now copy them to the next half of pages
      printf(&quot;Copying %d Pages %d times...&quot;, numOfPages/2, numOfTimes);
      fflush(stdout);
      startval = get_time_insec();
      for (x = 0; x &lt; numOfTimes; x++) {
          for (pagenum = 0; pagenum &lt; numOfPages; pagenum++) {
              memcpy((mymem + (numOfPages/2) * PAGESIZE) + (pagenum * PAGESIZE), mymem + (pagenum * PAGESIZE), PAGESIZE);
          }
      }
      printf(&quot;Time taken: %.6f sec\n&quot;, get_time_insec() - startval);
  
      free(mymem);
  
      return 0;
  }
  </code></pre>
  <p>And this are the results for running it with &quot;2000 200&quot;:</p>
  <p>Ubuntu:</p>
  <pre><code>Filling 2000 Pages 200 times...Time taken: 0.606000 sec
  Copying 1000 Pages 200 times...Time taken: 0.921000 sec
  </code></pre>
  <p>SLES:</p>
  <pre><code>Filling 2000 Pages 200 times...Time taken: 0.513000 sec
  Copying 1000 Pages 200 times...Time taken: 0.479000 sec
  </code></pre>
  <p>I can't get my head around why SLES is twice as fast then Ubuntu.
  I thought that writing into memory is a operation where the os doesn't do anything, apart from allocating the pages in the MMU beforehand.
  So why exactly could it be that SLES is twice as fast?
  Any kernel parameter? Any parameter in the virtual memory subsystem?</p>
  <p>I would love to solve this which is bugging me for weeks now!</p>
  <p>Thomas</p>