title: Memcpy takes the same time as memset
body: <p>I want to measure memory bandwidth using <code>memcpy</code>. I modified the code from this answer:<a href="https://stackoverflow.com/a/18159503/2542702">why vectorizing the loop does not have performance improvement</a> which used <code>memset</code> to measure the bandwidth. The problem is that <code>memcpy</code> is only slighly slower than <code>memset</code> when I expect it to be about two times slower since it operations on twice the memory.</p>
  
  <p>More specifically, I run over 1 GB arrays <code>a</code> and <code>b</code> (allocated will <code>calloc</code>) 100 times with the following operations.</p>
  
  <pre><code>operation             time(s)
  -----------------------------
  memset(a,0xff,LEN)    3.7
  memcpy(a,b,LEN)       3.9
  a[j] += b[j]          9.4
  memcpy(a,b,LEN)       3.8
  </code></pre>
  
  <p>Notice that <code>memcpy</code> is only slightly slower then <code>memset</code>. The operations <code>a[j] += b[j]</code> (where <code>j</code> goes over <code>[0,LEN)</code>) should take three times longer than <code>memcpy</code> because it operates on three times as much data.  However it's only about 2.5 as slow as <code>memset</code>. </p>
  
  <p>Then I initialized <code>b</code> to zero with <code>memset(b,0,LEN)</code> and test again:</p>
  
  <pre><code>operation             time(s)
  -----------------------------
  memcpy(a,b,LEN)       8.2
  a[j] += b[j]          11.5
  </code></pre>
  
  <p>Now we see that <code>memcpy</code> is about twice as slow as <code>memset</code> and <code>a[j] += b[j]</code> is about thrice as slow as <code>memset</code> like I expect.</p>
  
  <p>At the very least I would have expected that before <code>memset(b,0,LEN)</code> that <code>memcpy</code> would be <a href="https://stackoverflow.com/a/11579987/2542702">slower because the of lazy allocation (first touch)</a> on the first of the 100 iterations.</p>
  
  <p><strong>Why do I only get the time I expect after <code>memset(b,0,LEN)</code>?</strong></p>
  
  <p>test.c</p>
  
  <pre><code>#include &lt;time.h&gt;
  #include &lt;string.h&gt;
  #include &lt;stdio.h&gt;
  
  void tests(char *a, char *b, const int LEN){
      clock_t time0, time1;
      time0 = clock();
      for (int i = 0; i &lt; 100; i++) memset(a,0xff,LEN);
      time1 = clock();
      printf("%f\n", (double)(time1 - time0) / CLOCKS_PER_SEC);
  
      time0 = clock();
      for (int i = 0; i &lt; 100; i++) memcpy(a,b,LEN);
      time1 = clock();
      printf("%f\n", (double)(time1 - time0) / CLOCKS_PER_SEC);
  
      time0 = clock();
      for (int i = 0; i &lt; 100; i++) for(int j=0; j&lt;LEN; j++) a[j] += b[j];
      time1 = clock();
      printf("%f\n", (double)(time1 - time0) / CLOCKS_PER_SEC);
  
      time0 = clock();
      for (int i = 0; i &lt; 100; i++) memcpy(a,b,LEN);
      time1 = clock();
      printf("%f\n", (double)(time1 - time0) / CLOCKS_PER_SEC);
  
      memset(b,0,LEN);
      time0 = clock();
      for (int i = 0; i &lt; 100; i++) memcpy(a,b,LEN);
      time1 = clock();
      printf("%f\n", (double)(time1 - time0) / CLOCKS_PER_SEC);
  
      time0 = clock();
      for (int i = 0; i &lt; 100; i++) for(int j=0; j&lt;LEN; j++) a[j] += b[j];
      time1 = clock();
      printf("%f\n", (double)(time1 - time0) / CLOCKS_PER_SEC);
  }
  </code></pre>
  
  <p>main.c</p>
  
  <pre><code>#include &lt;stdlib.h&gt;
  
  int tests(char *a, char *b, const int LEN);
  
  int main(void) {
      const int LEN = 1 &lt;&lt; 30;    //  1GB
      char *a = (char*)calloc(LEN,1);
      char *b = (char*)calloc(LEN,1);
      tests(a, b, LEN);
  }
  </code></pre>
  
  <p>Compile with (gcc 6.2) <code>gcc -O3 test.c main.c</code>. Clang 3.8 gives essentially the same result.</p>
  
  <p>Test system: i7-6700HQ@2.60GHz (Skylake), 32 GB DDR4, Ubuntu 16.10. On my Haswell system the bandwidths make sense before <code>memset(b,0,LEN)</code> i.e. I only see a problem on my Skylake system.</p>
  
  <p>I first discovered this issue from the <code>a[j] += b[k]</code> operations <a href="https://stackoverflow.com/questions/42964820/why-is-this-simd-multiplication-not-faster-than-non-simd-multiplication/42972674#42972674">in this answer</a> which was overestimating the bandwidth.</p>
  
  <hr>
  
  <p>I came up with a simpler test</p>
  
  <pre><code>#include &lt;time.h&gt;
  #include &lt;string.h&gt;
  #include &lt;stdio.h&gt;
  
  void __attribute__ ((noinline))  foo(char *a, char *b, const int LEN) {
    for (int i = 0; i &lt; 100; i++) for(int j=0; j&lt;LEN; j++) a[j] += b[j];
  }
  
  void tests(char *a, char *b, const int LEN) {
      foo(a, b, LEN);
      memset(b,0,LEN);
      foo(a, b, LEN);
  }
  </code></pre>
  
  <p>This outputs. </p>
  
  <pre><code>9.472976
  12.728426
  </code></pre>
  
  <p>However, if I do <code>memset(b,1,LEN)</code> in main after <code>calloc</code> (see below) then it outputs </p>
  
  <pre><code>12.5
  12.5
  </code></pre>
  
  <p>This leads me to to think this is a OS allocation issue and not a compiler issue.</p>
  
  <pre><code>#include &lt;stdlib.h&gt;
  
  int tests(char *a, char *b, const int LEN);
  
  int main(void) {
      const int LEN = 1 &lt;&lt; 30;    //  1GB
      char *a = (char*)calloc(LEN,1);
      char *b = (char*)calloc(LEN,1);
      //GCC optimizes memset(b,0,LEN) away after calloc but Clang does not.
      memset(b,1,LEN);
      tests(a, b, LEN);
  }
  </code></pre>