title: Why does the speed of memcpy() drop dramatically every 4KB?
body: <p>I tested the speed of <code>memcpy()</code> noticing the speed drops dramatically at i*4KB. The result is as follow: the Y-axis is the speed(MB/second) and the X-axis is the size of buffer for <code>memcpy()</code>, increasing from 1KB to 2MB. Subfigure 2 and Subfigure 3 detail the part of 1KB-150KB and 1KB-32KB.</p>
  
  <p>Environment：</p>
  
  <p>CPU : Intel(R) Xeon(R) CPU E5620  @ 2.40GHz </p>
  
  <p>OS :  2.6.35-22-generic #33-Ubuntu</p>
  
  <p>GCC compiler flags : -O3 -msse4 -DINTEL_SSE4 -Wall -std=c99</p>
  
  <p><img src="https://i.stack.imgur.com/TJ4Xx.png" alt="Graphs of memcpy speed showing troughs every 4k"></p>
  
  <p>I guess it must be related to caches, but I can't find a reason from the following cache-unfriendly cases:</p>
  
  <ul>
  <li><p><a href="https://stackoverflow.com/questions/12264970/why-is-my-program-slow-when-looping-over-exactly-8192-elements">Why is my program slow when looping over exactly 8192 elements?</a></p></li>
  <li><p><a href="https://stackoverflow.com/questions/11413855/why-is-transposing-a-matrix-of-512x512-much-slower-than-transposing-a-matrix-of">Why is transposing a matrix of 512x512 much slower than transposing a matrix of 513x513?</a></p></li>
  </ul>
  
  <p>Since the performance degradation of these two cases are caused by unfriendly loops which read scattered bytes into the cache, wasting the rest of the space of a cache line.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>void memcpy_speed(unsigned long buf_size, unsigned long iters){
      struct timeval start,  end;
      unsigned char * pbuff_1;
      unsigned char * pbuff_2;
  
      pbuff_1 = malloc(buf_size);
      pbuff_2 = malloc(buf_size);
  
      gettimeofday(&amp;start, NULL);
      for(int i = 0; i &lt; iters; ++i){
          memcpy(pbuff_2, pbuff_1, buf_size);
      }   
      gettimeofday(&amp;end, NULL);
      printf("%5.3f\n", ((buf_size*iters)/(1.024*1.024))/((end.tv_sec - \
      start.tv_sec)*1000*1000+(end.tv_usec - start.tv_usec)));
      free(pbuff_1);
      free(pbuff_2);
  }
  </code></pre>
  
  <h1>UPDATE</h1>
  
  <p>Considering suggestions from @usr, @ChrisW and @Leeor, I redid the test more precisely and the graph below shows the results. The buffer size is from 26KB to 38KB, and I tested it every other 64B(26KB, 26KB+64B, 26KB+128B, ......, 38KB). Each test loops 100,000 times in about 0.15 second. The interesting thing is the drop not only occurs exactly in 4KB boundary, but also comes out in 4*i+2 KB, with a much less falling amplitude.</p>
  
  <p><img src="https://i.stack.imgur.com/qto0R.png" alt="More graphs showing performance drops"></p>
  
  <h1>PS</h1>
  
  <p>@Leeor offered a way to fill the drop, adding a 2KB dummy buffer between <code>pbuff_1</code> and <code>pbuff_2</code>. It works, but I am not sure about Leeor's explanation.</p>
  
  <p><img src="https://i.stack.imgur.com/NL3uy.png" alt="enter image description here"> </p>