title: Cache-Misses in OSU Unidirectional Bandwidth Test
body: <p>We are measuring the cache-misses in <code>MPI_Isend()+MPI_Waitall()</code> at the Sender and <code>MPI_Irecv()+MPI_Waitall()</code> at the receiver for the OSU unidrectional bandwidth benchmark. Suprisingly, the cache-misses at <code>8KB</code> message size exceed that at the <code>16KB</code> message size when using <code>Intel MPI 2017.4</code> and <code>Intel compiler 2017.4</code>. We capture cache-misses using the profiler TAU. Below are some graphs which capture this strange behaviour:
  <a href="https://i.stack.imgur.com/KkoZK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KkoZK.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/YWw98.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YWw98.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/oWGiw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oWGiw.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/vksr1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vksr1.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/NFzx7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NFzx7.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/xoTVW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xoTVW.png" alt="enter image description here" /></a></p>
  <p>We have ruled out the effect of Eager-Rendezvous switchover as the default <code>I_MPI_EAGER_THRESHOLD</code> is <code>256 KB</code> and the re-allocation of pre-registered buffers as the message size is less than the default size of pre-registered buffers (which is around 23.5 KB for Intel MPI).</p>
  <p>Any insight will be valuable. Thanks !</p>
  <p><strong>Addition 1</strong>: The communication is between nodes which are connected with Intel OPA having a maximum bandwidth of <code>100 Gbits/sec</code>.</p>
  <p><strong>Addition 2</strong>: We <em>suspect</em> that the default <code>Path MTU size of 8 KB</code> for Intel OPA has &quot;something&quot; to do with it but still the questions remains as to why cache-misses for a message of size 16 KB is less than that at 8 KB.</p>