title: Overlapping communications with computations in MPI (mvapich2) for large messages
body: <p>I have a very simple code, a data decomposition problem in which in a loop each process sends two large messages to the ranks before and after itself at each cycle. I run this code in a cluster of SMP nodes (AMD Magny cores, 32 core per node, 8 cores per socket). It's a while I'm in the process of optimizing this code. I have used pgprof and tau for profiling and it looks to me that the bottleneck is the communication. I have tried to overlap the communication with the computations in my code however it looks that the actual communication starts when the computations finish :(</p>
  
  <p>I use persistent communication in ready mode (MPI_Rsend_init) and in between the MPI_Start_all and MPI_Wait_all bulk of the computation is done. The code looks like this:</p>
  
  <pre><code>void main(int argc, char *argv[])
  {
    some definitions;
    some initializations;
  
    MPI_Init(&amp;argc, &amp;argv);
  
    MPI_Rsend_init( channel to the rank before );
    MPI_Rsend_init( channel to the rank after );
    MPI_Recv_init( channel to the rank before );
    MPI_Recv_init( channel to the rank after );
  
    for (timestep=0; temstep&lt;Time; timestep++)
    {
      prepare data for send;
      MPI_Start_all();
  
      do computations;
  
      MPI_Wait_all();
  
      do work on the received data;
    }
    MPI_Finalize();
  }
  </code></pre>
  
  <p>Unfortunately the actual data transfer does not start until the computations are done, I don't understand why. The network uses QDR InfiniBand Interconnect and mvapich2. each message size is 23MB (totally 46 MB message is sent). I tried to change the message passing to eager mode, since the memory in the system is large enough. I use the following flags in my job script:
      MV2_SMP_EAGERSIZE=46M<br>
      MV2_CPU_BINDING_LEVEL=socket<br>
      MV2_CPU_BINDING_POLICY=bunch  </p>
  
  <p>Which gives me an improvement of about 8%, probably because of better placement of the ranks inside the SMP nodes however still the problem with communication remains. My question is why can't I effectively overlap the communications with the computations? Is there any flag that I should use and I'm missing it? I know something is wrong, but whatever I have done has not been enough. </p>
  
  <p>By the order of ranks inside the SMP nodes the actual message sizes between the nodes is also 46MB (2x23MB) and the ranks are in a loop. Can you please help me? To see the flags that other users use I have checked /etc/mvapich2.conf however it is empty. </p>
  
  <p>Is there any other method that I should use? do you think one sided communication gives better performance? I feel there is a flag or something that I'm not aware of.</p>
  
  <p>Thanks alot.</p>