title: MPI_Allreduce is slow for 7 processes or more
body: <p>I've written this little <code>MPI_Allreduce</code> benchmark: <a href="https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb34a/mpi/bench_mpi.cxx" rel="nofollow noreferrer">bench_mpi.cxx</a>.</p>
  
  <p>It work well with <code>Open MPI 1.8.4</code> and <code>MPICH 1.4.1</code>.</p>
  
  <p>The results (1 column for the number of processors, and 1 columns for the corresponding wall clock time) are <a href="https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb34a/mpi/carla/conda-mpi4py-channel.openmpi.1.8.4.txt" rel="nofollow noreferrer">here</a> or <a href="https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb34a/mpi/carla/conda-default.mpich2.1.4.1p1.txt" rel="nofollow noreferrer">here</a>.</p>
  
  <p>With <code>MPICH 3.1.4</code>, the wall clock time increase for 7, 8 or more processes: results are <a href="https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb34a/mpi/carla/conda-mpi4py-channel.mpich.3.1.4.txt" rel="nofollow noreferrer">here</a>.</p>
  
  <p>In a real code (edit: a Computational Fluid Dynamic software), but for all of the 3 above MPI implementation, I observe the same problem for 7, 8 or more processes, while I expect my code to be scallable to at least 8 or 16 processes.</p>
  
  <p>So I'm trying to understand what could happen with the little benchmark and <code>MPICH 3.1.4</code>?</p>
  
  <h2>edit</h2>
  
  <p>Here is a zoom in the figure Rob Latham give in his answer.</p>
  
  <p><img src="https://i.stack.imgur.com/MXISQ.png" alt="idle"></p>
  
  <p>What does the code do during the green rectangle? The <code>Mpi_Allreduce</code> operation starts too late.</p>
  
  <h2>edit</h2>
  
  <p>I've posted another <a href="https://stackoverflow.com/questions/30039802/what-control-mpi-barrier-time-to-execute">question</a> on much more simpler code (just the time to execute <code>MPI_Barrier</code>).</p>