title: Limit to number of Broadcasts on a cluster?
body: <p>I have a problem with a loop containing mpi broadcast line. After 1000th iterations, the broadcast line stops or hangs the execution (waiting for the transmission presumably).</p>
  
  <p>See the following code:</p>
  
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;string.h&gt;
  #include "mpi.h"
  
  int main(int argc, char* argv[]){
      int  aa = 0, size = 10000;
      int  my_rank; /* rank of process */
      int  smessage = 9, rmessage = 9;        /* storage for message */
  
      MPI_Init(&amp;argc, &amp;argv);
      MPI_Comm_rank(MPI_COMM_WORLD, &amp;my_rank);
  
      if(my_rank !=0) {
          for(aa=1;aa&lt;size;aa++) {
              MPI_Bcast(&amp;rmessage, 1, MPI_INT, 0, MPI_COMM_WORLD);
              printf("rec message(i=%d)=%d\n", aa, rmessage);
          }
      }
      else {
          for(aa=1;aa&lt;size;aa++) {
              smessage=aa;
              printf("send message(i=%d)=%d\n", aa, smessage);
              MPI_Bcast(&amp;smessage, 1, MPI_INT, 0, MPI_COMM_WORLD);
          }
      }
      MPI_Finalize();
      return 0;
  }
  </code></pre>
  
  <p>when I run it with</p>
  
  <pre><code>mpicc -openmp -fopenmp example_code.c -o example_prog
  mpirun -n 3 example_prog
  </code></pre>
  
  <p>I get the expected output (up to some rearrangements)</p>
  
  <pre><code>sent message(i=1)=1
  rec message(i=1)=1
  rec message(i=1)=1
  ...
  sent message(i=9999)=9999
  rec message(i=9999)=9999
  rec message(i=9999)=9999
  </code></pre>
  
  <p>However, when I send it to my university cluster using SLURM's sbatch and a script containing</p>
  
  <pre><code>mpirun -n 3 --loadbalance --cpus-per-proc 24 ./example_prog
  </code></pre>
  
  <p>(I'm using OpenMPI 1.6.5) I get the program either hanged in the middle of the loop or gets automatically terminated by the job handler. The output is</p>
  
  <pre><code>sent message(i=1)=1
  rec message(i=1)=1
  rec message(i=1)=1
  ...
  
  rec message(i=9999)=999
  rec message(i=9999)=999
  sent message(i=9999)=1000
  </code></pre>
  
  <p>It clearly stops in the "middle" of the loop. Would you have any ideas as to what causes this error or how to avoid this error?</p>
  
  <p>Thank you so much!</p>