title: MPI: MPI_recv with MPI_ANY_SOURCE can not receive messages from some processes
body: <p>I want to implement a system, where there are one receiver and multiple senders. Each sender keeps sending data to the receiver. The receiver waits to receive data and process it. Here is the toy example.</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;cstdlib&gt;
  #include &lt;mpi.h&gt;
  using namespace std;
  
  int main(int argc, char *argv[]) {
      int _mpi_numWorkers, _mpi_rank;
  
      // Initialize openMPI
      MPI_Init(&amp;argc, &amp;argv);
  
      MPI_Comm_size(MPI_COMM_WORLD, &amp;_mpi_numWorkers);
      MPI_Comm_rank(MPI_COMM_WORLD, &amp;_mpi_rank);
  
      MPI_Barrier(MPI_COMM_WORLD);
  
      float *send_data = (float *)malloc(360 * 5000 * sizeof(float));
      MPI_Status receive_status;
  
      if (_mpi_rank == 0) {
          while (1) {
              MPI_Recv(send_data, 360 * 5000, MPI_FLOAT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &amp;receive_status);
  
              cout &lt;&lt; "Receive from " &lt;&lt; receive_status.MPI_SOURCE &lt;&lt; endl;
          }
      } else {
  
          while (1){
              MPI_Send(send_data, 360 * 5000, MPI_FLOAT, 0, 0, MPI_COMM_WORLD);
  
              //sleep(1);
          }
      }
  
      // Terminate
      MPI_Finalize();
      return 0;
  }
  </code></pre>
  
  <p>The issue is that MPI_recv can only receive messages from up to two processes no matter how many processes I set it to run (when no sleep). I have tested this code on one single machine, and on multiple machines cases:</p>
  
  <p><strong>Single Machine Case</strong></p>
  
  <p>I run this code through the following command:</p>
  
  <pre><code>mpiexec -n 5 ./test_mpi
  </code></pre>
  
  <p>Then, the receiver only receive from senders with rank 1 and 2.</p>
  
  <p><strong>Multiple Machine Case</strong></p>
  
  <p>I run 4 senders and 1 receiver on 5 homogeneous physical machines. All of them connect to a 100Mbps switch. In this case, the receiver also only receives data from a subset of senders. I use the tcpdump to check the packet, and observe that some senders do not even send the message. (Those senders are blocked at MPI_send, but no tcp sequence increases and no re-transmission.)</p>
  
  <p>For those two cases, if I make each sender sleep some time (decreasing the sending rate), the receiver can receive data from more senders.</p>
  
  <p>Can somebody help me to understand why this happens?</p>
  
  <p><strong>Environment</strong></p>
  
  <p>Debian testing with openmpi-1.6</p>
  
  <p><strong>Edit 2/4/16</strong></p>
  
  <p>I include <code>&lt;cstdlib&gt;</code> in the code to prevent any compilation issues.</p>