title: MPI error in communicating with many to many processors
body: <p>I am writing a code, where each processor must interact with multiple processors.</p>
  
  <p>Ex: I have 12 processors, so Processor 0 has to communicate to say 1,2,10 and 9. Lets call them as neighbours of Processor 0. Similarly I have
  Processor 1 has to communicate to say 5 ,3.
  Processor 2 has to communicate to 5,1,0,10,11
  and so on. 
  The flow of data is 2 ways, i.e Processor 0 must send data to 1,2,10 and 9 and also receive data from them.
  Also, there is no problem in Tag calculation.
  I have created a code which works like this:</p>
  
  <pre><code>for(all neighbours) 
  {
  store data in vector&lt;double&gt; x;
  
       MPI_Send(x)
  }
  MPI_BARRIER();
  for(all neighbours)
  {
  MPI_Recv(x);
  do work with x
  }
  </code></pre>
  
  <p>Now I testing this algorithm for different size of x and different arrangement of neighbours. The code works for some, but doesnot work for others, it simply resorts to deadlock.
  I have also tried:</p>
  
  <pre><code>for(all neighbours) 
  {
  store data in vector&lt;double&gt; x;
  
       MPI_ISend(x)
  }
  MPI_Test();
  for(all neighbours)
  {
  MPI_Recv(x);
  do work with x
  }
  </code></pre>
  
  <p>The result is same, although the deadlock is replcaed by NaN in result, as MPI_Test() tells me that some of the MPI_Isend() operation are not complete and it jumps immediately to MPI_Recv().</p>
  
  <p>Can anyone guide me in this matter, what am I dong wrong? Or is my fundamental approach itself is incorrect?</p>
  
  <p>EDIT: I am attaching code snippet for better understanding of the problem. I am basically workin on parallelizing an unstructured 3D-CFD solver</p>
  
  <p>I have attached one of the files, with some explanation. I am not broadcasting, I am looping over the neighbours of the parent processor to send the data across the interface( this can be defined as a boundary between two interfaces) . </p>
  
  <p>So, If say  I have 12 processors, and say Processor 0 has to communicate to say 1,2,10 and 9. So 0 is the parent processor and 1,2,10 and 9 are its neighbours.</p>
  
  <p>As the file was too long and a part of the solver, to make things simple, I have only kept the MPI function in it.</p>
  
  <pre><code>void Reader::MPI_InitializeInterface_Values() {
  double nbr_interface_id;
  Interface *interface;
  MPI_Status status;
  MPI_Request send_request, recv_request;
  int err, flag;
  int err2;
  char buffer[MPI_MAX_ERROR_STRING];
  int len;
  int count;
  
  
  for (int zone_no = 0; zone_no&lt;this-&gt;GetNumberOfZones(); zone_no++) { // Number of zone per processor is 1, so basically each zone is an independent processor
      UnstructuredGrid *zone = this-&gt;ZoneList[zone_no];
      int no_of_interface = zone-&gt;GetNumberOfInterfaces();
      // int count;
      long int count_send = 0;
      long int count_recv = 0;
      long int max_size = 10000; // can be set from test case later
      int max_size2 = 199;
  
      int proc_no = FlowSolution::processor_number;
      for (int interface_no = 0; interface_no &lt; no_of_interface; interface_no++) { // interface is defined as a boundary between two zones
  
  
          interface = zone-&gt;GetInterface(interface_no);
          int no_faces = interface-&gt;GetNumberOfFaces();
          if (no_faces != 0) {
  
              std::vector&lt; double &gt; Variable_send; // The vector which stores the data to be sent across the interface
              std::vector&lt; double &gt; Variable_recieve;
              int total_size = FlowSolution::VariableOrder.size() * no_faces;
              Variable_send.resize(total_size);
              Variable_recieve.resize(total_size);
              int nbr_proc_no = zone-&gt;GetInterface(interface_no)-&gt;GetNeighborZoneId(); // neighbour of parent processor
  
                  int j = 0;
                  nbr_interface_id = interface-&gt;GetShared_Interface_ID();
  
                  for (std::map&lt;VARIABLE, int&gt;::iterator iterator = FlowSolution::VariableOrder.begin(); iterator != FlowSolution::VariableOrder.end(); iterator++) {
  
                      for (int face_no = 0; face_no &lt; no_faces; face_no++) {
                          Face *face = interface-&gt;GetFace(face_no);
                          int owner_id = face-&gt;Getinterface_Original_face_owner_id();
                          double value_send = zone-&gt;GetInterface(interface_no)-&gt;GetFace(face_no)-&gt;GetCell(owner_id)-&gt;GetPresentFlowSolution()-&gt;GetVariableValue((*iterator).first);
                          Variable_send[j] = value_send;
                          j++;
                      }
                  }
                  count_send = nbr_proc_no * max_size + nbr_interface_id; // tag for data to be sent
                  err2 = MPI_Isend(&amp;Variable_send.front(), total_size, MPI_DOUBLE, nbr_proc_no, count_send, MPI_COMM_WORLD, &amp;send_request);
          }// end of sending
  
      } // all the processors have sent data to their corresponding neighbours
  
      MPI_Barrier(MPI_COMM_WORLD);
  
      for (int interface_no = 0; interface_no &lt; no_of_interface; interface_no++) { // loop over of neighbours of the current processor to receive data
  
          interface = zone-&gt;GetInterface(interface_no);
          int no_faces = interface-&gt;GetNumberOfFaces();
          if (no_faces != 0) {
              std::vector&lt; double &gt; Variable_recieve; // The vector which collects the data sent across the interface from 
              int total_size = FlowSolution::VariableOrder.size() * no_faces;
              Variable_recieve.resize(total_size);
              count_recv = proc_no * max_size + interface_no; // tag to receive data
              int nbr_proc_no = zone-&gt;GetInterface(interface_no)-&gt;GetNeighborZoneId();
              nbr_interface_id = interface-&gt;GetShared_Interface_ID();
                  MPI_Irecv(&amp;Variable_recieve.front(), total_size, MPI_DOUBLE, nbr_proc_no, count_recv, MPI_COMM_WORLD, &amp;recv_request);
  
                  /* Now some work is done using received data */
                  int j = 0;
                  for (std::map&lt;VARIABLE, int&gt;::iterator iterator = FlowSolution::VariableOrder.begin(); iterator != FlowSolution::VariableOrder.end(); iterator++) {
                      for (int face_no = 0; face_no &lt; no_faces; face_no++) {
                          double value_recieve = Variable_recieve[j];
                          j++;
                          Face *face = interface-&gt;GetFace(face_no);
                          int owner_id = face-&gt;Getinterface_Original_face_owner_id();
                          interface-&gt;GetFictitiousCell(face_no)-&gt;GetPresentFlowSolution()-&gt;SetVariableValue((*iterator).first, value_recieve);
                          double value1 = face-&gt;GetCell(owner_id)-&gt;GetPresentFlowSolution()-&gt;GetVariableValue((*iterator).first);
                          double face_value = 0.5 * (value1 + value_recieve);
                          interface-&gt;GetFace(face_no)-&gt;GetPresentFlowSolution()-&gt;SetVariableValue((*iterator).first, face_value);
                      }
                  }
                  // Variable_recieve.clear();
  
          }
  
      }// end of receiving
  </code></pre>
  
  <p>}</p>