title: MPI - How to avoid using two buffers in a non-blocking send & receive?
body: <p>I have a use-case where I need to transmit an Array of Structures (AoS) among a group of processors in a circular ("ring") fashion.
  I do that currently by using a non-blocking send and receive, and using two auxiliary buffers on the senders and receivers side respectively.</p>
  
  <p>Since the data I am transferring is extremely huge (~60 gb before partitioning), allocating two buffers seems inefficient to me. My question is -- Is there a better way to do it by avoiding "two buffers"? (Perhaps just using one buffer in some way?, so that I can reduce my memory requirement).</p>
  
  <p>Roughly my communication procedure looks like this:</p>
  
  <pre><code>//Pack AoS into send buffer (which is a vector&lt;char&gt;)        
          char *msg_ptr = &amp;(msg_send_buf_.front());
          for (auto&amp; param : local_primal_params_) {
            param.pack_to(msg_ptr);
            msg_ptr += PARAM_BYTENUM;
          }
  
  //Perform non-blocking send receive
          {
            MPI_Request send_request;
            MPI_Status send_stat, recv_stat;
            MPI_Isend(&amp;(msg_send_buf_.front()), local_primal_params_.size() * PARAM_BYTENUM, MPI_CHAR,
                      send_to, 0,
                      MPI_COMM_WORLD, &amp;send_request);
            MPI_Recv(&amp;(msg_recv_buf_.front()), local_primal_params_.size() * PARAM_BYTENUM, MPI_CHAR,
                      recv_from, 0,
                      MPI_COMM_WORLD, &amp;recv_stat);
            MPI_Wait(&amp;send_request, &amp;send_stat);
          }
  
  //Unpack receive buffer back into AoS on receiver side
          {
            char *msg_ptr = &amp;(msg_recv_buf_.front());
            for (auto&amp; param : local_primal_params_) {
              param.unpack_from(msg_ptr);
              msg_ptr += PARAM_BYTENUM;
            }
          }
  </code></pre>
  
  <p>Any recommendations will be very helpful.</p>