title: Questions about MPI Non-blocking details based on standard
body: <p>So more recently, I have been developing some asynchronous algorithms in my research. I was doing some parallel performance studies and I have been suspicious that I am not properly understanding some details about the various non-blocking MPI functions.</p>
  
  <p>I've seen some insightful posts on here, namely:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/10017301/mpi-blocking-vs-non-blocking">MPI: blocking vs non-blocking</a></li>
  <li><a href="https://stackoverflow.com/questions/22826470/mpi-non-blocking-irecv-didnt-receive-data">MPI Non-blocking Irecv didn&#39;t receive data?</a></li>
  </ul>
  
  <p>There's a few things I am uncertain about or just want to clarify related to working with non-blocking functionality that I think will help me potentially increase the performance of my current software.</p>
  
  <p>From the <em>Nonblocking Communication</em> part of the MPI 3.0 standard:</p>
  
  <blockquote>
    <p>A <strong>nonblocking send</strong> start call initiates
    the send operation, but <em>does not</em> complete it. The send start call can return before the message was copied out of the send buffer. A separate send complete call is needed to complete the communication, i.e., to verify that the data has been copied out of the send buffer. <em>With suitable hardware, the transfer of data out of the sender memory may proceed
    concurrently with computations done at the sender after the send was initiated and before it completed</em>.</p>
    
    <p>...</p>
    
    <p>If the send mode is <strong>standard</strong> then the send-complete call <em>may</em>
    return before a matching receive is posted, if the message is
    buffered. On the other hand, the receive-complete <em>may not</em> complete
    until a matching receive is posted, and the message was copied into
    the receive buffer.</p>
  </blockquote>
  
  <p>So as a first set of questions about the <code>MPI_Isend</code> (and similarly <code>MPI_Irecv</code>), it seems as though to ensure a non-blocking send finishes, I need to use some mechanism to <em>check</em> that it is complete because in the worst case, there <em>may not</em> be suitable hardware to transfer the data concurrently, right? So if I never use something like <code>MPI_Test</code> or <code>MPI_Wait</code> following the non-blocking send, the <code>MPI_Isend</code> may never actually get its message out, right?</p>
  
  <p>This question applies to some of my work because I am sending messages via <code>MPI_Isend</code> and not actually testing for completeness until I get the expected response message because I want to avoid the overhead of <code>MPI_Test</code> calls. While this approach has been working, it seems faulty based on my reading.</p>
  
  <p>Further, the second paragraph appears to say that for the standard non-blocking send, <code>MPI_Isend</code>, it <em>may not</em> even begin to send any of its data until the destination process has called a matching receive. Given the availability of <code>MPI_Probe</code>/<code>MPI_Iprobe</code>, does this mean an <code>MPI_Isend</code> call will <em>at least</em> send out some preliminary metadata of the message, such as size, source, and tag, so that the probe functions on the destination process can know a message wants to be sent there and so the destination process can actually post a corresponding receive?</p>
  
  <p>Related is a question about the probe. In the <em>Probe and Cancel</em> section, the standard says that</p>
  
  <blockquote>
    <p><code>MPI_IPROBE(source, tag, comm, flag, status)</code> returns <code>flag = true</code> if there is a message that can be received and that matches the pattern specifed by the arguments <code>source</code>, <code>tag</code>, and <code>comm</code>. The call matches the same message that would have been received by a call to <code>MPI_RECV(..., source, tag, comm, status)</code> executed at the same point in the program, and returns in status the same value that would have been returned by <code>MPI_RECV()</code>. Otherwise, the call returns <code>flag = false</code>, and leaves <code>status</code> undefined.</p>
  </blockquote>
  
  <p>Going off of the above passage, it is clear the probing will tell you whether there's an available message you can receive corresponding to the specified <code>source</code>, <code>tag</code>, and <code>comm</code>. My question is, should you assume that the data for the corresponding send from a successful probing has not actually been transferred yet? </p>
  
  <p>It seems reasonable to me now, after reading the standard, that indeed a message the probe is aware of need not be a message that the local process has actually fully received. Given the previous details about the standard non-blocking send, it seems you would need to post a receive after doing the probing to ensure the source non-blocking standard send will complete, because there might be times where the source is sending a large message that MPI does not want to copy into some internal buffer, right? And either way, it seems that posting the receive after a probing is how you ensure that you actually get the full data from the corresponding send to be sent. Is this correct?</p>
  
  <p>This latter question relates to one instance in my code where I am doing a <code>MPI_Iprobe</code> call and if it succeeds, I perform an <code>MPI_Recv</code> call to get the message. However, I think this could be problematic now because I was thinking in my mind that if the probe succeeds, that means it has gotten the whole message already. This implied to me that the <code>MPI_Recv</code> would run quickly, then, since the full message would already be in local memory somewhere. However, I am feeling this was an incorrect assumption now that some clarification on would be helpful.</p>