title: Asynchronous Finite Difference Scheme using MPI_Put
body: <p>A paper by Donzis &amp; Aditya suggests, that it is possible to use a finite difference scheme that might have a delay in the stencil. What does this mean? A FD scheme might be used to solve the heat equation and reads (or some simplification of it)</p>
  
  <pre><code>u[t+1,i] = u[t,i] + c (u[t,i-1]-u[t,i+1])
  </code></pre>
  
  <p>meaning, that the value at the next time step depends on the value at the same position and its neighbours at the previous time step. </p>
  
  <p>This problem can easily be parallized by splitting the (in our case 1D) domain onto the different processors. However, we need communication when computing the boundary nodes at a processor, since the element <code>u[t,i+-1]</code> is only available on another processor. </p>
  
  <p>The problem is illustrated in the following graphic, which is taken from the cited paper.</p>
  
  <p><img src="https://i.stack.imgur.com/aZ4Sn.png" alt="enter image description here"></p>
  
  <p>An MPI implementation might use <code>MPI_Send</code>and <code>MPI_Recv</code> for synchronous computation. 
  Since the computation itself is fairly easy, it is the communication which might become a possible bottleneck. </p>
  
  <p>A solution to the problem is given in the paper: </p>
  
  <p><strong>Instead of a synchronous process, just take the boundary note that is available, despite the fact that it might be the value of an earlier time step. The method then still converges (under some assumptions)</strong></p>
  
  <p>For my work, I would like to implement the asynchronous MPI case (which is not part of the paper). The synchronous part using <code>MPI_Send</code> and <code>MPI_Recv</code> is working correctly. I extended the memory by two elements as ghost cells for the neighbouring elements and send the needed values via send and receive. The code below is basically the implementation of the figure above and is performed during each time step prior to the computation.</p>
  
  <pre><code>MPI_Send(&amp;u[NpP],1,MPI_DOUBLE,RIGHT,rank,MPI_COMM_WORLD);
  MPI_Recv(&amp;u[0],1,MPI_DOUBLE,LEFT,LEFT,MPI_COMM_WORLD,MPI_STATUS_IGNORE);
  
  MPI_Send(&amp;u[1],1,MPI_DOUBLE,LEFT,rank,MPI_COMM_WORLD);
  MPI_Recv(&amp;u[NpP+1],1,MPI_DOUBLE,RIGHT,RIGHT,MPI_COMM_WORLD,MPI_STATUS_IGNORE);
  </code></pre>
  
  <p>Now, I'm by no means an MPI expert. I figured out, that <code>MPI_Put</code> might be what I need for the asynchronous case and reading a little bit, I came up with the following implementation.</p>
  
  <p>Before the time loop:</p>
  
  <pre><code>MPI_Win win;
  double *boundary;
  MPI_Alloc_mem(sizeof(double) * 2, MPI_INFO_NULL, &amp;boundary);
  MPI_Info info;
  MPI_Info_create(&amp;info);
  MPI_Info_set(info,"no_locks","true");
  MPI_Win_create(boundary, 2*sizeof(double), sizeof(double), info, MPI_COMM_WORLD, &amp;win);
  </code></pre>
  
  <p>Inside the time loop:</p>
  
  <pre><code>MPI_Put(&amp;u[1],1,MPI_DOUBLE,LEFT,1,1,MPI_DOUBLE,win);
  MPI_Put(&amp;u[NpP],1,MPI_DOUBLE,RIGHT,0,1,MPI_DOUBLE,win);
  MPI_Win_fence(0,win);
  u[0] = boundary[0];
  u[NpP+1] = boundary[1];
  </code></pre>
  
  <p>which puts the needed elements in the window, namely <code>boundary</code> (array with two elements) on the neighbouring processors and takes the values <code>u[0]</code> and <code>u[NpP+1]</code> from the <code>boundary</code> array itself.
  This implementation is working and I get the same result was with <code>MPI_Send/Recv</code>. However, this isn't really asynchronous since I'm still using <code>MPI_Win_fence</code>, which, as far as I understood, ensures synchronization.</p>
  
  <p>The problem is: If I take out the <code>MPI_Win_fence</code> the values inside <code>boundary</code> are never updated and stay the inital values. My understanding was, that without <code>MPI_Win_fence</code> you would take any value that is available inside <code>boundary</code> which might (or might not) have been updated by a neighbouring processor.</p>
  
  <p>Does anybody have an idea to avoid the use of <code>MPI_Win_fence</code> while also solving the problem, that the values inside <code>boundary</code> are never updated? </p>
  
  <p>I'm also not sure, if the code I provided is enough to understand my problem or to give any hints. If that is the case, feel free to ask, as I will try to add all the parts that are missing.</p>