title: Using Persistent Communication in Fortran MPI
body: <p>I am using persistent communication in my CFD code. I have the communications setup in another subroutine and in the main subroutine, where I have the do loop, I use the MPI_STARTALL(), MPI_WAITALL(). 
  In order to make it shorter, I am showing hte first part of the setup. The rest of the arrays are exactly the same. </p>
  
  <p>My setup subrotuine looks like: </p>
  
  <pre><code>Subroutine MPI_Subroutine
  use Variables
  use mpi
  implicit none
  
  !Starting up MPI
  call MPI_INIT(ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD,npes,ierr)
  call MPI_COMM_RANK(MPI_COMM_WORLD,MyRank,ierr)
  
  !Compute the size of local block (1D Decomposition)
  Jmax = JmaxGlobal
  Imax = ImaxGlobal/npes
  if (MyRank.lt.(ImaxGlobal - npes*Imax)) then
    Imax = Imax + 1
  end if
  if (MyRank.ne.0.and.MyRank.ne.(npes-1)) then
    Imax = Imax + 2
  Else
    Imax = Imax + 1
  endif
  
  ! Computing neighboars
  if (MyRank.eq.0) then
    Left = MPI_PROC_NULL
  else
    Left = MyRank - 1
  end if
  
  if (MyRank.eq.(npes -1)) then
    Right = MPI_PROC_NULL
  else
    Right = MyRank + 1
  end if
  
  
  ! Initializing the Arrays in each processor, according to the number of local nodes
  Call InitializeArrays
  
  !Creating the channel of communication for this computation,
  !Sending and receiving the u_old (Ghost cells)
  Call MPI_SEND_INIT(u_old(2,:),Jmax,MPI_DOUBLE_PRECISION,Left,tag,MPI_COMM_WORLD,req(1),ierr)
  Call MPI_RECV_INIT(u_old(Imax,:),jmax,MPI_DOUBLE_PRECISION,Right,tag,MPI_COMM_WORLD,req(2),ierr)
  Call MPI_SEND_INIT(u_old(Imax-1,:),Jmax,MPI_DOUBLE_PRECISION,Right,tag,MPI_COMM_WORLD,req(3),ierr)
  Call MPI_RECV_INIT(u_old(1,:),jmax,MPI_DOUBLE_PRECISION,Left,tag,MPI_COMM_WORLD,req(4),ierr)
  </code></pre>
  
  <p>Since I am debugging my code I am just checking these arrays. When I check my ghost cells are full of zeroes. Then I guess that I messing with the instruction.</p>
  
  <p>The main code, where I call the MPI_STARTALL, MPI_WAITALL looks like:</p>
  
  <pre><code>Program 
  use Variables
  use mpi
  implicit none
  open(32, file = 'error.dat')
  
  Call MPI_Subroutine
  
  
  !kk=kk+1
   DO kk=1, 2001
  ! A lot of calculation
  
  ! communicating the maximum error among the processes and delta t
  call MPI_REDUCE(eps,epsGlobal,1,MPI_DOUBLE_PRECISION,MPI_MAX,0,MPI_COMM_WORLD,ierr)
  call MPI_BCAST(epsGlobal,1,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,ierr)
  call MPI_REDUCE(delta_t,delta_tGlobal,1,MPI_DOUBLE_PRECISION,MPI_MIN,0,MPI_COMM_WORLD,ierr)
  
  
  if(MyRank.eq.0) delta_t = delta_tGlobal
  
  call MPI_BCAST(delta_t,1,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,ierr)
  
  
  if(MyRank.eq.0) then
    write(*,*) kk,epsGlobal,(kk*delta_t)
    write(32,*) kk,epsGlobal
  endif
  
  Call Swap
  Call MPI_STARTALL(4,req,ierr) !
  Call MPI_WAITALL(4,req,status,ierr) 
  enddo
  </code></pre>
  
  <p>The variables are set in another module. the MPI related variables looks like:</p>
  
  <pre><code>! MPI variables
  INTEGER :: npes, MyRank, ierr, Left, Right, tag
  INTEGER  :: status(MPI_STATUS_SIZE,4)
  INTEGER,dimension(4) :: req
  </code></pre>
  
  <p>I appreciate your time and suggestion in this problem. </p>