title: mmap on multiple nodes
body: <p>The script below writes in parallel to a memory mapped array with mmap. However it only works when all processes are on the same node -  otherwise it produces rows of 0 for processors not on the rank 0 node, or other stray zeros in the output. Why is this? I feel I am missing something about how mmap works.</p>
  <p>Edit: The same result occurs on both a NFS system, and on a parallel distributed system. A commenter below suggested it was to do with the page length of mmap. When the 'length' of my slice is exactly 4KiB the script still produces the  wrong output. The same also occurs when the slices are much longer than 4 KiB.</p>
  <pre><code>#!/usr/bin/python3
  
  from mpi4py import MPI
  import numpy as np
  
  comm = MPI.COMM_WORLD
  rank = comm.Get_rank()
  size = comm.Get_size()
  
  length = int(1e6)      # Edited to make test case longer.
  myfile = &quot;/tmp/map&quot;
  
  if rank == 0:
      fp = np.memmap(myfile, dtype=np.float32, mode='w+', shape=(size,length))
      del fp
  
  comm.Barrier()
  
  fp = np.memmap(myfile, dtype=np.float32, mode='r+', shape=(1,length),
                  offset=rank*length*4)
  fp[:,:] = np.full(length,rank)
  
  comm.Barrier()
  
  if rank == 0:
      out = np.memmap(myfile, dtype=np.float32, mode='r', shape=(size,length))
      print(out[:,:])
  </code></pre>
  <p>Correct output:</p>
  <pre><code>[[ 0.  0.  0.  0.]
   [ 1.  1.  1.  1.]
   [ 2.  2.  2.  2.]
   [ 3.  3.  3.  3.]
   [ 4.  4.  4.  4.]]
  </code></pre>
  <p>Incorrect output. Processors with rank 3 and 4 do not write.</p>
  <pre><code>[[ 0.  0.  0.  0.]
   [ 1.  1.  1.  1.]
   [ 2.  2.  2.  2.]
   [ 0.  0.  0.  0.]
   [ 0.  0.  0.  0.]]
  </code></pre>