title: Writing to multiple shared files with MPI-IO
body: <p>I'm running a simulation with thousands of MPI processes and need to write output data to a small set of files. For example, even though I might have 10,000 processes I only want to write out 10 files, with 1,000 writing to each one (at some appropriate offset). AFAIK the correct way to do this is to create a new communicator for the groups of processes that will be writing to the same files, open a shared file for that communicator with <code>MPI_File_open()</code>, and then write to it with <code>MPI_File_write_at_all()</code>. Is that correct? The following code is a toy example that I wrote up:</p>
  
  <pre><code>#include &lt;mpi.h&gt;
  #include &lt;math.h&gt;
  #include &lt;stdio.h&gt;
  
  const int MAX_NUM_FILES = 4;
  
  int main(){
      MPI_Init(NULL, NULL);
  
      int rank;
      MPI_Comm_rank(MPI_COMM_WORLD, &amp;rank);
  
      int numProcs;
      MPI_Comm_size(MPI_COMM_WORLD, &amp;numProcs);
  
      int numProcsPerFile = ceil(((double) numProcs) / MAX_NUM_FILES);
      int targetFile = rank / numProcsPerFile;
  
      MPI_Comm fileComm;
      MPI_Comm_split(MPI_COMM_WORLD, targetFile, rank, &amp;fileComm);
  
      int targetFileRank;
      MPI_Comm_rank(fileComm, &amp;targetFileRank);
  
      char filename[20]; // Sufficient for testing purposes
      snprintf(filename, 20, "out_%d.dat", targetFile);
      printf(
          "Proc %d: writing to file %s with rank %d\n", rank, filename,
          targetFileRank);
  
      MPI_File outFile;
      MPI_File_open(
          fileComm, filename, MPI_MODE_CREATE | MPI_MODE_WRONLY,
          MPI_INFO_NULL, &amp;outFile);
  
      char bufToWrite[4];
      snprintf(bufToWrite, 4, "%3d", rank);
  
      MPI_File_write_at_all(
          outFile, targetFileRank * 3,
          bufToWrite, 3, MPI_CHAR, MPI_STATUS_IGNORE);
  
      MPI_File_close(&amp;outFile);
      MPI_Finalize();
  }
  </code></pre>
  
  <p>I can compile with <code>mpicc file.c -lm</code> and run, say, 20 processes with <code>mpirun -np 20 a.out</code>, and I get the expected output (four files with five entries each), but I'm unsure whether this is the technically correct/most optimal way of doing it. Is there anything I should do differently?</p>