title: How to combine subarrays of different widths using only one array for send and receive in MPI
body: <p>I am having a hell of a time with this problem: Using MPI, I want to combine several contiguous, non-overlapping columnar chunks of a 2-dimensional array distributed in several MPI processes into one array residing at the root process.  The main condition being that the array must be the same for all the sending and receiving processes.  The second condition is that the columnar chunks sent by each process can be of different widths.  This seems to be a common problem in parallel programming since I have seen at least 6 question relating to this issue posted in StackOverflow.  None of the answers helped me, unfortunately.  I can resolve this project quite nicely when I divide the problem into row chunks, but not with columns.  I realize it has to do with the differing strides in the case of the columnar subarrays.  I have tried MPI vector and subarray types, both to no avail.</p>
  
  <p>Using the simplified version of my code, if I execute it with COLUMNS equals to 6, I get:</p>
  
  <pre>
      0:  1  1  1  2  2  2  
      1:  1  1  1  2  2  2  
      2:  1  1  1  2  2  2  
      3:  1  1  1  2  2  2  
      4:  1  1  1  2  2  2  
      5:  1  1  1  2  2  2  
      6:  1  1  1  2  2  2  
  </pre>
  
  <p>which is what I want.</p>
  
  <p>On the other hand, if I execute it with COLUMNS = 5, I expect to get:</p>
  
  <pre>
      0:  1  1  1  2  2
      1:  1  1  1  2  2
      2:  1  1  1  2  2
      3:  1  1  1  2  2
      4:  1  1  1  2  2
      5:  1  1  1  2  2
      6:  1  1  1  2  2
  </pre>
  
  <p>Instead, I get:</p>
  
  <pre>
      0:  1  1  1  2  2
      1:  2  1  1  2  2
      2:  2  1  1  2  2
      3:  2  1  1  2  2
      4:  2  1  1  2  2
      5:  1  1  1 -0 -0
      6:  1  1  1 -0 -0
  </pre>
  
  <p>Listing of the simplified code:</p>
  
  <pre><code>#include &lt;mpi.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;assert.h&gt;
  
  #define ROWS                    7
  #define COLUMNS                 6 // 5 or 6 only. I could pass this in the cmd line...
  #define NR_OF_PROCESSES         2
  
  void print_matrix (float ** X, int rows, int cols)
  {
      for (int i = 0; i &lt; rows; ++i) {
          printf ("%3d: ", i);
          for (int j = 0; j &lt; cols; ++j)
              printf ("%2.0f ", X[i][j]);
          printf ("\n");
      }
  }
  
  float **allocate_matrix (int rows, int cols)
  {
      float  *data   = (float  *) malloc (rows * cols * sizeof(float));
      float **matrix = (float **) malloc (rows * sizeof(float *));
      for (int i = 0; i &lt; rows; i++)
          matrix[i] = &amp; (data[i * cols]);
      return matrix;
  }
  
  int main (int argc, char *argv[])
  {
      int   num_procs, my_rank, i, j, root = 0, ncols, ndims = 2, strts;
      float **matrix;
      MPI_Datatype sendsubarray, recvsubarray, resizedrecvsubarray;
  
      assert (COLUMNS == 5 || COLUMNS == 6);
  
      MPI_Init (&amp;argc, &amp;argv);
      MPI_Comm_size (MPI_COMM_WORLD, &amp;num_procs);
      if (num_procs != NR_OF_PROCESSES) MPI_Abort (MPI_COMM_WORLD, -1);
      MPI_Comm_rank (MPI_COMM_WORLD, &amp;my_rank);
  
      ncols = (my_rank == root) ? 3 : COLUMNS - 3;
      strts = (my_rank == root) ? 0 : 3;
      int sizes[2]    = {ROWS, COLUMNS};
      int subsizes[2] = {ROWS, ncols};
      int starts[2]   = {0, strts};
  
      // Create and populate the matrix at each node (incl. the root):
      matrix = allocate_matrix (ROWS, COLUMNS);
      for (i = 0; i &lt; ROWS; i++)
          for (j = 0; j &lt; COLUMNS; j++)
              matrix[i][j] = my_rank * -1.0;
      for (i = starts[0]; i &lt; starts[0] + subsizes[0]; i++)
          for (j = starts[1]; j &lt; starts[1] + subsizes[1]; j++)
              matrix[i][j] = my_rank + 1.0;
  
      // Create the subarray type for use by each send node (incl. the root):
      MPI_Type_create_subarray (ndims, sizes, subsizes, starts, MPI_ORDER_C,
                                MPI_FLOAT, &amp;sendsubarray);
      MPI_Type_commit (&amp;sendsubarray);
  
      // Create the subarray type for use by the receive node (the root):
      if (my_rank == root) {
          MPI_Type_create_subarray (ndims, sizes, subsizes, starts, MPI_ORDER_C,
                                    MPI_FLOAT, &amp;recvsubarray);
          MPI_Type_commit (&amp;recvsubarray);
          MPI_Type_create_resized (recvsubarray, 0, 1 * sizeof(float),
                                   &amp;resizedrecvsubarray);
          MPI_Type_commit (&amp;resizedrecvsubarray);
      }
  
      // Gather the send matrices into the receive matrix:
      int counts[NR_OF_PROCESSES] = {3, COLUMNS - 3};
      int displs[NR_OF_PROCESSES] = {0, 3};
      MPI_Gatherv (matrix[0], 1, sendsubarray,
                   matrix[0], counts, displs, resizedrecvsubarray,
                   root, MPI_COMM_WORLD);
  
      // Have the root send the main array to the output:
      if (my_rank == root) print_matrix (matrix, ROWS, COLUMNS);
  
      // Free out all the allocations we created in this node...
      if (my_rank == 0) {
          MPI_Type_free (&amp;resizedrecvsubarray);
          MPI_Type_free (&amp;recvsubarray);
      }
      MPI_Type_free (&amp;sendsubarray);
      free (matrix);
  
      MPI_Finalize();
      return 0;
  }
  </code></pre>
  
  <p>I am thinking that maybe there is no straightforward solution to my little problem, as exemplified by the code above, and therefore I will have to settle for some convoluted many-step solution where I have to handle the different width subarrays in separate ways before gathering them into the receiving array in two or three steps, instead of just one.</p>
  
  <p>Any help will be much appreciated!</p>