title: MPI-Related Segmentation Fault for icpc But Not GCC
body: <p>I am trying to run a program compiled with Intel compilers. The program has both openMP and MPI code. The MPI code is a new addition to the program.</p>
  
  <ul>
  <li>Code compiled with gcc and OpenMPI (gcc-flavored, v1.8) runs and works correctly.</li>
  <li>Code compiled with icpc and Intel-Compiled OpenMPI seg faults with "Address Not Mapped" error</li>
  </ul>
  
  <p><strong>OpenMPI (compiled with Intel) Error</strong></p>
  
  <pre><code>[ida3c03:22018] 63 more processes have sent help message help-mpi-btl-openib.txt / reg mem limit low
  [ida3c03:22018] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
  done!
  [ida3c04:03329] *** Process received signal ***
  [ida3c04:03329] Signal: Segmentation fault (11)
  [ida3c04:03329] Signal code: Address not mapped (1)
  [ida3c04:03329] Failing at address: 0x10
  </code></pre>
  
  <ul>
  <li>Code compiled with icpc and Intel's own MPI seg faults as well with one of 2 errors</li>
  </ul>
  
  <p><strong>Error 1</strong></p>
  
  <pre><code> Could not find **ibv_destroy_qp in list of messages
   Could not find **ibv_destroy_qp in list of messages
   Could not find **ibv_destroy_qp in list of messages
   Could not find **ibv_destroy_qp in list of messages
   Could not find **ibv_destroy_qp in list of messages
   Could not find **vc_gen2_qp_finalize in list of messages
   Could not find **ibv_destroy_qp in list of messages
   Could not find **vc_gen2_qp_finalize in list of messages
   Could not find **vc_gen2_qp_finalize in list of messages
   Could not find **vc_gen2_qp_finalize in list of messages
   Could not find **vc_gen2_qp_finalize in list of messages
   Could not find **vc_gen2_qp_finalize in list of messages
   Fatal error in MPI_Finalize: Internal MPI error!, error stack:
   MPI_Finalize(311).................: MPI_Finalize failed
   MPI_Finalize(229).................:
   MPID_Finalize(140)................:
   MPIDI_CH3_Finalize(24)............:
   MPID_nem_finalize(63).............:
   MPID_nem_gen2_module_finalize(520):(unknown)(): Internal MPI error!
  </code></pre>
  
  <p><strong>Error 2</strong> </p>
  
  <pre><code>recv desc error, 128, 0x9d0240
  [mpiexec@ida3c03] control_cb (./pm/pmiserv/pmiserv_cb.c:717): assert (!closed) failed
  [mpiexec@ida3c03] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
  [mpiexec@ida3c03] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:435): error waiting for event
  [mpiexec@ida3c03] main (./ui/mpich/mpiexec.c:901): process manager error waiting for completion
  </code></pre>
  
  <p>The project is huge and would be difficult to show you all the MPI code, but I grepped out the MPI calls from one of the main files where I think the error occurs, in case that would be useful at all. I filled in actual variable names with "vector_#".</p>
  
  <pre><code>1172:  MPI_Init(&amp;argc,&amp;argv);
  1173:  MPI_Comm_size(MPI_COMM_WORLD, &amp;numranks);
  1174:  MPI_Comm_rank(MPI_COMM_WORLD,&amp;rank);
  1315:  MPI_Barrier(MPI_COMM_WORLD);
  1593:  MPI_Allgather(&amp;long_1,1,MPI_LONG,vector_1,1,MPI_LONG,MPI_COMM_WORLD);
  1594:  MPI_Allgather(&amp;long_2, 1,MPI_LONG,vector_2, 1,MPI_LONG,MPI_COMM_WORLD);    
  1624:  MPI_Barrier(MPI_COMM_WORLD);
  1655:  MPI_Bcast(buffer,long*[count],MPI_CHAR,count,MPI_COMM_WORLD);
  1661:  MPI_Allgatherv(dnas_chars_send, long_2, MPI_CHAR,  char*,int*, long*, MPI_CHAR,MPI_COMM_WORLD);
  1740:  MPI_Barrier(MPI_COMM_WORLD);
  2013:  MPI_Barrier(MPI_COMM_WORLD); 
  2064:  MPI_Allgather(&amp;int,1,MPI_INT,int*,1,MPI_INT,MPI_COMM_WORLD);
  2066:  MPI_Allgather(&amp;int,1,MPI_INT,int*,1,MPI_INT,MPI_COMM_WORLD);
  2068:  MPI_Allgather(&amp;int,1,MPI_INT,int*,1,MPI_INT,MPI_COMM_WORLD);
  2106:  MPI_Allgatherv(int*, int,MPI_INT,int* ,int*, int*, MPI_INT,MPI_COMM_WORLD);
  2107:  MPI_Allgatherv(int*, int,MPI_INT,int* ,int*, int*, MPI_INT,MPI_COMM_WORLD);
  2108:  MPI_Allgatherv(int*,int,MPI_INT,int*,int*,int*, MPI_INT,MPI_COMM_WORLD);
  2136:  MPI_Allgatherv(int*,int,MPI_INT,int*,int*,int*, MPI_INT,MPI_COMM_WORLD);
  2137:  MPI_Allgatherv(int*,int,MPI_INT,int*,int*,int*, MPI_INT,MPI_COMM_WORLD);
  2164:  MPI_Barrier(MPI_COMM_WORLD);
  2519:  MPI_Finalize();
  </code></pre>
  
  <p>But I don't understand why the code would be fine in gcc and then experience this error for icpc. The code is running on a cluster, and I don't have access to the backend nodes where the MPI processes are running.</p>