title: MVAPICH2-GDR has segmentation fault error
body: <p>I am trying to use MVAPICH2-GDR for the simple hello world program. Although it can compile the code successfully, it has segmentation fault error in runtime. My platform has Redhat 6.5 and CUDA 7.5. So I downloaded the rpm file mvapich2-gdr-cuda7.5-intel-2.2-0.3.rc1.el6.x86_64.rpm. </p>
  
  <p>The MPI code is the simple hello world program:</p>
  
  <pre><code>1 #include &lt;mpi.h&gt;
  2 #include &lt;stdio.h&gt;
  3
  4 int main(int argc, char** argv) {
  5     // Initialize the MPI environment
  6     MPI_Init(NULL, NULL);
  7     // Get the number of processes
  8     int world_size;
  9     MPI_Comm_size(MPI_COMM_WORLD, &amp;world_size);
  10
  11     // Get the rank of the process
  12     int world_rank;
  13     MPI_Comm_rank(MPI_COMM_WORLD, &amp;world_rank);
  14
  15     // Print off a hello world message
  16     printf("Hello world from %d out of %d\n", world_rank, world_size);
  17
  18     // Finalize the MPI environment.
  19     MPI_Finalize();
  20 }
  </code></pre>
  
  <p>To compile the program, I used the following command:</p>
  
  <pre><code>mpicc hello.c -o hello
  </code></pre>
  
  <p>To run the program:</p>
  
  <pre><code>mpirun -np 2 ./hello
  </code></pre>
  
  <p>The error message is as follows:</p>
  
  <pre><code>[localhost.localdomain:mpi_rank_1][error_sighandler] Caught error: Segmentation fault (signal 11)
  [localhost.localdomain:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11)                
  =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
  =   PID 188057 RUNNING AT localhost.localdomain
  =   EXIT CODE: 139
  =   CLEANING UP REMAINING PROCESSES
  =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES 
  
  YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
  </code></pre>
  
  <p>Because MVAPICH2-GDR did not open its source code, I really don't know where the error comes from. Has anyone successfully used MVAPICH2-GDR?</p>