title: Test MPI on a cluster
body: <p>I am learning OpenMPI on a cluster. Here is my first example. I expect the output would show response from different nodes, but they all respond from the same node node062. I just wonder why and how I can actually get report from different nodes to show MPI actually is distributing processes to different nodes? Thanks and regards!</p>
  
  <p>ex1.c  </p>
  
  <pre><code>/* test of MPI */  
  #include "mpi.h"  
  #include &lt;stdio.h&gt;  
  #include &lt;string.h&gt;  
  
  int main(int argc, char **argv)  
  {  
  char idstr[2232]; char buff[22128];  
  char processor_name[MPI_MAX_PROCESSOR_NAME];  
  int numprocs; int myid; int i; int namelen;  
  MPI_Status stat;  
  
  MPI_Init(&amp;argc,&amp;argv);  
  MPI_Comm_size(MPI_COMM_WORLD,&amp;numprocs);  
  MPI_Comm_rank(MPI_COMM_WORLD,&amp;myid);  
  MPI_Get_processor_name(processor_name, &amp;namelen);  
  
  if(myid == 0)  
  {  
    printf("WE have %d processors\n", numprocs);  
    for(i=1;i&lt;numprocs;i++)  
    {  
      sprintf(buff, "Hello %d", i);  
      MPI_Send(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD); }  
      for(i=1;i&lt;numprocs;i++)  
      {  
        MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &amp;stat);  
        printf("%s\n", buff);  
      }  
  }  
  else  
  {   
    MPI_Recv(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &amp;stat);  
    sprintf(idstr, " Processor %d at node %s ", myid, processor_name);  
    strcat(buff, idstr);  
    strcat(buff, "reporting for duty\n");  
    MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD);  
  }  
  MPI_Finalize();  
  
  }  
  </code></pre>
  
  <p>ex1.pbs  </p>
  
  <pre><code>#!/bin/sh  
  #  
  #This is an example script example.sh  
  #  
  #These commands set up the Grid Environment for your job:  
  #PBS -N ex1  
  #PBS -l nodes=10:ppn=1,walltime=1:10:00  
  #PBS -q dque    
  
  # export OMP_NUM_THREADS=4  
  
   mpirun -np 10 /home/tim/courses/MPI/examples/ex1  
  </code></pre>
  
  <p>compile and run:</p>
  
  <pre><code>[tim@user1 examples]$ mpicc ./ex1.c -o ex1   
  [tim@user1 examples]$ qsub ex1.pbs  
  35540.mgt  
  [tim@user1 examples]$ nano ex1.o35540  
  ----------------------------------------  
  Begin PBS Prologue Sat Jan 30 21:28:03 EST 2010 1264904883  
  Job ID:         35540.mgt  
  Username:       tim  
  Group:          Brown  
  Nodes:          node062 node063 node169 node170 node171 node172 node174 node175  
  node176 node177  
  End PBS Prologue Sat Jan 30 21:28:03 EST 2010 1264904883  
  ----------------------------------------  
  WE have 10 processors  
  Hello 1 Processor 1 at node node062 reporting for duty  
  Hello 2 Processor 2 at node node062 reporting for duty        
  Hello 3 Processor 3 at node node062 reporting for duty        
  Hello 4 Processor 4 at node node062 reporting for duty        
  Hello 5 Processor 5 at node node062 reporting for duty        
  Hello 6 Processor 6 at node node062 reporting for duty        
  Hello 7 Processor 7 at node node062 reporting for duty        
  Hello 8 Processor 8 at node node062 reporting for duty        
  Hello 9 Processor 9 at node node062 reporting for duty  
  
  ----------------------------------------  
  Begin PBS Epilogue Sat Jan 30 21:28:11 EST 2010 1264904891  
  Job ID:         35540.mgt  
  Username:       tim  
  Group:          Brown  
  Job Name:       ex1  
  Session:        15533  
  Limits:         neednodes=10:ppn=1,nodes=10:ppn=1,walltime=01:10:00  
  Resources:      cput=00:00:00,mem=420kb,vmem=8216kb,walltime=00:00:03  
  Queue:          dque  
  Account:  
  Nodes:  node062 node063 node169 node170 node171 node172 node174 node175 node176  
  node177  
  Killing leftovers...  
  
  End PBS Epilogue Sat Jan 30 21:28:11 EST 2010 1264904891  
  ----------------------------------------
  </code></pre>
  
  <hr>
  
  <p>UPDATE:</p>
  
  <p>I would like to run several background jobs in a single PBS script, so that the jobs can run at the same time. e.g. in the above example, I added another call to run ex1 and change both runs to be background in ex1.pbs</p>
  
  <pre><code>#!/bin/sh  
  #  
  #This is an example script example.sh  
  #  
  #These commands set up the Grid Environment for your job:  
  #PBS -N ex1  
  #PBS -l nodes=10:ppn=1,walltime=1:10:00  
  #PBS -q dque 
  
  echo "The first job starts!"  
  mpirun -np 5 --machinefile /home/tim/courses/MPI/examples/machinefile /home/tim/courses/MPI/examples/ex1 &amp;  
  echo "The first job ends!"  
  echo "The second job starts!"  
  mpirun -np 5 --machinefile /home/tim/courses/MPI/examples/machinefile /home/tim/courses/MPI/examples/ex1 &amp;  
  echo "The second job ends!" 
  </code></pre>
  
  <p>(1) The result is fine after qsub this script with previous compiled executable ex1. </p>
  
  <pre><code>The first job starts!  
  The first job ends!  
  The second job starts!  
  The second job ends!  
  WE have 5 processors  
  WE have 5 processors  
  Hello 1 Processor 1 at node node063 reporting for duty        
  Hello 2 Processor 2 at node node169 reporting for duty        
  Hello 3 Processor 3 at node node170 reporting for duty        
  Hello 1 Processor 1 at node node063 reporting for duty        
  Hello 4 Processor 4 at node node171 reporting for duty        
  Hello 2 Processor 2 at node node169 reporting for duty        
  Hello 3 Processor 3 at node node170 reporting for duty        
  Hello 4 Processor 4 at node node171 reporting for duty  
  </code></pre>
  
  <p>(2) However, I think the running time of ex1 is too quick and probably the two background jobs do not have much running time overlapping, which is not the case when I apply the same way to my real project. So I added sleep(30) to ex1.c to extend the running time of ex1 so that two jobs running ex1 in background will be running simultaneously almost all the time.</p>
  
  <pre><code>/* test of MPI */  
  #include "mpi.h"  
  #include &lt;stdio.h&gt;  
  #include &lt;string.h&gt;  
  #include &lt;unistd.h&gt;
  
  int main(int argc, char **argv)  
  {  
  char idstr[2232]; char buff[22128];  
  char processor_name[MPI_MAX_PROCESSOR_NAME];  
  int numprocs; int myid; int i; int namelen;  
  MPI_Status stat;  
  
  MPI_Init(&amp;argc,&amp;argv);  
  MPI_Comm_size(MPI_COMM_WORLD,&amp;numprocs);  
  MPI_Comm_rank(MPI_COMM_WORLD,&amp;myid);  
  MPI_Get_processor_name(processor_name, &amp;namelen);  
  
  if(myid == 0)  
  {  
    printf("WE have %d processors\n", numprocs);  
    for(i=1;i&lt;numprocs;i++)  
    {  
      sprintf(buff, "Hello %d", i);  
      MPI_Send(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD); }  
      for(i=1;i&lt;numprocs;i++)  
      {  
        MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &amp;stat);  
        printf("%s\n", buff);  
      }  
  }  
  else  
  {   
    MPI_Recv(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &amp;stat);  
    sprintf(idstr, " Processor %d at node %s ", myid, processor_name);  
    strcat(buff, idstr);  
    strcat(buff, "reporting for duty\n");  
    MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD);  
  }  
  
  sleep(30); // new added to extend the running time
  MPI_Finalize();  
  
  }  
  </code></pre>
  
  <p>But after recompilation and qsub again, the results seems not all right. There are processes aborted.
  in ex1.o35571:</p>
  
  <pre><code>The first job starts!  
  The first job ends!  
  The second job starts!  
  The second job ends!  
  WE have 5 processors  
  WE have 5 processors  
  Hello 1 Processor 1 at node node063 reporting for duty  
  Hello 2 Processor 2 at node node169 reporting for duty  
  Hello 3 Processor 3 at node node170 reporting for duty  
  Hello 4 Processor 4 at node node171 reporting for duty  
  Hello 1 Processor 1 at node node063 reporting for duty  
  Hello 2 Processor 2 at node node169 reporting for duty  
  Hello 3 Processor 3 at node node170 reporting for duty  
  Hello 4 Processor 4 at node node171 reporting for duty  
  4 additional processes aborted (not shown)  
  4 additional processes aborted (not shown)  
  </code></pre>
  
  <p>in ex1.e35571:</p>
  
  <pre><code>mpirun: killing job...  
  mpirun noticed that job rank 0 with PID 25376 on node node062 exited on signal 15 (Terminated).  
  mpirun: killing job...  
  mpirun noticed that job rank 0 with PID 25377 on node node062 exited on signal 15 (Terminated).  
  </code></pre>
  
  <p>I wonder why there are processes aborted? How can I qsub background jobs correctly in a PBS script?</p>