title: Running parallel Python threads that calls external MPI programme submitted using PBS
body: <p>I am pretty new to python and I'm unsure of what is the best way to implement a multithread/multiprocess code on a distributed cluster.</p>
  
  <p>I am trying to write a wrapper script using Python that calls an external MPI programme running on a large cluster using a PBS queuing system. A (very) simplified version of type of script I've been working on is given below, where the code moves into a specific directory, runs an external MPI programme and checks the results to see if there have been any large changes.</p>
  
  <pre><code>#!/local/python-2.7.1/bin/python2.7
  
  import os
  import subprocess as sp
  import coordinate_functions as coord_funcs
  
  os.chdir('/usr/work/cmurray/SeachTest/')
  print os.getcwd()
  
  # Gets nodefile and num procs (NP)
  cat_np = sp.Popen('cat $PBS_NODEFILE | wc -l', shell = True, stdout=sp.PIPE)
  NP = int(cat_np.communicate()[0])
  sp.call('cat $PBS_NODEFILE &gt; nodefile', shell = True)
  
  def run_mpi(np, nodefile):
          mpi_cmd = 'mpirun -machinefile %s -np %d mpipg &gt; calc.out' % (nodefile, np)
          sp.call(vasp_cmd, shell = True)
  
  
  def search_loop(calc_dir, t_total, nodefile, num_procs):
  
      os.chdir(calc_dir)
      no_events = True
      while no_events or t &lt; t_total:
          run_mpi(mynodefile, NP)
          num_events = coord_funcs.change_test('OUTFILE', 'INFILE', 0.01)
          if num_events &gt; 0:
              event = True
          else:
              t += 1
  
  search_loop('/usr/work/cmurray/SeachTest/calc_1/', 10, mynodefile, NP)
  </code></pre>
  
  <p>This is then submitted to the queue using:</p>
  
  <pre><code>qsub -l nodes=4 -N SeachTest ./SearchTest
  </code></pre>
  
  <p>What I want to do is run multiple versions of the <code>search_loop</code> function  in parallel in different directories (containing different starting positions for example) read from a list. The processes is very IO heavy with the MPI calculations taking maybe a few minutes to run each time they are called. </p>
  
  <p>Would the <code>threading</code> module be ok for this purpose or is the <code>multiprocessing</code> module a better choice? I will probably need to pass simple messages like the <code>event</code> boolean in the above example between threads/processes.</p>
  
  <p>Also, how do I make sure that the python script is not using processors that I've assigned to the MPI runs?</p>