Submitting jobs using python
Question: I am trying to submit a job in a cluster in our institute using python
scripts.

    
    
     compile_cmd = 'ifort -openmp ran_numbers.f90 ' + fname \
                      + ' ompscmf.f90 -o scmf.o'
     subprocess.Popen(compile_cmd, shell=True)
    
     Popen('qsub launcher',shell=True)
    

The problem is that , system is hanging at this point. Any obvious mistakes in
the above script? All the files mentioned in the code are available in that
directory ( I have cross checked that). _qsub_ is a command used to submit
jobs to our cluster. _fname_ is the name of a file that I created in the
process.


Answer: I have a script that I used to submit multiple jobs to our cluster using qsub.
qsub typically takes job submissions in the form

    
    
    qsub [qsub options] job
    

In my line of work, job is typically a bash (.sh) or python script (.py) that
actually calls the programs or code to be run on each node. If I wanted to
submit a job called "test_job.sh" with maximum walltime, I would do

    
    
    qsub -l walltime=72:00:00 test_job.sh
    

This amounts to the following python code

    
    
    from subprocess import call
    
    qsub_call = "qsub -l walltime=72:00:00 %s"
    call(qsub_call % "test_job.sh", shell=True)
    

* * *

Alternatively, what if you had a bash script that looked like

    
    
    #!/bin/bash
    
    filename="your_filename_here"
    ifort -openmp ran_numbers.f90 $filename ompscmf.f90 -o scmf.o
    

then submitted this via `qsub job.sh`?

* * *

**Edit:** Honestly, the most optimal job queueing scheme varies from cluster
to cluster. One simple way to simplify you job submissions scripts is to find
out how many CPUs are available at each node. Some of the more recent queueing
systems allow you to submit many single CPU jobs and they will submit these
together on as few nodes as possible; however, some older clusters won't do
that and submitting many individual jobs is frowned upon.

Say that each node in your cluster has 8 CPUs. You could write you script like

    
    
    #!/bin/bash
    #PBS -l nodes=1;ppn=8
    
    for ((i=0; i<8; i++))
    do
        ./myjob.sh filename_${i} &
    done
    wait
    

What this will do is submit 8 jobs on one node at once (`&` means do in
background) and wait until all 8 jobs are finished. This may be optimal for
clusters with many CPUs per node (for example, one cluster that I used has 48
CPUs per node).

Alternatively, if submitting many single core jobs is optimal and your
submission code above isn't working, you could use python to generate bash
scripts to pass to qsub.

    
    
    #!/usr/bin/env python
    import os
    from subprocess import call
    
    bash_lines = ['#!/bin/bash\n', '#PBS -l nodes=1;ppn=1\n']
    bash_name = 'myjob_%i.sh'
    job_call = 'ifort -openmp ran_numbers.f90 %s ompscmf.f90 -o scmf.o &\n'
    qsub_call = 'qsub myjob_%i.sh'
    
    filenames = [os.path.join(root, f) for root, _, files in os.walk(directory)
                                       for f in files if f.endswith('.txt')]
    for i, filename in enumerate(filenames):
        with open(bash_name%i, 'w') as bash_file:
            bash_file.writelines(bash_lines + [job_call%filename, 'wait\n'])
        call(qsub_call%i, shell=True)
    

