title: Interacting with slurm through python (to automatically chain jobs)
body: <p>I need to run 10,000 jobs on slurm (each taking 30 minutes let's say). Currently, the plan was to do it with a job array, using 250 cores in parallel, like so:</p>
  
  <pre><code>sbatch --array=0-10000%250 array_script.sh args
  </code></pre>
  
  <p>Unfortunately, the sys admin hasn't changed slurm's MaxArraySize (from the default 1001). To "circumvent" this, I was planning on slicing the overall job into 10 pieces, and somehow scheduling each piece so that it runs after the previous piece has finished. For example, I would start with:</p>
  
  <pre><code>sbatch --array=0-999%250 array_script.sh args
  </code></pre>
  
  <p>then when that is done, I would do:</p>
  
  <pre><code>sbatch --array=1000-1999%250 array_script.sh args
  </code></pre>
  
  <p>Now I need to somehow schedule this. I'm not that experienced with bash, and I had a python wrapper around everything so I thought I would do it with python (plus I'm using a python wrapper around the job array to do a lot of other stuff anyway). So how would I do this normally?</p>
  
  <p>Currently I have:</p>
  
  <pre><code>        for i in range(num_slices):
              command = 'sbatch --array={lower_end}-{upper_end}%250 array_script.sh args'.format(lower_end=i*1000, upper_end=min((i+1)*1000-1, num_targets-1), args=args)
              subprocess.run(command, shell=True)
              &lt;&lt; need to have a step that waits till the job is done &gt;&gt;
  </code></pre>
  
  <p>First of all, in the above, I run <code>sbatch</code> with <code>subprocess.run</code>, which means I currently don't know the <code>JOB_ID</code>. Is there a way to catch the output from <code>subprocess.run</code> or something that would allow me to find the <code>JOB_ID</code>. And how do I do the equivalent of <code>squeue</code> to check if the job is still running and decide whether to continue the loop?</p>