title: SLURM skipping some jobs in job array
body: <p>In my bash script which is made to run on SLURM scheduler, I specified that I want jobs <code>1-900%10</code> to be run on the cluster, so 10 jobs in parallel will be found at any time.</p>
  <p>I understand that based on <code>1-900%10</code> the first 10 must be run, then next 10, etc. Unless one of these finish before the other.</p>
  <p>However I am finding out that SLURM is missing some jobs. Here is a screenshot:</p>
  <p><a href="https://i.stack.imgur.com/LABDV.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LABDV.jpg" alt="enter image description here" /></a></p>
  <p>Notice that jobs <code>5,6,7,8</code> are not there. I am not sure what the reason is, also they are never computed later on.</p>
  <p>Here is a snippet of my bash script:</p>
  <pre><code>#!/usr/bin/env bash
  #SBATCH --job-name=fotrcont
  #SBATCH --account=anaccount
  #SBATCH --partition=normal
  #SBATCH --nodes=1
  #SBATCH --ntasks-per-node=1
  #SBATCH --cpus-per-task=2
  #SBATCH --mem=16000
  #SBATCH --mail-type=ALL
  #SBATCH --mail-user=someone@example.com
  #SBATCH --array=1-900%10
  </code></pre>
  <p><strong>UPDATE</strong>
  Turns out that for some reason the node the jobs are getting allocated to is drained for some reason. I have specified the node using <code>nodelist</code> as follows:</p>
  <pre><code>#!/usr/bin/env bash
  #SBATCH --job-name=fotrcont
  #SBATCH --account=anaccount
  #SBATCH --partition=normal
  #SBATCH --nodes=1
  #SBATCH --ntasks-per-node=1
  #SBATCH --cpus-per-task=2
  #SBATCH --mem=16000
  #SBATCH --mail-type=ALL
  #SBATCH --mail-user=someone@example.com
  #SBATCH --nodelist=onode09 #  HERE 
  #SBATCH --array=1-900%10
  </code></pre>
  <p>However its now running 5 jobs at a time rather than 10, although I have specified <code>1-900%10</code>. I have tried adding more nodes in <code>--nodelist</code>and increasing <code>--node</code> to 2 or 3 but it doesn't seem to work.</p>