title: How to make SLURM job step use the minimum number of nodes?
body: <p>I am trying to run many smaller SLURM job steps within one big multi-node allocation, but am struggling with how the tasks of the job steps are assigned to the different nodes. In general I would like to keep the tasks of one job step as local as possible (same node, same socket) and only spill over to the next node when not all tasks can be placed on a single node.</p>
  <p>The following example shows a case where I allocate 2 nodes with 4 tasks each and launch a job step asking for 4 tasks:</p>
  <pre><code>$ salloc -N 2 --ntasks-per-node=4 srun -l -n 4 hostname
  salloc: Granted job allocation 9677936
  0: compute-3-6.local
  1: compute-3-6.local
  2: compute-3-7.local
  3: compute-3-7.local
  salloc: Relinquishing job allocation 9677936
  </code></pre>
  <p>I would like these 4 tasks go to one of the nodes, so that a second job step could claim the other node, but this is not what happens: the first job step gets distributed evenly across the two nodes. If I launched a second job step with 4 tasks, it would be distributed across nodes too, causing lots of unnecessary inter-node network communication that could easily be avoided.</p>
  <p>I have already found out that I can force my job step to run on a single node by explicitly including <code>-N 1</code> for the job step launch:</p>
  <pre><code>$ salloc -N 2 --ntasks-per-node=4 srun -l -N 1 -n 4 hostname
  salloc: Granted job allocation 9677939
  0: compute-3-6.local
  1: compute-3-6.local
  2: compute-3-6.local
  3: compute-3-6.local
  salloc: Relinquishing job allocation 9677939
  </code></pre>
  <p>However, the number of job steps launched and the number of tasks per job step depends on user input in my case, so I can not just force <code>-N 1</code> for all of them. There may be job steps that have so many tasks that they can not be placed on a single node.</p>
  <p>Reading the <code>srun</code> manpage, I first thought that the <code>--distribute=block:block</code> option should work for me, but it does not. It seems that this option only comes into play after the decision on the number of nodes to be used by a job step has been made.</p>
  <p>Another idea that I had was that the job step might just be inheriting the <code>-N 2</code> argument from the allocation and was therefore also forced to use two nodes. I tried setting <code>-N 1-2</code> for the job step, in order to at least allow SLURM to assign the job step to a single node, but this does not have any effect for me, not even when combined with the <code>--use-min-nodes</code> flag.</p>
  <pre><code>$ salloc -N 2 --ntasks-per-node=4 srun -l -N 1-2 --use-min-nodes -n 4 hostname
  salloc: Granted job allocation 9677947
  0: compute-3-6.local
  1: compute-3-6.local
  2: compute-3-7.local
  3: compute-3-7.local
  salloc: Relinquishing job allocation 9677947
  </code></pre>
  <p><strong>How do I make a SLURM job step use the minimum number of nodes?</strong></p>