title: Running slurm script with multiple nodes, launch job steps with 1 task
body: <p>I am trying to launch a large number of job steps using a batch script. The different steps can be completely different programs and do need exactly one CPU each. First I tried doing this using the <code>--multi-prog</code> argument to <code>srun</code>. Unfortunately, when using all CPUs assigned to my job in this manner, performance degrades massively. The run time increases to almost its serialized value. By undersubscribing I could ameliorate this a little. I couldn't find anything online regarding this problem, so I assumed it to be a configuration problem of the cluster I am using.</p>
  
  <p>So I tried going a different route. I implemented the following script (launched via <code>sbatch my_script.slurm</code>):</p>
  
  <pre><code>#!/bin/bash
  #SBATCH -o $HOME/slurm/slurm_out/%j.%N.out
  #SBATCH --error=$HOME/slurm/slurm_out/%j.%N.err_out
  #SBATCH --get-user-env
  #SBATCH -J test
  #SBATCH -D $HOME/slurm
  #SBATCH --export=NONE
  #SBATCH --ntasks=48
  
  NR_PROCS=$(($SLURM_NTASKS))
  for PROC in $(seq 0 $(($NR_PROCS-1)));
  do
      #My call looks like this:
      #srun --exclusive -n1 bash $PROJECT/call_shells/call_"$PROC".sh &amp;
      srun --exclusive -n1 hostname &amp;
      pids[${PROC}]=$!    #Save PID of this background process
  done
  for pid in ${pids[*]};
  do
      wait ${pid} #Wait on all PIDs, this returns 0 if ANY process fails
  done
  </code></pre>
  
  <p>I am aware, that the <code>--exclusive</code> argument is not really needed in my case. The shell scripts called contain the different binaries and their arguments. The remaining part of my script relies on the fact that all processes have finished hence the <code>wait</code>. I changed the calling line to make it a minimal working example.</p>
  
  <p>At first this seemed to be the solution. Unfortunately when increasing the number of nodes used in my job allocation (for example by increasing <code>--ntasks</code> to a number larger than the number of CPUs per node in my cluster), the script does not work as expected anymore, returning</p>
  
  <pre><code>srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
  </code></pre>
  
  <p>and continuing using only one node (i.e. 48 CPUs in my case, which go through the job steps as fast as before, all processes on the other node(s) are subsequently killed).</p>
  
  <p>This seems to be the expected behaviour, but I can't really understand it. Why is it that every job step in a given allocation needs to include a minimum number of tasks equal to the number of nodes included in the allocation. I ordinarily really do not care at all about the number of nodes used in my allocation.</p>
  
  <p>How can I implement my batch script, so it can be used on multiple nodes reliably?</p>