title: Does SLURM submission scripts run on the compute node or the controller machine?
body: <p>I'm getting started with Slurm and I was assuming that the submission script I pass to <code>sbatch</code> runs on the controller and the steps that are marked with <code>srun</code> will run as a job <em>step</em> on a compute node. Consider example below:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  #SBATCH --cpus-per-task 12
  #SBATCH --gres=gpu:1
  #SBATCH --job-name=hello
  
  hostname
  srun hostname
  </code></pre>
  
  <p>I was expecting to see the hostname of the machine I'm submitting from first followed by the name of the compute node that's allocated for the job. Instead it seems the whole script is run on the compute node. I see the compute node's hostname in the log and then it fails because it can't find <code>srun</code> on that node:</p>
  
  <p><code>/var/spool/slurm/d/job00201/slurm_script: line 5: srun: command not found</code></p>
  
  <p>Am I missing something obvious?</p>