title: why does mpirun behave as it does when used with slurm?
body: <p>I am using Intel MPI and have encountered some confusing behavior when using <code>mpirun</code> in conjunction with slurm.</p>
  
  <p>If I run (in a login node) </p>
  
  <pre><code>mpirun -n 2 python -c "from mpi4py import MPI; print(MPI.COMM_WORLD.Get_rank())"
  </code></pre>
  
  <p>then I get as output the expected 0 and 1 printed out.</p>
  
  <p>If however I <code>salloc --time=30 --nodes=1</code> and run the same <code>mpirun</code> from the interactive compute node, I get two 0s printed out instead of the expected 0 and 1.</p>
  
  <p>Then, if I change <code>-n 2</code> to <code>-n 3</code> (still in compute node), I get a large error from slurm saying <code>srun: error: PMK_KVS_Barrier task count inconsistent (2 != 1)</code> (plus a load of other stuff), but I am not sure how to explain this either...</p>
  
  <p>Now, based on <a href="https://www.open-mpi.org/faq/?category=slurm" rel="nofollow noreferrer">this OpenMPI page</a>, it seems these kind of operations should be supported at least for OpenMPI:</p>
  
  <blockquote>
    <p>Specifically, you can launch Open MPI's mpirun in an interactive SLURM allocation (via the salloc command) or you can submit a script to SLURM (via the sbatch command), or you can "directly" launch MPI executables via srun. </p>
  </blockquote>
  
  <p>Maybe the Intel MPI implementation I was using just doesn't have the same support and is not designed to be used directly in a slurm environment (?), but I am still wondering: what is the underlying nature of <code>mpirun</code> and slurm (<code>salloc</code>) that this is the behavior produced? Why would it print two 0s in the first "case," and what are the inconsistent task counts it talks about in the second "case"?</p>