title: On the semantics of `srun ... >output_file` for parallel tasks
body: <p>Sorry, this question requires a lot of build-up, but in summary, it's about the conditions under which many parallel instances of <code>srun ... &gt;output_file</code> will or won't lead to the clobbering by some process/task of the output produced by some other process/task.</p>
  
  <hr>
  
  <p><strong>CASE 0: bash only (no SLURM)</strong></p>
  
  <p>Suppose that <code>prog-0.sh</code> is the following toy script:</p>
  
  <pre><code>#!/bin/bash
  
  hostname &gt;&amp;2
  
  if [[ $JOB_INDEX = 0 ]]
  then
      date
  fi
  </code></pre>
  
  <p>This script prints some output to <code>stderr</code>, and possibly prints the current date to <code>stdout</code>.</p>
  
  <p>The "driver" script <code>case-0.sh</code> shown below spawns <code>$NJOBS</code> processes, all writing to <code>prog-0-stdout.txt</code>:</p>
  
  <pre><code>#!/bin/bash
  
  for i in $( seq 0 $(( NJOBS - 1 )) )
  do  
      JOB_INDEX=$i ./prog-0.sh &gt;prog-0-stdout.txt &amp;
  done
  </code></pre>
  
  <p>After running</p>
  
  <pre><code>% NJOBS=100 ./case-0.sh 2&gt;prog-0-stderr.txt
  </code></pre>
  
  <p>...my expectation is that <code>prog-0-stderr.txt</code> will contain 100 lines, and that <code>prog-0-stdout.txt</code> will be <strong><em>empty</em></strong>.</p>
  
  <p>My expectation pans out:</p>
  
  <pre><code> % wc prog-0-std*.txt
    100  100 3000 prog-0-stderr.txt
      0    0    0 prog-0-stdout.txt
    100  100 3000 total
  </code></pre>
  
  <p>The explanation for these results is that, when <code>NJOBS</code> is sufficiently large, it is likely that, for some sufficiently high value of <code>$i</code>, the redirection <code>&gt;prog-0-stdout.txt</code> will be evaluated <em>after</em> the "designated job", the one <code>JOB_INDEX</code> 0 (and the only one that sends output to <code>stdout</code>) has written the date to <code>stdout</code>, and this will therefore clobber whatever output was earlier redirected by the "designated job" to <code>prog-0-stdout.txt</code>.</p>
  
  <p>BTW, the value of <code>NJOBS</code> needs to be high enough for the results to be as I've just described.  For example, if I use <code>NJOBS=2</code>:</p>
  
  <pre><code>% NJOBS=2 ./case-0.sh 2&gt;prog-0-stderr.txt
  </code></pre>
  
  <p>...then not only will <code>prog-0-stderr.txt</code> contain only 2 lines (not surprisingly), but <code>prog-0-stdout.txt</code> will contain a date:</p>
  
  <pre><code>% cat prog-0-stdout.txt
  Wed Oct  4 15:02:49 EDT 2017
  </code></pre>
  
  <p>In this case, all the <code>&gt;prog-0-stdout.txt</code> redirections have been evaluated before the designated job prints the date to <code>prog-0-stdout.txt</code>.</p>
  
  <hr>
  
  <p><strong>CASE 1: SLURM job arrays</strong></p>
  
  <p>Now, consider a very similar scenario, but using SLURM instead.  The script <code>prog-1.sh</code> is identical to <code>prog-0.sh</code>, except that it examines a different variable to decide whether or not to print the date to <code>stdout</code>:</p>
  
  <pre><code>#!/bin/bash
  
  hostname &gt;&amp;2
  
  if [[ $SLURM_ARRAY_TASK_ID = 0 ]]
  then
      date
  fi
  </code></pre>
  
  <p>And here's the corresponding "driver" script, <code>case-1.sh</code>:</p>
  
  <pre><code>#!/bin/bash
  #SBATCH -t 1
  #SBATCH -p test
  
  #SBATCH -e prog-1-%02a-stderr.txt
  #SBATCH -n 1
  #SBATCH -a 0-99
  
  srun ./prog-1.sh &gt;prog-1-stdout.txt
  </code></pre>
  
  <p>Like <code>case-0.sh</code>, this script redirects the output of its main step to a single file <code>./prog-1-stdout.txt</code>.</p>
  
  <p><em>Importantly</em>, this same file will be seen by all the nodes that run <code>./prog-1.sh</code> for this job.</p>
  
  <p>If I now run</p>
  
  <pre><code>sbatch case-1.sh
  </code></pre>
  
  <p>...I get 100 files <code>prog-1-00-stderr.txt</code> ... <code>prog-1-99-stderr.txt</code>, containing 1 line each, and an <em>empty</em> <code>prog-1-stdout.txt</code>.  I assume that the earlier explanation also explains why <code>prog-1-stdout.txt</code> is empty.</p>
  
  <p>So far so good.</p>
  
  <hr>
  
  <p><strong>CASE 2: SLURM tasks</strong></p>
  
  <p>Finally, consider one more SLURM-based case, this time using the core script <code>prog-2.sh</code> and the driver script <code>case-2.sh</code>.  Again, the only change in <code>prog-2.sh</code> is the variable it examines to decide whether or not to print the date to <code>stdout</code>:</p>
  
  <pre><code>#!/bin/bash
  
  hostname &gt;&amp;2
  
  if [[ $SLURM_PROCID = 1 ]]
  then
      date
  fi
  </code></pre>
  
  <p>And here is <code>case-2.sh</code>:</p>
  
  <pre><code>#!/bin/bash
  #SBATCH -t 1
  #SBATCH -p test
  
  #SBATCH -e prog-2-stderr.txt
  #SBATCH -N 10
  #SBATCH --tasks-per-node=10
  
  srun -l ./prog-2.sh &gt;prog-2-stdout.txt
  </code></pre>
  
  <p>As before, <code>prog-2-stdout.txt</code> is visible by all the nodes handling the job.</p>
  
  <p>Now, if I run <code>sbatch case-2.sh</code> and wait for the batch job to finish, then <code>prog-2-stderr.txt</code> contains 100 lines (as expected), but, <em>to my surprise</em>, <code>prog-2-stdout.txt</code> is <em>not</em> empty.  In fact, it contains a date:</p>
  
  <pre><code>% cat prog-2-stdout.txt
  01: Wed Oct  4 15:21:17 EDT 2017
  </code></pre>
  
  <p>The only explanation I can come up with is analogous to the one I gave earlier for the results I got when I ran</p>
  
  <pre><code>% NJOBS=2 ./case-0.sh 2&gt;prog-0-stderr.txt
  </code></pre>
  
  <p>If this explanation is correct, my concern is that the fact <code>case-2.sh</code> worked better than expected (i.e. prog-2-stdout.txt ends up with the right output) is just a coincidence, having to do with the relative timing of concurrent events.</p>
  
  <hr>
  
  <p>Now, at long last, my question is:</p>
  
  <p><strong>Q:</strong> does SLURM guarantee that a <code>prog-2-stdout.txt</code> file that contains the output generated by the designated task (i.e. the one that prints the date to <code>stdout</code>) will not be clobbered when the <code>&gt;prog-2-stdout.txt</code> redirection gets evaluated by one of the non-designated tasks?</p>