title: Slurm: how many times will failed jobs be --requeue'd
body: <p>I have a Slurm job array for which the job file includes a <code>--requeue</code> directive. Here is the full job file:</p>
  
  <pre><code>#!/bin/bash
  #SBATCH --job-name=catsss
  #SBATCH --output=logs/cats.log
  #SBATCH --array=1-10000
  #SBATCH --requeue
  #SBATCH --partition=scavenge
  #SBATCH --mem=32g
  #SBATCH --time=24:00:00
  #SBATCH --mail-type=FAIL
  #SBATCH --mail-user=douglas.duhaime@gmail.com
  module load Langs/Python/3.4.3
  python3 cats.py ${SLURM_ARRAY_TASK_ID} 'cats'
  </code></pre>
  
  <p>Several of the array values have restarted at least once. I would like to know, how many times will these jobs restart before they are finally cancelled by the scheduler? Will the restarts carry on indefinitely until a sysadmin manually cancels them, or do jobs like this have a maximum number of retries?</p>