title: Prevent GPU usage in SLURM when --gpus is not set
body: <p>We're using SLURM to manage a small on-premise cluster.  A key resource we are managing is GPUs.  When a user requests GPUs via <code>--gpus=2</code> the <code>CUDA_VISIBLE_DEVICES</code> environment variable is set with the GPUs SLURM allocates to the user.</p>
  
  <pre><code>$ srun --gpus=2 bash -c 'echo $CUDA_VISIBLE_DEVICES'
  0,1
  </code></pre>
  
  <p>We have a small team and can trust our users to not abuse the system (they could easily overwrite the environment variable) so this works great.  However, it's a bit too easy to bypass this accidentally because when <code>--gpus</code> isn't specified <code>$CUDA_VISIBLE_DEVICES</code> is left unset so the user can use any GPU (we're typically using PyTorch).</p>
  
  <p>In other words, the following command works fine (so long as it lands on a GPU node) but I would prefer that it fails (because no GPU was requested).</p>
  
  <pre><code>srun sudo docker run -e CUDA_VISIBLE_DEVICES --runtime=nvidia pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime python -c 'import torch; print(torch.tensor([1., 2.], device=torch.device("cuda:0")))'
  </code></pre>
  
  <p>It would fail if <code>$CUDA_VISIBLE_DEVICES</code> were set to <code>-1</code>.</p>
  
  <pre><code>$ CUDA_VISIBLE_DEVICES=-1 sudo docker run -e CUDA_VISIBLE_DEVICES --runtime=nvidia pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime python -c 'import torch; print(torch.tensor([1., 2.], device=torch.device("cuda:0")))'
  THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp line=51 error=38 : no CUDA-capable device is detected
  Traceback (most recent call last):
    File "&lt;string&gt;", line 1, in &lt;module&gt;
    File "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
      torch._C._cuda_init()
  RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp:51
  </code></pre>
  
  <p>How can I configure SLURM to set <code>CUDA_VISIBLE_DEVICES</code> to <code>-1</code> when <code>--gpus</code> is not specified?</p>