title: How to specify memory per process in an array job in slurm?
body: <p>I want to run thousands of identical single-thread simulations with different random seeds (which I pass to my program). Some of them have gone out of memory, yet I don't know why. I call <code>run_batch_job</code> as <code>sbatch --array=0-999%100 --mem=200M run_batch_job</code>, where <code>run_batch_job</code> contains:</p>
  
  <pre><code>#!/bin/env bash
  #SBATCH --ntasks=1  # Number of cores
  #SBATCH --nodes=1  # All cores on one machine
  
  srun my_program.out $SLURM_ARRAY_TASK_ID
  </code></pre>
  
  <p>For a single thread, 200M should be more than enough memory, yet for some simulations, I get the error:</p>
  
  <pre><code>slurmstepd: error: Exceeded step memory limit at some point.
  slurmstepd: error: Exceeded job memory limit at some point.
  srun: error: cluster-cn002: task 0: Out Of Memory
  slurmstepd: error: Exceeded job memory limit at some point.
  </code></pre>
  
  <p>Am I allocating 200M to each of the thousand threads, or am I doing something wrong?</p>
  
  <p>EDIT: I've tried specifying <code>--cpus-per-task=1</code> and <code>--mem-per-cpu=200M</code> instead of <code>--ntasks=1</code>, <code>--nodes=1</code> and <code>--mem=200M</code>, with the same results.</p>