title: Cannot enforce memory limits in SLURM
body: <p>I am using Slurm on a single node (control and compute) and I cannot seem to correctly limit memory. The script seems to call SBATCH with small memory values (3G), but I see values in top that exceed 25G. Sacct gives me the correct values:</p>
  <pre><code>squeue -o &quot;%C %m&quot;
  CPUS MIN_MEMORY
  2 3G
  </code></pre>
  <p>This is my slurm.conf:</p>
  <pre><code>#
  SlurmctldHost=schopenhauer
  #
  #DisableRootJobs=NO
  #EnforcePartLimits=NO
  #Epilog=
  #EpilogSlurmctld=
  #FirstJobId=1
  #MaxJobId=999999
  #GresTypes=
  #GroupUpdateForce=0
  #GroupUpdateTime=600
  #JobFileAppend=0
  #JobRequeue=1
  #JobSubmitPlugins=1
  #KillOnBadExit=0
  #LaunchType=launch/slurm
  #Licenses=foo*4,bar
  #MailProg=/bin/mail
  #MaxJobCount=5000
  #MaxStepCount=40000
  #MaxTasksPerNode=128
  MpiDefault=none
  #MpiParams=ports=#-#
  #PluginDir=
  #PlugStackConfig=
  #PrivateData=jobs
  #ProctrackType=proctrack/cgroup
  ProctrackType=proctrack/linuxproc
  #Prolog=
  #PrologFlags=
  #PrologSlurmctld=
  #PropagatePrioProcess=0
  #PropagateResourceLimits=
  #PropagateResourceLimitsExcept=
  #RebootProgram=
  ReturnToService=1
  #SallocDefaultCommand=
  SlurmctldPidFile=/var/run/slurmctld.pid
  SlurmctldPort=6817
  SlurmdPidFile=/var/run/slurmd.pid
  SlurmdPort=6818
  SlurmdSpoolDir=/var/spool/slurmd
  SlurmUser=slurm
  #SlurmdUser=root
  #SrunEpilog=
  #SrunProlog=
  StateSaveLocation=/var/spool/slurmd
  SwitchType=switch/none
  #TaskEpilog=
  TaskPlugin=task/affinity
  TaskPluginParam=Sched
  #TaskProlog=
  #TopologyPlugin=topology/tree
  #TmpFS=/tmp
  #TrackWCKey=no
  #TreeWidth=
  #UnkillableStepProgram=
  #UsePAM=0
  #
  #
  # TIMERS
  #BatchStartTimeout=10
  #CompleteWait=0
  #EpilogMsgTime=2000
  #GetEnvTimeout=2
  #HealthCheckInterval=0
  #HealthCheckProgram=
  InactiveLimit=0
  KillWait=30
  #MessageTimeout=10
  #ResvOverRun=0
  MinJobAge=300000
  #OverTimeLimit=0
  SlurmctldTimeout=120
  SlurmdTimeout=300
  #UnkillableStepTimeout=60
  #VSizeFactor=0
  Waittime=0
  #
  #
  # SCHEDULING
  #DefMemPerCPU=0
  #MaxMemPerCPU=0
  #SchedulerTimeSlice=30
  SchedulerType=sched/backfill
  SelectType=select/cons_tres
  SelectTypeParameters=CR_Core
  #
  #
  # JOB PRIORITY
  #PriorityFlags=
  #PriorityType=priority/basic
  #PriorityDecayHalfLife=
  #PriorityCalcPeriod=
  #PriorityFavorSmall=
  #PriorityMaxAge=
  #PriorityUsageResetPeriod=
  #PriorityWeightAge=
  #PriorityWeightFairshare=
  #PriorityWeightJobSize=
  #PriorityWeightPartition=
  #PriorityWeightQOS=
  #
  #
  # LOGGING AND ACCOUNTING
  #AccountingStorageEnforce=0
  #AccountingStorageHost=
  AccountingStorageLoc=/var/log/slurm/slurm_jobacct.log
  #AccountingStoragePass=
  #AccountingStoragePort=
  AccountingStorageType=accounting_storage/filetxt
  #AccountingStorageUser=
  AccountingStoreJobComment=YES
  ClusterName=cluster
  #DebugFlags=
  #JobCompHost=
  JobCompLoc=/var/log/slurm/slurm_jobcomp.log
  #JobCompPass=
  #JobCompPort=
  JobCompType=jobcomp/filetxt
  #JobCompUser=
  #JobContainerType=job_container/none
  JobAcctGatherFrequency=30
  JobAcctGatherType=jobacct_gather/linux
  SlurmctldDebug=info
  SlurmctldLogFile=/var/log/slurm/slurmctld.log
  SlurmdDebug=debug5
  SlurmdLogFile=/var/log/slurm/slurmd.log
  #SlurmSchedLogFile=
  #SlurmSchedLogLevel=
  #
  #
  # POWER SAVE SUPPORT FOR IDLE NODES (optional)
  #SuspendProgram=
  #ResumeProgram=
  #SuspendTimeout=
  #ResumeTimeout=
  #ResumeRate=
  #SuspendExcNodes=
  #SuspendExcParts=
  #SuspendRate=
  #SuspendTime=
  #
  #
  # COMPUTE NODES
  NodeName=schopenhauer CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=500000 State=UNKNOWN
  PartitionName=short Nodes=schopenhauer Default=YES MaxTime=INFINITE State=UP
  </code></pre>
  <p>Did I misunderstand something? Why does it say minimum memory when I want to make that minimum and maximum as well?</p>
  <p>EDIT: I just noticed by setting required memory to a larger one that this doesn't work as a minimum either i.e. many tasks were started even though there was enough RAM for only 12 of them (I requested 40G and I have 500G). Is this the same problem?</p>