title: How do I specify a minimum number of schedulable pods in kubernetes?
body: <p>I want to run multiple jobs in a kubernetes cluster, but the total resource requirements exceed the size of the cluster, and the requirements of one job span multiple nodes. How do I avoid a livelock where all jobs have some resources, but none have enough to complete?</p>
  <p>For example, suppose I have 4 nodes, each with 1 GB of memory available. I want to submit 2 jobs, each of which requires 3 GB of memory to complete, split across 3 pods that require 1 GB each. The correct solution here would be to run the jobs sequentially, how do I ensure this happens?</p>
  <p>I want to avoid the situation where both jobs schedule two pods each, using up the entire cluster, while the remaining pod of each job is stuck in the <code>Pending</code> state, as no more resources are available. Because the jobs cannot complete using only 2 GB of memory, the system is now incapable of making progress.</p>
  <h1>Related Features</h1>
  <p>Some features I've looked at that don't seem to be suitable:</p>
  <ol>
  <li><a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/" rel="nofollow noreferrer">Pod Disruption Budget</a> - this is for ensuring that the number of pods never goes below X, but doesn't have any effect when scheduling the pods initially</li>
  <li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="nofollow noreferrer">Pod Affinity</a> - this can ensure I schedule pods in a region where a matching pod is running, but I can't require two or more pods. I'm also not sure if affinity would be satisfied if no pods are running but they are scheduled.</li>
  <li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/" rel="nofollow noreferrer">Pod Topology Spread Constraints</a> - This is to ensure that the numbers of pods scheduled in multiple regions is always within N of each other, but again I can't specify a required minimum.</li>
  </ol>
  <h1>Possible Solution</h1>
  <p>It looks like a custom scheduler is needed. <a href="https://github.com/kubernetes-sigs/kube-batch/blob/master/doc/usage/tutorial.md" rel="nofollow noreferrer">Kube Batch</a> looks like a possible solution for this, supporting a <code>minMember</code> attribute. I will test this and submit it as a self-answer, unless anyone can chime in with more detail.</p>