title: Using podAntiAffinity rules to ensure pods run on different pre-emptible nodes
body: <p>I have a 3-node cluster running on GKE. All the nodes are pre-emptible meaning they can be killed at any time and generally do not live longer than 24 hours. In the event a node is killed the autoscaler spins up a new node to replace it. This usually takes a minute or so when this happens.</p>
  
  <p>In my cluster I have a deployment with its replicas set to 3. My intention is that each pod will be spread across all the nodes such that my application will still run as long as at least one node in my cluster is alive.</p>
  
  <p>I've used the following affinity configuration such that pods prefer running on hosts different to ones already running pods for that deployment:</p>
  
  <pre><code>spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - my-app
            topologyKey: kubernetes.io/hostname
          weight: 100
  </code></pre>
  
  <p>When I scale my application from 0 this seems to work as intended. But in practice the following happens:</p>
  
  <ol>
  <li>Lets say pods belonging to the <code>my-app</code> replicaset <code>A</code>, <code>B</code> and <code>C</code> are running on nodes <code>1</code>, <code>2</code> and <code>3</code> respectively. So state would be:</li>
  </ol>
  
  <pre><code>  1 -&gt; A
    2 -&gt; B
    3 -&gt; C
  </code></pre>
  
  <ol start="2">
  <li>Node 3 is killed taking pod C with it, resulting in 2 running pods in the replicaset.</li>
  <li>The scheduler automatically starts to schedule a new pod to bring the replicaset back up to 3.</li>
  <li>It looks for a node without any pods for <code>my-app</code>. As the autoscalar is still in the process of starting a replacement node (<code>4</code>), only <code>1</code> and <code>2</code> are available.</li>
  <li>It schedules the new pod <code>D</code> on node <code>1</code></li>
  <li>Node <code>4</code> eventually comes online but as <code>my-app</code> has all its pods scheduled it doesn't have any pods running on it. Resultant state is</li>
  </ol>
  
  <pre><code>  1 -&gt; A, D
    2 -&gt; B
    4 -&gt; -
  </code></pre>
  
  <p>This is not the ideal configuration. The problem arises because there's a delay creating the new node and the schedular is not aware that it'll be available very soon.</p>
  
  <p>Is there a better configuration that can ensure the pods will always be distributed across the node? I was thinking a directive like <code>preferredDuringSchedulingpreferredDuringExecution</code> might do it but that doesn't exist.</p>