<p dir="auto">I have a simple setup with 1 master and 2 nodes running a couple of pods on Kubernetes v0.16.0.<br>
When I simulate a crash on one of them, pods keep being scheduled on it indefinitely.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get nodes

NAME                                        LABELS    STATUS
ip-10-0-21-131.eu-west-1.compute.internal   &lt;none&gt;    Ready
ip-10-0-21-132.eu-west-1.compute.internal   &lt;none&gt;    NotReady"><pre class="notranslate"><code class="notranslate">$ kubectl get nodes

NAME                                        LABELS    STATUS
ip-10-0-21-131.eu-west-1.compute.internal   &lt;none&gt;    Ready
ip-10-0-21-132.eu-west-1.compute.internal   &lt;none&gt;    NotReady
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get pods
web-4a3cq       10.144.71.11     ip-10-0-21-131.eu-west-1.compute.internal/10.0.21.131   app=web      Running   23 hours 
web-st0du                        ip-10-0-21-132.eu-west-1.compute.internal/              app=web      Pending   2 seconds"><pre class="notranslate"><code class="notranslate">$ kubectl get pods
web-4a3cq       10.144.71.11     ip-10-0-21-131.eu-west-1.compute.internal/10.0.21.131   app=web      Running   23 hours 
web-st0du                        ip-10-0-21-132.eu-west-1.compute.internal/              app=web      Pending   2 seconds
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ journalctl -u kube-controller-manager
nodecontroller.go:431] node ip-10-0-21-132.eu-west-1.compute.internal hasn't been updated for 15m30.391884952s. Last ready condition is: {Type:Ready Status:Unknown LastHeartbeatTime:2015-05-01 15:30:47 +0000 UTC LastTransitionTime:2015-05-01 15:31:29 +0000 UTC Reason:Kubelet stopped posting node status. Message:}
nodecontroller.go:504] Evicting pods2: 2015-05-01 15:46:19.911780538 +0000 UTC is later than 2015-05-01 15:46:19.911779449 +0000 UTC + -10s
nodecontroller.go:175] Error syncing cloud: Request has expired. (RequestExpired)nodecontroller.go:431] node ip-10-0-21-132.eu-west-1.compute.internal hasn't been updated for 15m35.402783207s. Last ready condition is: {Type:Ready Status:Unkn
own LastHeartbeatTime:2015-05-01 15:30:47 +0000 UTC LastTransitionTime:2015-05-01 15:31:29 +0000 UTC Reason:Kubelet stopped posting node status. Message:}
nodecontroller.go:504] Evicting pods2: 2015-05-01 15:46:24.924422488 +0000 UTC is later than 2015-05-01 15:46:24.924421484 +0000 UTC + -10snodecontroller.go:584] Delete all pods from ip-10-0-21-132.eu-west-1.compute.internal
nodecontroller.go:595] Delete pod api-seu4e
nodecontroller.go:595] Delete pod web-mbpos
replication_controller.go:253] Too few &quot;api&quot; replicas, creating 1
replication_controller.go:253] Too few &quot;web&quot; replicas, creating 1
endpoints_controller.go:314] Failed to find an IP for pod default/web-hcxtd
endpoints_controller.go:314] Failed to find an IP for pod default/api-rb12s
endpoints_controller.go:314] Failed to find an IP for pod default/web-hcxtd
endpoints_controller.go:314] Failed to find an IP for pod default/api-rb12s"><pre class="notranslate"><code class="notranslate">$ journalctl -u kube-controller-manager
nodecontroller.go:431] node ip-10-0-21-132.eu-west-1.compute.internal hasn't been updated for 15m30.391884952s. Last ready condition is: {Type:Ready Status:Unknown LastHeartbeatTime:2015-05-01 15:30:47 +0000 UTC LastTransitionTime:2015-05-01 15:31:29 +0000 UTC Reason:Kubelet stopped posting node status. Message:}
nodecontroller.go:504] Evicting pods2: 2015-05-01 15:46:19.911780538 +0000 UTC is later than 2015-05-01 15:46:19.911779449 +0000 UTC + -10s
nodecontroller.go:175] Error syncing cloud: Request has expired. (RequestExpired)nodecontroller.go:431] node ip-10-0-21-132.eu-west-1.compute.internal hasn't been updated for 15m35.402783207s. Last ready condition is: {Type:Ready Status:Unkn
own LastHeartbeatTime:2015-05-01 15:30:47 +0000 UTC LastTransitionTime:2015-05-01 15:31:29 +0000 UTC Reason:Kubelet stopped posting node status. Message:}
nodecontroller.go:504] Evicting pods2: 2015-05-01 15:46:24.924422488 +0000 UTC is later than 2015-05-01 15:46:24.924421484 +0000 UTC + -10snodecontroller.go:584] Delete all pods from ip-10-0-21-132.eu-west-1.compute.internal
nodecontroller.go:595] Delete pod api-seu4e
nodecontroller.go:595] Delete pod web-mbpos
replication_controller.go:253] Too few "api" replicas, creating 1
replication_controller.go:253] Too few "web" replicas, creating 1
endpoints_controller.go:314] Failed to find an IP for pod default/web-hcxtd
endpoints_controller.go:314] Failed to find an IP for pod default/api-rb12s
endpoints_controller.go:314] Failed to find an IP for pod default/web-hcxtd
endpoints_controller.go:314] Failed to find an IP for pod default/api-rb12s
</code></pre></div>
<p dir="auto">I am running my cluster on aws. The setups has a etcd cluster with 3 etcd nodes, 1 master and 4 minions.<br>
I am running a simple hello-world  nodejs script using docker to test my application. For skydns i am using the steps mentioned <a href="https://github.com/GoogleCloudPlatform/kubernetes/blob/ebcc45b11c64d17b589c0592426646c7e977f3c6/cluster/addons/dns/README.md#how-do-i-configure-it%5D">here</a></p>
<p dir="auto">I shutdown all minions, while my master and etcd were running.<br>
A kubectl get nodes on the master showed that two nodes were not ready and the other two as ready</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="core@ip-172-31-24-64 ~ $ ./kubectl get nodes
NAME           LABELS        STATUS
172.31.25.11   Schedulable   name=dns      NotReady
172.31.25.12   Schedulable   name=dns      NotReady
172.31.26.95   Schedulable   name=minion   Ready
172.31.27.92   Schedulable   name=minion   Ready"><pre class="notranslate"><code class="notranslate">core@ip-172-31-24-64 ~ $ ./kubectl get nodes
NAME           LABELS        STATUS
172.31.25.11   Schedulable   name=dns      NotReady
172.31.25.12   Schedulable   name=dns      NotReady
172.31.26.95   Schedulable   name=minion   Ready
172.31.27.92   Schedulable   name=minion   Ready
</code></pre></div>
<p dir="auto"><strong>Is it expected that the LABEL for a shut-down node is Schedulable?</strong><br>
I now scheduled my pods using replicationControllers and created the associated services. One of the pods was scheduled on a "NotReady" node.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" core@ip-172-31-24-64 ~ $ ./kubectl get pods 
 POD               IP           CONTAINER(S)   IMAGE(S)                               HOST            LABELS           STATUS    CREATED
 replicont-0cj5l   172.17.0.4   basicnodejs    basicnodejs   172.31.26.95/    name=replicont   Running   19 minutes
 replicont-gl3wm   172.17.0.4   basicnodejs    basicnodejs   172.31.27.92/    name=replicont   Running   19 minutes
 skydns-1ad74      172.17.0.5   etcd           quay.io/coreos/etcd:latest             172.31.27.92/   name=skydns      Running   18 minutes
                                kube2sky       kubernetes/kube2sky:1.0                                                           
                                skydns         kubernetes/skydns:2014-12-23-001                                                  
 skydns-oib7w                   etcd           quay.io/coreos/etcd:latest             172.31.25.12/   name=skydns      Pending   7 minutes
                                kube2sky       kubernetes/kube2sky:1.0                                                           
                                skydns         kubernetes/skydns:2014-12-23-001  "><pre class="notranslate"><code class="notranslate"> core@ip-172-31-24-64 ~ $ ./kubectl get pods 
 POD               IP           CONTAINER(S)   IMAGE(S)                               HOST            LABELS           STATUS    CREATED
 replicont-0cj5l   172.17.0.4   basicnodejs    basicnodejs   172.31.26.95/    name=replicont   Running   19 minutes
 replicont-gl3wm   172.17.0.4   basicnodejs    basicnodejs   172.31.27.92/    name=replicont   Running   19 minutes
 skydns-1ad74      172.17.0.5   etcd           quay.io/coreos/etcd:latest             172.31.27.92/   name=skydns      Running   18 minutes
                                kube2sky       kubernetes/kube2sky:1.0                                                           
                                skydns         kubernetes/skydns:2014-12-23-001                                                  
 skydns-oib7w                   etcd           quay.io/coreos/etcd:latest             172.31.25.12/   name=skydns      Pending   7 minutes
                                kube2sky       kubernetes/kube2sky:1.0                                                           
                                skydns         kubernetes/skydns:2014-12-23-001  
</code></pre></div>
<p dir="auto">I ran a kubectl delete pod on the problematic pod and the new pod was scheduled on another node which is not up.<br>
I waited for some time(may have been 5-10 minutes) and the pod was rescheduled on a live node. So I killed it and a new one was scheduled on a shutdown node.</p>