title: Kubernetes - How to remove Pod from Service load balancers when the node hosting the Pod gets unreachable
body: <p>having a local Kubernetes cluster (Master and 2 Worker Nodes) on my VirtualBox deployed with kubeadm:</p>
  <pre><code>NAME     STATUS   ROLES    AGE     VERSION
  master   Ready    master   2d21h   v1.19.2
  node1    Ready    &lt;none&gt;   2d21h   v1.19.2
  node2    Ready    &lt;none&gt;   2d21h   v1.19.2
  </code></pre>
  <p>the containerized application is a trivial node.js that displays &quot;HelloWorld&quot; from the URL http://ExternalIP:8080</p>
  <pre><code>NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
  helloworld-service   LoadBalancer   10.111.237.33   192.168.1.163   8080:30317/TCP   124m
  kubernetes           ClusterIP      10.96.0.1       &lt;none&gt;          443/TCP          2d21h
  </code></pre>
  <p>The Deployment configured has replicas=2. Supposing that one Pod is running on node1 and the other on node2, if i run the command:</p>
  <pre><code> curl http://ExternalIP:8080
  </code></pre>
  <p>in a for loop, from an external machine outside the cluster, the application gives the &quot;HelloWorld&quot; for every received request, as expected. Here the describe Service:</p>
  <pre><code> Name:                     helloworld-service
   Namespace:                default
   Labels:                   &lt;none&gt;
   Annotations:              &lt;none&gt;
   Selector:                 app=hello-world-app
   Type:                     LoadBalancer
   IP:                       10.111.237.33
   External IPs:             192.168.1.163
   Port:                     &lt;unset&gt;  8080/TCP
   TargetPort:               8081/TCP
   NodePort:                 &lt;unset&gt;  30317/TCP
   Endpoints:                10.36.0.1:8081,10.44.0.1:8081
   Session Affinity:         None
   External Traffic Policy:  Cluster
   Events:                   &lt;none&gt;
  </code></pre>
  <p>Now, having the for loop active from the external machine, i want to test downtime of the application turning off one of the Worker Node. The result is:</p>
  <pre><code> - about 30 seconds having no response and 503 service unavailable from application
   - the application starts working fine only after the Master declares the turned-off Node as NotRunning 
  </code></pre>
  <p>at this point the output of the cluster is:</p>
  <pre><code>NAME     STATUS      ROLES    AGE     VERSION
  master   Ready       master   2d21h   v1.19.2
  node1    Ready       &lt;none&gt;   2d21h   v1.19.2
  node2    NotReady    &lt;none&gt;   2d21h   v1.19.2
  </code></pre>
  <p>and application replies to every request.</p>
  <p>From kubernetes.io documentation:</p>
  <p><em>The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.</em></p>
  <p>I changed my yaml configuration in the Deployment object, but still getting those 30 seconds of application downtime:</p>
  <pre><code>  apiVersion: v1
    kind: Service
    metadata:
      name: helloworld-service
  
    spec:
      selector:
        app: hello-world-app
  
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8081
  
      type: LoadBalancer
      clusterIP: 10.111.237.33
  
      externalIPs:
      - 192.168.1.163
  
    status:
      loadBalancer:
        ingress:
        - ip: 192.168.1.224
  </code></pre>
  <hr />
  <pre><code>  apiVersion: apps/v1
  
    kind: Deployment
  
    metadata:
      name: helloworld-deployment
  
    spec:
      selector:
        matchLabels:
          app: hello-world-app
      
      replicas: 2
  
      template:
        metadata:
          labels:
            app: hello-world-app
  
        spec:
          containers:
          - name: hello-world-container
            image: marcoif81/testmarcoif:latest
  
            ports:
            - containerPort: 8081
  
            livenessProbe:
              tcpSocket:
                port: 8081
              initialDelaySeconds: 15
              periodSeconds: 2
              failureThreshold: 2
            readinessProbe:
              tcpSocket:
                port: 8081
              initialDelaySeconds: 15
              periodSeconds: 2
              failureThreshold: 2
  </code></pre>
  <p>from the command here, execute during application downtime:</p>
  <pre><code>  # kubectl describe pod &quot;pod_name&quot;
  </code></pre>
  <p>in the Events section no output is showned about probing:</p>
  <pre><code>Events:
  Type     Reason        Age   From               Message
  ----     ------        ----  ----               -------
  Normal   Scheduled     89s   default-scheduler  Successfully assigned default/helloworld-deployment-7456bb6569-qrfnj to node2
  Warning  FailedMount   88s   kubelet            MountVolume.SetUp failed for volume &quot;default-token-hjq76&quot; : failed to sync secret cache: timed out waiting for the condition
  Normal   Pulling       87s   kubelet            Pulling image &quot;marcoif81/testmarcoif:latest&quot;
  Normal   Pulled        86s   kubelet            Successfully pulled image &quot;marcoif81/testmarcoif:latest&quot; in 1.581369887s
  Normal   Created       86s   kubelet            Created container hello-world-container
  Normal   Started       85s   kubelet            Started container hello-world-container
  Warning  NodeNotReady  1s    node-controller    Node is not ready
  </code></pre>