title: k8s pod readiness probed failed: read tcp xxx -> yyy: read: connection reset by peer
body: <p>I'm running Fargate on EKS and I have about 20~30 pods running. After about a few days (5 ~ 7 days; experienced two times), they begin to refuse Readiness probe HTTP requests. I captured the pod's description at that time. I want to point out the first event - <code>connection reset by peer</code>.</p>
  <p>I've come across <a href="https://github.com/istio/istio/issues/15881" rel="nofollow noreferrer">this issue</a> in Istio and the root cause can be the same. However, I don't use Istio so I'm stuck where to go. I'm going to attach partial data of my ingress, service, and deployment below.</p>
  <pre><code>Events:
    Type     Reason             Age                  From     Message
    ----     ------             ----                 ----     -------
    Warning  Unhealthy          56m                  kubelet  Readiness probe failed: Get &quot;http://10.104.4.xxx:20001/health_readiness&quot;: read tcp 169.254.175.xxx:36978-&gt;10.104.4.xxx:20001: read: connection reset by peer
    Warning  Unhealthy          55m (x3 over 56m)    kubelet  Liveness probe failed: dial tcp 10.104.4.xxx:20001: connect: connection refused
    Normal   Killing            55m                  kubelet  Container hybrid-server-logic failed liveness probe, will be restarted
    Warning  FailedPreStopHook  55m                  kubelet  Exec lifecycle hook ([/bin/bash -c kill -SIGTERM $(ps -ef | grep node | grep -v grep | awk '{print $1}')]) for Container &quot;hybrid-server-logic&quot; in Pod &quot;hybrid-server-logic-745bf8ffc4-479x6_jpj-prod(c4acfaef-a8a6-41e8-9d89-3c03336388b3)&quot; failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to create exec &quot;e92f0b6c6f1dcfa680a03ed3d2dc9b5176980d7b6dce371a8bcbb2c5eb2368fe&quot;: mkdir /run/containerd/io.containerd.grpc.v1.cri/containers/hybrid-server-logic/io/168763600: no space left on device, message: &quot;&quot;
    Warning  Unhealthy          72s (x331 over 56m)  kubelet  Readiness probe failed: Get &quot;http://10.104.4.xxx:20001/health_readiness&quot;: dial tcp 10.104.4.xxx:20001: connect: connection refused
  </code></pre>
  <pre><code>//ingress
  http {
          path {
            path = &quot;/*&quot;
            backend {
              service_name = &quot;my-app-service&quot;
              service_port = 20001
            }
          }
  }
  
  </code></pre>
  <pre><code>// serivce
  name = my-app-service
  spec {
      port {
        port        = 20001
        protocol    = &quot;TCP&quot;
        target_port = &quot;my-app-port&quot;
      }
      selector = {
        &quot;app&quot; = &quot;my-app&quot;
      }
      type = &quot;NodePort&quot;
  }
  </code></pre>
  <pre><code>// deployment
  ...
  ports:
          - containerPort: 20001
            name: logic-port
            protocol: TCP
  ...
  readinessProbe: # on failure, k8s will not forward traffic.
            httpGet:
              path: /health_readiness
              port: my-app-port
            initialDelaySeconds: 20
            periodSeconds: 10
            timeoutSeconds: 5
  livenessProbe: # on failure, k8s will restarts the server.
            tcpSocket:
              port: my-app-port
            initialDelaySeconds: 10
            periodSeconds: 20
            timeoutSeconds: 5
  </code></pre>