title: Why does Kubernetes not terminating PODs after a node crash?
body: <p>I am running a self managed Kubernetes cluster 1.18.1. I have deployed some pods with persistence volumes (based on the longhorn project).
  Now after doing some testing I observe the following behavior:</p>
  <p>If I simulate a hard shutdown of one node, after a while (5 minutes) Kubernetes is recognizing the loss and starts rescheduling PODs form the dead node to another.</p>
  <p>Because of the fact that my nodes had persistence volumes the new POD will never start. The reason is that the old pod (on the dead node) is now durable in the the status terminating.</p>
  <p>The fact that pods that reside on a crashed node did not terminate seems to be an well known Kubernetes limitation. See also the problem description <a href="https://www.ibm.com/support/knowledgecenter/SSRQ8T_1.0.0/csi_ug_node_crash.html" rel="nofollow noreferrer">here</a>.</p>
  <p>My question is: Why does Kubernetes not provide a function to automatically terminate old PODs and resources like persistence volumes. Why do I have to intervene manually as an administrator? To me, this behavior seems not logical regarding to the promises that Kubernetes makes.</p>
  <p>This is for example how my yaml file looks like:</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: db
    labels: 
      app: db
  spec:
    replicas: 1
    selector: 
      matchLabels:
        app: db
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: db
      spec:
        containers:
        - env:
          - name: POSTGRES_DB
            value: office
          image: postgres:9.6.1
          name: db
  
          livenessProbe:
            tcpSocket:
              port: 5432
            initialDelaySeconds: 30
            periodSeconds: 10
  
          ports:
            - containerPort: 5432        
          resources: {}
          volumeMounts:
          - mountPath: /var/lib/postgresql/data
            name: dbdata
            subPath: postgres
        restartPolicy: Always
        volumes:
        - name: dbdata
          persistentVolumeClaim:
            claimName: office-demo-dbdata-pvc
  
  
  # Storage
  ---
  kind: PersistentVolume
  apiVersion: v1
  metadata:
    name: office-demo-dbdata-pv
  spec:
    capacity:
      storage: 2Gi
    volumeMode: Filesystem
    accessModes:
      - ReadWriteOnce
    claimRef:
      namespace: default
      name: office-demo-dbdata-pvc
    csi:
      driver: io.rancher.longhorn 
      fsType: ext4
      volumeHandle: office-demo-dbdata
    storageClassName: longhorn-durable
  ---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: office-demo-dbdata-pvc
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: longhorn-durable
    resources:
      requests:
        storage: 2Gi
    volumeName: &quot;office-demo-dbdata-pv&quot;
  </code></pre>
  <p>As explained the volume is created on Longhorn. But the attachement is not released even after kubernetes starts to reschedule the pod to another node.</p>
  <p>The pod hanging in the termination status can be released if I manually delete the 'volumeattachment'</p>
  <pre><code>$ kubectl delete volumeattachment csi-08d9842e.......
  </code></pre>
  <p>But in any case this is a manual action.</p>