title: Liveness probe on batch Job restarts Pod after completion
body: <p>Is there a way to configure a liveness probe to stop running when a Pod has successfully completed?</p>
  
  <p>I'm using a liveness probe to ensure that batch Jobs (which are expected to run to completion over the course of a few minutes to weeks) are responsive and running properly. However, when a Pod completes successfully, there seems to be a delay between when the Pod stops serving the liveness probe (in this case, touch-ing a file) and when the Pod is deleted after completing successfully. During this delay, the liveness probe fails enough times to trigger Kubernetes to restart the Pod.</p>
  
  <p>Aside from increasing the liveness probe's failure threshold or period, or decreasing the Pod's termination grace period, I haven't come across any possible mitigations, and no robust solutions, for this issue. In fact, I haven't found any mention in the Kubernetes' docs of using a liveness probe in a batch Job.</p>
  
  <p>The events log from <code>kubectl describe pod &lt;pod&gt;</code> is below. Of particular interest to me, and what's guiding my thinking that the liveness probe is failing during the Pod's completion, is the message <code>Liveness probe failed: OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown</code>.</p>
  
  <pre><code>Events:
    Type     Reason     Age                From              Message
    ----     ------     ----               ----              -------
    Warning  Unhealthy  55m                kubelet, pascal0  Liveness probe failed: OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown
    Normal   Created    55m (x8 over 21h)  kubelet, pascal0  Created container
    Normal   Pulled     55m (x7 over 18h)  kubelet, pascal0  Container image "&lt;image&gt;" already present on machine
    Normal   Started    55m (x8 over 21h)  kubelet, pascal0  Started container
  </code></pre>
  
  <p>Some relevant Job configuration values are included below.</p>
  
  <pre><code>backoffLimit: 10
  restartPolicy: OnFailure
  livenessProbe:
    exec:
      command:
        - test
        - $(stat -c %Y /tmp/healthy) -gt $(($(date +%s) - 10))
    initialDelaySeconds: 30
    periodSeconds: 60
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 3
  </code></pre>