title: how can I alert on Pod Eviction or Failed && Evicted pods in kubernetes
body: <p>I can see from the pod description that my pod &quot;Failed&quot; due to being &quot;Evicted&quot; due to Memory Pressure. but how can i test for too many &quot;Failed &amp;&amp; Evicted&quot; pods with prometheus alert rules or some other means?</p>
  <p>I have Prometheus Operator installed, I can see metrics for Failed Pods but not Failed and Evicted</p>
  <p>kubectl describe pod gives:</p>
  <pre><code>Name:         besteffort-evictme-001
  Namespace:    skyfii
  Priority:     0
  Node:         ip-172-17-2-169.ap-southeast-2.compute.internal/
  Start Time:   Fri, 24 Sep 2021 15:28:53 +1000
  Labels:       &lt;none&gt;
  Annotations:  kubernetes.io/psp: eks.privileged
  Status:       Failed
  Reason:       Evicted
  Message:      The node was low on resource: memory. Container termination-demo-container was using 17165108Ki, which exceeds its request of 0. 
  IP:           
  IPs:          &lt;none&gt;
  Containers:
  
  </code></pre>
  <p>the prometheus rule :</p>
  <pre><code>kube_pod_status_phase{phase=&quot;Failed&quot;} &gt; 0
  </code></pre>
  <p>shows the failed pod</p>
  <pre><code>kube_pod_status_phase{endpoint=&quot;http&quot;,instance=&quot;172.17.3.141:8080&quot;,job=&quot;kube-state-metrics&quot;,namespace=&quot;skyfii&quot;,phase=&quot;Failed&quot;,pod=&quot;besteffort-evictme-001&quot;,service=&quot;prometheus-kube-state-metrics&quot;}
  </code></pre>
  <p>but nothing shows up for</p>
  <pre><code>kube_pod_container_status_terminated_reason{reason=&quot;Evicted&quot;} &gt; 0
  </code></pre>
  <p>Any Ideas?</p>
  <p>Thanks
  Karl</p>