title: CPU usage less than a threshold and more than one node present - Prometheus
body: <p>I am trying to create a alert rule in prometheus so that it fires an alert when the avg CPU usage of all nodes with label agentpool="worker" for last 3 mins has been below 30%.</p>
  
  <p>Right now, I am able to get alert for ANY node with CPU usage less than 30% for last 3 mins using</p>
  
  <pre><code>- alert: NodeCPUUtilizationLow
      expr: instance:node_cpu:rate:sum * 100 &lt; 30
      labels:
          severity: none
      annotations:
          description: CPU utilization has been lower than 30% for last 3 minutes (current value is {{$value}})
  </code></pre>
  
  <p>What should be the expr so that it alerts only for nodes having agentpool="worker" label? I am able to list the nodes with label agentpool="worker" using <code>kube_node_labels{label_agentpool="worker"}</code>.</p>
  
  <p>How can I do the same for pods? I want a similar alert: if avg CPU usage is below a threshold for all the pods/containers having label <code>app=web</code></p>
  
  <p>I am running prometheus on kubernetes. It was installed using helm charts from here: <a href="https://github.com/coreos/prometheus-operator/tree/master/helm/" rel="nofollow noreferrer">https://github.com/coreos/prometheus-operator/tree/master/helm/</a></p>
  
  <p>Output:</p>
  
  <p><strong>instance:node_cpu:rate:sum * 100 &lt; 30</strong></p>
  
  <pre><code>{instance="10.240.0.187:9100"}  26.466666666668715
  </code></pre>
  
  <p><strong>kube_node_labels{label_agentpool="worker"}</strong></p>
  
  <pre><code>kube_node_labels{endpoint="kube-state-metrics",instance="10.240.0.187:8080",job="kube-prometheus-exporter-kube-state",label_agentpool="worker",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_instance_type="Standard_B2s",label_beta_kubernetes_io_os="linux",label_failure_domain_beta_kubernetes_io_region="southcentralus",label_failure_domain_beta_kubernetes_io_zone="0",label_kubernetes_azure_com_cluster="dev-kube-cluster",label_kubernetes_io_hostname="k8s-worker-22588695-1",label_kubernetes_io_role="agent",namespace="monitoring",node="k8s-worker-22588695-1",pod="kube-prometheus-exporter-kube-state-854f846569-8lnk2",service="kube-prometheus-exporter-kube-state"}   1
  kube_node_labels{endpoint="kube-state-metrics",instance="10.240.0.187:8080",job="kube-prometheus-exporter-kube-state",label_agentpool="worker",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_instance_type="Standard_B2s",label_beta_kubernetes_io_os="linux",label_failure_domain_beta_kubernetes_io_region="southcentralus",label_failure_domain_beta_kubernetes_io_zone="1",label_kubernetes_azure_com_cluster="dev-kube-cluster",label_kubernetes_io_hostname="k8s-worker-22588695-0",label_kubernetes_io_role="agent",namespace="monitoring",node="k8s-worker-22588695-0",pod="kube-prometheus-exporter-kube-state-854f846569-8lnk2",service="kube-prometheus-exporter-kube-state"}   1
  </code></pre>