title: Duplicate metrics with multiple instances of kube-state-metrics
body: <p><strong>Problem</strong>:</p>
  
  <p>Duplicate data when querying from prometheus for metrics from <em>kube-state-metrics</em>.</p>
  
  <p>Sample query and result with 3 instances of <em>kube-state-metrics</em> running:</p>
  
  <p>Query:</p>
  
  <pre><code>kube_pod_container_resource_requests_cpu_cores{namespace="ns-dummy"}
  </code></pre>
  
  <p>Metrics</p>
  
  <pre><code>kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.142:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-34-25.ec2.internal",pod="app1-appname-6bd9d8d978-gfk7f",service="prom-kube-state-metrics"}
  1
  kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.142:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-35-22.ec2.internal",pod="app2-appname-ccbdfc7c8-g9x6s",service="prom-kube-state-metrics"}
  1
  kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.17:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-34-25.ec2.internal",pod="app1-appname-6bd9d8d978-gfk7f",service="prom-kube-state-metrics"}
  1
  kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.17:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-35-22.ec2.internal",pod="app2-appname-ccbdfc7c8-g9x6s",service="prom-kube-state-metrics"}
  1
  kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.37.171:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-34-25.ec2.internal",pod="app1-appname-6bd9d8d978-gfk7f",service="prom-kube-state-metrics"}
  1
  kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.37.171:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-35-22.ec2.internal",pod="app2-appname-ccbdfc7c8-g9x6s",service="prom-kube-state-metrics"}
  </code></pre>
  
  <p><strong>Observation</strong>:</p>
  
  <p>Every metric is coming up Nx when N pods are running for <em>kube-state-metrics</em>. If it's a single pod running, we get the correct info.</p>
  
  <p><strong>Possible solutions</strong>:</p>
  
  <ol>
  <li>Scale down to single instance of kube-state-metrics. (Reduced availability is a concern)</li>
  <li>Enable sharding. (Solves duplication problem, still less available)</li>
  </ol>
  
  <p>According to the <a href="https://github.com/kubernetes/kube-state-metrics#horizontal-scaling-sharding" rel="nofollow noreferrer">docs</a>, for horizontal scaling we have to pass sharding arguments to the pods.</p>
  
  <p>Shards are zero indexed. So we have to pass the index and total number of shards for each pod.</p>
  
  <p>We are using <a href="https://github.com/helm/charts/tree/master/stable/kube-state-metrics" rel="nofollow noreferrer">Helm chart</a> and it is deployed as a deployment. </p>
  
  <p><strong>Questions</strong>:</p>
  
  <ol>
  <li>How can we pass different arguments to different pods in this scenario, if its possible?</li>
  <li>Should we be worried about availability of the <em>kube-state-metrics</em> considering the self-healing nature of k8s workloads? </li>
  <li>When should we really scale it to multiple instances and how?</li>
  </ol>