title: Prometheus Adapter Custom Metrics for Libvirt in a K8S Cluster
body: <p>I have a K8S cluster which is also managing VMs via virtlet.  This K8S cluster is running K8S v1.13.2, with prometheus and the prometheus-adapter, and a custom-metrics server.  I have written a custom metrics exporter for libvirtd which pulls in VM metrics and have configured prometheus to scrape that exporter for those VM metrics -- this is working and working well.  </p>
  
  <p>What I need to do next, is to have the prometheus-adapter push those metrics into K8S.  Nothing I have done is working.  Funny thing is, I can see the metrics in prometheus, but I am unable to present them to the custom metrics API.</p>
  
  <p><strong>Example metric visible in prometheus:</strong></p>
  
  <pre><code>libvirt_cpu_stats_cpu_time_nanosecs{app="prometheus-lex",domain="virtlet-c91822c8-5e82-beta-deflect",instance="192.168.2.32:9177",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_pod_name="prometheus-lex-866694b884-9z8v6",name="prometheus-lex",pod_template_hash="866694b884"}
  </code></pre>
  
  <p><strong>Prometheus Adapter configuration for this metric:</strong></p>
  
  <pre><code>    - seriesQuery: 'libvirt_cpu_stats_cpu_time_nanosecs{job="kubernetes-pods", app="prometheus-lex"}'
        seriesFilters: []
        resource:
          overrides:
            kubernetes_pod_name:
              resource: pod
            kubernetes_namespace:
              resource: namespace
        name:
          matches: libvirt_cpu_stats_cpu_time_nanosecs
          as: libvirt_cpu_stats_cpu_time_rate
        metricsQuery: rate(libvirt_cpu_stats_cpu_time_nanosecs{job="kubernetes-pods", app="prometheus-lex", &lt;&lt;.LabelMatchers&gt;&gt;}[5m])
  </code></pre>
  
  <p>When I query the custom metrics API, I do not see what I am looking for:</p>
  
  <pre><code>kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|grep libvirt
  
  returns nothing
  </code></pre>
  
  <p>Additionally, I can see the prometheus-adapter is able to query the series from prometheus.   So I know that side of the adapter is working.   I am just trying to figure out why it's not presenting them to the custom metrics server.</p>
  
  <p>From the prometheus-adapter</p>
  
  <pre><code>I0220 19:12:58.442937       1 api.go:74] GET http://prometheus-server.default.svc.cluster.local:80/api/v1/series?match%5B%5D=libvirt_cpu_stats_cpu_time_nanosecs%7Bkubernetes_namespace%21%3D%22%22%2Ckubernetes_pod_name%21%3D%22%22%7D&amp;start=1550689948.392 200 OK
  </code></pre>
  
  <p>Any ideas what I am missing here?</p>
  
  <p>Update::</p>
  
  <p>I have also tried the following new configuration, and it's still not working.  </p>
  
  <pre><code>- seriesQuery: 'libvirt_cpu_stats_cpu_time_nanosecs{kubernetes_namespace!="",kubernetes_pod_name!=""}'
    seriesFilters: []
    resource:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_pod_name: {resource: "pod"}
    name:
      matches: 'libvirt_cpu_stats_cpu_time_nanosecs'
      as: 'libvirt_cpu_stats_cpu_time_rate'
    metricsQuery: 'sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[2m])) by (&lt;&lt;.GroupBy&gt;&gt;)'
  </code></pre>