title: prometheus to monitor eks (aws) node
body: <p>I am trying to setup prometheus to monitor kubernetes node (eks, Version 1.10) with following <code>scrape_configs</code></p>
  
  <pre><code>  - job_name: 'kubernetes-nodes'
      kubernetes_sd_configs:
      - api_servers:
        - 'https://kubernetes.default.svc.cluster.local'
        in_cluster: true
        role: node
  
      tls_config:
        insecure_skip_verify: true
  
      relabel_configs:
      - target_label: __scheme__
        replacement: https
  </code></pre>
  
  <p><strong>node-exporter</strong> :</p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: DaemonSet
  metadata:
    name: node-exporter
    namespace: kube-system
    labels:
      name: node-exporter
  spec:
    template:
      metadata:
        labels:
          name: node-exporter
        annotations:
           prometheus.io/scrape: "true"
           prometheus.io/port: "9100"
      spec:
        hostPID: true
        hostIPC: true
        hostNetwork: true
        containers:
          - ports:
              - containerPort: 9100
                protocol: TCP
            resources:
              requests:
                cpu: 0.15
            securityContext:
              privileged: true
            image: prom/node-exporter:v0.15.2
            args:
              - --path.procfs
              - /host/proc
              - --path.sysfs
              - /host/sys
              - --collector.filesystem.ignored-mount-points
              - '"^/(sys|proc|dev|host|etc)($|/)"'
            name: node-exporter
            volumeMounts:
              - name: dev
                mountPath: /host/dev
              - name: proc
                mountPath: /host/proc
              - name: sys
                mountPath: /host/sys
              - name: rootfs
                mountPath: /rootfs
        volumes:
          - name: proc
            hostPath:
              path: /proc
          - name: dev
            hostPath:
              path: /dev
          - name: sys
            hostPath:
              path: /sys
          - name: rootfs
            hostPath:
              path: /
  </code></pre>
  
  <p>With this config prometheus pod fails with :</p>
  
  <pre><code>level=error ts=2019-05-15T15:18:45.472Z caller=main.go:717 
  err="error loading config from \"/etc/prometheus/prometheus.yml\": 
  couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file 
  /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n  line 33: field api_servers not found in type kubernetes.plain\n  
  line 35: field in_cluster not found in type kubernetes.plain"
  </code></pre>
  
  <p>Update :</p>
  
  <p>Corrected <code>scrape_configs</code> to </p>
  
  <pre><code>  - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: http
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
  
    - job_name: 'kubernetes-nodes'
      scheme: http
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
  </code></pre>
  
  <p>node metric not showing up in the prometheus UI</p>