title: Prometheus - Kubernetes RBAC
body: <p>I upgrade my GKE API server to 1.6, and am in the process of upgrading nodes to 1.6, but ran into a snag...</p>
  
  <p>I've got a prometheus server (version 1.5.2) running in a pod managed by a Kubernetes deployment with a couple of nodes running version 1.5.4 Kubelet, with a single new node running 1.6.</p>
  
  <p>Prometheus can't connect to the new node--it's metrics endpoint is returning 401 Unauthorized.</p>
  
  <p>This seems to be a RBAC issue, but I'm not sure how to proceed. I can't find docs on what roles the Prometheus server needs, or even how to grant them to the server.</p>
  
  <p>From the coreos/prometheus-operator repo I was able to piece together a configuration that I might expect to work:</p>
  
  <pre><code>apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: prometheus
  ---
  
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRole
  metadata:
    name: prometheus
  rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
    - configmaps
    verbs: ["get"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
  ---
  
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: prometheus
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: prometheus
  subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: default
  ---
  
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: prometheus
    namespace: default
  secrets:
  - name: prometheus-token-xxxxx
  
  ---
  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    labels:
      app: prometheus-prometheus
      component: server
      release: prometheus
    name: prometheus-server
    namespace: default
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: prometheus-prometheus
        component: server
        release: prometheus
    strategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 1
      type: RollingUpdate
    template:
      metadata:
        labels:
          app: prometheus-prometheus
          component: server
          release: prometheus
      spec:
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        serviceAccount: prometheus
        serviceAccountName: prometheus
        ...
  </code></pre>
  
  <p>But Prometheus is still getting 401s.</p>
  
  <p>UPDATE: seems like a kubernetes authentication issue as Jordan said. See new, more focused question here; <a href="https://serverfault.com/questions/843751/kubernetes-node-metrics-endpoint-returns-401">https://serverfault.com/questions/843751/kubernetes-node-metrics-endpoint-returns-401</a></p>