title: kubernetes dashboard error : 'Metric client health check failed: the server could not find the requested resource (get services heapster).'
body: <p>I'm new in kubernetes world, so forgive me if i'm writing mistake. I'm trying to deploy kubernetes dashboard </p>
  
  <p>My cluster is containing three masters and 3 workers drained and not schedulable in order to install dashboard to masters nodes :</p>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl get nodes
  
  NAME            STATUS                     ROLES    AGE    VERSION
  pp-tmp-test20   Ready                      master   2d2h   v1.15.2
  pp-tmp-test21   Ready                      master   37h    v1.15.2
  pp-tmp-test22   Ready                      master   37h    v1.15.2
  pp-tmp-test23   Ready,SchedulingDisabled   worker   36h    v1.15.2
  pp-tmp-test24   Ready,SchedulingDisabled   worker   36h    v1.15.2
  pp-tmp-test25   Ready,SchedulingDisabled   worker   36h    v1.15.2
  
  </code></pre>
  
  <p>I'm trying to deploy kubernetes dashboard via this url : </p>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
  </code></pre>
  
  <ul>
  <li><p>After this, a pod <code>kubernetes-dashboard-5698d5bc9-ql6q8</code> is scheduled on my master node <code>pp-tmp-test20/172.31.68.220</code></p></li>
  <li><p>the pod</p></li>
  </ul>
  
  <pre><code>kube-system   kubernetes-dashboard-5698d5bc9-ql6q8  /1     Running   1          7m11s   10.244.0.7      pp-tmp-test20   &lt;none&gt;        &lt;none&gt;
  </code></pre>
  
  <ul>
  <li>the pod's logs</li>
  </ul>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl logs kubernetes-dashboard-5698d5bc9-ql6q8 -n kube-system
  
  2019/08/14 10:14:57 Starting overwatch
  2019/08/14 10:14:57 Using in-cluster config to connect to apiserver
  2019/08/14 10:14:57 Using service account token for csrf signing
  2019/08/14 10:14:58 Successful initial request to the apiserver, version: v1.15.2
  2019/08/14 10:14:58 Generating JWE encryption key
  2019/08/14 10:14:58 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
  2019/08/14 10:14:58 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
  2019/08/14 10:14:59 Initializing JWE encryption key from synchronized object
  2019/08/14 10:14:59 Creating in-cluster Heapster client
  2019/08/14 10:14:59 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  2019/08/14 10:14:59 Auto-generating certificates
  2019/08/14 10:14:59 Successfully created certificates
  2019/08/14 10:14:59 Serving securely on HTTPS port: 8443
  2019/08/14 10:15:29 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  2019/08/14 10:15:59 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  </code></pre>
  
  <ul>
  <li>the describe of the pod</li>
  </ul>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl describe pob kubernetes-dashboard-5698d5bc9-ql6q8 -n kube-system
  
  Name:           kubernetes-dashboard-5698d5bc9-ql6q8
  Namespace:      kube-system
  Priority:       0
  Node:           pp-tmp-test20/172.31.68.220
  Start Time:     Wed, 14 Aug 2019 16:58:39 +0200
  Labels:         k8s-app=kubernetes-dashboard
                  pod-template-hash=5698d5bc9
  Annotations:    &lt;none&gt;
  Status:         Running
  IP:             10.244.0.7
  Controlled By:  ReplicaSet/kubernetes-dashboard-5698d5bc9
  Containers:
    kubernetes-dashboard:
      Container ID:  docker://40edddf7a9102d15e3b22f4bc6f08b3a07a19e4841f09360daefbce0486baf0e
      Image:         k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
      Image ID:      docker-pullable://k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747
      Port:          8443/TCP
      Host Port:     0/TCP
      Args:
        --auto-generate-certificates
      State:          Running
        Started:      Wed, 14 Aug 2019 16:58:43 +0200
      Last State:     Terminated
        Reason:       Error
        Exit Code:    1
        Started:      Wed, 14 Aug 2019 16:58:41 +0200
        Finished:     Wed, 14 Aug 2019 16:58:42 +0200
      Ready:          True
      Restart Count:  1
      Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
      Environment:    &lt;none&gt;
      Mounts:
        /certs from kubernetes-dashboard-certs (rw)
        /tmp from tmp-volume (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-ptw78 (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             True
    ContainersReady   True
    PodScheduled      True
  Volumes:
    kubernetes-dashboard-certs:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  kubernetes-dashboard-certs
      Optional:    false
    tmp-volume:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:
      SizeLimit:  &lt;unset&gt;
    kubernetes-dashboard-token-ptw78:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  kubernetes-dashboard-token-ptw78
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  dashboard=true
  Tolerations:     node-role.kubernetes.io/master:NoSchedule
                   node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
  Events:
    Type    Reason     Age                    From                                                     Message
    ----    ------     ----                   ----                                                     -------
    Normal  Scheduled  2m41s                  default-scheduler                                        Successfully assigned kube-system/kubernetes-dashboard-5698d5bc9-ql6q8 to pp-tmp-test20.tec.prj.in.phm.education.gouv.fr
    Normal  Pulled     2m38s (x2 over 2m40s)  kubelet, pp-tmp-test20  Container image "k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1" already present on machine
    Normal  Created    2m37s (x2 over 2m39s)  kubelet, pp-tmp-test20  Created container kubernetes-dashboard
    Normal  Started    2m37s (x2 over 2m39s)  kubelet, pp-tmp-test20  Started container kubernetes-dashboard
  </code></pre>
  
  <ul>
  <li>the describe of the dashboard service</li>
  </ul>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl describe svc/kubernetes-dashboard -n kube-system
  
  Name:              kubernetes-dashboard
  Namespace:         kube-system
  Labels:            k8s-app=kubernetes-dashboard
  Annotations:       &lt;none&gt;
  Selector:          k8s-app=kubernetes-dashboard
  Type:              ClusterIP
  IP:                10.110.236.88
  Port:              &lt;unset&gt;  443/TCP
  TargetPort:        8443/TCP
  Endpoints:         10.244.0.7:8443
  Session Affinity:  None
  Events:            &lt;none&gt;
  </code></pre>
  
  <ul>
  <li>the docker ps on my master running the pod</li>
  </ul>
  
  <pre><code>[root@pp-tmp-test20 ~]# Docker ps
  
  CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
  40edddf7a910        f9aed6605b81           "/dashboard --inse..."   7 minutes ago       Up 7 minutes                            k8s_kubernetes-dashboard_kubernetes-dashboard-5698d5bc9-ql6q8_kube-system_f785d4bd-2e67-4daa-9f6c-19f98582fccb_1
  e7f3820f1cf2        k8s.gcr.io/pause:3.1   "/pause"                 7 minutes ago       Up 7 minutes                            k8s_POD_kubernetes-dashboard-5698d5bc9-ql6q8_kube-system_f785d4bd-2e67-4daa-9f6c-19f98582fccb_0
  
  [root@pp-tmp-test20 ~]# docker logs 40edddf7a910
  2019/08/14 14:58:43 Starting overwatch
  2019/08/14 14:58:43 Using in-cluster config to connect to apiserver
  2019/08/14 14:58:43 Using service account token for csrf signing
  2019/08/14 14:58:44 Successful initial request to the apiserver, version: v1.15.2
  2019/08/14 14:58:44 Generating JWE encryption key
  2019/08/14 14:58:44 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
  2019/08/14 14:58:44 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
  2019/08/14 14:58:44 Initializing JWE encryption key from synchronized object
  2019/08/14 14:58:44 Creating in-cluster Heapster client
  2019/08/14 14:58:44 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  2019/08/14 14:58:44 Auto-generating certificates
  2019/08/14 14:58:44 Successfully created certificates
  2019/08/14 14:58:44 Serving securely on HTTPS port: 8443
  2019/08/14 14:59:14 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  2019/08/14 14:59:44 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  2019/08/14 15:00:14 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  </code></pre>
  
  <p>1/ On my master I start the proxy</p>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl proxy
  Starting to serve on 127.0.0.1:8001
  </code></pre>
  
  <p>2/ I launch firefox with x11 redirect from my master and  hit this url</p>
  
  <pre><code>http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login
  </code></pre>
  
  <p>this is the error message I get in the browser</p>
  
  <pre><code>Error: 'dial tcp 10.244.0.7:8443: connect: no route to host'
  Trying to reach: 'https://10.244.0.7:8443/'
  </code></pre>
  
  <p>In the same time i got these errors from the console where  I launched the proxy</p>
  
  <pre><code>I0814 16:10:05.836114   20240 log.go:172] http: proxy error: context canceled
  I0814 16:10:06.198701   20240 log.go:172] http: proxy error: context canceled
  I0814 16:13:21.708190   20240 log.go:172] http: proxy error: unexpected EOF
  I0814 16:13:21.708229   20240 log.go:172] http: proxy error: unexpected EOF
  I0814 16:13:21.708270   20240 log.go:172] http: proxy error: unexpected EOF
  I0814 16:13:39.335483   20240 log.go:172] http: proxy error: context canceled
  I0814 16:13:39.716360   20240 log.go:172] http: proxy error: context canceled
  </code></pre>
  
  <p>but after refresh n times (randomly) the browser I'm able to reach the login interface to enter the token (created before)</p>
  
  <p><a href="https://i.stack.imgur.com/ArNRb.png" rel="noreferrer">Dashboard_login</a></p>
  
  <p>But... the same error occur again</p>
  
  <p><a href="https://i.stack.imgur.com/1ANks.png" rel="noreferrer">Dashboard_login_error</a></p>
  
  <p>After hit n times the 'sign in' button I'm able to get the dashboard.. for few seconds. </p>
  
  <p><a href="https://i.stack.imgur.com/iSd2x.png" rel="noreferrer">dashboard_interface_1</a></p>
  
  <p><a href="https://i.stack.imgur.com/vNXHM.png" rel="noreferrer">dashboard_interface_2</a></p>
  
  <p>after that the dashboard start to produce the same errors when I'm am exploring the interface:</p>
  
  <p><a href="https://i.stack.imgur.com/Rdl08.png" rel="noreferrer">dashboard_interface_error_1</a></p>
  
  <p><a href="https://i.stack.imgur.com/qEOEP.png" rel="noreferrer">dashboard_interface_error_2</a></p>
  
  <p>I looked the pod logs, we can see some trafic :</p>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl logs kubernetes-dashboard-5698d5bc9-ql6q8  -n kube-system
  2019/08/14 14:16:56 Getting list of all services in the cluster
  2019/08/14 14:16:56 [2019-08-14T14:16:56Z] Outcoming response to 10.244.0.1:56140 with 200 status code
  2019/08/14 14:17:01 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.244.0.1:56140: {}
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Outcoming response to 10.244.0.1:56140 with 200 status code
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.244.0.1:56140: {}
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Outcoming response to 10.244.0.1:56140 with 200 status code
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.244.0.1:56140: { contents hidden }
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Outcoming response to 10.244.0.1:56140 with 200 status code
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Incoming HTTP/2.0 GET /api/v1/settings/global/cani request from 10.244.0.1:56140: {}
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Outcoming response to 10.244.0.1:56140 with 200 status code
  2019/08/14 14:17:22 [2019-08-14T14:17:22Z] Incoming HTTP/2.0 GET /api/v1/settings/global request from 10.244.0.1:56140: {}
  2019/08/14 14:17:22 Cannot find settings config map: configmaps "kubernetes-dashboard-settings" not found
  </code></pre>
  
  <p>and again the pod logs</p>
  
  <pre><code>[root@pp-tmp-test20 ~]# kubectl logs kubernetes-dashboard-5698d5bc9-ql6q8  -n kube-system
  Error from server: Get https://172.31.68.220:10250/containerLogs/kube-system/kubernetes-dashboard-5698d5bc9-ql6q8/kubernetes-dashboard: Forbidden
  </code></pre>
  
  <p>What I'm doing wrong ? Could you please tell me some investigating way ?</p>
  
  <p>EDIT :</p>
  
  <p>my service account that I used </p>
  
  <pre><code># cat dashboard-adminuser.yaml
  
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: admin-user
    namespace: kube-system
  
  # cat dashboard-adminuser-ClusterRoleBinding.yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: admin-user
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
  subjects:
  - kind: ServiceAccount
    name: admin-user
    namespace: kube-system
  </code></pre>