title: Kubernetes RBAC - forbidden attempt to grant extra privileges
body: <p>I'm using <code>Kubernetes</code> v1.8.14 on custom built <code>CoreOS</code> cluster:</p>
  
  <pre><code>$ kubectl version --short 
  Client Version: v1.10.5
  Server Version: v1.8.14+coreos.0
  </code></pre>
  
  <p>When trying to create the following <code>ClusterRole</code>:</p>
  
  <pre><code>$ cat ClusterRole.yml 
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:coredns
  rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    - pods
    - namespaces
    verbs:
    - list
    - watch
  </code></pre>
  
  <p>I get the following error:</p>
  
  <pre><code>$ kubectl create -f ClusterRole.yml 
  Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&amp;{cluster-admin  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
  </code></pre>
  
  <p>As far as I can tell I'm connecting as <code>cluster-admin</code>, therefore should have sufficient permissions for what I'm trying to achieve. Below are relevant <code>cluster-admin</code> config:</p>
  
  <pre><code>$ cat ~/.kube/config
  apiVersion: v1
  kind: Config
  current-context: dev
  preferences:
    colors: true
  
  clusters:
  - cluster:
      certificate-authority: cluster-ca.pem
      server: https://k8s.loc:4430
    name: dev
  
  contexts:
  - context:
      cluster: dev
      namespace: kube-system
      user: cluster-admin
    name: dev
  
  users:
  - name: cluster-admin
    user:
      client-certificate: cluster.pem
      client-key: cluster-key.pem
  
  
  $ kubectl get clusterrole cluster-admin -o yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: 2018-07-30T14:44:44Z
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: cluster-admin
    resourceVersion: "1164791"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
    uid: 196ffecc-9407-11e8-bd67-525400ac0b7d
  rules:
  - apiGroups:
    - '*'
    resources:
    - '*'
    verbs:
    - '*'
  - nonResourceURLs:
    - '*'
    verbs:
    - '*'
  
  
  $ kubectl get clusterrolebinding cluster-admin -o yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: 2018-07-30T14:44:45Z
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: cluster-admin
    resourceVersion: "1164832"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
    uid: 19e516a6-9407-11e8-bd67-525400ac0b7d
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
  subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: system:masters
  
  
  $ kubectl get serviceaccount cluster-admin -o yaml
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    creationTimestamp: 2018-07-30T13:32:13Z
    name: cluster-admin
    namespace: kube-system
    resourceVersion: "1158783"
    selfLink: /api/v1/namespaces/kube-system/serviceaccounts/cluster-admin
    uid: f809e079-93fc-11e8-8b85-525400546bcd
  secrets:
  - name: cluster-admin-token-t7s4c
  </code></pre>
  
  <p>I understand this is RBAC problem, but have no idea how further debug this.</p>
  
  <h3>Edit-1.</h3>
  
  <p>I tried the suggested, no joy unfortunately...</p>
  
  <pre><code>$ kubectl get clusterrolebinding cluster-admin-binding -o yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    creationTimestamp: 2018-07-31T09:21:34Z
    name: cluster-admin-binding
    resourceVersion: "1252260"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin-binding
    uid: 1e1c0647-94a3-11e8-9f9b-525400ac0b7d
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
  subjects:
  - kind: ServiceAccount
    name: cluster-admin
    namespace: default
  
  
  $ kubectl describe secret $(kubectl get secret | awk '/cluster-admin/{print $1}')
  Name:         cluster-admin-token-t7s4c
  Namespace:    kube-system
  Labels:       &lt;none&gt;
  Annotations:  kubernetes.io/service-account.name=cluster-admin
                kubernetes.io/service-account.uid=f809e079-93fc-11e8-8b85-525400546bcd
  
  Type:  kubernetes.io/service-account-token
  
  Data
  ====
  ca.crt:     1785 bytes
  namespace:  11 bytes
  token:      eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
  </code></pre>
  
  <p>Then amended <code>~/.kube/config</code>:</p>
  
  <pre><code>$ cat ~/.kube/config
  apiVersion: v1
  kind: Config
  current-context: dev
  preferences:
    colors: true
  
  clusters:
  - cluster:
      certificate-authority: cluster-ca.pem
      server: https://k8s.loc:4430
    name: dev
  
  contexts:
  - context:
      cluster: dev
      namespace: kube-system
      user: cluster-admin-2
    name: dev
  
  users:
  - name: cluster-admin
    user:
      client-certificate: cluster.pem
      client-key: cluster-key.pem
  - name: cluster-admin-2
    user:
      token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXRva2VuLXQ3czRjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmODA5ZTA3OS05M2ZjLTExZTgtOGI4NS01MjU0MDA1NDZiY2QiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbiJ9.rC1x9Or8GArkhC3P0s-l_Pc0e6TEUwfbJtXAN2w-cOaRUCNCo6r4WxXKu32ngOg86TXqCho2wBopXtbJ2CparIb7FWDXzri6O6LPFzHWNzZo3b-TON2yxHMWECGjpbbqjDgkPKDEldkdxJehDBJM_GFAaUdNyYpFFsP1_t3vVIsf2DpCjeMlOBSprYRcEKmDiE6ehF4RSn1JqB7TVpvTZ_WAL4CRZoTJtZDVoF75AtKIADtVXTxVv_ewznDCKUWDupg5Jk44QSMJ0YiG30QYYM699L5iFLirzD5pj0EEPAoMeOqSjdp7KvDzIM2tBiu8YYl6Fj7pG_53WjZrvlSk5pgPLS-jPKOkixFM9FfB2eeuP0eWwLO5wvU5s--a2ekkEhaqHTXgigeedudDA_5JVIJTS0m6V9gcbE4_kYRpU7_QD_0TR68C5yxUL83KfOzj6A_S6idOZ-p7Ni6ffE_KlGqqcgUUR2MTakJgimjn0gYHNaIqmHIu4YhrT-jffP0-5ZClbI5srj-aB4YqGtCH9w5_KBYD4S2y6Rjv4kO00nZyvi0jAHlZ6el63TQPWYkjyPL2moF_P8xcPeoDrF6o8bXDzFqlXLqda2Nqyo8LMhLxjpe_wFeGuwzIUxwwtH1RUR6BISRUf86041aa2PeJMqjTfaU0u_SvO-yHMGxZt3o
  </code></pre>
  
  <p>And then tried to apply the same <code>ClusterRole</code>, which rendered the same error:</p>
  
  <pre><code>$ kubectl apply -f ClusterRole.yml 
  Error from server (Forbidden): error when creating "ClusterRole.yml": clusterroles.rbac.authorization.k8s.io "system:coredns" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}] user=&amp;{system:serviceaccount:kube-system:cluster-admin f809e079-93fc-11e8-8b85-525400546bcd [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
  </code></pre>
  
  <p>Below are the flags which I use to start <code>apiserver</code>:</p>
  
  <pre><code>  containers:
      - name: kube-apiserver
        image: quay.io/coreos/hyperkube:${K8S_VER}
        command:
          - /hyperkube
          - apiserver
          - --bind-address=0.0.0.0
          - --etcd-servers=${ETCD_ENDPOINTS}
          - --allow-privileged=true
          - --service-cluster-ip-range=${SERVICE_IP_RANGE}
          - --secure-port=443
          - --advertise-address=${ADVERTISE_IP}
          - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
          - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
          - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --client-ca-file=/etc/kubernetes/ssl/ca.pem
          - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --runtime-config=extensions/v1beta1/networkpolicies=true
          - --anonymous-auth=false
          - --authorization-mode=AlwaysAllow,RBAC,Node
  </code></pre>
  
  <p>And here are the scripts, which I use to generate my <code>tls</code> certs:</p>
  
  <p><strong>root ca</strong>:</p>
  
  <pre><code>openssl genrsa -out ca-key.pem 4096
  openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=kube-ca"
  </code></pre>
  
  <p><strong>apiserver</strong>:</p>
  
  <pre><code>cat &gt; openssl.cnf &lt;&lt;EOF
  [req]
  req_extensions = v3_req
  distinguished_name = req_distinguished_name
  
  [req_distinguished_name]
  
  [v3_req]
  basicConstraints = CA:FALSE
  keyUsage = nonRepudiation, digitalSignature, keyEncipherment
  subjectAltName = @alt_names
  
  [alt_names]
  DNS.1 = kubernetes
  DNS.2 = kubernetes.default
  DNS.3 = kubernetes.default.svc
  DNS.4 = kubernetes.default.svc.cluster.local
  DNS.5 = ${MASTER_LB_DNS}
  IP.1 = ${K8S_SERVICE_IP}
  IP.2 = ${MASTER_HOST}
  EOF
  
  openssl genrsa -out apiserver-key.pem 4096
  openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
  openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 3650 -extensions v3_req -extfile openssl.cnf
  </code></pre>
  
  <p><strong>cluster-admin</strong>:</p>
  
  <pre><code>openssl genrsa -out cluster-admin-key.pem 4096
  openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin"
  openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
  </code></pre>
  
  <p>I hope this gives you more insight, what's wrong with my system.</p>
  
  <h3>Edit-2.</h3>
  
  <p>I noted a slight discrepancy between my system configuration and what @MarcinRomaszewicz suggested, thus the <em>namespace</em> of the <em>cluster-admin</em> <code>ServiceAccount</code>, in my case it is in the <code>kube-system</code> as opposed to the <code>default</code> <em>namespace</em>:</p>
  
  <pre><code>$ kubectl delete clusterrolebinding cluster-admin-binding 
  clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" deleted
  
  $ kubectl create clusterrolebinding cluster-admin-binding \
   --clusterrole=cluster-admin --serviceaccount=kube-system:cluster-admin
  clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-binding" created
  
  $ kubectl apply -f ClusterRole.yml 
  clusterrole.rbac.authorization.k8s.io "system:coredns" created
  </code></pre>
  
  <p>However it still doesn't work with my certificates... </p>
  
  <h3>Edit-3.</h3>
  
  <p>As suggested in the comments, in order for <code>apiserver</code> to recognize the user as the <code>cluster-admin</code>, Subject line in the certificate of that user must contain the following items: <code>Subject: CN = cluster-admin, O = system:masters</code>. One way to generate such a certificate is as follows:</p>
  
  <pre><code>openssl genrsa -out cluster-admin-key.pem 4096
  openssl req -new -key cluster-admin-key.pem -out cluster-admin.csr -subj "/CN=cluster-admin/O=system:masters"
  openssl x509 -req -in cluster-admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cluster-admin.pem -days 3650
  </code></pre>