title: Unable to fetch Vault Token for Pod Service Account
body: <p>I am using Vault CSI Driver on Charmed Kubernetes v1.19 where I'm trying to retrieve secrets from Vault for a pod running in a separate namespace (<code>webapp</code>) with its own service account (<code>webapp-sa</code>) following the steps in the <a href="https://www.hashicorp.com/blog/retrieve-hashicorp-vault-secrets-with-kubernetes-csi" rel="nofollow noreferrer">blog</a>.</p>
  <p>As I have been able to understand so far, the Pod is trying authenticate to the Kubernetes API, so that later it can generate a Vault token to access the secret from Vault.</p>
  <pre><code>$ kubectl get po webapp
  NAME     READY   STATUS              RESTARTS   AGE
  webapp   0/1     ContainerCreating   0          22m
  </code></pre>
  <p>It appears to me there's some issue authenticating with the Kubernetes API.</p>
  <p>The pod remains stuck in the Container Creating state with the message - <code>failed to create a service account token for requesting pod</code></p>
  <pre><code>Events:
    Type     Reason       Age                       From               Message
    ----     ------       ----                      ----               -------
    Normal   Scheduled    35m                       default-scheduler  Successfully assigned webapp/webapp to host-03
  
    Warning  FailedMount  4m38s (x23 over 35m)      kubelet            MountVolume.SetUp failed for volume &quot;secrets-store-inline&quot; : rpc error: code = Unknown desc = failed to mount secrets store objects for pod webapp/webapp, err: rpc error: code = Unknown desc = error making mount request: **failed to create a service account token for requesting pod** {webapp xxxx webapp webapp-sa}: the server could not find the requested resource
  </code></pre>
  <p>I can get the vault token using cli in the pod namespace:</p>
  <pre><code>$ vault write auth/kubernetes/login role=database jwt=$SA_JWT_TOKEN
  Key                                       Value
  ---                                       -----
  token                                     &lt;snipped&gt;
  </code></pre>
  <p>I do get the vault token using the API as well:</p>
  <pre><code>$ curl --request POST --data @payload.json https://127.0.0.1:8200/v1/auth/kubernetes/login
      {
         &quot;request_id&quot;:&quot;1234&quot;,
         &lt;snipped&gt;
         &quot;auth&quot;:{
            &quot;client_token&quot;:&quot;XyZ&quot;,
            &quot;accessor&quot;:&quot;abc&quot;,
            &quot;policies&quot;:[
               &quot;default&quot;,
               &quot;webapp-policy&quot;
            ],
            &quot;token_policies&quot;:[
               &quot;default&quot;,
               &quot;webapp-policy&quot;
            ],
            &quot;metadata&quot;:{
               &quot;role&quot;:&quot;database&quot;,
               &quot;service_account_name&quot;:&quot;webapp-sa&quot;,
               &quot;service_account_namespace&quot;:&quot;webapp&quot;,
               &quot;service_account_secret_name&quot;:&quot;webapp-sa-token-abcd&quot;,
               &quot;service_account_uid&quot;:&quot;123456&quot;
            },
            &lt;snipped&gt;    
         }
      }
  </code></pre>
  <p>Reference: <a href="https://www.vaultproject.io/docs/auth/kubernetes" rel="nofollow noreferrer">https://www.vaultproject.io/docs/auth/kubernetes</a></p>
  <p>As per the vault documentation, I've configured Vault with the Token Reviewer SA as follows:</p>
  <pre><code>$ cat vault-auth-service-account.yaml
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: role-token-review-binding
    namespace: vault
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:auth-delegator
  subjects:
    - kind: ServiceAccount
      name: vault-auth
      namespace: vault
  </code></pre>
  <p>Vault is configured with JWT from the Token Reviewer SA as follows:</p>
  <pre><code>$ vault write auth/kubernetes/config \
      token_reviewer_jwt=&quot;&lt; TOKEN Reviewer service account JWT&gt;&quot; \
      kubernetes_host=&quot;https://$KUBERNETES_PORT_443_TCP_ADDR:443&quot; 
      kubernetes_ca_cert=@ca.crt
  </code></pre>
  <p>I have defined a Vault Role to allow the <code>webapp-sa</code> access to the secret:</p>
  <pre><code>$ vault write auth/kubernetes/role/database \
      bound_service_account_names=webapp-sa \
      bound_service_account_namespaces=webapp \
      policies=webapp-policy \
      ttl=72h
  Success! Data written to: auth/kubernetes/role/database
  </code></pre>
  <p>The <code>webapp-sa</code> is allowed access to the secret as per the Vault Policy defined as follows:</p>
  <pre><code>$ vault policy write webapp-policy - &lt;&lt;EOF
  &gt; path &quot;secret/data/db-pass&quot; {
  &gt;    capabilities = [&quot;read&quot;]
  &gt; }
  &gt; EOF
  Success! Uploaded policy: webapp-policy
  </code></pre>
  <p>Pod and it's SA is defined as follows:</p>
  <pre><code>$ cat webapp-sa-and-pod.yaml
  kind: ServiceAccount
  apiVersion: v1
  metadata:
    name: webapp-sa
  ---
  kind: Pod
  apiVersion: v1
  metadata:
    name: webapp
  spec:
    serviceAccountName: webapp-sa
    containers:
    - image: registry/jweissig/app:0.0.1
      name: webapp
      volumeMounts:
      - name: secrets-store-inline
        mountPath: &quot;/mnt/secrets-store&quot;
        readOnly: true
    volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            providerName: vault
            secretProviderClass: &quot;vault-database&quot;
  </code></pre>
  <ul>
  <li>Does anyone have any clue as to why the Pod won't authenticate with
  the Kubernetes API?</li>
  <li>Do I have to enable flags on the kube-apiserver for Token Review API
  to work?</li>
  <li>Is it enabled by default on Charmed Kubernetes v1.19?</li>
  </ul>
  <p>Would be grateful for any help.</p>
  <p>Regards,
  Sana</p>