title: How Do I Properly Set --kubelet-certificate-authority apiserver parameter?
body: <p>I am using <strong>KubeSpray</strong> to provision a two node cluster on AWS. By default, the <code>--kubelet-certificate-authority</code> parameter is not used. However, I would like to set it.</p>
  <p>I do not know the correct setting for <code>--kubelet-certificate-authority</code>. When I set it to <code>/etc/kubernetes/pki/ca.crt</code> I see messages like the following in my logs:</p>
  <pre><code>TLS handshake error from 10.245.207.223:59278: remote error: tls: unknown certificate authority
  </code></pre>
  <p>In order to create an isolated environment for testing, I SSH'ed to my controller node to run this command which runs apiserver on the non-standard port of <code>4667</code>. I copied these values directly from <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code>. You'll need to adjust the value to match your own cluster. I purposely am running the container in interactive mode so that I can see the log messages.</p>
  <pre class="lang-sh prettyprint-override"><code>sudo docker run \
    --name api-server-playground \
    -it \
    --rm \
    --network host \
    --volume /etc/kubernetes:/etc/kubernetes:ro \
    --volume /etc/pki:/etc/pki:ro \
    --volume /etc/ssl:/etc/ssl/:ro \
    k8s.gcr.io/kube-apiserver:v1.18.9 \
    kube-apiserver \
      --advertise-address=10.245.207.223 \
      --allow-privileged=true \
      --anonymous-auth=True \
      --apiserver-count=1 \
      --authorization-mode=Node,RBAC \
      --bind-address=0.0.0.0 \
      --client-ca-file=/etc/kubernetes/ssl/ca.crt \
      --cloud-config=/etc/kubernetes/cloud_config \
      --cloud-provider=aws \
      --enable-admission-plugins=NodeRestriction \
      --enable-aggregator-routing=False \
      --enable-bootstrap-token-auth=true \
      --endpoint-reconciler-type=lease \
      --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
      --etcd-certfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal.pem \
      --etcd-keyfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal-key.pem \
      --etcd-servers=https://10.245.207.119:2379 \
      --event-ttl=1h0m0s \
      --insecure-port=0 \
      --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt \
      --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key \
      --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP \
      --profiling=False \
      --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt \
      --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key \
      --request-timeout=1m0s \
      --requestheader-allowed-names=front-proxy-client \
      --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt \
      --requestheader-extra-headers-prefix=X-Remote-Extra- \
      --requestheader-group-headers=X-Remote-Group \
      --requestheader-username-headers=X-Remote-User \
      --secure-port=6447 \
      --service-account-key-file=/etc/kubernetes/ssl/sa.pub \
      --service-cluster-ip-range=10.233.0.0/18 \
      --service-node-port-range=30000-32767 \
      --storage-backend=etcd3 \
      --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt \
      --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
  </code></pre>
  <p>Now it's possible to SSH into the controller again and use <code>curl</code> to interact with the custom apiserver container.</p>
  <ul>
  <li>Set the APISERVER variable to point to the controller node. I'm using the value of the <code>advertise-address</code> parameter above.</li>
  </ul>
  <pre><code>APISERVER=https://10.245.207.223:6447
  </code></pre>
  <ul>
  <li>Get a token.</li>
  </ul>
  <pre class="lang-sh prettyprint-override"><code>TOKEN=$(kubectl get secrets -o jsonpath=&quot;{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}&quot;|base64 --decode); echo &quot;TOKEN=$TOKEN&quot;
  </code></pre>
  <ul>
  <li>Make an api request. This request will fail because &quot;Peer's Certificate issuer is not recognized.&quot;.</li>
  </ul>
  <pre class="lang-sh prettyprint-override"><code>curl --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
  </code></pre>
  <p>An error message will appear in the docker log. It will look like this:</p>
  <pre><code>I0921 14:39:07.662368       1 log.go:172] http: TLS handshake error 
  from 10.245.207.223:59278: remote error: tls: unknown certificate authority
  </code></pre>
  <ul>
  <li>Now use the <code>-k</code> curl parameter to bypass the recognition issue.</li>
  </ul>
  <pre><code>curl -k --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
  {
    &quot;kind&quot;: &quot;APIVersions&quot;,
    &quot;versions&quot;: [
      &quot;v1&quot;
    ],
    &quot;serverAddressByClientCIDRs&quot;: [
      {
        &quot;clientCIDR&quot;: &quot;0.0.0.0/0&quot;,
        &quot;serverAddress&quot;: &quot;10.245.207.223:6447&quot;
      }
    ]
  }
  </code></pre>
  <p>You'll see the request works correctly. However, I don't want to use the <code>-k</code> parameter. So I tried to use the certificate authority from the apiserver.</p>
  <ul>
  <li>Get the certificate authority from the apiserver.</li>
  </ul>
  <pre class="lang-sh prettyprint-override"><code>echo | \
      openssl s_client -connect $APISERVER 2&gt;/dev/null | \
      openssl x509 -text | \
      sed -n &quot;/BEGIN CERTIFICATE/,/END CERTIFICATE/p&quot; \
      &gt; apiserver.ca.crt
  </code></pre>
  <ul>
  <li>Use the certificate authority file for the api request.</li>
  </ul>
  <pre class="lang-sh prettyprint-override"><code>curl --cacert apiserver.ca.crt --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
  </code></pre>
  <p><strong>UPDATE</strong></p>
  <p>Following a thought prompted by Wiktor's response, I added /etc/kubernetes/ssl/ca.crt as the certificate authority. And used that file in my <code>curl</code> command.</p>
  <pre class="lang-sh prettyprint-override"><code>curl --cacert /etc/kubernetes/ssl/ca.crt --header &quot;Authorization: Bearer $TOKEN&quot; -X GET $APISERVER/api
  </code></pre>
  <p>This worked.</p>