<p dir="auto">While working on a webhook auth server, I've found that the <code class="notranslate">kube-apiserver</code> logs don't include useful info when the API server fails to parse the response from the auth server.  For example, the following response looks okay (the problem was the missing Content-Type header), but then returns a 403 to the client with no explanation.  This was captured with <code class="notranslate">--v=8</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0927 01:59:54.425034       1 round_trippers.go:296] POST https://127.0.0.1:4000/
I0927 01:59:54.425043       1 round_trippers.go:303] Request Headers:
I0927 01:59:54.425048       1 round_trippers.go:306]     Accept: application/json, */*
I0927 01:59:54.425053       1 round_trippers.go:306]     Content-Type: application/json
I0927 01:59:54.439002       1 round_trippers.go:321] Response Status: 200 OK  in 13 milliseconds
I0927 01:59:54.439020       1 round_trippers.go:324] Response Headers:
I0927 01:59:54.439026       1 round_trippers.go:327]     Server: WEBrick/1.3.1 (Ruby/2.3.1/2016-04-26) OpenSSL/1.0.2h
I0927 01:59:54.439031       1 round_trippers.go:327]     Date: Tue, 27 Sep 2016 01:59:54 GMT
I0927 01:59:54.439036       1 round_trippers.go:327]     Content-Length: 211
I0927 01:59:54.439040       1 round_trippers.go:327]     Connection: Keep-Alive
I0927 01:59:54.477173       1 request.go:891] Response Body: {&quot;kind&quot;:&quot;SubjectAccessReview&quot;,&quot;apiVersion&quot;:&quot;authorization.k8s.io/v1beta1&quot;,&quot;spec&quot;:{&quot;nonResourceAttributes&quot;:{&quot;path&quot;:&quot;/api&quot;,&quot;verb&quot;:&quot;get&quot;},&quot;user&quot;:&quot;kevin.burk@k8sdev&quot;},&quot;status&quot;:{&quot;allowed&quot;:true,&quot;reason&quot;:&quot;Allow all.&quot;}}
I0927 01:59:54.477443       1 handlers.go:165] GET /api: (53.098111ms) 403 [[kubectl/v1.3.0+2831379 (darwin/amd64) kubernetes/2831379] 10.52.9.58:51513]
I0927 01:59:54.500444       1 request.go:558] Request Body: {&quot;kind&quot;:&quot;SubjectAccessReview&quot;,&quot;apiVersion&quot;:&quot;authorization.k8s.io/v1beta1&quot;,&quot;spec&quot;:{&quot;nonResourceAttributes&quot;:{&quot;path&quot;:&quot;/apis&quot;,&quot;verb&quot;:&quot;get&quot;},&quot;user&quot;:&quot;kevin.burk@k8sdev&quot;},&quot;status&quot;:{&quot;allowed&quot;:false}}"><pre class="notranslate"><code class="notranslate">I0927 01:59:54.425034       1 round_trippers.go:296] POST https://127.0.0.1:4000/
I0927 01:59:54.425043       1 round_trippers.go:303] Request Headers:
I0927 01:59:54.425048       1 round_trippers.go:306]     Accept: application/json, */*
I0927 01:59:54.425053       1 round_trippers.go:306]     Content-Type: application/json
I0927 01:59:54.439002       1 round_trippers.go:321] Response Status: 200 OK  in 13 milliseconds
I0927 01:59:54.439020       1 round_trippers.go:324] Response Headers:
I0927 01:59:54.439026       1 round_trippers.go:327]     Server: WEBrick/1.3.1 (Ruby/2.3.1/2016-04-26) OpenSSL/1.0.2h
I0927 01:59:54.439031       1 round_trippers.go:327]     Date: Tue, 27 Sep 2016 01:59:54 GMT
I0927 01:59:54.439036       1 round_trippers.go:327]     Content-Length: 211
I0927 01:59:54.439040       1 round_trippers.go:327]     Connection: Keep-Alive
I0927 01:59:54.477173       1 request.go:891] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","spec":{"nonResourceAttributes":{"path":"/api","verb":"get"},"user":"kevin.burk@k8sdev"},"status":{"allowed":true,"reason":"Allow all."}}
I0927 01:59:54.477443       1 handlers.go:165] GET /api: (53.098111ms) 403 [[kubectl/v1.3.0+2831379 (darwin/amd64) kubernetes/2831379] 10.52.9.58:51513]
I0927 01:59:54.500444       1 request.go:558] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","spec":{"nonResourceAttributes":{"path":"/apis","verb":"get"},"user":"kevin.burk@k8sdev"},"status":{"allowed":false}}
</code></pre></div>
<p dir="auto">Better logs would be very nice to have!</p>
<p dir="auto">My best guess is that it's happening <a href="https://github.com/kubernetes/kubernetes/blob/5f9447a91a8ed9286026d9c112aee541167603c3/plugin/pkg/auth/authorizer/webhook/webhook.go#L178">here</a>, but I'm not familiar enough with the codebase to say for sure.</p>
<p dir="auto">And my system info (OSX local; CoreOS server):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.0+2831379&quot;, GitCommit:&quot;283137936a498aed572ee22af6774b6fb6e9fd94&quot;, GitTreeState:&quot;not a git tree&quot;, BuildDate:&quot;2016-07-05T15:40:13Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.5+coreos.1&quot;, GitCommit:&quot;4fc64a21f8a593d97b75296acd1e979391518b28&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-22T17:15:28Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0+2831379", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"not a git tree", BuildDate:"2016-07-05T15:40:13Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5+coreos.1", GitCommit:"4fc64a21f8a593d97b75296acd1e979391518b28", GitTreeState:"clean", BuildDate:"2016-08-22T17:15:28Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto">Today I set up an apiserver with <code class="notranslate">--authorization-mode=Webhook,ABAC</code> and (accidentally) pointed the webhook at an endpoint that did not return a proper <code class="notranslate">SubjectAccessReview</code> object. The apiserver seemed to send the request payload to the server, ignore the incorrect response then move on to the ABAC authorizer. I would expect to see an error in the apiserver logs about this, but I did not see one.</p>
<p dir="auto">There should be an error logged each time the Webhook endpoint does not return a proper JSON response.</p>