title: GKE ESP Failed to fetch service account token
body: <p>I got gRPC service in GKE with 3 pods. Each pod got attached ESP container with rollout_strategy=managed and backend pointing to gRPC.</p>
  
  <p>Sometimes when I hit http endpoint I got</p>
  
  <pre><code>{
   "code": 13,
   "message": "Failed to fetch service account token",
   "details": [
    {
     "@type": "type.googleapis.com/google.rpc.DebugInfo",
     "stackEntries": [],
     "detail": "internal"
    }
   ]
  }
  </code></pre>
  
  <p>Every third request failed. When I deleted every pod, it started to works. I assume it will fail after one hour when the token will be expired.</p>
  
  <p>Deployment ESP container</p>
  
  <pre><code> containers:
        - args:
          - --http_port=80
          - --backend=grpc://127.0.0.1:6000
          - --service=service.endpoints.cluster.cloud.goog
          - --rollout_strategy=managed
          image: gcr.io/endpoints-release/endpoints-runtime:1
          imagePullPolicy: IfNotPresent
          name: esp
          ports:
          - containerPort: 80
            protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
  </code></pre>
  
  <p>In container logs there are errors like this:</p>
  
  <pre><code>2019/07/17 07:51:05[error]10#10: Failed to call https://servicecontrol.googleapis.com/v1/services/service.endpoints.cluster.cloud.goog:report, Error: UNAUTHORIZED: server response status code: 401, Response body: �Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
  [libprotobuf ERROR external/servicecontrol_client_git/src/service_control_client_impl.cc:182] Failed in Report call: Service control request failed with HTTP response code 401
  </code></pre>
  
  <p>I found the same issue here <a href="https://groups.google.com/forum/#!topic/google-cloud-endpoints/0EEY88X8oIY" rel="nofollow noreferrer">https://groups.google.com/forum/#!topic/google-cloud-endpoints/0EEY88X8oIY</a>
  But it's just workaround by providing explicitly service account.</p>
  
  <p>Edit: I tried service account, but there is still problem when pod is trying to refresh token</p>
  
  <pre><code>2019-07-19T11:45:48.031715293Z INFO:Refreshing access_token
   E 
  2019-07-19T11:45:48.071914451Z Traceback (most recent call last):
    File ".bootstrap/_pex/pex.py", line 365, in execute
    File ".bootstrap/_pex/pex.py", line 293, in _wrap_coverage
    File ".bootstrap/_pex/pex.py", line 325, in _wrap_profiling
    File ".bootstrap/_pex/pex.py", line 408, in _execute
    File ".bootstrap/_pex/pex.py", line 466, in execute_entry
    File ".bootstrap/_pex/pex.py", line 471, in execute_module
    File "/usr/lib/python2.7/runpy.py", line 192, in run_module
      fname, loader, pkg_name)
    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
      exec code in run_globals
    File "start_esp/start_esp.py", line 1061, in &lt;module&gt;
    File "start_esp/start_esp.py", line 343, in fetch_service_config
    File "start_esp/fetch_service_config.py", line 151, in make_access_token
    File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 663, in get_access_token
   E 
  2019-07-19T11:45:48.074420362Z     self.refresh(http)
   E 
  2019-07-19T11:45:48.074424955Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 545, in refresh
   E 
  2019-07-19T11:45:48.074430139Z     self._refresh(http)
   E 
  2019-07-19T11:45:48.074434982Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 749, in _refresh
   E 
  2019-07-19T11:45:48.074440214Z     self._do_refresh_request(http)
   E 
  2019-07-19T11:45:48.074444687Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 780, in _do_refresh_request
   E 
  2019-07-19T11:45:48.074450391Z     body=body, headers=headers)
   E 
  2019-07-19T11:45:48.074455280Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/transport.py", line 282, in request
   E 
  2019-07-19T11:45:48.074460859Z     connection_type=connection_type)
   E 
  2019-07-19T11:45:48.074465988Z   File "/root/.pex/install/httplib2-0.13.0-py2-none-any.whl.f7b2ff9b447b5741fa19aab52032ab1c7db17f35/httplib2-0.13.0-py2-none-any.whl/httplib2/__init__.py", line 2135, in request
   E 
  2019-07-19T11:45:48.074482470Z     cachekey,
   E 
  2019-07-19T11:45:48.074487590Z   File "/root/.pex/install/httplib2-0.13.0-py2-none-any.whl.f7b2ff9b447b5741fa19aab52032ab1c7db17f35/httplib2-0.13.0-py2-none-any.whl/httplib2/__init__.py", line 1796, in _request
   E 
  2019-07-19T11:45:48.074492698Z     conn, request_uri, method, body, headers
   E 
  2019-07-19T11:45:48.074497648Z   File "/root/.pex/install/httplib2-0.13.0-py2-none-any.whl.f7b2ff9b447b5741fa19aab52032ab1c7db17f35/httplib2-0.13.0-py2-none-any.whl/httplib2/__init__.py", line 1737, in _conn_request
   E 
  2019-07-19T11:45:48.074502516Z     response = conn.getresponse()
   E 
  2019-07-19T11:45:48.074506952Z   File "/usr/lib/python2.7/httplib.py", line 1108, in getresponse
   E 
  2019-07-19T11:45:48.074511855Z     raise ResponseNotReady()
   E 
  2019-07-19T11:45:48.074516567Z httplib.ResponseNotReady
   E 
  </code></pre>
  
  <p>Afterwards pod is restarted and everything works for 1 hour.</p>