title: Stackdriver API metrics show many 429 errors, while quota is not exceeded
body: <p>Earlier this month we've enabled Stackdriver Monitoring in 3 our projects on GCP. </p>
  
  <p>Recently we've found that Stackdriver API metrics show around 85% of errors:</p>
  
  <p><a href="https://i.stack.imgur.com/hDfEE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hDfEE.png" alt="enter image description here"></a></p>
  
  <p>On graphs, these error codes are 429:</p>
  
  <p><a href="https://i.stack.imgur.com/I37rO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/I37rO.png" alt="enter image description here"></a></p>
  
  <p>I've checked Quotas, everything seems fine:</p>
  
  <p><a href="https://i.stack.imgur.com/Iljwl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Iljwl.png" alt="enter image description here"></a></p>
  
  <p>Next metrics graph tells us what method causing errors:</p>
  
  <p><a href="https://i.stack.imgur.com/cpoMK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cpoMK.png" alt="enter image description here"></a></p>
  
  <p>Using the other graph "Errors by credential" I found out that API requests made by our GKE service account. We have custom service account for GKE instances, and as far as we know it has all required permissions for monitoring:</p>
  
  <ul>
  <li><code>roles/logging.logWriter</code></li>
  <li><code>roles/monitoring.metricWriter</code></li>
  <li><code>roles/stackdriver.resourceMetadata.writer</code> (as noted in <a href="https://github.com/Stackdriver/kubernetes-configs/issues/25#issuecomment-493431063" rel="nofollow noreferrer">this issue</a>)</li>
  </ul>
  
  <p>Also, <code>stackdriver-metadata-agent</code> pods in GKE cluster logs related error every minute:</p>
  
  <pre><code>stackdriver-metadata-agent-cluster-level-d6556b55-2bkbc metadata-agent I0203 15:03:16.911940       1 binarylog.go:265] rpc: flushed binary log to ""
  stackdriver-metadata-agent-cluster-level-d6556b55-2bkbc metadata-agent W0203 15:03:56.495034       1 kubernetes.go:118] Failed to publish resource metadata: rpc error: code = ResourceExhausted desc = Resource has been exhausted (e.g. check quota).
  stackdriver-metadata-agent-cluster-level-d6556b55-2bkbc metadata-agent I0203 15:04:16.912272       1 binarylog.go:265] rpc: flushed binary log to ""
  stackdriver-metadata-agent-cluster-level-d6556b55-2bkbc metadata-agent W0203 15:04:56.657831       1 kubernetes.go:118] Failed to publish resource metadata: rpc error: code = ResourceExhausted desc = Resource has been exhausted (e.g. check quota).
  </code></pre>
  
  <p>Aside from that I haven't found any logs related to the issue yet, and I cannot figure out who does 2 requests per second to Stackdriver API receiving 429 errors.</p>
  
  <p>I should add that everything above is true for all 3 projects.</p>
  
  <p>Can someone suggest how can we solve the issue? </p>
  
  <p>Is this still an excess of the quota? If yes, why request metrics for quotas are ok <code>Quota exceeded errors count</code> contains no data?</p>
  
  <p>Are we missing any permissions on our GKE service account?</p>
  
  <p>What else can be related?</p>
  
  <p>Thanks in advance.</p>