title: Timed out waiting for cluster initialization, Auto upgrade of nodes fail / or Run with error
body: <p>I have few clusters with 3 node in each cluster's nodepools in my GCP project and has auto-upgrade and repair enabled. </p>
  
  <p>The auto upgrade began approximately 3 days ago and is still running for the GKE version: 1.12.10-gke.17.</p>
  
  <p>Now as my clusters are opted in for the auto-upgrade and auto repair, few clusters are getting upgraded without issues and few others are running update/upgrade with issues</p>
  
  <p>ON my first cluster, few of my pods went unschedulable and the suggested possible actions by GCP is to </p>
  
  <ul>
  <li>Enable Autoscaling in one or more node pools that have autoscaling disabled.</li>
  <li>Increase size of one or more node pools manually.</li>
  </ul>
  
  <p>when I run 
   "gcloud container clusters describe "clustername" "zone" " </p>
  
  <p>I get details of the cluster. however, under the nodepools section</p>
  
  <pre><code> status: RUNNING_WITH_ERROR
    statusMessage: 'asia-south1-a: Timed out waiting for cluster initialization; cluster
      API may not be available: k8sclient: 7 - 404 status code returned. Requested resource
      not found.'
    version: 1.12.10-gke.17
  </code></pre>
  
  <p>NOTE:</p>
  
  <p>I also see that the GCP suggests to </p>
  
  <ul>
  <li>Enable autoscaling in one or more node pools that have autoscaling
  disabled. </li>
  <li>Shrink one or more node pools manually.</li>
  </ul>
  
  <p>because there  is low resource requests. </p>
  
  <p>Please let me know what other logs I can provide to resolve this issue. </p>
  
  <p><a href="https://i.stack.imgur.com/hLwNk.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hLwNk.jpg" alt="Error Description and Activity"></a></p>
  
  <p>UPDATE: </p>
  
  <p>We went through these logs and google support believes that it could be that the kubelet might be failing to submit a Certificate Signing Request (CSR) or that it might have old invalid credentials. To assist on the troubleshooting, might you answer these questions:</p>
  
  <ol>
  <li>sudo journalctl -u kubelet > kubelet.log</li>
  <li>sudo journalctl -u kube-node-installation > kube-node-installation.log</li>
  <li>sudo journalctl -u kube-node-configuration > kube-node-configuration.log</li>
  <li>sudo journalctl -u node-problem-detector > node-problem-detector.log</li>
  <li>sudo journalctl -u docker > docker.log</li>
  <li>sudo journalctl -u cloud-init > cloud-init.log</li>
  </ol>
  
  <p>Any node that starts running 1.13.12-gke.13 fails to connect to master. Anything else that's happening to nodes (e.g. recreation) is because they are trying to fix them in a repair loop and doesn't seem to be causing additional problems.</p>