title: What causes pods to be slow in kubernetes?
body: <p>Certain pods on my cluster are extremely slow in almost all aspects. Startup time, network, i/o.</p>
  
  <p>I have minimized the application code in these containers and it seems to have no effect, these are basically minimal containers running a simple webapi with a health check endpoint.</p>
  
  <p>I'm wondering someone can help me figure out what's wrong or debug this.</p>
  
  <p>When I say slow in all aspects I mean a couple things</p>
  
  <ol>
  <li><p>Very slow startup.  I actually have to change my readiness probe initial delay to near 5 minutes.</p></li>
  <li><p>Inside the container running any command is slow.  Running an <code>apt-get update</code> takes near 5 minutes, even if the container has been running for hours.</p></li>
  <li><p>Any connections to an RDS database will timeout for at least the first 10 minutes the pod is running, after that it's hit or miss, sometimes normal speed, sometimes we'll start getting connection timeouts again (mainly if the pod hasn't been used/requested for awhile).</p></li>
  </ol>
  
  <p>On nearly identical pods with same base image the container will start in less than a couple seconds, and running an <code>apt-get update</code> will take maybe 3 seconds.  I cannot for the life of me see what is different between the pods that causes some to be 'good pods', and others to be 'bad pods'.</p>
  
  <p>Running any of these images locally they will start in no time (less than a second or so).</p>
  
  <h2>My Environment</h2>
  
  <p><strong>Cluster (AWS)</strong> </p>
  
  <ul>
  <li>1 c4.large master</li>
  <li>3 c4.xlarge nodes</li>
  <li>~10-20 pods per node</li>
  <li>provisioned with kops using 'standard' settings (I haven't done anything tricky)</li>
  </ul>
  
  <p><strong>Things I've checked/tried</strong></p>
  
  <ul>
  <li><p>too many pods</p>
  
  <p>My first thought was maybe i'm running too many pods. I've launched up brand new nodes for this (c4.xlarge) and had this pod the only pod running in the cluster, issue still seen.</p></li>
  <li><p>node resources</p>
  
  <p>Checking every node level metric I could nothing looks out of the ordinary (also tried on several brand new pretty high powered nodes)  </p></li>
  <li><p>Deployment/Pod Metrics</p>
  
  <p>I'm happy to show whatever metric anyone can think of here, nothing looks out of the norm.  I have Prometheus running and have looked into every metric I could think to check.  I can't see difference between a 'good' running pod and a 'bad' one.</p></li>
  <li><p>cluster itself</p>
  
  <p>I actually have 2 clusters, both provisioned with kops, this is seen on both clusters (though not always the same applications, which is odd).</p></li>
  </ul>
  
  <p>Any help here is appreciated</p>