title: High disk IO in Kubernetes nodes
body: <p>Deployed 3 node K8s cluster on CentOS 7 VMs(vsphear) with <code>kubeadm</code>.</p>
  
  <pre><code>Master - 4GB    
  Minion1 - 6GB RAM
  Minion2 - 6GB RAM
  K8s Version - v1.11.2
  </code></pre>
  
  <p>Before starting kubelet daemon on each node, I did <code>swapoff -a</code> to disable swap memory which is recommended for K8s</p>
  
  <p>Deployed <a href="https://www.spinnaker.io/" rel="nofollow noreferrer">Spinnaker</a> on K8s cluster and I see all spinnaker components are spring boot java applications. </p>
  
  <p>I noticed that after some time, I'm losing ssh connection(able to ping) and nodes become <code>NotReady</code> state. After force restart of nodes, I started <code>kubelet</code>(after <code>swapoff -a</code>) while looking I/O states, <code>iotop</code>. Then I saw that java application doing lot of Disk Reads but not much CPU and memory. Also noticed that, sometimes nodes alive for long times(Hours). Verified same scenario few times in vsphear UI graph(sudden increase of disk reads).</p>
  
  <p>Now my question is, too many disk reads/IO because of <code>swapoff -a</code>? or java application doesn't like disabling swap memory?
  I see some similar issue in <a href="https://github.com/kubernetes/kubernetes/issues/47928" rel="nofollow noreferrer">k8s github repo</a>. </p>
  
  <p><a href="https://i.stack.imgur.com/JvIR2.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JvIR2.jpg" alt="enter image description here"></a></p>
  
  <p>Any help would be appreciated. </p>