title: no space left on device, which is unexpected. MountVolume.SetUp failed for volume in kubernetes cluster
body: <p>Today my kubernetes cluster(v1.15.2) node disk full and cause pods give this tips:</p>
  
  <pre><code>Update plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.261578065: no space left on device, which is unexpected.
  MountVolume.SetUp failed for volume "default-token-xnrwt" : mkdir /opt/k8s/k8s/kubelet/pods/67eaa71b-adf4-4365-a1c7-42045d5e9426: no space left on device
  </code></pre>
  
  <p>I login into server and find the disk usage is 100%, so I remove some log file and release 10GB + disk space, but now it seems pod is not recovery automaticlly and still have this error tips:</p>
  
  <p><a href="https://i.stack.imgur.com/bTlcL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bTlcL.png" alt="enter image description here"></a></p>
  
  <p>what should I do to fix this problem? I am try to restart all pods, and all pods works fine. But finally I found the error tips message still give me tips no space left and did not disappear automaticlly. I check the node status and find the node has no disk pressure. How to make the error tips disappear?</p>