title: microk8s containerd - failed to reserve sandbox name
body: <p>Have microk8s running on two nodes. Recently it got into a state where the master node fails to go into <code>Ready</code> status because <code>microk8s.daemon-containerd</code> service fails to start. This started happening after trying to get <code>cert-manager</code> configuration running in the k8s cluster.</p>
  <p>As far as I can see <code>cert-manager-webhook</code> pod is running on second node okay.</p>
  <p>I have tried <code>microk8s stop</code>/<code>microk8s start</code>. I have even tried <code>microk8s reset</code> at this point but containerd always shows same error.</p>
  <p>Outputs:</p>
  <pre><code>$ kubectl get node
  NAME        STATUS     ROLES    AGE   VERSION
  pi-k8s-00   NotReady   &lt;none&gt;   77d   v1.18.6-1+b4f4cb0b7fe3c1
  pi-k8s-01   Ready      &lt;none&gt;   77d   v1.19.2-34+37bbd8cebecb60
  </code></pre>
  <pre><code>$ kubectl get pod -n cert-manager
  NAME                                       READY   STATUS    RESTARTS   AGE
  cert-manager-676b755d5f-6bjxv              1/1     Running   0          12m
  cert-manager-cainjector-795f67b984-tsmw9   1/1     Running   3          12m
  cert-manager-webhook-86c4dcd4b5-bgrmb      1/1     Running   0          12m
  </code></pre>
  <pre><code>$ sudo journalctl -u snap.microk8s.daemon-containerd
  ...
  Oct 17 10:42:33 pi-k8s-00 microk8s.daemon-containerd[44363]: time=&quot;2020-10-17T10:42:33.848409047Z&quot; level=fatal msg=&quot;Failed to run CRI service&quot; error=&quot;failed to recover state: failed to reserve sandbox name \&quot;cert-manager-webhook&gt;
  Oct 17 10:42:33 pi-k8s-00 systemd[1]: snap.microk8s.daemon-containerd.service: Main process exited, code=exited, status=1/FAILURE
  Oct 17 10:42:33 pi-k8s-00 systemd[1]: snap.microk8s.daemon-containerd.service: Failed with result 'exit-code'.
  Oct 17 10:42:34 pi-k8s-00 systemd[1]: snap.microk8s.daemon-containerd.service: Scheduled restart job, restart counter is at 5.
  Oct 17 10:42:34 pi-k8s-00 systemd[1]: Stopped Service for snap application microk8s.daemon-containerd.
  Oct 17 10:42:34 pi-k8s-00 systemd[1]: snap.microk8s.daemon-containerd.service: Start request repeated too quickly.
  Oct 17 10:42:34 pi-k8s-00 systemd[1]: snap.microk8s.daemon-containerd.service: Failed with result 'exit-code'.
  Oct 17 10:42:34 pi-k8s-00 systemd[1]: Failed to start Service for snap application microk8s.daemon-containerd.
  </code></pre>
  <pre><code>$ uname -a
  Linux pi-k8s-00 5.4.0-1021-raspi #24-Ubuntu SMP PREEMPT Mon Oct 5 09:59:23 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
  </code></pre>
  <p>How can I get the master node back in a good running/ready state?</p>
  <p>--- UPDATE ---</p>
  <p>Output:</p>
  <pre><code>$ less /var/snap/microk8s/current/inspection-report/snap.microk8s.daemon-containerd/journal.log
  Oct 18 14:48:03 pi-k8s-00 microk8s.daemon-containerd[239043]: time=&quot;2020-10-18T14:48:03.936439781Z&quot; level=fatal msg=&quot;Failed to run CRI service&quot; error=&quot;failed to recover state: failed to reserve sandbox name \&quot;cert-manager-webhook-64b9b4fdfd-9d6tm_cert-manager_81fb08ac-7e87-42bd-9123-b0b8b098fe50_3\&quot;: name \&quot;cert-manager-webhook-64b9b4fdfd-9d6tm_cert-manager_81fb08ac-7e87-42bd-9123-b0b8b098fe50_3\&quot; is reserved for \&quot;149b0aa92e3eb042f87353ead44a7247e756c8071f804bfbec3b781a5565e52c\&quot;&quot;
  </code></pre>
  <p>This last log shows that the sandbox name is reserved for a given id.</p>
  <p>What id would that be? And where do I go and what should one do to free things up?</p>
  <p>Looking through comments in <a href="https://github.com/containerd/cri/issues/1014" rel="nofollow noreferrer">'failed to reserve sandbox name' error after hard reboot #1014</a> I tried:</p>
  <pre><code>$ sudo ctr -n=k8s.io containers info 149b0aa92e3eb042f87353ead44a7247e756c8071f804bfbec3b781a5565e52c
  ctr: container &quot;149b0aa92e3eb042f87353ead44a7247e756c8071f804bfbec3b781a5565e52c&quot; in namespace &quot;k8s.io&quot;: not found
  </code></pre>
  <p>But as can be seen from the output no container with that id exists?</p>