title: Pods stuck in ContainerCreating with "failed to assign an IP address to container"
body: <p>Multiple pods of a 600 pod deployment stuck in <code>ContainerCreating</code> after a rolling update with the message:</p>
  
  <blockquote>
    <p>Failed create pod sandbox: rpc error: code = Unknown desc =
    NetworkPlugin cni failed to set up pod  network: add cmd:
    failed to assign an IP address to container</p>
  </blockquote>
  
  <p>What I have tried:</p>
  
  <ul>
  <li>Upgraded to v1.12 on EKS and CNI 1.5.0.  <a href="https://github.com/aws/amazon-vpc-cni-k8s/issues/123" rel="noreferrer">This issue was closed stating CNI 1.5.0 solved the issue.</a>  It did not for us.  In another thread <a href="https://github.com/aws/amazon-vpc-cni-k8s/issues/69" rel="noreferrer">leaking ENIs was blamed</a> but was also closed due to CNI upgrade.</li>
  <li><a href="https://aws.amazon.com/blogs/opensource/cni-metrics-helper/" rel="noreferrer">Installed cni-metrics-helper</a> and this is a snapshot of the output:</li>
  </ul>
  
  <pre><code>maxIPAddresses, value: 759.000000
  ipamdActionInProgress, value: 1.000000
  addReqCount, value: 16093.000000
  awsAPILatency, value: 564.000000
  delReqCount, value: 32337.000000
  eniMaxAvailable, value: 69.000000
  assignIPAddresses, value: 558.000000
  totalIPAddresses, value: 682.000000
  eniAllocated, value: 69.000000
  </code></pre>
  
  <p>Do the CNI metrics output suggest there's an issue?  Seems like there are enough IPs.</p>
  
  <p>What else can I try to debug?</p>