title: The ECS servive with aws_vpc cannot start due to ENI issues
body: <p>I have got one service. I have ECS cluster with 2 instances of t3.small.</p>
  
  <p>I cannot start the ECS task. I have ECS task with 2 containers(NGINX and PHP-FPM). NGINX exposes port 80 and PHP-FPM exposes ports 9000, 9001, 9002.</p>
  
  <p>Error I can see:</p>
  
  <pre><code>dev-cluster/ecs-agents i-12345678901234567 2019-09-15T13:20:48Z [ERROR] Task engine [arn:aws:ecs:us-east-1:123456789012:task/ea1d6e4b-ff9f-4e0a-b77a-1698721faa5c]: unable to configure pause container namespace: cni setup: invoke bridge plugin failed: bridge ipam ADD: failed to execute plugin: ecs-ipam: getIPV4AddressFromDB commands: failed to get available ip from the db: getAvailableIP ipstore: failed to find available ip addresses in the subnet
  </code></pre>
  
  <p>ECS agent: 1.29. </p>
  
  <p>Do you know How Can I figure out what is wrong?</p>
  
  <p>Here is logs snippet: <a href="https://pastebin.com/my620Kip" rel="nofollow noreferrer">https://pastebin.com/my620Kip</a></p>
  
  <p>Task definition: <a href="https://pastebin.com/C5khX9Zy" rel="nofollow noreferrer">https://pastebin.com/C5khX9Zy</a></p>
  
  <h2>UPDATE: My observations</h2>
  
  <p>Edited because my post below was deleted...</p>
  
  <p>I recreated cluster, then the problem disappears. <br>
  Then I removed the application image from the ECR and I was seeing an error in AWS web console: </p>
  
  <p><code>CannotPullContainerError: Error response from daemon: manifest for 123456789123.dkr.ecr.us-east-1.amazonaws.com/application123:development-716b4e55dd3235f6548d645af9e463e744d3785f not found</code></p>
  
  <p>Then I waited a few hours until the original issue happened again. <br>
  Then I restarted instance manually with <code>systemctl reboot</code> and the problem disappeared again only for restarted instance.</p>
  
  <p>This issue appears when On the cluster is hundred(s) awsvpc task which cannot start.</p>
  
  <p>I think this is a bug in ECS agent. And When We are trying to create too many containers with requires ENI it is trying to use all free IPs in the subnet. (255) I think after restart/recreate EC2 instance some cache is cleared and the problem is solved.</p>
  
  <p>Here is similar solution I found today: <a href="https://github.com/aws/amazon-ecs-cni-plugins/issues/93#issuecomment-502099642" rel="nofollow noreferrer">https://github.com/aws/amazon-ecs-cni-plugins/issues/93#issuecomment-502099642</a></p>
  
  <p>What do you think about it?</p>
  
  <p>I am opened for suggestions.</p>