title: Docker service stuck in New state (Swarm)
body: <p>I'm facing a strange issue with my Docker Swarm (a cluster of 3 managers and 5 workers). I have many running services right now and when I approach around 100 services (and with replications more than 110 services), the new services I want to run won't start.</p>
  
  <p>When I list the service, I have this:</p>
  
  <pre><code>ID            NAME            IMAGE       NODE  DESIRED STATE  CURRENT STATE     ERROR  PORTS
  alam7whfn1xe  service_name.1  some_image        Running        New 22 hours ago
  </code></pre>
  
  <p>You can see <code>CURRENT STATE == New 22 hours ago</code>. If I try to inspect the logs, they're empty. Inspecting the service won't help either (nothing relevant).</p>
  
  <p>If I stop some of my services, the service tagged with <code>New</code> state may start by itself after the first retry. It seems that I reached a limit by any way.</p>
  
  <p>I followed up some documentation on the web and there is nothing clear about this issue. <strong>You'll be welcome if you can point me some links.</strong></p>
  
  <p>Today, in my opinion, I suspect that the networks I created in the Swarm (<code>--driver=overlay</code>) have an insufficient IP range and can't give enough IP to containers. These networks are <code>/24</code> subnets. Is there any way to "flush" the IP reservations in order to re-initialize the networks without recreation Docker networks?</p>
  
  <p>After investigation, there are two types of services that can reach this <code>New</code> state and they're on 2 same networks.</p>
  
  <p>The result of <code>docker network inspect</code>:</p>
  
  <pre><code>[
      {
          "Name": "network_name",
          "Id": "okbrl5twyheq32ht3zw5l00gs",
          "Created": "0001-01-01T00:00:00Z", &lt;- this is the real date, strange isn't it?
          "Scope": "swarm",
          "Driver": "overlay",
          "EnableIPv6": false,
          "IPAM": {
              "Driver": "default",
              "Options": null,
              "Config": [
                  {
                      "Subnet": "172.16.2.0/24",
                      "Gateway": "172.16.2.1"
                  }
              ]
          },
          "Internal": false,
          "Attachable": true,
           "Ingress": false,
          "ConfigFrom": {
              "Network": ""
          },
          "ConfigOnly": false,
          "Containers": null,
          "Options": {
              "com.docker.network.driver.overlay.vxlanid_list": "4097"
          },
          "Labels": null
      }
  ]
  </code></pre>
  
  <p>Additionnaly, <code>docker version</code>:</p>
  
  <pre><code>Client:
   Version:      17.06.2-ce
   API version:  1.30
   Go version:   go1.8.3
   Git commit:   cec0b72
   Built:        Tue Sep  5 20:00:06 2017
   OS/Arch:      linux/amd64
  
  Server:
   Version:      17.06.2-ce
   API version:  1.30 (minimum version 1.12)
   Go version:   go1.8.3
   Git commit:   cec0b72
   Built:        Tue Sep  5 19:58:57 2017
   OS/Arch:      linux/amd64
   Experimental: false
  </code></pre>
  
  <p><strong>N.B.:</strong> I don't want to update Docker in this moment.</p>
  
  <p><strong>EDIT 1:</strong></p>
  
  <p>I read again the <a href="https://docs.docker.com/engine/reference/commandline/network_create/#overlay-network-limitations" rel="noreferrer">Docker documentation about networks</a> and they are mentionning an opened issue on Moby's Github Project <a href="https://github.com/moby/moby/issues/30820" rel="noreferrer">Swarm Mode at Scale #30820</a>.</p>
  
  <blockquote>
    <h1>Overlay network limitations</h1>
    
    <p>You should create overlay networks with /24 blocks (the default), which limits you to 256 IP addresses, when you create networks using the default VIP-based endpoint-mode. This recommendation addresses <a href="https://github.com/moby/moby/issues/30820" rel="noreferrer">limitations with swarm mode</a>. If you need more than 256 IP addresses, do not increase the IP block size. You can either use dnsrr endpoint mode with an external load balancer, or use multiple smaller overlay networks. See <a href="https://docs.docker.com/engine/swarm/networking/#configure-service-discovery" rel="noreferrer">Configure service discovery</a> for more information about different endpoint modes.</p>
    
    <p>-- <a href="https://docs.docker.com/engine/reference/commandline/network_create/#overlay-network-limitations" rel="noreferrer">https://docs.docker.com/engine/reference/commandline/network_create/#overlay-network-limitations</a></p>
  </blockquote>
  
  <p><strong>EDIT 2:</strong></p>
  
  <p>Based on <a href="https://github.com/moby/moby/issues/30820#issuecomment-386654376" rel="noreferrer">Flavio 'fcrisciani' Crisciani's comment</a> on the issue <a href="https://github.com/moby/moby/issues/30820" rel="noreferrer">Swarm Mode at Scale #30820</a>, I'll try to add the option <code>--endpoint-mode=dnsrr</code> on my services.</p>