title: Courier Fetch: shards failed
body: <p>Why do I get these warnings after adding more data to my elasticsearch?
  And the warnings are different every time I browse the dashboard.</p>
  
  <p>"Courier Fetch: 30 of 60 shards failed."</p>
  
  <p><img src="https://i.stack.imgur.com/PMFjq.png" alt="Example 1"></p>
  
  <p><img src="https://i.stack.imgur.com/II5Ly.png" alt="Example 2"></p>
  
  <p>More details:</p>
  
  <p>It's a sole node on a CentOS 7.1</p>
  
  <p><strong>/etc/elasticsearch/elasticsearch.yml</strong></p>
  
  <pre><code>index.number_of_shards: 3
  index.number_of_replicas: 1
  
  bootstrap.mlockall: true
  
  threadpool.bulk.queue_size: 1000
  indices.fielddata.cache.size: 50%
  threadpool.index.queue_size: 400
  index.refresh_interval: 30s
  
  index.number_of_shards: 5
  index.number_of_replicas: 1
  </code></pre>
  
  <p><strong>/usr/share/elasticsearch/bin/elasticsearch.in.sh</strong></p>
  
  <pre><code>ES_HEAP_SIZE=3G
  
  #I use this Garbage Collector instead of the default one.
  
  JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
  </code></pre>
  
  <p><strong>cluster status</strong></p>
  
  <pre><code>{
    "cluster_name" : "my_cluster",
    "status" : "yellow",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 61,
    "active_shards" : 61,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 61
  }
  </code></pre>
  
  <p><strong>cluster details</strong></p>
  
  <pre><code>{
    "cluster_name" : "my_cluster",
    "nodes" : {
      "some weird number" : {
        "name" : "ES 1",
        "transport_address" : "inet[localhost/127.0.0.1:9300]",
        "host" : "some host",
        "ip" : "150.244.58.112",
        "version" : "1.4.4",
        "build" : "c88f77f",
        "http_address" : "inet[localhost/127.0.0.1:9200]",
        "process" : {
          "refresh_interval_in_millis" : 1000,
          "id" : 7854,
          "max_file_descriptors" : 65535,
          "mlockall" : false
        }
      }
    }
  }
  </code></pre>
  
  <p>I'm curious about the <strong>"mlockall" : false</strong> because on the yml I did write <strong>bootstrap.mlockall: true</strong></p>
  
  <p><strong>logs</strong></p>
  
  <p>lots of lines like:</p>
  
  <pre><code>org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@a9a34f5
  </code></pre>