title: ElasticSearch Sorting performance problems
body: <p><strong>Details</strong></p>
  
  <p>4-node cluster: 28GB RAM, 4 Cores, 2 512GB SSDs on each
  2 - master/data
  2- data-only</p>
  
  <p>Using ElasticSearch 1.5.2 (could this be a bug that was since fixed?)</p>
  
  <p><strong>Summary</strong></p>
  
  <p>I am trying to debug some issues we are seeing with our cluster responding slow to queries. It has not always been so slow, so we suspected it was just a scale issue (sitting right under 2 billion documents). </p>
  
  <p>However, after some poking around with the index I noticed that our sorting was taking up 99% of the response time.  Here are 2 example queries I have been running. The first is with sorting and the second is without.</p>
  
  <p><strong>With sorting</strong></p>
  
  <p>This call takes anywhere from 45 seconds to 60+ seconds (our nginx server timeout is set to 60s and it times out a lot) and only returns 644 documents. This means it appears to take 45+ seconds to sort a mere 644 items</p>
  
  <pre><code>POST /backup/entity/_search?routing=3f9d219f-6eef-4880-ad5f-0494492c2fd6
  {
    "from": 0,
    "size": 100,
    "sort": [
      {
        "entityName.raw": {
          "order": "asc"
        }
      }
    ],
    "highlight": {
      "pre_tags": [
        "&lt;span class=\"highlighter\"&gt;"
      ],
      "post_tags": [
        "&lt;/span&gt;"
      ],
      "fields": {
        "entityName": {},
        "friendlyUrl": {},
        "sentBy": {}
      },
      "require_field_match": true
    },
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "serviceId": "3f9d219f-6eef-4880-ad5f-0494492c2fd6"
            }
          },
          {
            "term": {
              "subscriptionType": 1
            }
          },
          {
            "terms": {
              "entityType": [
                2
              ]
            }
          }
        ],
        "should": []
      }
    }
  }
  </code></pre>
  
  <p><strong>Without sorting</strong></p>
  
  <p>This call on the other hand returns in 100-200ms. Same 644 documents, just not sorted.</p>
  
  <pre><code>POST /backup/entity/_search?routing=3f9d219f-6eef-4880-ad5f-0494492c2fd6
  {
    "from": 0,
    "size": 100,
    "highlight": {
      "pre_tags": [
        "&lt;span class=\"highlighter\"&gt;"
      ],
      "post_tags": [
        "&lt;/span&gt;"
      ],
      "fields": {
        "entityName": {},
        "friendlyUrl": {},
        "sentBy": {}
      },
      "require_field_match": true
    },
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "serviceId": "3f9d219f-6eef-4880-ad5f-0494492c2fd6"
            }
          },
          {
            "term": {
              "subscriptionType": 1
            }
          },
          {
            "terms": {
              "entityType": [
                2
              ]
            }
          }
        ],
        "should": []
      }
    }
  }
  </code></pre>
  
  <p>It seems hard to believe that it is truly taking this long to run a sorting algorithm on these documents.  Even if I inspect all 4 of our nodes at the same time, the CPU and JVM do not ever exceed 60% usage at any given time, so its not like some machine is computing the sort at 100% CPU for all 45+ seconds.  Something else must be going on here.</p>