title: Search Performance of Elasticsearch index with large documents (PDF’s)
body: <p>I'm fairly new to Elasticsearch and would like some help with tuning and tips on how to increase the performance of my index.</p>
  
  <p>Currently i have around 4500 documents in the index and a size on disc about 34GB that consists of PDF's with some metadata. The PDF's are indexed using the Mapper Attachment plugin and are from 10MB to 150MB each, some are bigger up to 250MB.</p>
  
  <p>My problem is that search operations take long time, sometimes up to multiple seconds, i'm filtering on up to 0 - 7 fields, ordering on 2 plus a Query String query against the documents text (base64 encoded), title, and some other meta fields. Im also using pagination for all of the results (up to 450 pages of 10 documents each) and highlighting to show what part that was hit. I guess this is part of my problem but i can't really get away from it.</p>
  
  <p>The server got 8GB of RAM and ElasticSearch has the ES_HEAP_SIZE set to 2GB, i'm guessing this is the other part of my problem and that the bottleneck is here, right? Don't know how much i can increase it either since it's running a web server as well. Server can of course be upgraded.</p>
  
  <p>I haven't changed any settings regarding shards from the default values. It's currently hosted in Azure but i don't know right now if i got SSD's or spinning disc's</p>
  
  <p>I'm not surprised that it does but i would like to understand why </p>
  
  <p>What can i do to improve my performance?</p>
  
  <p>As requested this is a sample query.</p>
  
  <pre><code>{
  "index": "publications",
  "body": {
      "query": {
          "filtered": {
              "filter": {
                  "bool": {
                      "must": [
                          [{
                              "range": {
                                  "pubdate": {
                                      "gte": "2010-01-01T00:00:00",
                                      "lte": "2016-01-01T00:00:00"
                                  }
                              }
                          }, {
                              "term": {
                                  "author": "daniel"
                              }
                          }, {
                              "term": {
                                  "title": "rock"
                              }
                          }]
                      ]
                  }
              },
              "query": {
                  "query_string": {
                      "fields": [
                          ["title", "author", "files.data", "articleId"]
                      ],
                      "query": "Hard"
                  }
              }
          }
      },
      "highlight": {
          "fields": {
              "title": {},
              "author": {},
              "articleId": {},
              "exact_articleId": {},
              "files.data": {}
          }
      },
      "sort": {
          "date": {
              "order": "desc"
          },
          "_score": {
              "order": "desc"
          }
      },
      "size": 10,
      "from": 0,
      "fields": ["id", "title", "pubdate", "orderable", "articleId", "author", "languages", "types", "exact_title", "files.file", "files.name", "bibas_date"],
      "_source": ["files.file", "files.name"]
  }
  </code></pre>
  
  <p>}</p>
  
  <p>And this is my mapping </p>
  
  <pre><code>{
  "publications": {
      "aliases": {},
      "mappings": {
          "publication": {
              "properties": {
                  "articleId": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_standard_whitespace"
                  },
                  "author": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_standard",
                      "fields": {
                          "raw": {
                              "type": "string",
                              "index": "not_analyzed"
                          }
                      }
                  },
                  "date": {
                      "type": "date",
                      "store": true,
                      "format": "dateOptionalTime"
                  },
                  "created": {
                      "type": "date",
                      "store": true,
                      "format": "dateOptionalTime"
                  },
                  "exact_articleId": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_keyword"
                  },
                  "exact_title": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_keyword"
                  },
                  "files": {
                      "properties": {
                          "data": {
                              "type": "attachment",
                              "path": "full",
                              "fields": {
                                  "data": {
                                      "type": "string",
                                      "store": true,
                                      "term_vector": "with_positions_offsets",
                                      "analyzer": "analyzer_standard_whitespace"
                                  },
                                  "author": {
                                      "type": "string"
                                  },
                                  "title": {
                                      "type": "string"
                                  },
                                  "name": {
                                      "type": "string"
                                  },
                                  "date": {
                                      "type": "date",
                                      "format": "dateOptionalTime"
                                  },
                                  "keywords": {
                                      "type": "string"
                                  },
                                  "content_type": {
                                      "type": "string"
                                  },
                                  "content_length": {
                                      "type": "integer"
                                  },
                                  "language": {
                                      "type": "string"
                                  }
                              }
                          },
                          "description": {
                              "type": "string",
                              "store": true
                          },
                          "file": {
                              "type": "string",
                              "store": true,
                              "analyzer": "analyzer_keyword"
                          },
                          "name": {
                              "type": "string",
                              "store": true,
                              "analyzer": "analyzer_keyword"
                          }
                      }
                  },
                  "id": {
                      "type": "integer",
                      "store": true
                  },
                  "keywords": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_keyword"
                  },
                  "languages": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_keyword"
                  },
                  "orderable": {
                      "type": "boolean",
                      "store": true
                  },
                  "pubdate": {
                      "type": "date",
                      "store": true,
                      "format": "dateOptionalTime"
                  },
                  "title": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_standard",
                      "fields": {
                          "raw": {
                              "type": "string",
                              "index": "not_analyzed"
                          }
                      }
                  },
                  "types": {
                      "type": "string",
                      "store": true,
                      "analyzer": "analyzer_keyword"
                  },
                  "updated": {
                      "type": "date",
                      "store": true,
                      "format": "dateOptionalTime"
                  }
              }
          }
      },
      "settings": {
          "index": {
              "creation_date": "1451478916341",
              "analysis": {
                  "analyzer": {
                      "analyzer_standard_whitespace": {
                          "filter": "lowercase",
                          "tokenizer": "whitespace"
                      },
                      "analyzer_standard": {
                          "filter": "lowercase",
                          "tokenizer": "standard"
                      },
                      "analyzer_keyword": {
                          "filter": "lowercase",
                          "tokenizer": "keyword"
                      }
                  }
              },
              "number_of_shards": "5",
              "uuid": "PjlrtJUUT1CrGduRpxPETw",
              "version": {
                  "created": "1070399"
              },
              "number_of_replicas": "1"
          }
      },
      "warmers": {}
  }
  </code></pre>
  
  <p>}</p>