title: Elasticsearch index size is 40% larger in 5.x than in 1.x
body: <p>I have an old cluster running Elasticsearch <code>1.4.4</code>. 
  My cluster is containing ~11 billion documents and the size of all primaries is around <code>4TB</code>.</p>
  
  <p>I am now in the process of upgrading to Elasticsearch <code>5.2.2</code>, which of course means reindexing my data. I have a separate cluster where this is happening at the moment. I am reindexing from my source database, since I have <code>_all</code> and <code>_source</code> disabled on the original index. </p>
  
  <p>I have now reindexed around 750 million documents and noticed that my new index size is already <code>350GB</code>. I did some math and it looks like the index will grow to around <code>5.5TB</code> when fully indexed. That's <strong>1.5TB more</strong> than the <code>1.4.4</code> index. I wasn't expecting this. On the contrary I was expecting a decrease in size, since I've removed several attributes. Is this a normal thing or did I do something wrong? Is there a different default settings in <code>5.2.2</code> that can contribute to this growth?</p>
  
  <p><strong>1.4.4 index settings:</strong></p>
  
  <pre><code>{
    "index": {
      "refresh_interval": "30s",
      "number_of_shards": "20",
      "creation_date": "1426251049131",
      "analysis": {
        "analyzer": {
          "default": {
            "filter": [
              "icu_folding",
              "icu_normalizer"
            ],
            "type": "custom",
            "tokenizer": "icu_tokenizer"
          }
        }
      },
      "uuid": "WdgnCLyITgmpb4DROegV3Q",
      "version": {
        "created": "1040499"
      },
      "number_of_replicas": "1"
    }
  }
  </code></pre>
  
  <p><strong>1.4.4 index mapping:</strong></p>
  
  <pre><code>{
    "article": {
      "_source": {
        "enabled": false
      },
      "_all": {
        "enabled": false
      },
      "properties": {
        "date": {
          "format": "dateOptionalTime",
          "type": "date",
          "doc_values": true
        },
        "has_enclosures": {
          "type": "boolean"
        },
        "feed_subscribers": {
          "type": "integer",
          "doc_values": true
        },
        "feed_language": {
          "index": "not_analyzed",
          "type": "string"
        },
        "author": {
          "norms": {
            "enabled": false
          },
          "analyzer": "keyword",
          "type": "string"
        },
        "has_pictures": {
          "type": "boolean"
        },
        "title": {
          "norms": {
            "enabled": false
          },
          "type": "string"
        },
        "content": {
          "norms": {
            "enabled": false
          },
          "type": "string"
        },
        "has_video": {
          "type": "boolean"
        },
        "url": {
          "index": "not_analyzed",
          "type": "string"
        },
        "feed_canonical": {
          "type": "boolean"
        },
        "feed_id": {
          "type": "integer",
          "doc_values": true
        }
      }
    }
  }
  </code></pre>
  
  <p><strong>5.2.2 index settings:</strong></p>
  
  <pre><code>{
    "articles": {
      "settings": {
        "index": {
          "refresh_interval": "-1",
          "number_of_shards": "40",
          "provided_name": "articles",
          "creation_date": "1489604158595",
          "analysis": {
            "analyzer": {
              "default": {
                "filter": [
                  "icu_folding",
                  "icu_normalizer"
                ],
                "type": "custom",
                "tokenizer": "icu_tokenizer"
              }
            }
          },
          "number_of_replicas": "0",
          "uuid": "LOeOcZb_TMCX6E_86uMyXQ",
          "version": {
            "created": "5020299"
          }
        }
      }
    }
  }
  </code></pre>
  
  <p><strong>5.2.2 index mapping:</strong></p>
  
  <pre><code>{
    "articles": {
      "mappings": {
        "article": {
          "_all": {
            "enabled": false
          },
          "_source": {
            "enabled": false
          },
          "properties": {
            "author": {
              "type": "text",
              "norms": false,
              "analyzer": "keyword"
            },
            "content": {
              "type": "text",
              "norms": false
            },
            "date": {
              "type": "date"
            },
            "feed_canonical": {
              "type": "boolean"
            },
            "feed_id": {
              "type": "integer"
            },
            "feed_subscribers": {
              "type": "integer"
            },
            "title": {
              "type": "text",
              "norms": false
            },
            "url": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
  </code></pre>
  
  <p>Any help will be much appreciated since full reindexing on this cluster takes about 30 days... Thanks!</p>