<p dir="auto">I have a an index with 30M documents, spread across 4 nodes, 30 shards, with 2 replicas each. Each document has a nested mapping type "a", representing a transaction log entry. Each transaction log entry has a positive or negative dollar value and a timestamp. When I attempt to bucket the nested objects using the date range and sum aggregations, the sum aggregation appears to break on transactions with both positive and negative numbers.</p>
<p dir="auto">For example, consider the following query. I include a reverse nested agg to show that the nested objects are different than the output of the sum agg.</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;size&quot;: 0,
    &quot;query&quot;: {
        &quot;filtered&quot;: {
            &quot;query&quot;: {
                &quot;match_all&quot;: {}
            },
            &quot;filter&quot;: {
                &quot;bool&quot;: {
                    &quot;must&quot;: [
                        {
                            &quot;range&quot;: {
                                &quot;timestamp&quot;: {
                                    &quot;gte&quot;: &quot;2011-10-01&quot;
                                }
                            }
                        },
                        {
                            &quot;term&quot;: {
                                &quot;bar.name&quot;: &quot;ASpecialName&quot;
                            }
                        }
                    ]
                }
            }
        }
    },
    &quot;aggs&quot;: {
        &quot;BAR&quot;: {
            &quot;terms&quot;: {
                &quot;field&quot;: &quot;bar.name&quot;,
                &quot;size&quot;: 2,
                &quot;order&quot;: {
                    &quot;NESTED&gt;TOTAL&quot;: &quot;asc&quot;
                }
            },
            &quot;aggs&quot;: {
                &quot;NESTED&quot;: {
                    &quot;nested&quot;: {
                        &quot;path&quot;: &quot;a&quot;
                    },
                    &quot;aggs&quot;: {
                        &quot;TOTAL&quot;: {
                            &quot;sum&quot;: {
                                &quot;field&quot;: &quot;a.obligatedamount&quot;
                            }
                        },
                        &quot;DATES&quot;: {
                            &quot;date_range&quot;: {
                                &quot;field&quot;: &quot;a.signeddate&quot;,
                                &quot;keyed&quot;: true,
                                &quot;ranges&quot;: [
                                    {
                                        &quot;key&quot;: &quot;FY2012&quot;,
                                        &quot;from&quot;: &quot;2011-10-01&quot;,
                                        &quot;to&quot;: &quot;2012-09-30&quot;
                                    },
                                    {
                                        &quot;key&quot;: &quot;FY2013&quot;,
                                        &quot;from&quot;: &quot;2012-10-01&quot;,
                                        &quot;to&quot;: &quot;2013-09-30&quot;
                                    },
                                    {
                                        &quot;key&quot;: &quot;FY2014&quot;,
                                        &quot;from&quot;: &quot;2013-10-01&quot;,
                                        &quot;to&quot;: &quot;2014-09-30&quot;
                                    },
                                    {
                                        &quot;key&quot;: &quot;FY2015&quot;,
                                        &quot;from&quot;: &quot;2014-10-01&quot;,
                                        &quot;to&quot;: &quot;2015-09-30&quot;
                                    }
                                ]
                            },
                            &quot;aggs&quot;: {
                                &quot;DATEBUCKET_SUBTOTAL&quot;: {
                                    &quot;sum&quot;: {
                                        &quot;field&quot;: &quot;a.obligatedamount&quot;
                                    }
                                },
                                &quot;HITS_REVERSE&quot;: {
                                    &quot;reverse_nested&quot;: {},
                                    &quot;aggs&quot;: {
                                        &quot;HITS&quot;: {
                                            &quot;top_hits&quot;: {
                                                &quot;_source&quot;: {
                                                    &quot;include&quot;: [
                                                        &quot;a.obligatedamount&quot;,
                                                        &quot;a.signeddate&quot;
                                                    ]
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}"><pre class="notranslate">{
    <span class="pl-ent">"size"</span>: <span class="pl-c1">0</span>,
    <span class="pl-ent">"query"</span>: {
        <span class="pl-ent">"filtered"</span>: {
            <span class="pl-ent">"query"</span>: {
                <span class="pl-ent">"match_all"</span>: {}
            },
            <span class="pl-ent">"filter"</span>: {
                <span class="pl-ent">"bool"</span>: {
                    <span class="pl-ent">"must"</span>: [
                        {
                            <span class="pl-ent">"range"</span>: {
                                <span class="pl-ent">"timestamp"</span>: {
                                    <span class="pl-ent">"gte"</span>: <span class="pl-s"><span class="pl-pds">"</span>2011-10-01<span class="pl-pds">"</span></span>
                                }
                            }
                        },
                        {
                            <span class="pl-ent">"term"</span>: {
                                <span class="pl-ent">"bar.name"</span>: <span class="pl-s"><span class="pl-pds">"</span>ASpecialName<span class="pl-pds">"</span></span>
                            }
                        }
                    ]
                }
            }
        }
    },
    <span class="pl-ent">"aggs"</span>: {
        <span class="pl-ent">"BAR"</span>: {
            <span class="pl-ent">"terms"</span>: {
                <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>bar.name<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"size"</span>: <span class="pl-c1">2</span>,
                <span class="pl-ent">"order"</span>: {
                    <span class="pl-ent">"NESTED&gt;TOTAL"</span>: <span class="pl-s"><span class="pl-pds">"</span>asc<span class="pl-pds">"</span></span>
                }
            },
            <span class="pl-ent">"aggs"</span>: {
                <span class="pl-ent">"NESTED"</span>: {
                    <span class="pl-ent">"nested"</span>: {
                        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>a<span class="pl-pds">"</span></span>
                    },
                    <span class="pl-ent">"aggs"</span>: {
                        <span class="pl-ent">"TOTAL"</span>: {
                            <span class="pl-ent">"sum"</span>: {
                                <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>a.obligatedamount<span class="pl-pds">"</span></span>
                            }
                        },
                        <span class="pl-ent">"DATES"</span>: {
                            <span class="pl-ent">"date_range"</span>: {
                                <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>a.signeddate<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"keyed"</span>: <span class="pl-c1">true</span>,
                                <span class="pl-ent">"ranges"</span>: [
                                    {
                                        <span class="pl-ent">"key"</span>: <span class="pl-s"><span class="pl-pds">"</span>FY2012<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"from"</span>: <span class="pl-s"><span class="pl-pds">"</span>2011-10-01<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"to"</span>: <span class="pl-s"><span class="pl-pds">"</span>2012-09-30<span class="pl-pds">"</span></span>
                                    },
                                    {
                                        <span class="pl-ent">"key"</span>: <span class="pl-s"><span class="pl-pds">"</span>FY2013<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"from"</span>: <span class="pl-s"><span class="pl-pds">"</span>2012-10-01<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"to"</span>: <span class="pl-s"><span class="pl-pds">"</span>2013-09-30<span class="pl-pds">"</span></span>
                                    },
                                    {
                                        <span class="pl-ent">"key"</span>: <span class="pl-s"><span class="pl-pds">"</span>FY2014<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"from"</span>: <span class="pl-s"><span class="pl-pds">"</span>2013-10-01<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"to"</span>: <span class="pl-s"><span class="pl-pds">"</span>2014-09-30<span class="pl-pds">"</span></span>
                                    },
                                    {
                                        <span class="pl-ent">"key"</span>: <span class="pl-s"><span class="pl-pds">"</span>FY2015<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"from"</span>: <span class="pl-s"><span class="pl-pds">"</span>2014-10-01<span class="pl-pds">"</span></span>,
                                        <span class="pl-ent">"to"</span>: <span class="pl-s"><span class="pl-pds">"</span>2015-09-30<span class="pl-pds">"</span></span>
                                    }
                                ]
                            },
                            <span class="pl-ent">"aggs"</span>: {
                                <span class="pl-ent">"DATEBUCKET_SUBTOTAL"</span>: {
                                    <span class="pl-ent">"sum"</span>: {
                                        <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>a.obligatedamount<span class="pl-pds">"</span></span>
                                    }
                                },
                                <span class="pl-ent">"HITS_REVERSE"</span>: {
                                    <span class="pl-ent">"reverse_nested"</span>: {},
                                    <span class="pl-ent">"aggs"</span>: {
                                        <span class="pl-ent">"HITS"</span>: {
                                            <span class="pl-ent">"top_hits"</span>: {
                                                <span class="pl-ent">"_source"</span>: {
                                                    <span class="pl-ent">"include"</span>: [
                                                        <span class="pl-s"><span class="pl-pds">"</span>a.obligatedamount<span class="pl-pds">"</span></span>,
                                                        <span class="pl-s"><span class="pl-pds">"</span>a.signeddate<span class="pl-pds">"</span></span>
                                                    ]
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}</pre></div>
<p dir="auto">The following result is returned. Note the odd number in the DATEBUCKET_SUBTOTAL in FY2012 for "ASpecialPlace". This appears to only be an issue when the transaction log contains both negative and positive numbers. (Response slightly truncated...)</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;aggregations&quot;: {
        &quot;BAR&quot;: {
            &quot;doc_count_error_upper_bound&quot;: 0,
            &quot;sum_other_doc_count&quot;: 0,
            &quot;buckets&quot;: [
                {
                    &quot;key&quot;: &quot;ASpecialPlace&quot;,
                    &quot;doc_count&quot;: 2,
                    &quot;NESTED&quot;: {
                        &quot;doc_count&quot;: 3,
                        &quot;DATES&quot;: {
                            &quot;buckets&quot;: {
                                &quot;FY2012&quot;: {
                                    &quot;from_as_string&quot;: &quot;2011-10-01T00:00:00.000Z&quot;,
                                    &quot;to_as_string&quot;: &quot;2012-09-30T00:00:00.000Z&quot;,
                                    &quot;doc_count&quot;: 1,
                                    &quot;HITS_REVERSE&quot;: {
                                        &quot;doc_count&quot;: 1,
                                        &quot;HITS&quot;: {
                                            &quot;hits&quot;: {
                                                &quot;total&quot;: 1,
                                                &quot;max_score&quot;: 1,
                                                &quot;hits&quot;: [
                                                    {
                                                        &quot;_index&quot;: &quot;myIndex&quot;,
                                                        &quot;_type&quot;: &quot;myType&quot;,
                                                        &quot;_id&quot;: &quot;AG9J61P110025&quot;,
                                                        &quot;_score&quot;: 1,
                                                        &quot;_source&quot;: {
                                                            &quot;a&quot;: [
                                                                {
                                                                    &quot;signeddate&quot;: &quot;2011-02-01T00:00:00+0000&quot;,
                                                                    &quot;obligatedamount&quot;: 4000
                                                                },
                                                                {
                                                                    &quot;signeddate&quot;: &quot;2012-07-11T00:00:00+0000&quot;,
                                                                    &quot;obligatedamount&quot;: -1694
                                                                }
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                    &quot;DATEBUCKET_SUBTOTAL&quot;: {
                                        &quot;value&quot;: -8.365e-321
                                    }
                                },
                                &quot;FY2013&quot;: {
                                    &quot;from_as_string&quot;: &quot;2012-10-01T00:00:00.000Z&quot;,
                                    &quot;to&quot;: 1380499200000,
                                    &quot;doc_count&quot;: 0,
                                    &quot;HITS_REVERSE&quot;: {
                                        &quot;doc_count&quot;: 0,
                                        &quot;HITS&quot;: {
                                            &quot;hits&quot;: {
                                                &quot;total&quot;: 0,
                                                &quot;max_score&quot;: null,
                                                &quot;hits&quot;: []
                                            }
                                        }
                                    },
                                    &quot;DATEBUCKET_SUBTOTAL&quot;: {
                                        &quot;value&quot;: 0
                                    }
                                },
                                &quot;FY2014&quot;: {
                                    &quot;from_as_string&quot;: &quot;2013-10-01T00:00:00.000Z&quot;,
                                    &quot;to_as_string&quot;: &quot;2014-09-30T00:00:00.000Z&quot;,
                                    &quot;doc_count&quot;: 1,
                                    &quot;HITS_REVERSE&quot;: {
                                        &quot;doc_count&quot;: 1,
                                        &quot;HITS&quot;: {
                                            &quot;hits&quot;: {
                                                &quot;total&quot;: 1,
                                                &quot;max_score&quot;: 1,
                                                &quot;hits&quot;: [
                                                    {
                                                        &quot;_index&quot;: &quot;myIndex&quot;,
                                                        &quot;_type&quot;: &quot;myType&quot;,
                                                        &quot;_id&quot;: &quot;AG04GGP140011&quot;,
                                                        &quot;_score&quot;: 1,
                                                        &quot;_source&quot;: {
                                                            &quot;a&quot;: [
                                                                {
                                                                    &quot;signeddate&quot;: &quot;2013-12-19T00:00:00+0000&quot;,
                                                                    &quot;obligatedamount&quot;: 3449
                                                                }
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                    &quot;DATEBUCKET_SUBTOTAL&quot;: {
                                        &quot;value&quot;: 3449
                                    }
                                },
                                &quot;FY2015&quot;: {
                                    &quot;from_as_string&quot;: &quot;2014-10-01T00:00:00.000Z&quot;,
                                    &quot;to_as_string&quot;: &quot;2015-09-30T00:00:00.000Z&quot;,
                                    &quot;doc_count&quot;: 0,
                                    &quot;HITS_REVERSE&quot;: {
                                        &quot;doc_count&quot;: 0,
                                        &quot;HITS&quot;: {
                                            &quot;hits&quot;: {
                                                &quot;total&quot;: 0,
                                                &quot;max_score&quot;: null,
                                                &quot;hits&quot;: []
                                            }
                                        }
                                    },
                                    &quot;DATEBUCKET_SUBTOTAL&quot;: {
                                        &quot;value&quot;: 0
                                    }
                                }
                            }
                        },
                        &quot;TOTAL&quot;: {
                            &quot;value&quot;: 3449
                        }
                    }
                }
            ]
        }
    }
}"><pre class="notranslate">{
    <span class="pl-ent">"aggregations"</span>: {
        <span class="pl-ent">"BAR"</span>: {
            <span class="pl-ent">"doc_count_error_upper_bound"</span>: <span class="pl-c1">0</span>,
            <span class="pl-ent">"sum_other_doc_count"</span>: <span class="pl-c1">0</span>,
            <span class="pl-ent">"buckets"</span>: [
                {
                    <span class="pl-ent">"key"</span>: <span class="pl-s"><span class="pl-pds">"</span>ASpecialPlace<span class="pl-pds">"</span></span>,
                    <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">2</span>,
                    <span class="pl-ent">"NESTED"</span>: {
                        <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">3</span>,
                        <span class="pl-ent">"DATES"</span>: {
                            <span class="pl-ent">"buckets"</span>: {
                                <span class="pl-ent">"FY2012"</span>: {
                                    <span class="pl-ent">"from_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2011-10-01T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"to_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2012-09-30T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">1</span>,
                                    <span class="pl-ent">"HITS_REVERSE"</span>: {
                                        <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">1</span>,
                                        <span class="pl-ent">"HITS"</span>: {
                                            <span class="pl-ent">"hits"</span>: {
                                                <span class="pl-ent">"total"</span>: <span class="pl-c1">1</span>,
                                                <span class="pl-ent">"max_score"</span>: <span class="pl-c1">1</span>,
                                                <span class="pl-ent">"hits"</span>: [
                                                    {
                                                        <span class="pl-ent">"_index"</span>: <span class="pl-s"><span class="pl-pds">"</span>myIndex<span class="pl-pds">"</span></span>,
                                                        <span class="pl-ent">"_type"</span>: <span class="pl-s"><span class="pl-pds">"</span>myType<span class="pl-pds">"</span></span>,
                                                        <span class="pl-ent">"_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>AG9J61P110025<span class="pl-pds">"</span></span>,
                                                        <span class="pl-ent">"_score"</span>: <span class="pl-c1">1</span>,
                                                        <span class="pl-ent">"_source"</span>: {
                                                            <span class="pl-ent">"a"</span>: [
                                                                {
                                                                    <span class="pl-ent">"signeddate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2011-02-01T00:00:00+0000<span class="pl-pds">"</span></span>,
                                                                    <span class="pl-ent">"obligatedamount"</span>: <span class="pl-c1">4000</span>
                                                                },
                                                                {
                                                                    <span class="pl-ent">"signeddate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2012-07-11T00:00:00+0000<span class="pl-pds">"</span></span>,
                                                                    <span class="pl-ent">"obligatedamount"</span>: <span class="pl-c1">-1694</span>
                                                                }
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                    <span class="pl-ent">"DATEBUCKET_SUBTOTAL"</span>: {
                                        <span class="pl-ent">"value"</span>: <span class="pl-c1">-8.365e-321</span>
                                    }
                                },
                                <span class="pl-ent">"FY2013"</span>: {
                                    <span class="pl-ent">"from_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2012-10-01T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"to"</span>: <span class="pl-c1">1380499200000</span>,
                                    <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">0</span>,
                                    <span class="pl-ent">"HITS_REVERSE"</span>: {
                                        <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">0</span>,
                                        <span class="pl-ent">"HITS"</span>: {
                                            <span class="pl-ent">"hits"</span>: {
                                                <span class="pl-ent">"total"</span>: <span class="pl-c1">0</span>,
                                                <span class="pl-ent">"max_score"</span>: <span class="pl-c1">null</span>,
                                                <span class="pl-ent">"hits"</span>: []
                                            }
                                        }
                                    },
                                    <span class="pl-ent">"DATEBUCKET_SUBTOTAL"</span>: {
                                        <span class="pl-ent">"value"</span>: <span class="pl-c1">0</span>
                                    }
                                },
                                <span class="pl-ent">"FY2014"</span>: {
                                    <span class="pl-ent">"from_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2013-10-01T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"to_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2014-09-30T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">1</span>,
                                    <span class="pl-ent">"HITS_REVERSE"</span>: {
                                        <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">1</span>,
                                        <span class="pl-ent">"HITS"</span>: {
                                            <span class="pl-ent">"hits"</span>: {
                                                <span class="pl-ent">"total"</span>: <span class="pl-c1">1</span>,
                                                <span class="pl-ent">"max_score"</span>: <span class="pl-c1">1</span>,
                                                <span class="pl-ent">"hits"</span>: [
                                                    {
                                                        <span class="pl-ent">"_index"</span>: <span class="pl-s"><span class="pl-pds">"</span>myIndex<span class="pl-pds">"</span></span>,
                                                        <span class="pl-ent">"_type"</span>: <span class="pl-s"><span class="pl-pds">"</span>myType<span class="pl-pds">"</span></span>,
                                                        <span class="pl-ent">"_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>AG04GGP140011<span class="pl-pds">"</span></span>,
                                                        <span class="pl-ent">"_score"</span>: <span class="pl-c1">1</span>,
                                                        <span class="pl-ent">"_source"</span>: {
                                                            <span class="pl-ent">"a"</span>: [
                                                                {
                                                                    <span class="pl-ent">"signeddate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2013-12-19T00:00:00+0000<span class="pl-pds">"</span></span>,
                                                                    <span class="pl-ent">"obligatedamount"</span>: <span class="pl-c1">3449</span>
                                                                }
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                    <span class="pl-ent">"DATEBUCKET_SUBTOTAL"</span>: {
                                        <span class="pl-ent">"value"</span>: <span class="pl-c1">3449</span>
                                    }
                                },
                                <span class="pl-ent">"FY2015"</span>: {
                                    <span class="pl-ent">"from_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2014-10-01T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"to_as_string"</span>: <span class="pl-s"><span class="pl-pds">"</span>2015-09-30T00:00:00.000Z<span class="pl-pds">"</span></span>,
                                    <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">0</span>,
                                    <span class="pl-ent">"HITS_REVERSE"</span>: {
                                        <span class="pl-ent">"doc_count"</span>: <span class="pl-c1">0</span>,
                                        <span class="pl-ent">"HITS"</span>: {
                                            <span class="pl-ent">"hits"</span>: {
                                                <span class="pl-ent">"total"</span>: <span class="pl-c1">0</span>,
                                                <span class="pl-ent">"max_score"</span>: <span class="pl-c1">null</span>,
                                                <span class="pl-ent">"hits"</span>: []
                                            }
                                        }
                                    },
                                    <span class="pl-ent">"DATEBUCKET_SUBTOTAL"</span>: {
                                        <span class="pl-ent">"value"</span>: <span class="pl-c1">0</span>
                                    }
                                }
                            }
                        },
                        <span class="pl-ent">"TOTAL"</span>: {
                            <span class="pl-ent">"value"</span>: <span class="pl-c1">3449</span>
                        }
                    }
                }
            ]
        }
    }
}</pre></div>
<p dir="auto">If I set a filter at the nested level to remove any nested "a" objects with a negative value, everything works fine. Note, this behavior is also apparent if using a stats agg instead of just a sum.</p>
<p dir="auto">Today, a new field can be added to the local mapping of two shards simultaneously. If the detected field type is different, then each shard can end up with a different field type.  These shards then send their new mapping to the master, but only of the mappings will win.</p>
<p dir="auto">This can result in incorrect results and even data loss, eg: one shard thinks that the field is a string, and the other shard (and the master) thinks that it is a number.  In this case, sorting and aggregations will be wrong (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="48793648" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8485" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8485/hovercard" href="https://github.com/elastic/elasticsearch/issues/8485">#8485</a>).  Then, when the string-shard is allocated to a new node, that node will receive the "number" mapping from the master.  Replaying the transaction log can cause shard failures (eg <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="50293007" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8684" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8684/hovercard" href="https://github.com/elastic/elasticsearch/issues/8684">#8684</a>).  Or new numeric doc values are written but, when we try to merge segments, it fails with a doc values exception (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="45091629" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8009" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8009/hovercard" href="https://github.com/elastic/elasticsearch/issues/8009">#8009</a>).</p>
<p dir="auto">The only way to ensure that dynamic fields are the same on all shards is to wait for the master to acknowledge the mapping changes before indexing the document, so:</p>
<ul dir="auto">
<li>parse the document</li>
<li>update local mapping</li>
<li><strong>if changed, send to master and wait for ack</strong></li>
<li>index document</li>
</ul>
<p dir="auto">This will potentially slow down indexing when many dynamic fields are being added, but it is the only way to automatically protect against data loss due to mapping conflicts.</p>
<p dir="auto">Should a user wish to disable waiting for the master (and they are certain that their dynamic mapping rules are good enough to prevent these problems) then we should allow them to opt-out by setting <code class="notranslate">"dynamic": "unsafe"</code></p>