<p dir="auto">If you manually move a shard using the Reroute API, it appears that the Cluster node stats API breaks for the duration of the shard reassignment (500 Internal Server error, message at bottom of the gist).  It begins working again once the shard has been fully relocated and intialized.</p>
<p dir="auto">Here is a recreation.  The index in question is "test".  The cluster has three data nodes (S1, S2, S3) and one non-data client node (C1)</p>
<p dir="auto">Edit:  this is with ES Version 0.20.2</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
  &quot;commands&quot;:[
    {
      &quot;move&quot;:{
        &quot;index&quot;:&quot;test&quot;,
        &quot;shard&quot;:2,
        &quot;from_node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
        &quot;to_node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;
      }
    }
  ]
}' 

{
  &quot;ok&quot;:true,
  &quot;state&quot;:{
    &quot;master_node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
    &quot;blocks&quot;:{

    },
    &quot;nodes&quot;:{
      &quot;ez-rzcnfSESrVYP6zTWksA&quot;:{
        &quot;name&quot;:&quot;S1&quot;,
        &quot;transport_address&quot;:&quot;inet[/144.76.3.102:9300]&quot;,
        &quot;attributes&quot;:{

        }
      },
      &quot;-VNWEiiaSG2IRlwM6kzcxg&quot;:{
        &quot;name&quot;:&quot;C1&quot;,
        &quot;transport_address&quot;:&quot;inet[/144.76.8.228:9300]&quot;,
        &quot;attributes&quot;:{
          &quot;data&quot;:&quot;false&quot;
        }
      },
      &quot;kH152vsLTL-y20mcLFs9GQ&quot;:{
        &quot;name&quot;:&quot;S3&quot;,
        &quot;transport_address&quot;:&quot;inet[/144.76.2.205:9300]&quot;,
        &quot;attributes&quot;:{

        }
      },
      &quot;J47gdOIwQMq2GTmzzmzJBA&quot;:{
        &quot;name&quot;:&quot;S2&quot;,
        &quot;transport_address&quot;:&quot;inet[/144.76.3.103:9300]&quot;,
        &quot;attributes&quot;:{

        }
      }
    },
    &quot;routing_table&quot;:{
      &quot;indices&quot;:{
        &quot;test&quot;:{
          &quot;shards&quot;:{
            &quot;0&quot;:[
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:true,
                &quot;node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:0,
                &quot;index&quot;:&quot;test&quot;
              },
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:false,
                &quot;node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:0,
                &quot;index&quot;:&quot;test&quot;
              }
            ],
            &quot;1&quot;:[
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:true,
                &quot;node&quot;:&quot;ez-rzcnfSESrVYP6zTWksA&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:1,
                &quot;index&quot;:&quot;test&quot;
              },
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:false,
                &quot;node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:1,
                &quot;index&quot;:&quot;test&quot;
              }
            ],
            &quot;2&quot;:[
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:false,
                &quot;node&quot;:&quot;ez-rzcnfSESrVYP6zTWksA&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:2,
                &quot;index&quot;:&quot;test&quot;
              },
              {
                &quot;state&quot;:&quot;RELOCATING&quot;,
                &quot;primary&quot;:true,
                &quot;node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
                &quot;relocating_node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
                &quot;shard&quot;:2,
                &quot;index&quot;:&quot;test&quot;
              }
            ]
          }
        },
        &quot;test123&quot;:{
          &quot;shards&quot;:{
            &quot;0&quot;:[
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:false,
                &quot;node&quot;:&quot;ez-rzcnfSESrVYP6zTWksA&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:0,
                &quot;index&quot;:&quot;test123&quot;
              },
              {
                &quot;state&quot;:&quot;STARTED&quot;,
                &quot;primary&quot;:true,
                &quot;node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
                &quot;relocating_node&quot;:null,
                &quot;shard&quot;:0,
                &quot;index&quot;:&quot;test123&quot;
              }
            ]
          }
        }
      }
    },
    &quot;routing_nodes&quot;:{
      &quot;unassigned&quot;:[

      ],
      &quot;nodes&quot;:{
        &quot;ez-rzcnfSESrVYP6zTWksA&quot;:[
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:true,
            &quot;node&quot;:&quot;ez-rzcnfSESrVYP6zTWksA&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:1,
            &quot;index&quot;:&quot;test&quot;
          },
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:false,
            &quot;node&quot;:&quot;ez-rzcnfSESrVYP6zTWksA&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:2,
            &quot;index&quot;:&quot;test&quot;
          },
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:false,
            &quot;node&quot;:&quot;ez-rzcnfSESrVYP6zTWksA&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:0,
            &quot;index&quot;:&quot;test123&quot;
          }
        ],
        &quot;kH152vsLTL-y20mcLFs9GQ&quot;:[
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:true,
            &quot;node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:0,
            &quot;index&quot;:&quot;test&quot;
          },
          {
            &quot;state&quot;:&quot;RELOCATING&quot;,
            &quot;primary&quot;:true,
            &quot;node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
            &quot;relocating_node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
            &quot;shard&quot;:2,
            &quot;index&quot;:&quot;test&quot;
          },
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:true,
            &quot;node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:0,
            &quot;index&quot;:&quot;test123&quot;
          }
        ],
        &quot;J47gdOIwQMq2GTmzzmzJBA&quot;:[
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:false,
            &quot;node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:0,
            &quot;index&quot;:&quot;test&quot;
          },
          {
            &quot;state&quot;:&quot;STARTED&quot;,
            &quot;primary&quot;:false,
            &quot;node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
            &quot;relocating_node&quot;:null,
            &quot;shard&quot;:1,
            &quot;index&quot;:&quot;test&quot;
          },
          {
            &quot;state&quot;:&quot;INITIALIZING&quot;,
            &quot;primary&quot;:true,
            &quot;node&quot;:&quot;J47gdOIwQMq2GTmzzmzJBA&quot;,
            &quot;relocating_node&quot;:&quot;kH152vsLTL-y20mcLFs9GQ&quot;,
            &quot;shard&quot;:2,
            &quot;index&quot;:&quot;test&quot;
          }
        ]
      }
    },
    &quot;allocations&quot;:[

    ]
  }
}

##For the duration of the move, this API returns a 500 error
$ curl -XGET localhost:9200/_cluster/nodes/stats?all=true
{
    &quot;error&quot;: &quot;ArithmeticException[Value cannot fit in an int: -2562047788015]&quot;,
    &quot;status&quot;: 500
}"><pre class="notranslate">curl -XPOST <span class="pl-s"><span class="pl-pds">'</span>localhost:9200/_cluster/reroute<span class="pl-pds">'</span></span> -d <span class="pl-s"><span class="pl-pds">'</span>{</span>
<span class="pl-s">  "commands":[</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "move":{</span>
<span class="pl-s">        "index":"test",</span>
<span class="pl-s">        "shard":2,</span>
<span class="pl-s">        "from_node":"kH152vsLTL-y20mcLFs9GQ",</span>
<span class="pl-s">        "to_node":"J47gdOIwQMq2GTmzzmzJBA"</span>
<span class="pl-s">      }</span>
<span class="pl-s">    }</span>
<span class="pl-s">  ]</span>
<span class="pl-s">}<span class="pl-pds">'</span></span> 

{
  <span class="pl-s"><span class="pl-pds">"</span>ok<span class="pl-pds">"</span></span>:true,
  <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:{
    <span class="pl-s"><span class="pl-pds">"</span>master_node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>blocks<span class="pl-pds">"</span></span>:{

    },
    <span class="pl-s"><span class="pl-pds">"</span>nodes<span class="pl-pds">"</span></span>:{
      <span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>:{
        <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>S1<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>transport_address<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>inet[/144.76.3.102:9300]<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>attributes<span class="pl-pds">"</span></span>:{

        }
      },
      <span class="pl-s"><span class="pl-pds">"</span>-VNWEiiaSG2IRlwM6kzcxg<span class="pl-pds">"</span></span>:{
        <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>C1<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>transport_address<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>inet[/144.76.8.228:9300]<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>attributes<span class="pl-pds">"</span></span>:{
          <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>false<span class="pl-pds">"</span></span>
        }
      },
      <span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>:{
        <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>S3<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>transport_address<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>inet[/144.76.2.205:9300]<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>attributes<span class="pl-pds">"</span></span>:{

        }
      },
      <span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>:{
        <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>S2<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>transport_address<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>inet[/144.76.3.103:9300]<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>attributes<span class="pl-pds">"</span></span>:{

        }
      }
    },
    <span class="pl-s"><span class="pl-pds">"</span>routing_table<span class="pl-pds">"</span></span>:{
      <span class="pl-s"><span class="pl-pds">"</span>indices<span class="pl-pds">"</span></span>:{
        <span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>:{
          <span class="pl-s"><span class="pl-pds">"</span>shards<span class="pl-pds">"</span></span>:{
            <span class="pl-s"><span class="pl-pds">"</span>0<span class="pl-pds">"</span></span>:[
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
              },
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
              }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>:[
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:1,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
              },
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:1,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
              }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>2<span class="pl-pds">"</span></span>:[
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:2,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
              },
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>RELOCATING<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:2,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
              }
            ]
          }
        },
        <span class="pl-s"><span class="pl-pds">"</span>test123<span class="pl-pds">"</span></span>:{
          <span class="pl-s"><span class="pl-pds">"</span>shards<span class="pl-pds">"</span></span>:{
            <span class="pl-s"><span class="pl-pds">"</span>0<span class="pl-pds">"</span></span>:[
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test123<span class="pl-pds">"</span></span>
              },
              {
                <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
                <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
                <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
                <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test123<span class="pl-pds">"</span></span>
              }
            ]
          }
        }
      }
    },
    <span class="pl-s"><span class="pl-pds">"</span>routing_nodes<span class="pl-pds">"</span></span>:{
      <span class="pl-s"><span class="pl-pds">"</span>unassigned<span class="pl-pds">"</span></span>:[

      ],
      <span class="pl-s"><span class="pl-pds">"</span>nodes<span class="pl-pds">"</span></span>:{
        <span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>:[
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:1,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          },
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:2,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          },
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>ez-rzcnfSESrVYP6zTWksA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test123<span class="pl-pds">"</span></span>
          }
        ],
        <span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>:[
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          },
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>RELOCATING<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:2,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          },
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test123<span class="pl-pds">"</span></span>
          }
        ],
        <span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>:[
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:0,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          },
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>STARTED<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:false,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:null,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:1,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          },
          {
            <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>INITIALIZING<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>primary<span class="pl-pds">"</span></span>:true,
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>J47gdOIwQMq2GTmzzmzJBA<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>relocating_node<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>kH152vsLTL-y20mcLFs9GQ<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>shard<span class="pl-pds">"</span></span>:2,
            <span class="pl-s"><span class="pl-pds">"</span>index<span class="pl-pds">"</span></span>:<span class="pl-s"><span class="pl-pds">"</span>test<span class="pl-pds">"</span></span>
          }
        ]
      }
    },
    <span class="pl-s"><span class="pl-pds">"</span>allocations<span class="pl-pds">"</span></span>:[

    ]
  }
}

<span class="pl-c"><span class="pl-c">#</span>#For the duration of the move, this API returns a 500 error</span>
$ curl -XGET localhost:9200/_cluster/nodes/stats<span class="pl-k">?</span>all=true
{
    <span class="pl-s"><span class="pl-pds">"</span>error<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>ArithmeticException[Value cannot fit in an int: -2562047788015]<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>status<span class="pl-pds">"</span></span>: 500
}</pre></div>
<p dir="auto">To reproduce, start a single node and as soon as node starts run the following script:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Create and delete index a few times
for i in {1..4}
do
    curl -XDELETE localhost:9200/test
    curl -XPUT localhost:9200/test -d '{
        &quot;settings&quot;: {
            &quot;index.number_of_shards&quot;: 1,
            &quot;index.number_of_replicas&quot;: 0
        }
    }'
    curl -XPUT localhost:9200/test/doc/1 -d '{&quot;foo&quot;: &quot;bar&quot;}'
    curl -XPUT localhost:9200/test/doc/2 -d '{&quot;foo&quot;: &quot;bar&quot;}'
    curl -XPUT localhost:9200/test/doc/3 -d '{&quot;foo&quot;: &quot;bar&quot;}'
    curl -XPOST localhost:9200/test/_refresh
    # Do search to fill filter cache
    curl &quot;localhost:9200/test/doc/_search&quot;
done
# Sleep for indices.cache.filter.clean_interval
# Reduce it to make this repro run faster
echo
echo &quot;Waiting 1 min for cache to be cleaned&quot;
sleep 60
curl -s -XGET 'http://localhost:9200/_nodes/stats?pretty=true'"><pre class="notranslate"><code class="notranslate"># Create and delete index a few times
for i in {1..4}
do
    curl -XDELETE localhost:9200/test
    curl -XPUT localhost:9200/test -d '{
        "settings": {
            "index.number_of_shards": 1,
            "index.number_of_replicas": 0
        }
    }'
    curl -XPUT localhost:9200/test/doc/1 -d '{"foo": "bar"}'
    curl -XPUT localhost:9200/test/doc/2 -d '{"foo": "bar"}'
    curl -XPUT localhost:9200/test/doc/3 -d '{"foo": "bar"}'
    curl -XPOST localhost:9200/test/_refresh
    # Do search to fill filter cache
    curl "localhost:9200/test/doc/_search"
done
# Sleep for indices.cache.filter.clean_interval
# Reduce it to make this repro run faster
echo
echo "Waiting 1 min for cache to be cleaned"
sleep 60
curl -s -XGET 'http://localhost:9200/_nodes/stats?pretty=true'
</code></pre></div>
<p dir="auto">The result of this script is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="...
        &quot;cache&quot; : {
          &quot;filter_count&quot; : -2,
          &quot;filter_evictions&quot; : 3,
          &quot;filter_size&quot; : &quot;-48b&quot;,
          &quot;filter_size_in_bytes&quot; : -48,
          &quot;id_cache_size&quot; : &quot;0b&quot;,
          &quot;id_cache_size_in_bytes&quot; : 0
        }
..."><pre class="notranslate"><code class="notranslate">...
        "cache" : {
          "filter_count" : -2,
          "filter_evictions" : 3,
          "filter_size" : "-48b",
          "filter_size_in_bytes" : -48,
          "id_cache_size" : "0b",
          "id_cache_size_in_bytes" : 0
        }
...
</code></pre></div>
<p dir="auto">Analysis:</p>
<ul dir="auto">
<li><a href="https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/indices/cache/filter/IndicesFilterCache.java#L165">IndicesFilterCache</a> is using index name to dispatch removal notifications.</li>
<li>When index closes, it doesn't clean the cache immediately. So, it's possible that the cache for the old index will be cleaned when a new index with the same name is already created.</li>
<li>As a result, it's possible for WeightedFilterCache to receive notifications targeted for the previous incarnations of the index with the same name.</li>
<li>WeightedFilterCache decrements stats counters without checking if removal notification was sent for this index or for one of its previous incarnations.</li>
</ul>