<p dir="auto">The goal of this issue is to agree on basic design principles and expectations from this refactoring.</p>
<p dir="auto"><strong>Goals</strong></p>
<ul dir="auto">
<li>Speed up critical places in query processing by skipping bound checks, which <code class="notranslate">ByteBuffer</code> always performs when making any read or write. Bound checks could be hoisted outside the hot loops manually or enabled via assertions/static final boolean flag, <a href="http://pveentjer.blogspot.mx/2017/01/final-static-boolean-jit.html" rel="nofollow">that is effectively free with some precautions</a>.</li>
<li>Get rid or 2 GB limitation, use <code class="notranslate">long</code> indexes. See <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="193670759" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/3743" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/3743/hovercard" href="https://github.com/apache/druid/pull/3743">#3743</a>.</li>
<li>Don't make a lot of Object copies when a <code class="notranslate">Memory</code> object has to be "sliced", "converted to read-only", etc. See <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="191492016" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/3716" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/3716/hovercard?comment_id=274658045&amp;comment_type=issue_comment" href="https://github.com/apache/druid/pull/3716#issuecomment-274658045">#3716 (comment)</a>.</li>
<li><span>+</span> Eliminate current Endian craziness in Druid by standardizing on native-endian. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="204122452" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/3892" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/3892/hovercard?comment_id=276235357&amp;comment_type=issue_comment" href="https://github.com/apache/druid/issues/3892#issuecomment-276235357">#3892 (comment)</a></li>
</ul>
<p dir="auto"><strong>Design</strong></p>
<ul dir="auto">
<li>Based on DataSketches's <a href="https://github.com/DataSketches/sketches-core/blob/master/memory/src/main/java/com/yahoo/memory/Memory.java">Memory</a>.</li>
<li><strike>It is moved to Druid source tree, not used as a dependency.</strike> See <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="204122452" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/3892" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/3892/hovercard?comment_id=276589924&amp;comment_type=issue_comment" href="https://github.com/apache/druid/issues/3892#issuecomment-276589924">#3892 (comment)</a></li>
<li><code class="notranslate">Memory</code> object is immutable. "position and limit", if needed, are passed along with <code class="notranslate">Memory</code> as two primitive longs.</li>
<li>Upon creation a mutable <code class="notranslate">Memory</code> object has a cached immutable "view" object (which implements read methods and throws exception from write methods), this object is always returned when "as read only" is needed.</li>
<li>Explicit <code class="notranslate">close()</code> or <code class="notranslate">free()</code> on <code class="notranslate">Memory</code> is possible, but not strictly required, there is a <code class="notranslate">sun.misc.Cleaner</code> safety net, as well as in <code class="notranslate">DirectByteBuffer</code>.</li>
<li>While transition from <code class="notranslate">ByteBuffer</code> to <code class="notranslate">Memory</code> is in progress (<strike>not expected to be done all at once, but subsystem by subsystem, class by class</strike> see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="204122452" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/3892" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/3892/hovercard?comment_id=276548114&amp;comment_type=issue_comment" href="https://github.com/apache/druid/issues/3892#issuecomment-276548114">#3892 (comment)</a>), and also when we need to interop with external libraries which require <code class="notranslate">ByteBuffer</code>, conversion from <code class="notranslate">Memory</code> to <code class="notranslate">ByteBuffer</code> and vice versa is possible. Likely it requires to
<ul dir="auto">
<li>Use <code class="notranslate">DirectByteBuffer</code>-compatible format of <code class="notranslate">Cleaner</code> inside <code class="notranslate">Memory</code></li>
<li>Be able to access private <code class="notranslate">DirectByteBuffer</code> constructors <a href="https://github.com/OpenHFT/Java-Lang/blob/7f6ca8aceb4dbcaea83f62d1d85542d6f374515a/lang/src/main/java/net/openhft/lang/io/ByteBufferReuse.java">via <code class="notranslate">MagicAccessorImpl</code></a>.</li>
</ul>
</li>
<li><span>+</span> "Global" (immutable) <code class="notranslate">Memory</code>'s bounds are checked optionally via assertions/guarded by static final boolean flag, "local" position and limit are check explicitly manually, with helper methods or versions of read and write methods of Memory, which accept read/write position and "local" limits. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="204122452" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/3892" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/3892/hovercard?comment_id=276185704&amp;comment_type=issue_comment" href="https://github.com/apache/druid/issues/3892#issuecomment-276185704">#3892 (comment)</a></li>
</ul>
<p dir="auto">Objections, additions, corrections, questions are welcome.<br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/leerho/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/leerho">@leerho</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cheddar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cheddar">@cheddar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/weijietong/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/weijietong">@weijietong</a>  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/akashdw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/akashdw">@akashdw</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/himanshug/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/himanshug">@himanshug</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fjy/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fjy">@fjy</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/niketh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/niketh">@niketh</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/michaelschiff/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/michaelschiff">@michaelschiff</a>  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gianm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gianm">@gianm</a><br>
Druid version  0.8.2<br>
Segment Granularity: HOUR<br>
My problem is very similar with this closed issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="14319958" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/145" data-hovercard-type="issue" data-hovercard-url="/apache/druid/issues/145/hovercard" href="https://github.com/apache/druid/issues/145">#145</a>,<br>
Around 40 minutes after an hour, I added a longSum metric and restarted the node.<br>
when merging index,will throw NPE</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="java.lang.NullPointerException
   at io.druid.query.aggregation.LongSumAggregator.combineValues(LongSumAggregator.java:39) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.query.aggregation.LongSumAggregatorFactory.combine(LongSumAggregatorFactory.java:78) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.segment.IndexMerger$RowboatMergeFunction.apply(IndexMerger.java:1260) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.segment.IndexMerger$RowboatMergeFunction.apply(IndexMerger.java:1236) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.collections.CombiningIterator.next(CombiningIterator.java:78) ~[druid-common-0.8.2.jar:0.8.2]
   at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:747) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.segment.IndexMerger.merge(IndexMerger.java:351) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:227) ~[druid-processing-0.8.2.jar:0.8.2]
   at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:216) ~[druid-processing-0.8.2.jar:0.8.2]"><pre class="notranslate"><span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">NullPointerException</span>
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">query</span>.<span class="pl-s1">aggregation</span>.<span class="pl-s1">LongSumAggregator</span>.<span class="pl-en">combineValues</span>(<span class="pl-smi">LongSumAggregator</span>.<span class="pl-smi">java</span>:<span class="pl-c1">39</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-smi">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">query</span>.<span class="pl-s1">aggregation</span>.<span class="pl-s1">LongSumAggregatorFactory</span>.<span class="pl-s1">combine</span>(<span class="pl-smi">LongSumAggregatorFactory</span>.<span class="pl-smi">java</span>:<span class="pl-c1">78</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">segment</span>.<span class="pl-s1">IndexMerger$RowboatMergeFunction</span>.<span class="pl-en">apply</span>(<span class="pl-smi">IndexMerger</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1260</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">segment</span>.<span class="pl-s1">IndexMerger$RowboatMergeFunction</span>.<span class="pl-en">apply</span>(<span class="pl-smi">IndexMerger</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1236</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">collections</span>.<span class="pl-s1">CombiningIterator</span>.<span class="pl-en">next</span>(<span class="pl-smi">CombiningIterator</span>.<span class="pl-smi">java</span>:<span class="pl-c1">78</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">common</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">segment</span>.<span class="pl-s1">IndexMerger</span>.<span class="pl-en">makeIndexFiles</span>(<span class="pl-smi">IndexMerger</span>.<span class="pl-smi">java</span>:<span class="pl-c1">747</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">segment</span>.<span class="pl-s1">IndexMerger</span>.<span class="pl-en">merge</span>(<span class="pl-smi">IndexMerger</span>.<span class="pl-smi">java</span>:<span class="pl-c1">351</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">segment</span>.<span class="pl-s1">IndexMerger</span>.<span class="pl-en">mergeQueryableIndex</span>(<span class="pl-smi">IndexMerger</span>.<span class="pl-smi">java</span>:<span class="pl-c1">227</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]
   <span class="pl-s1">at</span> <span class="pl-s1">io</span>.<span class="pl-s1">druid</span>.<span class="pl-s1">segment</span>.<span class="pl-s1">IndexMerger</span>.<span class="pl-en">mergeQueryableIndex</span>(<span class="pl-smi">IndexMerger</span>.<span class="pl-smi">java</span>:<span class="pl-c1">216</span>) ~[<span class="pl-s1">druid</span>-<span class="pl-s1">processing</span>-<span class="pl-c1">0.8</span>.<span class="pl-c1">2.</span><span class="pl-s1">jar</span>:<span class="pl-c1">0.8</span>.<span class="pl-c1">2</span>]</pre></div>
<p dir="auto">So I think this issue hasn't been resolved.</p>
<p dir="auto">look into the following code of IndexMerger line number from 657 to 688.</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" boats.add(
          new MMappedIndexRowIterable(
              Iterables.transform(
                  indexes.get(i).getRows(),
                  new Function&lt;Rowboat, Rowboat&gt;()
                  {
                    @Override
                    public Rowboat apply(@Nullable Rowboat input)
                    {
                      int[][] newDims = new int[mergedDimensions.size()][];
                      int j = 0;
                      for (int[] dim : input.getDims()) {
                        newDims[dimLookup[j]] = dim;
                        j++;
                      }

                      Object[] newMetrics = new Object[mergedMetrics.size()];
                      j = 0;
                      for (Object met : input.getMetrics()) {
                        newMetrics[metricLookup[j]] = met;
                        j++;
                      }

                      return new Rowboat(
                          input.getTimestamp(),
                          newDims,
                          newMetrics,
                          input.getRowNum()
                      );
                    }
                  }
              )"><pre class="notranslate"> <span class="pl-s1">boats</span>.<span class="pl-s1">add</span>(
          <span class="pl-k">new</span> <span class="pl-s1">MMappedIndexRowIterable</span>(
              <span class="pl-smi">Iterables</span>.<span class="pl-en">transform</span>(
                  <span class="pl-s1">indexes</span>.<span class="pl-en">get</span>(<span class="pl-s1">i</span>).<span class="pl-en">getRows</span>(),
                  <span class="pl-k">new</span> <span class="pl-smi">Function</span>&lt;<span class="pl-smi">Rowboat</span>, <span class="pl-smi">Rowboat</span>&gt;()
                  {
                    <span class="pl-c1">@</span><span class="pl-c1">Override</span>
                    <span class="pl-k">public</span> <span class="pl-smi">Rowboat</span> <span class="pl-en">apply</span>(<span class="pl-c1">@</span><span class="pl-c1">Nullable</span> <span class="pl-smi">Rowboat</span> <span class="pl-s1">input</span>)
                    {
                      <span class="pl-smi">int</span>[][] <span class="pl-s1">newDims</span> = <span class="pl-k">new</span> <span class="pl-smi">int</span>[<span class="pl-s1">mergedDimensions</span>.<span class="pl-en">size</span>()][];
                      <span class="pl-smi">int</span> <span class="pl-s1">j</span> = <span class="pl-c1">0</span>;
                      <span class="pl-k">for</span> (<span class="pl-smi">int</span>[] <span class="pl-s1">dim</span> : <span class="pl-s1">input</span>.<span class="pl-en">getDims</span>()) {
                        <span class="pl-s1">newDims</span>[<span class="pl-s1">dimLookup</span>[<span class="pl-s1">j</span>]] = <span class="pl-s1">dim</span>;
                        <span class="pl-s1">j</span>++;
                      }

                      <span class="pl-smi">Object</span>[] <span class="pl-s1">newMetrics</span> = <span class="pl-k">new</span> <span class="pl-smi">Object</span>[<span class="pl-s1">mergedMetrics</span>.<span class="pl-en">size</span>()];
                      <span class="pl-s1">j</span> = <span class="pl-c1">0</span>;
                      <span class="pl-k">for</span> (<span class="pl-smi">Object</span> <span class="pl-s1">met</span> : <span class="pl-s1">input</span>.<span class="pl-en">getMetrics</span>()) {
                        <span class="pl-s1">newMetrics</span>[<span class="pl-s1">metricLookup</span>[<span class="pl-s1">j</span>]] = <span class="pl-s1">met</span>;
                        <span class="pl-s1">j</span>++;
                      }

                      <span class="pl-k">return</span> <span class="pl-k">new</span> <span class="pl-smi">Rowboat</span>(
                          <span class="pl-s1">input</span>.<span class="pl-en">getTimestamp</span>(),
                          <span class="pl-s1">newDims</span>,
                          <span class="pl-s1">newMetrics</span>,
                          <span class="pl-s1">input</span>.<span class="pl-en">getRowNum</span>()
                      );
                    }
                  }
              )</pre></div>
<p dir="auto">Build newMetrics base on the metric of  input row. the previous indexes don't have new addition metric, So in the newMetrics,the value of new addition metric is null</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="  for (Object met : input.getMetrics()) {
      newMetrics[metricLookup[j]] = met;
      j++;
  }"><pre class="notranslate">  <span class="pl-k">for</span> (<span class="pl-smi">Object</span> <span class="pl-s1">met</span> : <span class="pl-s1">input</span>.<span class="pl-en">getMetrics</span>()) {
      <span class="pl-s1">newMetrics</span>[<span class="pl-s1">metricLookup</span>[<span class="pl-s1">j</span>]] = <span class="pl-s1">met</span>;
      <span class="pl-s1">j</span>++;
  }</pre></div>
<p dir="auto">modify the code of LongSumAggregator ,first check for not null,it work well.</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" static long combineValues(Object lhs, Object rhs) {
    if(lhs == null){
        lhs = 0L;
    }
    if(rhs == null){
        rhs = 0L;
    }
    return ((Number) lhs).longValue() + ((Number) rhs).longValue();
  }"><pre class="notranslate"> <span class="pl-k">static</span> <span class="pl-smi">long</span> <span class="pl-s1">combineValues</span>(<span class="pl-smi">Object</span> <span class="pl-s1">lhs</span>, <span class="pl-smi">Object</span> <span class="pl-s1">rhs</span>) {
    <span class="pl-k">if</span>(<span class="pl-s1">lhs</span> == <span class="pl-c1">null</span>){
        <span class="pl-s1">lhs</span> = <span class="pl-c1">0L</span>;
    }
    <span class="pl-k">if</span>(<span class="pl-s1">rhs</span> == <span class="pl-c1">null</span>){
        <span class="pl-s1">rhs</span> = <span class="pl-c1">0L</span>;
    }
    <span class="pl-k">return</span> ((<span class="pl-smi">Number</span>) <span class="pl-s1">lhs</span>).<span class="pl-en">longValue</span>() + ((<span class="pl-smi">Number</span>) <span class="pl-s1">rhs</span>).<span class="pl-en">longValue</span>();
  }</pre></div>
<p dir="auto">I found  that pull requst <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="126262873" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/2257" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/2257/hovercard" href="https://github.com/apache/druid/pull/2257">#2257</a> can solve NPE when adding metric.<br>
But I add following unit test on Build 0.9.1-rc1,still throw NPE</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="@Test
  public void testAddMetrics() throws IOException
  {
    IncrementalIndex index1 = IncrementalIndexTest.createIndex(new AggregatorFactory[]{
        new LongSumAggregatorFactory(&quot;A&quot;, &quot;A&quot;)
    });
    closer.closeLater(index1);
    long timestamp  = System.currentTimeMillis();
    index1.add(
            new MapBasedInputRow(
                timestamp,
                Arrays.asList(&quot;dim1&quot;, &quot;dim2&quot;),
                ImmutableMap.&lt;String, Object&gt;of(&quot;dim1&quot;, &quot;1&quot;, &quot;dim2&quot;, &quot;2&quot;,&quot;A&quot;,5)
            )
        );

    IncrementalIndex index2 = IncrementalIndexTest.createIndex(new AggregatorFactory[]{
        new LongSumAggregatorFactory(&quot;A&quot;, &quot;A&quot;),
        new LongSumAggregatorFactory(&quot;C&quot;, &quot;C&quot;)
    });

    index2.add(
            new MapBasedInputRow(
                timestamp,
                Arrays.asList(&quot;dim1&quot;, &quot;dim2&quot;),
                ImmutableMap.&lt;String, Object&gt;of(&quot;dim1&quot;, &quot;1&quot;, &quot;dim2&quot;, &quot;2&quot;,&quot;A&quot;,5,&quot;C&quot;,6)
            )
    );

    closer.closeLater(index2);

    Interval interval = new Interval(0, new DateTime().getMillis());
    RoaringBitmapFactory factory = new RoaringBitmapFactory();
    ArrayList&lt;IndexableAdapter&gt; toMerge = Lists.&lt;IndexableAdapter&gt;newArrayList(
        new IncrementalIndexAdapter(interval, index1, factory),
        new IncrementalIndexAdapter(interval, index2, factory)

    );

    final File tmpDirMerged = temporaryFolder.newFolder();

    File merged = INDEX_MERGER.merge(
        toMerge,
        new AggregatorFactory[]{
            new LongSumAggregatorFactory(&quot;A&quot;, &quot;A&quot;),
            new LongSumAggregatorFactory(&quot;C&quot;, &quot;C&quot;)
        },
        tmpDirMerged,
        indexSpec
    );
    final QueryableIndexStorageAdapter adapter = new QueryableIndexStorageAdapter(closer.closeLater(INDEX_IO.loadIndex(merged)));
    Assert.assertEquals(ImmutableSet.of(&quot;A&quot;,&quot;C&quot;), ImmutableSet.copyOf(adapter.getAvailableMetrics()));

  }"><pre class="notranslate"><span class="pl-c1">@</span><span class="pl-c1">Test</span>
  <span class="pl-k">public</span> <span class="pl-smi">void</span> <span class="pl-s1">testAddMetrics</span>() <span class="pl-k">throws</span> <span class="pl-s1">IOException</span>
  {
    <span class="pl-smi">IncrementalIndex</span> <span class="pl-s1">index1</span> = <span class="pl-smi">IncrementalIndexTest</span>.<span class="pl-en">createIndex</span>(<span class="pl-k">new</span> <span class="pl-smi">AggregatorFactory</span>[]{
        <span class="pl-k">new</span> <span class="pl-smi">LongSumAggregatorFactory</span>(<span class="pl-s">"A"</span>, <span class="pl-s">"A"</span>)
    });
    <span class="pl-s1">closer</span>.<span class="pl-en">closeLater</span>(<span class="pl-s1">index1</span>);
    <span class="pl-smi">long</span> <span class="pl-s1">timestamp</span>  = <span class="pl-smi">System</span>.<span class="pl-en">currentTimeMillis</span>();
    <span class="pl-s1">index1</span>.<span class="pl-en">add</span>(
            <span class="pl-k">new</span> <span class="pl-smi">MapBasedInputRow</span>(
                <span class="pl-s1">timestamp</span>,
                <span class="pl-smi">Arrays</span>.<span class="pl-en">asList</span>(<span class="pl-s">"dim1"</span>, <span class="pl-s">"dim2"</span>),
                <span class="pl-smi">ImmutableMap</span>.&lt;<span class="pl-smi">String</span>, <span class="pl-smi">Object</span>&gt;<span class="pl-en">of</span>(<span class="pl-s">"dim1"</span>, <span class="pl-s">"1"</span>, <span class="pl-s">"dim2"</span>, <span class="pl-s">"2"</span>,<span class="pl-s">"A"</span>,<span class="pl-c1">5</span>)
            )
        );

    <span class="pl-smi">IncrementalIndex</span> <span class="pl-s1">index2</span> = <span class="pl-smi">IncrementalIndexTest</span>.<span class="pl-en">createIndex</span>(<span class="pl-k">new</span> <span class="pl-smi">AggregatorFactory</span>[]{
        <span class="pl-k">new</span> <span class="pl-smi">LongSumAggregatorFactory</span>(<span class="pl-s">"A"</span>, <span class="pl-s">"A"</span>),
        <span class="pl-k">new</span> <span class="pl-smi">LongSumAggregatorFactory</span>(<span class="pl-s">"C"</span>, <span class="pl-s">"C"</span>)
    });

    <span class="pl-s1">index2</span>.<span class="pl-en">add</span>(
            <span class="pl-k">new</span> <span class="pl-smi">MapBasedInputRow</span>(
                <span class="pl-s1">timestamp</span>,
                <span class="pl-smi">Arrays</span>.<span class="pl-en">asList</span>(<span class="pl-s">"dim1"</span>, <span class="pl-s">"dim2"</span>),
                <span class="pl-smi">ImmutableMap</span>.&lt;<span class="pl-smi">String</span>, <span class="pl-smi">Object</span>&gt;<span class="pl-en">of</span>(<span class="pl-s">"dim1"</span>, <span class="pl-s">"1"</span>, <span class="pl-s">"dim2"</span>, <span class="pl-s">"2"</span>,<span class="pl-s">"A"</span>,<span class="pl-c1">5</span>,<span class="pl-s">"C"</span>,<span class="pl-c1">6</span>)
            )
    );

    <span class="pl-s1">closer</span>.<span class="pl-en">closeLater</span>(<span class="pl-s1">index2</span>);

    <span class="pl-smi">Interval</span> <span class="pl-s1">interval</span> = <span class="pl-k">new</span> <span class="pl-smi">Interval</span>(<span class="pl-c1">0</span>, <span class="pl-k">new</span> <span class="pl-smi">DateTime</span>().<span class="pl-en">getMillis</span>());
    <span class="pl-smi">RoaringBitmapFactory</span> <span class="pl-s1">factory</span> = <span class="pl-k">new</span> <span class="pl-smi">RoaringBitmapFactory</span>();
    <span class="pl-smi">ArrayList</span>&lt;<span class="pl-smi">IndexableAdapter</span>&gt; <span class="pl-s1">toMerge</span> = <span class="pl-smi">Lists</span>.&lt;<span class="pl-smi">IndexableAdapter</span>&gt;<span class="pl-en">newArrayList</span>(
        <span class="pl-k">new</span> <span class="pl-smi">IncrementalIndexAdapter</span>(<span class="pl-s1">interval</span>, <span class="pl-s1">index1</span>, <span class="pl-s1">factory</span>),
        <span class="pl-k">new</span> <span class="pl-smi">IncrementalIndexAdapter</span>(<span class="pl-s1">interval</span>, <span class="pl-s1">index2</span>, <span class="pl-s1">factory</span>)

    );

    <span class="pl-k">final</span> <span class="pl-smi">File</span> <span class="pl-s1">tmpDirMerged</span> = <span class="pl-s1">temporaryFolder</span>.<span class="pl-en">newFolder</span>();

    <span class="pl-smi">File</span> <span class="pl-s1">merged</span> = <span class="pl-smi">INDEX_MERGER</span>.<span class="pl-en">merge</span>(
        <span class="pl-s1">toMerge</span>,
        <span class="pl-k">new</span> <span class="pl-smi">AggregatorFactory</span>[]{
            <span class="pl-k">new</span> <span class="pl-smi">LongSumAggregatorFactory</span>(<span class="pl-s">"A"</span>, <span class="pl-s">"A"</span>),
            <span class="pl-k">new</span> <span class="pl-smi">LongSumAggregatorFactory</span>(<span class="pl-s">"C"</span>, <span class="pl-s">"C"</span>)
        },
        <span class="pl-s1">tmpDirMerged</span>,
        <span class="pl-s1">indexSpec</span>
    );
    <span class="pl-k">final</span> <span class="pl-smi">QueryableIndexStorageAdapter</span> <span class="pl-s1">adapter</span> = <span class="pl-k">new</span> <span class="pl-smi">QueryableIndexStorageAdapter</span>(<span class="pl-s1">closer</span>.<span class="pl-en">closeLater</span>(<span class="pl-smi">INDEX_IO</span>.<span class="pl-en">loadIndex</span>(<span class="pl-s1">merged</span>)));
    <span class="pl-smi">Assert</span>.<span class="pl-en">assertEquals</span>(<span class="pl-smi">ImmutableSet</span>.<span class="pl-en">of</span>(<span class="pl-s">"A"</span>,<span class="pl-s">"C"</span>), <span class="pl-smi">ImmutableSet</span>.<span class="pl-en">copyOf</span>(<span class="pl-s1">adapter</span>.<span class="pl-en">getAvailableMetrics</span>()));

  }</pre></div>
<p dir="auto">can reproduce NPE</p>