<p dir="auto">It's possible to make segment merging more in-memory and reduce the time needed for segment merging by 50%, but it doesn't work (failing with OOM errors) when v8 segment format is build first, and then converted to v9.</p>
<p dir="auto">I don't see a way to keep the ability to merge segments in v8 format and make the improvement without massive code duplication (because v8 and v9 use the same classes on certain level).</p>
<p dir="auto">If we remove ability to build v8, on the contrary, some code could be removed. But not all v8-related, because segments in this format should be still readable.</p>
<h3 dir="auto">Affected Version</h3>
<p dir="auto">The master branch</p>
<h3 dir="auto">Description</h3>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="800230698" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/10843" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/10843/hovercard" href="https://github.com/apache/druid/pull/10843">#10843</a> added support for segment granularity for auto compaction. This change can make auto compaction to fail in finding candidate segments for compaction when those segments have mixed versions, especially when you change segment granularity from something small to something large. When you change segment granularity, auto compaction internally creates another timeline which is populated based on the new segment granularity. Here is a code snippet of <a href="https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/NewestSegmentFirstIterator.java#L135-L148">how we populate the new timeline</a>.</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="              DataSegment segmentsForCompact = segment.withShardSpec(new NumberedShardSpec(partitionNum, partitions));
              // PartitionHolder can only holds chunks of one partition space
              // However, partition in the new timeline (timelineWithConfiguredSegmentGranularity) can be hold multiple
              // partitions of the original timeline (when the new segmentGranularity is larger than the original
              // segmentGranularity). Hence, we group all the segments of the original timeline into intervals bucket
              // by the new configuredSegmentGranularity. We then convert each segment into a new partition space so that
              // there is no duplicate partitionNum across all segments of each new Interval. We will have to save the
              // original ShardSpec to convert the segment back when returning from the iterator.
              originalShardSpecs.put(new Pair&lt;&gt;(interval, segmentsForCompact.getId()), segment.getShardSpec());
              timelineWithConfiguredSegmentGranularity.add(
                  interval,
                  segmentsForCompact.getVersion(),
                  NumberedPartitionChunk.make(partitionNum, partitions, segmentsForCompact)
              );"><pre class="notranslate">              <span class="pl-smi">DataSegment</span> <span class="pl-s1">segmentsForCompact</span> = <span class="pl-s1">segment</span>.<span class="pl-en">withShardSpec</span>(<span class="pl-k">new</span> <span class="pl-smi">NumberedShardSpec</span>(<span class="pl-s1">partitionNum</span>, <span class="pl-s1">partitions</span>));
              <span class="pl-c">// PartitionHolder can only holds chunks of one partition space</span>
              <span class="pl-c">// However, partition in the new timeline (timelineWithConfiguredSegmentGranularity) can be hold multiple</span>
              <span class="pl-c">// partitions of the original timeline (when the new segmentGranularity is larger than the original</span>
              <span class="pl-c">// segmentGranularity). Hence, we group all the segments of the original timeline into intervals bucket</span>
              <span class="pl-c">// by the new configuredSegmentGranularity. We then convert each segment into a new partition space so that</span>
              <span class="pl-c">// there is no duplicate partitionNum across all segments of each new Interval. We will have to save the</span>
              <span class="pl-c">// original ShardSpec to convert the segment back when returning from the iterator.</span>
              <span class="pl-s1">originalShardSpecs</span>.<span class="pl-en">put</span>(<span class="pl-k">new</span> <span class="pl-smi">Pair</span>&lt;&gt;(<span class="pl-s1">interval</span>, <span class="pl-s1">segmentsForCompact</span>.<span class="pl-en">getId</span>()), <span class="pl-s1">segment</span>.<span class="pl-en">getShardSpec</span>());
              <span class="pl-s1">timelineWithConfiguredSegmentGranularity</span>.<span class="pl-en">add</span>(
                  <span class="pl-s1">interval</span>,
                  <span class="pl-s1">segmentsForCompact</span>.<span class="pl-en">getVersion</span>(),
                  <span class="pl-smi">NumberedPartitionChunk</span>.<span class="pl-en">make</span>(<span class="pl-s1">partitionNum</span>, <span class="pl-s1">partitions</span>, <span class="pl-s1">segmentsForCompact</span>)
              );</pre></div>
<p dir="auto">As shown in the snippet, we use the segment version directly when populating the new timeline. Since the <code class="notranslate">interval</code> in the snippet is a time chunk based on new segment granularity, those segments of mixed versions can be added into the same time chunk in the new timeline. Finally, we replace the shardSpec of those segments with a new one that has <code class="notranslate">partitions</code> of the number of segments in the new time chunk. As a result, all those segments of mixed versions will not be visible since there will be always less number of non-overshadowed segments than <code class="notranslate">partitions</code> in that time chunk.</p>