<p dir="auto">I've been using Druid to ingest data from Kafka. The messages in Kafka are thrift-serialized versions of custom records. In order to parse these serialized messages, I've written a custom Druid extension, which extends the ByteBufferInputRowParser class in Druid.</p>
<p dir="auto">This has been working fine. However, one issue I've encountered is that the parse(T input) method of  ByteBufferInputRowParser lets me convert one input to only on Druid row. I</p>
<p dir="auto"><strong>Question:</strong><br>
Is there a way for me to map a given input into multiple InputRow objects? Each of my Kafka messages has a list of items and I need to store each item as a separate Druid row.</p>
<p dir="auto">It would benefit some users if kafka-indexing service supported the capability to let users put multiple druid InputRow inside single kafka record.<br>
This allows users to do batching while still using kafka sync producer which allows only one kafka record at a time.</p>
<p dir="auto">I would imagine adding following method to <code class="notranslate">InputRowParser</code> interface.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  default List&lt;InputRow&gt; parseBatch(T input)
  {
    return ImmutableList.of(parse(input));
  }"><pre class="notranslate"><code class="notranslate">  default List&lt;InputRow&gt; parseBatch(T input)
  {
    return ImmutableList.of(parse(input));
  }
</code></pre></div>
<p dir="auto">current <code class="notranslate">InputRow parse(T input)</code> method would be deprecated and all of druid code would be adjusted to use <code class="notranslate">parseBatch(input)</code> instead.</p>
<p dir="auto">kafka-indexing service will need to persist &lt;offset,row-number-in-record&gt; pair instead of just offset to support exactly once. I believe, this will end up being the biggest change really.</p>
<p dir="auto">This approach would add batching support in general and not just for kafka-indexing .</p>