title: Understanding offset jumps in topic produced to by Kafka Streams
body: <p>This question is <em>similar</em> to <a href="https://stackoverflow.com/a/54637004">https://stackoverflow.com/a/54637004</a> but I am not 100% sure that it is the exact same situation - it presents slightly differently at least.</p>
  <p>I noticed that a topic that I am writing to using Kafka Streams, as in the above question, does not have a continuous value for the offsets in any given partition. This is <em>expected</em> when using exactly-once semantics, but the &quot;gaps&quot; seem a little confusing to me.</p>
  <p>The behaviour I'm seeing similar is the following:</p>
  <pre><code>$ kcat -C -b &lt;broker&gt; -t &lt;topic&gt; -o beginning -f '%o\n' -p 0
  0
  1
  2
  3
  4
  5
  7  &lt;-- Gap in offsets after 6 records
  8
  ...
  13
  14
  16 &lt;-- Gap in offsets after 8 records
  17
  ...
  24
  25
  27 &lt;-- Gap in offsets after 10 records
  ...
  </code></pre>
  <p>(note that kcat uses the librdkafka default, <code>READ_COMMITTED</code>, when consuming)</p>
  <p>As opposed to the picture in the above question, the &quot;commit marker offset&quot; (i.e. the gap) is at very different locations.</p>
  <p>I tried to check whether there might be any aborted messages in the topic, but did not find any when reading with <code>READ_UNCOMMITTED</code> isolation level, which leads me to believe that the gaps are indeed due to the commit marker.</p>
  <p>Am I understanding correctly that the different gap sizes between offsets is due to Kafka Streams producing different numbers of messages in a given transaction batch (through e.g. <code>max.message.bytes</code>)?</p>