title: Apache Kafka on Confluent Cloud - Incoherent offsets in partitioned topic and consumer lag
body: <p>I am finding a strange behaviour when using Kafka on Confluent Cloud. I have created a topic with the default partitioning value: 6.</p>
  
  <p>My system consists of a Java Producer application that sends a message to that topic and a Kafka Streams application that reads from it and performs an operation per message.</p>
  
  <pre><code>-----------------------          --------            -----------
  | Kafka Java Producer |  ----&gt;  | topic | ----&gt;      | KStream |
  -----------------------          --------            -----------
  </code></pre>
  
  <p>At the moment I am starting only one instance of the Kafka Streams application, so the consumer group has one member.</p>
  
  <p>This is what I've observed:</p>
  
  <ol>
  <li>The producer sends a message and it is recorded in the event topic with <strong>offset 0</strong>:</li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/toQ50.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/toQ50.png" alt="enter image description here"></a></p>
  
  <ol start="2">
  <li>The message reaches the KStream, being processed correctly as I can see in the KStream log trace: </li>
  </ol>
  
  <p><strong>KStream</strong></p>
  
  <pre><code>events.foreach { key, value -&gt;
      logger.info("--------&gt; Processing TimeMetric {}", value)
      //Store in DB
  </code></pre>
  
  <p><strong>Log</strong></p>
  
  <p><code>[-StreamThread-1] uration$$EnhancerBySpringCGLIB$$e72e3f00 : --------&gt; Processing Event {"...</code></p>
  
  <ol start="3">
  <li>In the Confluent Cloud consumer lag I can see all consumer groups and their state. There's one for the KStream called <code>events-processor-19549050-d8b0-4b39...</code>. As stated before, this group has only one member (the only instance of the KStream). However, if shows that this group is behind one message in the partition 2. Besides, note that current offset seems to be 1 and end offset 2):</li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/goQol.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/goQol.png" alt="enter image description here"></a></p>
  
  <ol start="4">
  <li>If I send another message in the producer, it is recorded again in the topic but this time <strong>with offset 2 instead of 1</strong>:</li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/vpUXp.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vpUXp.png" alt="enter image description here"></a></p>
  
  <ol start="5">
  <li>The message reaches the KStream and is processed normally again: </li>
  </ol>
  
  <p><code>[-StreamThread-1] uration$$EnhancerBySpringCGLIB$$e72e3f00 : --------&gt; Processing Event {</code></p>
  
  <ol start="6">
  <li>And going back to the consumer lag of the consumer group it is still one message behind, still with some strange offsets (current 3, end 4):</li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/G9Mpa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/G9Mpa.png" alt="enter image description here"></a></p>
  
  <p>Although the processing seems to be fine, the state showed above doesn't make much sense. Can you explain the reasons why:</p>
  
  <ol>
  <li>the message offsets are incremented +2 instead of +1?</li>
  <li>the consumer group seems to be 1 message behind even though it processed the messages correctly?</li>
  </ol>