title: How does the offset for a topic work in Kafka (Kafka_net)
body: <p>I have a basic producer app and a consumer app. if I run both and have both start consuming on their respective topics, I have a great working system. My thought was that if I started the producer and sent a message that I would be able to then start the consumer and have it pick up that message. I was wrong.</p>
  <p>Unless both are up and running, I lose messages (or they do not get consumed).</p>
  <p>my consumer app looks like this for comsuming...</p>
  <pre><code>Uri uri = new Uri(&quot;http://localhost:9092&quot;);
  
  KafkaOptions options = new KafkaOptions(uri);                
  BrokerRouter brokerRouter = new BrokerRouter(options);
  Consumer consumer = new Consumer(new ConsumerOptions(receiveTopic, brokerRouter));
               
  </code></pre>
  <hr />
  <pre><code>List&lt;OffsetResponse&gt; offset = consumer.GetTopicOffsetAsync(receiveTopic, 100000).Result;
  
  IEnumerable&lt;OffsetPosition&gt; t = from x in offset select new OffsetPosition(x.PartitionId, x.Offsets.Max());
  consumer.SetOffsetPosition(t.ToArray());
  IEnumerable&lt;KafkaNet.Protocol.Message&gt; msgs = consumer.Consume();
  </code></pre>
  <hr />
  <pre><code>foreach (KafkaNet.Protocol.Message msg in msgs)
  {
      do some stuff here based on the message received
  }
  </code></pre>
  <p>unless I have the code between the lines, it starts at the beginning every time I start the application.
  What is the proper way to manage topic offsets so messages are consumed after a disconnect happens?</p>
  <p>If I run</p>
  <pre><code>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message-reply-XXX consumer-property fetch-size=40000000 --from-beginning 
  </code></pre>
  <p>I can see the messages, but when I connect my application to that topic, the consumer.Consume() does not pick up the messages it has not already seen. I have tried this with and without runing the above bat file to see if that makes any difference. When I look at the consumer.SetOffsetPosition(t.ToArray()) call (t specifically) it shows that the offset is the count of all messages for the topic.</p>
  <p>Please help,</p>