<p dir="auto">When a Kafka index task successfully publishes some segments then it persists the latest consumed offsets info in the metadata store. Now if the supervisor is stopped or stops because of some reason and restarted at a later point and the earliest msg offset in Kafka is greater than persisted offset in the metadata (probably because earlier messages were dropped by kafka brokers or something else) then Kafka Index task will get <code class="notranslate">OffsetOutOfRangeException</code> and keep on retrying the same offset indefinitely.</p>
<p dir="auto">One way to solve this issue is to reset the consumer offset if the persisted offset is less than the earliest offset at the Kafka broker. Although I think in this case the persisted offset information would need to be delete from metadata store otherwise the new tasks will fail to publish segment because of consecutive offset check. Any thoughts ?</p>
<p dir="auto">The Kafka indexing service can get into a stuck state where it is trying to read the Kafka offset following the last one recorded in the dataSource metadata table but can't read it because Kafka's retention period has elapsed and that message is no longer available. We probably don't want to automatically jump to a valid Kafka offset since we would have missed events and would no longer have ingested each message exactly once, so currently we start throwing exceptions and the user needs to acknowledge what happened by removing the last committed offset from the dataSource table.</p>
<p dir="auto">It would be nice to include an API that will help them by either removing the dataSource table entry or setting it to a valid Kafka offset, but would still require a manual action/acknowledgment by the user.</p>