title: Kafka Consumer is getting few (not all) old messages (that was already processed earlier)
body: <p>We have topics with retention set as 7 days (168 hours). Messages are consumed in real-time as and when the producer sends the message. Everything is working as expected. However recently on a production server, Devops changed the time zone from PST to EST  accidentally as part of OS patch. </p>
  
  <p>After Kafka server restart, we saw few (not all of them, but random) old messages being consumed by the consumers. We asked Devops to change it back to PST and restart. Again the old messages re-appeared this weekend as well.</p>
  
  <p>We have not seen this problem in lower environments (Dev, QA, Stage etc).</p>
  
  <p>Kafka version: kafka_2.12-0.11.0.2</p>
  
  <p>Any help is highly appreciated.</p>
  
  <p>Adding more info... Recently our CentOS had a patch update and somehow, admins changed from PST timezone to EST and started Kafka servers... After that our consumers started seeing messages from offset 0. After debugging, I found the timezone change and admins changed back from EST to PST after 4 days. Our message producers were sending messages before and after timezone changes regularly. After timezone change from EST back to PST, Kafka servers were restarted and I am seeing the bellow warning.</p>
  
  <p>This log happened when we chnaged back from EST to PST : (server.log)
  [2018-06-13 18:36:34,430] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (/app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.index) has non-zero size but the last offset is 2076 which is no larger than the base offset 2076.}. deleting /app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.timeindex, /app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.index, and /app/kafka_2.12-0.11.0.2/data/__consumer_offsets-21/00000000000000002076.txnindex and rebuilding index... (kafka.log.Log)</p>
  
  <p>We restarted consumers after 3 days of timezone change back from EST to PST and started seeing consumer messages with offset 0 again.</p>