title: Kafka broker unable to restart due to corrupt file
body: <p>How do I recover from a corrupt file in Kafka?</p>
  
  <p>We are running a three node cluster with replication factor of 2 and ISR=1. Recently we had near-simultaneous failure of all brokers are the same time. This resulted in a situation where broker id 102 is down while the other two brokers recovered. Unfortunately at least one partition of one topic had 102 as the leader with isr of only 102 as well. This means that the other brokers are missing some (unknown) amount of data from this partition and that they thus refuse to receive/send data from this topic.</p>
  
  <p>Since I would like to recover my cluster and my data I am trying to restart broker 102. But it fails on some unknown file with this mesage</p>
  
  <p><code>
  [2018-07-18 14:44:44,806] ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.KafkaException: java.io.EOFException: Failed to read `log header` from file channel `sun.nio.
  ch.FileChannelImpl@375a9d12`. Expected to read 17 bytes, but reached end of file after reading 0 bytes. Started read from position 2147483631. (kafka.log.LogManager)
  [2018-07-18 14:44:44,809] ERROR [KafkaServer id=102] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
  org.apache.kafka.common.KafkaException: java.io.EOFException: Failed to read `log header` from file channel `sun.nio.ch.FileChannelImpl@375a9d12`. Expected to read 17 bytes, but reached end of file after reading
   0 bytes. Started read from position 2147483631.
  </code></p>
  
  <p>Unfortunately this does not tell me which file is broken. I have repeatedly tried restarting broker 102 in the hope that all the reindexing it is doing would recover the files somehow, but no luck.</p>
  
  <p>My guess is that the offending file is <strong>not</strong> from the partition for which 102 is the dead leader. So I am thinking</p>
  
  <p>a) Can I delete all the log files on 102 for the partitions where 102 is not the leader and when it comes back online it will simply resync without problems?</p>
  
  <p>b) Can I make 102 restart if I somehow locate the right file and remove it?</p>
  
  <p>c) is there any way of figuring out which file Kafka chokes on?</p>