title: Kafka stream app failing to fetch offsets for partition
body: <p>I created a kafka cluster with 3 brokers and following details:</p>
  
  <ol>
  <li>Created 3 topics, each one with replication factor=3 and partitions=2.</li>
  <li>Created 2 producers each one writing to one of the topics.</li>
  <li>Created a Streams application to process messages from 2 topics and write to the 3rd topic.</li>
  </ol>
  
  <p>It was all running fine till now but I suddenly started getting the following warning when starting the Streams application:</p>
  
  <pre><code>[WARN ] 2018-06-08 21:16:49.188 [Stream3-4f7403ad-aba6-4d34-885d-60114fc9fcff-StreamThread-1] org.apache.kafka.clients.consumer.internals.Fetcher [Consumer clientId=Stream3-4f7403ad-aba6-4d34-885d-60114fc9fcff-StreamThread-1-restore-consumer, groupId=] Attempt to fetch offsets for partition Stream3-KSTREAM-OUTEROTHER-0000000005-store-changelog-0 failed due to: Disk error when trying to access log file on the disk.
  </code></pre>
  
  <p>Due to this warning, Streams application is not processing anything from the 2 topics. </p>
  
  <p>I tried following things:</p>
  
  <ol>
  <li>Stopped all brokers, deleted kafka-logs directory for each broker and restarted the brokers. It didn't solve the issue.</li>
  <li>Stopped zookeeper and all brokers, deleted zookeeper logs as well as kafka-logs for each broker, restarted zookeeper and brokers and created the topics again. This too didn't solve the issue.</li>
  </ol>
  
  <p>I am not able to find anything related to this error on official docs or web. Does anyone have an idea of why am I getting this error suddenly?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>Out of 3 brokers, 2 brokers(broker-0 and broker-2) continously emit these logs:</p>
  
  <p>Broker-0 logs:</p>
  
  <pre><code>[2018-06-09 02:03:08,750] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial11_topic-1 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
  [2018-06-09 02:03:08,750] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial12_topic-0 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
  </code></pre>
  
  <p>Broker-2 logs:</p>
  
  <pre><code>[2018-06-09 02:04:46,889] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial11_topic-1 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
  [2018-06-09 02:04:46,889] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Retrying leaderEpoch request for partition initial12_topic-0 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
  </code></pre>
  
  <p>Broker-1 shows following logs:</p>
  
  <pre><code>[2018-06-09 01:21:26,689] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
  [2018-06-09 01:31:26,689] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
  [2018-06-09 01:39:44,667] ERROR [KafkaApi-1] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
  [2018-06-09 01:41:26,689] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
  </code></pre>
  
  <p>I again stopped zookeeper and brokers, deleted their logs and restarted. As soon as I create the topics again, I start getting the above logs. </p>
  
  <p>Topic details:</p>
  
  <pre><code>    [zk: localhost:2181(CONNECTED) 3] get /brokers/topics/initial11_topic
  {"version":1,"partitions":{"1":[1,0,2],"0":[0,2,1]}}
  cZxid = 0x53
  ctime = Sat Jun 09 01:25:42 EDT 2018
  mZxid = 0x53
  mtime = Sat Jun 09 01:25:42 EDT 2018
  pZxid = 0x54
  cversion = 1
  dataVersion = 0
  aclVersion = 0
  ephemeralOwner = 0x0
  dataLength = 52
  numChildren = 1
  [zk: localhost:2181(CONNECTED) 4] get /brokers/topics/initial12_topic
  {"version":1,"partitions":{"1":[2,1,0],"0":[1,0,2]}}
  cZxid = 0x61
  ctime = Sat Jun 09 01:25:47 EDT 2018
  mZxid = 0x61
  mtime = Sat Jun 09 01:25:47 EDT 2018
  pZxid = 0x62
  cversion = 1
  dataVersion = 0
  aclVersion = 0
  ephemeralOwner = 0x0
  dataLength = 52
  numChildren = 1
  [zk: localhost:2181(CONNECTED) 5] get /brokers/topics/final11_topic
  {"version":1,"partitions":{"1":[0,1,2],"0":[2,0,1]}}
  cZxid = 0x48
  ctime = Sat Jun 09 01:25:32 EDT 2018
  mZxid = 0x48
  mtime = Sat Jun 09 01:25:32 EDT 2018
  pZxid = 0x4a
  cversion = 1
  dataVersion = 0
  aclVersion = 0
  ephemeralOwner = 0x0
  dataLength = 52
  numChildren = 1
  </code></pre>
  
  <p>Any clue?</p>