title: Kafka consumer leaves group after failing to heartbeat
body: <p>We are facing an issue with a set of kafka-consumers. Whenever there is activity on the kafka-cluster like rebooting the brokers(rolling restarts) or reboot of VM's running the broker, our kafka consumers <code>LeaveGroup</code> after failing to heartbeat. The below logs repeat exactly for one minute and correspond to the <code>commitSync</code> call being done in the application code as a part of consuming messages from the topic</p>
  <pre><code>[Consumer clientId=consumer-13, groupId=delay-group-PostEnrollmentAppManagement] Offset commit failed on partition &lt;topic-name&gt; at offset 455700: The coordinator is loading and hence can't process requests.
  </code></pre>
  <p>This interval corresponds to the default time for which retries are done in the Kafka consumer client <code>commitSync</code> java API.</p>
  <p>Post this there are no logs for the next 5 minutes.</p>
  <p>Thereafter I see the following</p>
  <pre><code>[Consumer clientId=consumer-13, groupId=delay-group-PostEnrollmentAppManagement] Attempt to heartbeat failed since group is rebalancing
  [Consumer clientId=consumer-13, groupId=delay-group-PostEnrollmentAppManagement] Member consumer-13-837563e4-49e9-4bd1-aee4-cb21263e176a sending LeaveGroup request to coordinator &lt;broker-host-name&gt; (id: 2147483646 rack: null)
  [Consumer clientId=consumer-13, groupId=delay-group-PostEnrollmentAppManagement] This member will leave the group because consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
  </code></pre>
  <p>After this the messages pile up in the topic and we notice the lag obviously increasing with no consumers. We restart the app hosting the consumers to start consumption again.</p>
  <p>What can we do to avoid this? Should anything be done on the application consumer side to handle this?</p>
  <p>Note : For this particular consumer we have used the <a href="https://kafka.apache.org/26/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html" rel="nofollow noreferrer">apache kafka client</a> library. We normally use the <code>spring-kafka</code> library to build our consumers. We used the apache one as we wanted to use <code>pause</code> and <code>resume</code> functions of the kafka consumers which are not supported in the version of the <code>spring-kafka</code> we use</p>