title: Kafka consumer groups suddenly stopped balancing messages among instances
body: <p>We have a microservice architecture communicated by Kafka on Confluent where each service is set in its own consumer group in order to balance message delivery between the multiple instances.</p>
  
  <p><strong>For example:</strong></p>
  
  <pre><code>SERVICE_A_INSTANCE_1 (CONSUMER_GROUP_A)
  SERVICE_A_INSTANCE_2 (CONSUMER_GROUP_A)
  SERVICE_A_INSTANCE_3 (CONSUMER_GROUP_A)
  
  SERVICE_B_INSTANCE_1 (CONSUMER_GROUP_B)
  SERVICE_B_INSTANCE_2 (CONSUMER_GROUP_B)
  </code></pre>
  
  <p>When a message is emitted it should only be consumed by one instance of each consumer group. </p>
  
  <p>This worked fine until two days ago. All of the sudden, each message is being delivered to all the instances, so each message is processed multiple times. Basically, the consumer-group stopped working and messages are not being distributed.</p>
  
  <p><strong>Important points:</strong></p>
  
  <ul>
  <li>We use Kafka paas in Confluent on GCP.</li>
  <li>We tested this in a different environment and everything worked as expected</li>
  <li>No changes have been made on our consumers</li>
  <li>No changes have been made on our part to the cluster (we cant know if Confluent changed something)</li>
  </ul>
  
  <p>We suspect it might be a problem on Confluent or an update that is not compatible with our current configuration. <a href="https://www.apache.org/dist/kafka/2.2.0/RELEASE_NOTES.html" rel="nofollow noreferrer">Kafka 2.2.0 was recently released</a> and it has some changes to consumer groups behavior.</p>
  
  <p>We are currently working on migrating to AWS MSK to see if the issue prevails. </p>
  
  <p>Any ideas on what could be causing this? </p>