title: Kafka partition lag differences between machines
body: <p>I have a service that reports on Kafka lag values out on a Kubernetes pod -- it adds up all the lag values from the partitions on a topic. Failing to find an API to handle this, the Java code makes an outside process call to kafka-consumer-groups (found as part of kafka_2.12-2.2.0) and then takes the output and tallies the "LAG" column. This value is monitored and alerted on if it begins to get too high.</p>
  
  <p>We have noticed that, after a while, the lag value jumps up and re-stabilizes at this higher amount. However, if we run kafka-consumer-groups from a laptop command line, the lag values are more in line with the original, lower, expected lag values. If we go to a command line on our pod running the service and execute kafka-consumer-groups there, we see the same, higher lag values that are being monitored. If we restart the pod, our monitored lag values again return to the expected, lower amounts.</p>
  
  <p>What could explain the discrepancy? And more importantly, how can we prevent/fix this so that we don't get woken up at 2am by monitor alerts? If someone can point me to an API call that could simplify the retrieval of lag value (and possibly fix the inflated amount issue in the process!), that would be great. But mostly, fixing the issue of inflated values is what I'm concerned about.</p>