title: how to get count of unread messages in a Kafka topic for a certain group
body: <p>I know, Kafka is meant to be dealt with like an infinite stream of events, and getting the remaining messages count isn't a built-in functionality. but I have to somehow <em><strong>monitor</strong></em> how my consumer processes are doing and if I'm providing enough resources for them.
  overall scenario is basic Kafka usage, a couple of producers on different servers insert into a topic, consumers in <code>group_a</code> read messages, perform some AI stuff and insert into another topic for further processing.
  rate of incoming messages is in no way constant or predictable, so I need to have a measure of checking if my consumers are lagging behind (let's say if <code>group_a</code> has more than 1000 <em>unread</em> messages in my input topic).</p>
  <p>considering I have full control over Kafka setup, as well as consumer and producer code, what are my options here ?</p>
  <ul>
  <li>I assume reading the whole topic without any heavy processing and counting messages is neither clean nor efficient (?)</li>
  <li>if there was only a single pair of producer/consumer I could count the number of produced and consumed messages and calculate the remaining ones, but I'm running a multi-server setup and it's not quite feasible. is it possible to use the Kafka itself as a shared data store and keep count of all the produced and consumed messages?</li>
  <li>to avoid the <em><strong>XY problem</strong></em> is counting the number of unread messages the right way to know if my consumers are given &quot;enough&quot; resources?</li>
  </ul>
  <p><em><strong>EDIT:
  if you need to implement this in python yourself, I posted my solution as an answer <a href="https://stackoverflow.com/a/73670700/9940814">below</a></strong></em></p>