title: LibrdKafkaError: Broker: Unknown member after running around about 2 hours randomly
body: <p>Right now, i want to implement <code>node-rdkafka</code> into our service, but i faced this error many times <code>Broker: Unknown member</code>.
  The same issue on github was <a href="https://github.com/confluentinc/confluent-kafka-dotnet/issues/1464" rel="nofollow noreferrer">https://github.com/confluentinc/confluent-kafka-dotnet/issues/1464</a>. they say our consumer using same group id to retry or delay. but i didn't find any retry and delay on my code.
  or <a href="https://github.com/confluentinc/confluent-kafka-python/issues/1004" rel="nofollow noreferrer">https://github.com/confluentinc/confluent-kafka-python/issues/1004</a>, but i have recheck all consumer group id and it was unique.</p>
  <p>The config of <code>node-rdkafka</code> producer as follows:</p>
  <pre><code>      this.producer = new Producer({
          &quot;client.id&quot;: this.cliendID,
          &quot;metadata.broker.list&quot;: this.brokerList,
          'compression.codec': &quot;lz4&quot;,
          'retry.backoff.ms': 200,
          'socket.keepalive.enable': true,
          'queue.buffering.max.messages': 100000,
          'queue.buffering.max.ms': 1000,
          'batch.num.messages': 1000000,
          &quot;transaction.timeout.ms&quot;: 2000,
          &quot;enable.idempotence&quot;: false,
          &quot;max.in.flight.requests.per.connection&quot;: 1,
          &quot;debug&quot;: this.debug,
          'dr_cb': true,
          &quot;retries&quot;: 0,
          &quot;log_cb&quot;: (_: any) =&gt; console.log(`log_cb =&gt;`, _),
          &quot;sasl.username&quot;: this.saslUsername,
          &quot;sasl.password&quot;: this.saslPassword,
          &quot;sasl.mechanism&quot;: this.saslMechanism,
          &quot;security.protocol&quot;: this.securityProtocol
        }, {
          &quot;acks&quot;: -1
        })
  </code></pre>
  <p>The config of <code>node-rdkafka</code> consumer as follows:</p>
  <pre><code>this.consumer = new KafkaConsumer({
          'group.id': this.groupID,
          'metadata.broker.list': this.brokerList,
          &quot;sasl.username&quot;: this.saslUsername,
          &quot;sasl.password&quot;: this.saslPassword,
          &quot;enable.auto.commit&quot;: false,
          &quot;auto.commit.interval.ms&quot;: 2000,
          &quot;session.timeout.ms&quot;: 45000,
          &quot;max.poll.interval.ms&quot;: 300000,
          &quot;heartbeat.interval.ms&quot;: 3000,
          &quot;api.version.request.timeout.ms&quot;: 10000,
          &quot;max.in.flight.requests.per.connection&quot;: 1,
          &quot;debug&quot;: this.debug,
          &quot;sasl.mechanism&quot;: this.saslMechanism,
          &quot;log.connection.close&quot;: true,
          &quot;log.queue&quot;: true,
          &quot;log_level&quot;: 7,
          &quot;log.thread.name&quot;: true,
          &quot;isolation.level&quot;: &quot;read_committed&quot;,
          &quot;ssl.ca.location&quot;: &quot;/etc/ssl/certs/&quot;,
          &quot;log_cb&quot;: (_: any) =&gt; console.log(`log_cb =&gt;`, _),
          &quot;security.protocol&quot;: this.securityProtocol
        }, {})
  
        await new Promise(resolve =&gt; {
  
          this.consumer?.connect()
          this.consumer?.on('ready', () =&gt; {
            try {
              this.consumer?.subscribe(subscriptions)
              this.consumer?.consume()
              console.log('[SUCCESS] Subscribe Event =&gt; all event')
            } catch (err) {
              console.log('[FAILED] Subscribe =&gt; all event')
              console.log(err)
            }
  
            resolve(this.consumer)
          }).on('data', async (data) =&gt; {
  
            this.topicFunctionMap[data.topic]({
              partition: data.partition,
              topic: data.topic,
              message: {
                key: data.key,
                offset: data.offset.toString(),
                size: data.size,
                value: data.value,
                timestamp: data.timestamp?.toString()
              }
            } as ISubsCallbackParam)
  
            this.consumer?.commitSync({
              topic: data.topic,
              offset: data.offset,
              partition: data.partition
            })
  
  
          })
        })
  </code></pre>
  <p>Using those configuration, the consumer is able to receive event but its not last for long. around 2hours more it randomly gives those error.
  I am not sure if it because manual commit or our function tooks long because i have tried both <code>async</code> and <code>sync</code> commit so the commitSync its not depend on our function.</p>
  <p>Let says it because the our function tooks long, and it make our cosumer kicked from the group. maybe its the suspect after i found additional error Broker: Specified group generation id is not valid</p>
  <p>source: <a href="https://github.com/confluentinc/confluent-kafka-dotnet/issues/1155" rel="nofollow noreferrer">https://github.com/confluentinc/confluent-kafka-dotnet/issues/1155</a></p>
  <p>Its says i need to increase the session time out, then i tried to increase it to <code>&quot;session.timeout.ms&quot;: 300000</code> or 5min, and the heartbeat <code>&quot;heartbeat.interval.ms&quot;:3000</code>, i found in github issue, that the heartbeat should less than = (timeout/3). so i think 3sec will fine.</p>
  <p>Using <code>&quot;session.timeout.ms&quot;: 300000</code> and <code>&quot;heartbeat.interval.ms&quot;:3000</code>
  the consumer is able to consume and last for long but the problems is:</p>
  <ul>
  <li>first time using those config, its fine running around 0-2sec to receive</li>
  <li>after a while, its received, but tooks 1-10sec to receive the message</li>
  </ul>
  <p>The detail errors:</p>
  <pre><code>received event =&gt; onCustomerServiceRegister
  [COMMIT_ERR]  LibrdKafkaError: Broker: Unknown member
      at Function.createLibrdkafkaError [as create] (/src/app/node_modules/node-rdkafka/lib/error.js:454:10)
      at KafkaConsumer.Client._errorWrap (/src/app/node_modules/node-rdkafka/lib/client.js:481:29)
      at KafkaConsumer.commitSync (/src/app/node_modules/node-rdkafka/lib/kafka-consumer.js:560:8)
      at KafkaRDConnect.&lt;anonymous&gt; (/src/app/dist/events/connectors/kafkaRD.js:240:110)
      at step (/src/app/dist/events/connectors/kafkaRD.js:53:23)
      at Object.next (/src/app/dist/events/connectors/kafkaRD.js:34:53)
      at /src/app/dist/events/connectors/kafkaRD.js:28:71
      at new Promise (&lt;anonymous&gt;)
      at __awaiter (/src/app/dist/events/connectors/kafkaRD.js:24:12)
      at KafkaConsumer.&lt;anonymous&gt; (/src/app/dist/events/connectors/kafkaRD.js:213:72)
      at KafkaConsumer.emit (node:events:376:20)
      at KafkaConsumer.EventEmitter.emit (node:domain:470:12)
      at /src/app/node_modules/node-rdkafka/lib/kafka-consumer.js:488:12 {
  
  </code></pre>