title: Kafka broker hangs for unknown reason
body: <p>I'm using a single node setup of Kafka 2.0.0 and experiencing weird hangs in specific times during its life-cycle.</p>
  
  <p>I have a consumer connected 24/7 and at some point it drops throwing these error messages:</p>
  
  <pre><code>2018/11/27 22:34:33 Consumer error: kafkaa:9094/1001: 1 request(s) timed out: disconnect (&lt;nil&gt;)
  %3|1543358073.459|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: 1/1 brokers are down: Local: All broker connections are down
  runtime stack:
  fatal error: unexpected signal during runtime execution
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x7fb3c5fdedb4]
  
  runtime.throw(0x9749e0, 0x2a)
  runtime.sigpanic()
  goroutine 1 [syscall]:
      /usr/local/go/src/runtime/signal_unix.go:374 +0x2f2
  </code></pre>
  
  <p>Although nothing is being done on the application side, the consumer is getting disconnected unexpectedly.</p>
  
  <p>I traced back the Kafka logs to the times where the disconnections start occurring:</p>
  
  <pre><code>[2018-11-27 22:30:37,905] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
  [2018-11-27 22:33:36,037] TRACE [Controller id=1001] Leader imbalance ratio for broker 1001 is 0.0 (kafka.controller.KafkaController)
  [2018-11-27 22:33:36,037] TRACE [Controller id=1001] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
  [2018-11-27 22:33:36,037] DEBUG [Controller id=1001] Topics not in preferred replica for broker 1001 Map() (kafka.controller.KafkaController)
  [2018-11-27 22:33:36,037] DEBUG [Controller id=1001] Preferred replicas by broker Map(1001 -&gt; Map(__consumer_offsets-22 -&gt; Vector(1001), __consumer_offsets-30 -&gt; Vector(1001), __consumer_offsets-8 -&gt; Vector(1001), __consumer_offsets-21 -&gt; Vector(1001), __consumer_offsets-4 -&gt; Vector(1001), __consumer_offsets-27 -&gt; Vector(1001), __consumer_offsets-7 -&gt; Vector(1001), pcap-input-0 -&gt; Vector(1001), __consumer_offsets-9 -&gt; Vector(1001), __consumer_offsets-46 -&gt; Vector(1001), __consumer_offsets-25 -&gt; Vector(1001), __consumer_offsets-35 -&gt; Vector(1001), __consumer_offsets-41 -&gt; Vector(1001), __consumer_offsets-33 -&gt; Vector(1001), __consumer_offsets-23 -&gt; Vector(1001), __consumer_offsets-49 -&gt; Vector(1001), _schemas-0 -&gt; Vector(1001), pcap-output-0 -&gt; Vector(1001), __consumer_offsets-47 -&gt; Vector(1001), __consumer_offsets-16 -&gt; Vector(1001), __consumer_offsets-28 -&gt; Vector(1001), __consumer_offsets-31 -&gt; Vector(1001), __consumer_offsets-36 -&gt; Vector(1001), __consumer_offsets-42 -&gt; Vector(1001), __consumer_offsets-3 -&gt; Vector(1001), __consumer_offsets-18 -&gt; Vector(1001), __consumer_offsets-37 -&gt; Vector(1001), __consumer_offsets-15 -&gt; Vector(1001), __consumer_offsets-24 -&gt; Vector(1001), pcap-input-error-0 -&gt; Vector(1001), __consumer_offsets-38 -&gt; Vector(1001), __consumer_offsets-17 -&gt; Vector(1001), __consumer_offsets-48 -&gt; Vector(1001), __confluent.support.metrics-0 -&gt; Vector(1001), __consumer_offsets-19 -&gt; Vector(1001), __consumer_offsets-11 -&gt; Vector(1001), __consumer_offsets-13 -&gt; Vector(1001), __consumer_offsets-2 -&gt; Vector(1001), __consumer_offsets-43 -&gt; Vector(1001), __consumer_offsets-6 -&gt; Vector(1001), __consumer_offsets-14 -&gt; Vector(1001), __consumer_offsets-20 -&gt; Vector(1001), __consumer_offsets-0 -&gt; Vector(1001), __consumer_offsets-44 -&gt; Vector(1001), pcaps-output-failures-memsql-0 -&gt; Vector(1001), __consumer_offsets-39 -&gt; Vector(1001), __consumer_offsets-12 -&gt; Vector(1001), __consumer_offsets-45 -&gt; Vector(1001), __consumer_offsets-1 -&gt; Vector(1001), __consumer_offsets-5 -&gt; Vector(1001), __consumer_offsets-26 -&gt; Vector(1001), __consumer_offsets-29 -&gt; Vector(1001), __consumer_offsets-34 -&gt; Vector(1001), __consumer_offsets-10 -&gt; Vector(1001), pcaps-output-failures-elastic-0 -&gt; Vector(1001), __consumer_offsets-32 -&gt; Vector(1001), __consumer_offsets-40 -&gt; Vector(1001))) (kafka.controller.KafkaController)
  [2018-11-27 22:48:10,422] WARN Attempting to send response via channel for which there is no open connection, connection id 172.17.0.28:9094-172.17.0.27:40266-160 (kafka.network.Processor)
  [2018-11-27 22:48:10,422] WARN Client session timed out, have not heard from server in 849630ms for sessionid 0x10001e9d5c00006 (org.apache.zookeeper.ClientCnxn)
  </code></pre>
  
  <p>Resource metrics show the broker spikes up to 100% CPU at this point (22:34 to be exact).</p>
  
  <p>What is the broker doing here? how does this justify a 100% CPU consumption?</p>