Title: Kafka broker request queue spikes, causing Streams timeout exception
Tags: apache-kafka;jmx;apache-kafka-streams
Question: I've been monitoring the metrics on a Kafka cluster of 4 machines. I have an input application writing messages into Kafka and a couple of Kafka Streams applications processing those messages and writing them back into a new Kafka topic partitioned by a geolocation variable.
The cluster will run without any problems for an indeterminate amount of time (usually two or three days), with nothing suspicious being reported in the metrics, then out of nowhere the metric kafka.network:type=RequestChannel,name=RequestQueueSize will spike up from a max value of no more than 10 requests to 50 or 60 requests, but only on a single broker. This ultimately causes the producer request queue in Kafka Streams to build up and timeout within a matter of minutes (at the moment I'm not replicating topics).
Furthermore, if I restart the Streams app, the broker request queue quickly builds up again.
It looks like it involves particular requests but not all of them based on a high 99th percentile for
kafka.network:type=RequestMetrics,name=RequestQueueTimeMs (on the order of 2 seconds) but a low mean (on the order of .3 milliseconds).
CPU usage is normal, i.e. not hitting against a hard limit.
What are some reasons the broker might become unhealthy in this way? Are there additional metrics that I should be looking at?

Here is the accepted answer: If you're experiencing a sudden drop in performance or getting timeouts with CPU to spare, you're probably dealing with an IO problem.
One of the best metrics to look at is kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs. If you see your log flush rate or log flush latency increasing that means that Kafka is having a problem writing to disk.
In our case, our page cache was being flushed too frequently, causing our write iops to spike while our average io request size fell. Since we were using EBS instances with burst balances, the repeated writes were draining our burst bucket and causing our request queue to build up.


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2017:00:12/07/17', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<apache-kafka><jmx><apache-kafka-streams>', 'Title': 'Kafka broker request queue spikes, causing Streams timeout exception', 'ViewCount': 1991, 'id': 47685901, 'user_AboutMe': '', 'user_AccountId': 3945996.0, 'user_DisplayName': 'kellanburket', 'user_Reputation': 11482.0, 'user_Views': 420.0, 'user_id': 3258600.0}}