title: Kafka Consumers Reading From Disk Vs Cache
body: <p>I have configured a two node Kafka cluster on AWS, and am currently testing its performance attributes. </p>
  
  <p>I used the kafka-consumer-perf-test.sh to read 50 million messages from a Kafka topic using a single thread. </p>
  
  <p>I observed the following while testing consumer throughput.</p>
  
  <p><strong>Observation 1</strong></p>
  
  <p>Single consumer on a m4.large EC2 instance - Read throughput 40.2MB/sec</p>
  
  <p>Three consumers on three seperate m4.large EC2 instances - Individual read throughput - 40.25MB /sec.</p>
  
  <p>No disk reads or writes were reported on the Kafka broker side ( running on two seperate m4.2xlarge EC2 instances backed by 2 EBS volumes )</p>
  
  <p>I ran the tool again after some time. </p>
  
  <p><strong>Observation 2</strong></p>
  
  <p>Three consumers on three seperate m4.large EC2 instances - Individual read throughput dropped to - 34.25MB /sec.</p>
  
  <p>In this case I observed a considerable rate of disk reads in Kafka broker nodes.</p>
  
  <p>I would appreciate it if you could help me clarify the following.</p>
  
  <ol>
  <li><p>In observation 1, since I did not observe any disk reads, could all data have been fetched from the Memory ( where it is cached ) ?. </p></li>
  <li><p>In observation 2, I assume performance dropped due to the disk reads. Although it read from the disk, still enough free memory was available as revealed by nmon reports. </p></li>
  </ol>
  
  <p>What could have been the reason for consumers to read from disk instead of memory? For how long does data produced by producers stay in the cache? </p>
  
  <ol start="3">
  <li>I assume maximum 40MB/s read throughput is due to the network bandwidth limitation for m4.large EC2 instance. Is this assumption correct?</li>
  </ol>
  
  <p>Thank you.</p>