title: How to minimize the latency involved in kafka messaging framework?
body: <p><strong>Scenario</strong>: I have a low-volume topic (~150msgs/sec) for which we would like to have a
  low propagation delay from producer to consumer.</p>
  
  <p>I added a time stamp from a producer and read it at consumer to record the propagation delay, with default configurations the msg (of 20 bytes) showed a propagation delay of 1960ms to 1230ms. No network delay is involved since, I tried on a 1 producer and 1 simple consumer on the same machine.</p>
  
  <p>When I have tried adjusting the topic flush interval to 20ms, it drops
  to 1100ms to 980ms. Then I tried adjusting the consumers <code>"fetcher.backoff.ms"</code> to 10ms, it dropped to 1070ms - 860ms.</p>
  
  <p><strong>Issue</strong>: For a 20 bytes of a msg, I would like to have a propagation delay as low as possible and ~950ms is a higher figure. </p>
  
  <p><strong>Question</strong>: Anything I am missing out in configuration?
  I do welcome comments, delay which you got as minimum.</p>
  
  <p><strong>Assumption</strong>: The Kafka system involves the disk I/O before the consumer get the msg from the producer and this goes with the hard disk RPM and so on..
  <hr><strong>Update</strong>:
  Tried to tune the <strong>Log Flush Policy</strong> for Durability &amp; Latency.<br/>Following is the configuration:</p>
  
  <pre><code># The number of messages to accept before forcing a flush of data to disk
  log.flush.interval=10
  # The maximum amount of time a message can sit in a log before we force a flush
  log.default.flush.interval.ms=100
  # The interval (in ms) at which logs are checked to see if they need to be 
  # flushed to disk.
  log.default.flush.scheduler.interval.ms=100
  </code></pre>
  
  <p>For the same msg of 20 bytes, the delay was 740ms -880ms. </p>
  
  <p>The following statements are made clear in the configuration itself.<br/>
  <strong>There are a few important trade-offs:</strong><br/></p>
  
  <ol>
  <li><strong>Durability</strong>: Unflushed data is at greater risk of loss in the event of a crash.<br/></li>
  <li><strong>Latency</strong>: Data is not made available to consumers until it is flushed (which adds latency).<br/></li>
  <li><strong>Throughput</strong>: The flush is generally the most expensive operation.<br/></li>
  </ol>
  
  <p>So, I believe there is no way to come down to a mark of 150ms - 250ms. (without hardware upgrade) .</p>