title: Which gets the measurements right, JMeter or Apache ab?
body: <p>I started writing some basic tests in JMeter and was surprised that the measurements are so different from those from Apache ab. </p>
  
  <p>I have a gigabit LAN connecting an Intel i7 server running Nginx and an i5 test machine running JMeter or ab. Initially, I am simply testing the out-of-the box Nginx home page response rate.</p>
  
  <pre><code>ab -c 1 -n 100 http://testserver.local/
  </code></pre>
  
  <p>gives </p>
  
  <pre><code>Document Path:          /
  Document Length:        151 bytes
  
  Concurrency Level:      1
  Time taken for tests:   0.078 seconds
  Complete requests:      100
  Failed requests:        0
  Write errors:           0
  Total transferred:      38400 bytes
  HTML transferred:       15100 bytes
  Requests per second:    1280.77 [#/sec] (mean)
  Time per request:       0.781 [ms] (mean)
  Time per request:       0.781 [ms] (mean, across all concurrent requests)
  Transfer rate:          480.29 [Kbytes/sec] received
  </code></pre>
  
  <p>This result is consistently reproducible, +/- a few percent.</p>
  
  <p><br/></p>
  
  <p>In JMeter, I have a 1-user 100-loop thread group containing:</p>
  
  <ul>
  <li>an HTTP header manager setting Accept-Encoding: gzip</li>
  <li>an HTTP Get / sampler</li>
  <li>a summary report listener</li>
  </ul>
  
  <p>With only 100 samples, this gives wildly inconsistent results each time I run it. But the most startling fact is that the throughput is reported as low as 40 requests per second (not 1280). The highest recorded rate was 1030, and this was achieved only when I increased to 10,000 samples.</p>
  
  <p>Am I right in thinking that JMeter is the wrong tool for simple load tests because its overheads are too high to allow accurate measurements?</p>