title: How do you reason about fluctuations in benchmarking data?
body: <p>Suppose you're trying to optimize a function and using some benchmarking framework (like Google Benchmark) for measurement. You run the benchmarks on the original function 3 times and see average wall clock time/CPU times of 100 ms, 110 ms, 90 ms. Then you run the benchmarks on the &quot;optimized&quot; function 3 times and see 80 ms, 95 ms, 105 ms. (I made these numbers up). Do you conclude that your optimizations were successful?</p>
  <p>Another problem I often run into is that I'll go do something else and run the benchmarks later in the day and get numbers that are further away than the delta between the original and optimized earlier in the day (say, 80 ms, 85 ms, 75 ms for the original function).</p>
  <p>I know there are statistical methods to determine whether the improvement is &quot;significant&quot;. Do software engineers actually use these formal calculations in practice?</p>
  <p>I'm looking for some kind of process to follow when optimizing code.</p>