Imagine I am working on a C compiler. I have made some changes to the compiler's code, and am interested to know what effect these changes have on the performance of generated binaries.I have a benchmark program, written in C, which runs for a minute or so and outputs a megaFLOPS value. I can compile this benchmark using both the original and changed compiler.How should I run the resulting binaries in order to compare their performance? Should I simply run each binary multiple times and compare the mean megaFLOPS values? Or perhaps maximum megaFLOPS values?The above would allow me to state by what percentage one binary is faster/slower than the other. Is there any way to use the variance of the megaFLOPS values in order to quantify confidence in my conclusion? Ideally I would like to be able to say, for example, My changes make the benchmark run 4% faster (95% confidence).
Can I use statistical methods to compare benchmark performance figures?
performance;statistics;benchmarking