title: Clean comparison of the performence of different approaches with a same goal in R?
body: <p>In <code>R</code>, how can I cleanly compare different solutions to a same problem, being "fair" between each of them?
  Could running a resource-consuming solution before others alter the performences of the latter ones? 
  How could one 'clean' the state of the machine between each test?</p>
  
  <hr>
  
  <p>Suppose I want to compute the mean over columns of a matrix, I could do it the easy or the complicated way:</p>
  
  <pre><code>set.seed(9)
  N = 1e7
  ncol = 1e3
  myT = matrix(runif(N), ncol = ncol)
  
  func1 &lt;- function(mat) {
    colMeans(mat)
  }
  
  func2 &lt;- function(mat) {
    apply(mat, 2, function(x) sum(x)/length(x))
  }
  
  func3 &lt;- function(mat) {
    nrows = c()
    for (i in 1:nrow(mat)) {
      nrows = c(nrows, 1) # yes, this is very stupid ;-)
    }
    colSums(mat) / sum(nrows)
  }
  
  
  system.time( replicate(1, t1 &lt;- func1(myT)))
  # user  system elapsed 
  # 0.012   0.000   0.011 
  system.time( replicate(1, t2 &lt;- func2(myT)))
  # user  system elapsed 
  # 0.136   0.036   0.170
  system.time( replicate(1, t3 &lt;- func3(myT)))
  # user  system elapsed 
  # 0.140   0.032   0.170
  </code></pre>
  
  <p>Running several times the <code>system.time()</code> execution can give different results for a same test (possibly altering the conclusions). I noticed it was especially the case for more complicated, resource-sonsuming solutions, while the cleanest ones tend to have a more consistent execution time - what is the reason for this? How to avoid big changes between executions of the same expression, and how to prevent them to interfere with each other?</p>
  
  <p>Is a call to <code>gc()</code> between tests useful, and is it enough?</p>
  
  <p>I also know about the <code>microbenchmark</code> package, but I am looking for something more 'manual' in order to understand what happens.</p>
  
  <p>I am working with <code>RStudio</code>, in case it matters...</p>