title: Fast ANOVA computation in R
body: <p>I have a dataframe with the following dimensions:</p>
  
  <pre><code>dim(b)  
  [1]    974 433685
  </code></pre>
  
  <p>The columns represent variables that I want to run ANOVAs on (i.e., I want to run 433,685 ANOVAs). Sample size is 974. The last column is the 'group' variable. </p>
  
  <p>I've come up with 3 different methods, but all are too slow due to the number of tests. </p>
  
  <p>First, let's generate a small practice dataset to play with:</p>
  
  <pre><code>dat = as.data.frame(matrix(runif(10000*500), ncol = 10000, nrow = 500))
  dat$group = rep(letters[1:10], 5000)
  </code></pre>
  
  <p><strong>Method 1 (based on <a href="https://stackoverflow.com/questions/6621188/anova-for-loop-to-apply-function/6626740#6626740">'sapply'</a>):</strong></p>
  
  <pre><code>system.time(sapply(dat[,-length(dat)], function(x) aov(x~group, data=dat) ))
  
     user  system elapsed 
   143.76    0.33  151.79 
  </code></pre>
  
  <p><strong>Methods 2 (based on 'mclapply' from 'parallel' package):</strong></p>
  
  <pre><code>library(parallel)
  options(mc.cores=3)
  system.time(mclapply(dat[,-length(dat)], function(x) aov(x~group, data=dat) ))
  
     user  system elapsed 
   141.76    0.21  142.58 
  </code></pre>
  
  <p><strong>Methods 3 (based on <a href="https://stackoverflow.com/questions/26889240/looping-many-one-sided-anova-in-r/26890713#26890713">'cbind'-ing the LHS</a>):</strong></p>
  
  <pre><code>formula = as.formula( paste0("cbind(", paste(names(dat)[-length(dat)],collapse=","), ")~group") ) 
  system.time(aov(formula, data=dat))
  
    user  system elapsed 
    10.00    0.22   10.25 
  </code></pre>
  
  <p>In the practice dataset, Method 3 is a clear winner.  However, when I do this on my actual data, computing on just 10 (of 433,685) columns using Method 3 takes this long:</p>
  
  <pre><code>   user  system elapsed
  119.028   5.430 124.414
  </code></pre>
  
  <p>Not sure why it takes substantially longer on my actual data. I have access to a Linux cluster with upwards of 16 cores and 72GB of RAM. </p>
  
  <p>Is there any way to compute this faster? </p>