title: R: speeding up "group by" operations
body: <p>I have a simulation that has a huge aggregate and combine step right in the middle. I prototyped this process using plyr's ddply() function which works great for a huge percentage of my needs. But I need this aggregation step to be faster since I have to run 10K simulations. I'm already scaling the simulations in parallel but if this one step were faster I could greatly decrease the number of nodes I need. </p>
  
  <p>Here's a reasonable simplification of what I am trying to do:</p>
  
  <pre><code>library(Hmisc)
  
  # Set up some example data
  year &lt;-    sample(1970:2008, 1e6, rep=T)
  state &lt;-   sample(1:50, 1e6, rep=T)
  group1 &lt;-  sample(1:6, 1e6, rep=T)
  group2 &lt;-  sample(1:3, 1e6, rep=T)
  myFact &lt;-  rnorm(100, 15, 1e6)
  weights &lt;- rnorm(1e6)
  myDF &lt;- data.frame(year, state, group1, group2, myFact, weights)
  
  # this is the step I want to make faster
  system.time(aggregateDF &lt;- ddply(myDF, c("year", "state", "group1", "group2"),
                       function(df) wtd.mean(df$myFact, weights=df$weights)
                                   )
             )
  </code></pre>
  
  <p>All tips or suggestions are appreciated! </p>