title: Big data: generalized linear mixed-effects models
body: <p>I'm looking for suggestions for a strategy of fitting <em>generalized linear mixed-effects models</em> for a relative large data-set.</p>
  <p>Consider I have data on  8 milllion US basketball passes on about 300 teams in 10 years. The data looks something like this:</p>
  <pre><code>data &lt;- data.frame(count = c(1,1,2,1,1,5),
                 length_pass= c(1,2,5,7,1,3),
                 year= c(1,1,1,2,2,2),
                 mean_length_pass_team= c(15,15,9,14,14,8),
                 team= c('A', 'A', 'B', 'A', 'A', 'B'))
  data
   count length_pass year mean_length_pass_team team
  1     1           1    1                    15    A
  2     1           2    1                    15    A
  3     2           5    1                     9    B
  4     1           7    2                    14    A
  5     1           1    2                    14    A
  6     5           3    2                     8    B
  </code></pre>
  <p>I'm want to explain the <code>count</code> of steps a player takes before passing the ball. I have theoretical motivations to assume there are team-level differences between <code>count</code> and <code>length_pass</code>, so a multi-level (i.e. mixed effects) model seems appropriate.</p>
  <p>My individual level control variables are <code>length_pass</code> and <code>year</code>.</p>
  <p>On the team-level I have <code>mean_length_pass_team</code>. This should help me to avoid ecological fallacies, according to <a href="https://link.springer.com/referenceworkentry/10.1007%2F978-3-642-04898-2_387" rel="nofollow noreferrer">Snijders, 2011</a>.</p>
  <p>I have been using the <code>lme4</code> and <code>brms</code> packages to estimate these models but it takes days/weeks to fit these models on my local 12-core 128GB machine.</p>
  <pre><code>library(lme4)
  model_a &lt;- glmer(count ~ length_pass + year + mean_length_pass_team + (1 | team),
                   data=data,
                   family= &quot;poisson&quot;,
                   control=glmerControl(optCtrl=list(maxfun=2e8))) 
  
  library(brms)
  options (mc.cores=parallel::detectCores ())
  model_b &lt;- brm(count ~ length_pass + year + mean_length_pass_team + (1 | team),
                   data=data,
                   family= &quot;poisson&quot;)
  </code></pre>
  <p>I am looking for suggestions to speed up the fitting process or new techniques to fit a generalized linear mixed-effects model:</p>
  <ul>
  <li>(How) Can I improve the speed on the <code>lme4</code> and <code>brms</code> fits?</li>
  <li>Are there other packages to consider?</li>
  <li>Are there <em>step-wise</em> procedures that can help increase the speed of fitting models?</li>
  <li>Are there interesting options outside the R environment that can help me fit this?</li>
  </ul>
  <p>Any pointers are much appreciated!</p>