Title: Powerful statistical inference for nested data using sufficient summary statistics,
Abstract: Hierarchically-organized data arise naturally in many psychology and
neuroscience studies. As the standard assumption of independent and identically
distributed samples does not hold for such data, two important problems are to
accurately estimate group-level effect sizes, and to obtain powerful
statistical tests against group-level null hypotheses. A common approach is to
summarize subject-level data by a single quantity per subject, which is often
the mean or the difference between class means, and treat these as samples in a
group-level t-test. This 'naive' approach is, however, suboptimal in terms of
statistical power, as it ignores information about the intra-subject variance.
To address this issue, we review several approaches to deal with nested data,
with a focus on methods that are easy to implement. With what we call the
sufficient-summary-statistic approach, we highlight a computationally efficient
technique that can improve statistical power by taking into account
within-subject variances, and we provide step-by-step instructions on how to
apply this approach to a number of frequently-used measures of effect size. The
properties of the reviewed approaches and the potential benefits over a
group-level t-test are quantitatively assessed on simulated data and
demonstrated on EEG data from a simulated-driving experiment.