Title: Cluster Failure Revisited: Impact of First Level Design and Data Quality on Cluster False Positive Rates,
Abstract: Methodological research rarely generates a broad interest, yet our work on
the validity of cluster inference methods for functional magnetic resonance
imaging (fMRI) created intense discussion on both the minutia of our approach
and its implications for the discipline. In the present work, we take on
various critiques of our work and further explore the limitations of our
original work. We address issues about the particular event-related designs we
used, considering multiple event types and randomisation of events between
subjects. We consider the lack of validity found with one-sample permutation
(sign flipping) tests, investigating a number of approaches to improve the
false positive control of this widely used procedure. We found that the
combination of a two-sided test and cleaning the data using ICA FIX resulted in
nominal false positive rates for all datasets, meaning that data cleaning is
not only important for resting state fMRI, but also for task fMRI. Finally, we
discuss the implications of our work on the fMRI literature as a whole,
estimating that at least 10% of the fMRI studies have used the most problematic
cluster inference method (P = 0.01 cluster defining threshold), and how
individual studies can be interpreted in light of our findings. These
additional results underscore our original conclusions, on the importance of
data sharing and thorough evaluation of statistical methods on realistic null
data.