Title: Evaluating Predictive Models of Student Success: Closing the Methodological Gap,
Abstract: Model evaluation -- the process of making inferences about the performance of
predictive models -- is a critical component of predictive modeling research in
learning analytics. We survey the state of the practice with respect to model
evaluation in learning analytics, which overwhelmingly uses only naive methods
for model evaluation or statistical tests which are not appropriate for
predictive model evaluation. We conduct a critical comparison of both null
hypothesis significance testing (NHST) and a preferred Bayesian method for
model evaluation. Finally, we apply three methods -- the na{ï}ve average
commonly used in learning analytics, NHST, and Bayesian -- to a predictive
modeling experiment on a large set of MOOC data. We compare 96 different
predictive models, including different feature sets, statistical modeling
algorithms, and tuning hyperparameters for each, using this case study to
demonstrate the different experimental conclusions these evaluation techniques
provide.