Title: Differential Testing for Variational Analyses: Experience from Developing KConfigReader,
Abstract: Differential testing to solve the oracle problem has been applied in many
scenarios where multiple supposedly equivalent implementations exist, such as
multiple implementations of a C compiler. If the multiple systems disagree on
the output for a given test input, we have likely discovered a bug without
every having to specify what the expected output is. Research on variational
analyses (or variability-aware or family-based analyses) can benefit from
similar ideas. The goal of most variational analyses is to perform an analysis,
such as type checking or model checking, over a large number of configurations
much faster than an existing traditional analysis could by analyzing each
configuration separately. Variational analyses are very suitable for
differential testing, since the existence nonvariational analysis can provide
the oracle for test cases that would otherwise be tedious or difficult to
write. In this experience paper, I report how differential testing has helped
in developing KConfigReader, a tool for translating the Linux kernel's kconfig
model into a propositional formula. Differential testing allows us to quickly
build a large test base and incorporate external tests that avoided many
regressions during development and made KConfigReader likely the most precise
kconfig extraction tool available.