Title: Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack,
Abstract: Many approaches for testing configurable software systems start from the same
assumption: it is impossible to test all configurations. This motivated the
definition of variability-aware abstractions and sampling techniques to cope
with large configuration spaces. Yet, there is no theoretical barrier that
prevents the exhaustive testing of all configurations by simply enumerating
them, if the effort required to do so remains acceptable. Not only this: we
believe there is lots to be learned by systematically and exhaustively testing
a configurable system. In this case study, we report on the first ever
endeavour to test all possible configurations of an industry-strength, open
source configurable software system, JHipster, a popular code generator for web
applications. We built a testing scaffold for the 26,000+ configurations of
JHipster using a cluster of 80 machines during 4 nights for a total of 4,376
hours (182 days) CPU time. We find that 35.70% configurations fail and we
identify the feature interactions that cause the errors. We show that sampling
strategies (like dissimilarity and 2-wise): (1) are more effective to find
faults than the 12 default configurations used in the JHipster continuous
integration; (2) can be too costly and exceed the available testing budget. We
cross this quantitative analysis with the qualitative assessment of JHipster's
lead developers.