Title: Test Prioritization in Continuous Integration Environments,
Abstract: Two heuristics namely diversity-based (DBTP) and history-based test
prioritization (HBTP) have been separately proposed in the literature. Yet,
their combination has not been widely studied in continuous integration (CI)
environments. The objective of this study is to catch regression faults
earlier, allowing developers to integrate and verify their changes more
frequently and continuously. To achieve this, we investigated six open-source
projects, each of which included several builds over a large time period.
Findings indicate that previous failure knowledge seems to have strong
predictive power in CI environments and can be used to effectively prioritize
tests. HBTP does not necessarily need to have large data, and its effectiveness
improves to a certain degree with larger history interval. DBTP can be used
effectively during the early stages, when no historical data is available, and
also combined with HBTP to improve its effectiveness. Among the investigated
techniques, we found that history-based diversity using NCD Multiset is
superior in terms of effectiveness but comes with relatively higher overhead in
terms of method execution time. Test prioritization in CI environments can be
effectively performed with negligible investment using previous failure
knowledge, and its effectiveness can be further improved by considering
dissimilarities among the tests.