Regression testing is performed to provide confidence that changes in a part of software do not affect other parts of the software. An execution of all existing test cases is the best way to re-establish this confidence. However, regression testing is an expensive process---there might be insufficient resources (e.g., time, workforce) to allow for the re-execution of all test cases. Regression test prioritization techniques attempt to re-order a regression test suite based on some criteria so that highest priority test cases are executed earlier. In this study, we want to prioritize test cases for regression testing based on the dependency network of faults. In software testing, it is common that some faults are consequences of other faults (leading faults). Moreover, dependent faults can be removed if and only if the leading faults have been removed. Our goal is to prioritize test cases so that test cases that exposed leading faults (the most central faults in the fault dependency network) in the system testing phase, are executed first in regression testing. We present ComReg, a test case prioritization technique based on the dependency network of faults. We model a fault dependency network as a directed graph and identify leading faults to prioritize test cases for regression testing. We use a centrality aggregation technique which considers six network representative centrality metrics to identify leading faults in the fault dependency network. We also discuss the use of fault communities to select an arbitrary percentage of the test cases from a prioritized regression test suite. We conduct a case study that evaluates the effectiveness and applicability of the proposed method.
@cite_28: This paper presents an empirical comparison of four different techniques for filtering large test suites: test suite minimization, prioritization by additional coverage, cluster filtering with one-per-cluster sampling, and failure pursuit sampling. The first two techniques are based on selecting subsets that maximize code coverage as quickly as possible, while the latter two are based on analyzing the distribution of the tests' execution profiles. These techniques were compared with data sets obtained from three large subject programs: the GCC, Jikes, and javac compilers. The results indicate that distribution-based techniques can be as efficient or more efficient for revealing defects than coverage-based techniques, but that the two kinds of techniques are also complementary in the sense that they find different defects. Accordingly, some simple combinations of these techniques were evaluated for use in test case prioritization. The results indicate that these techniques can create more efficient prioritizations than those generated using prioritization by additional coverage.
@cite_48: During development and testing, changes made to a system to repair a detected fault can often inject a new fault into the code base. These injected faults may not be in the same files that were just changed, since the effects of a change in the code base can have ramifications in other parts of the system. We propose a methodology for determining the effect of a change and then prioritizing regression test cases by gathering software change records and analyzing them through singular value decomposition. This methodology generates clusters of files that historically tend to change together. Combining these clusters with test case information yields a matrix that can be multiplied by a vector representing a new system modification to create a prioritized list of test cases. We performed a post hoc case study using this technique with three minor releases of a software product at IBM. We found that our methodology suggested additional regression tests in 50 of test runs and that the highest-priority suggested test found an additional fault 60 of the time.
@cite_21: Web applications have rapidly become a critical part of business for many organizations. However, increased usage of Web applications has not been reciprocated with corresponding increases in reliability. Unique characteristics, such as quick turnaround time, coupled with growing popularity motivate the need for efficient and effective Web application testing strategies. In this paper, we propose several new test suite prioritization strategies for Web applications and examine whether these strategies can improve the rate of fault detection for three Web applications and their preexisting test suites. We prioritize test suites by test lengths, frequency of appearance of request sequences, and systematic coverage of parameter-values and their interactions. Experimental results show that the proposed prioritization criteria often improve the rate of fault detection of the test suites when compared to random ordering of test cases. In general, the best prioritization metrics either (1) consider frequency of appearance of sequences of requests or (2) systematically cover combinations of parameter-values as early as possible.
@cite_50: Regression test prioritization techniques re-order the execution of a test suite in an attempt to ensure that defects are revealed earlier in the test execution phase. In prior work, test suites were prioritized with respect to their ability to satisfy control flow-based and mutation-based test adequacy criteria. In this paper, we propose an approach to regression test prioritization that leverages the all-DUs test adequacy criterion that focuses on the definition and use of variables within the program under test. Our prioritization scheme is motivated by empirical studies that have shown that (i) tests fulfilling the all-DUs test adequacy criteria are more likely to reveal defects than those that meet the control flow-based criteria, (ii) there is an unclear relationship between all-DUs and mutation-based criteria, and (iii) mutation-based testing is significantly more expensive than testing that relies upon all-DUs.In support of our prioritization technique, we provide a formal statement of the algorithms and equations that we use to instrument the program under test, perform test suite coverage monitoring, and calculate test adequacy. Furthermore, we examine the architecture of a tool that implements our novel prioritization scheme and facilitates experimentation. The use of this tool in a preliminary experimental evaluation indicates that, for three case study applications, our prioritization can be performed with acceptable time and space overheads. Finally, these experiments also demonstrate that the prioritized test suite can have an improved potential to identify defects earlier during the process of test execution.
@cite_49: Test case prioritization is an effective and practical technique that helps to increase the rate of regression fault detection when software evolves. Numerous techniques have been reported in the literature on prioritizing test cases for regression testing. However, existing prioritization techniques implicitly assume that source or binary code is available when regression testing is performed, and therefore cannot be implemented when there is no program source or binary code to be analyzed. In this paper, we presented a new technique for black box regression testing, and we performed an experiment to measure our technique. Our results show that the new technique is helpful to improve the effectiveness of fault detection when performing regression test in black box environment.
@cite_46: Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible. The sizes of test suites grow as software evolves. Due to resource constraints, it is important to prioritize the execution of test cases so as to increase chances of early detection of faults. Prior techniques for test case prioritization are based on the total number of coverage requirements exercised by the test cases. In this paper, we present a new approach to prioritize test cases based on the coverage requirements present in the relevant slices of the outputs of test cases. We present experimental results comparing the effectiveness of our prioritization approach with that of existing techniques that only account for total requirement coverage, in terms of ability to achieve high rate of fault detection. Our results present interesting insights into the effectiveness of using relevant slices for test case prioritization.
Researchers used a number of other criteria to prioritize test cases. prioritized test cases based on historical change records in @cite_48 . They proposed a methodology for determining the effect of a software feature change and then prioritized regression test cases by gathering software change records and analyzing them through singular value decomposition. @cite_28 introduced distribution-based filtering and prioritized test cases based on the distribution of the profiles of test cases in the multi-dimensional profile space. @cite_21 prioritized test cases for web applications. They prioritized test suites by test lengths, frequency of appearance of request sequences and systematic coverage of parameter-values and their interactions. @cite_50 introduced a prioritization technique based on data-flow analysis. They focused on the definition and use of program variables for the data-flow analysis. @cite_46 prioritized test cases using relevant slices. @cite_49 prioritized test cases in a black box environment.