Title: Attention Please: Consider Mockito when Evaluating Newly Proposed Automated Program Repair Techniques,
Abstract: Automated program repair (APR) has attracted widespread attention in recent
years with substantial techniques being proposed. Meanwhile, a number of
benchmarks have been established for evaluating the performances of APR
techniques, among which Defects4J is one of the most wildly used benchmark.
However, bugs in Mockito, a project augmented in a later-version of Defects4J,
do not receive much attention by recent researches. In this paper, we aim at
investigating the necessity of considering Mockito bugs when evaluating APR
techniques. Our findings show that: 1) Mockito bugs are not more complex for
repairing compared with bugs from other projects; 2) the bugs repaired by the
state-of-the-art tools share the same repair patterns compared with those
patterns required to repair Mockito bugs; however, 3) the state-of-the-art
tools perform poorly on Mockito bugs (Nopol can only correctly fix one bug
while SimFix and CapGen cannot fix any bug in Mockito even if all the buggy
locations have been exposed). We conclude from these results that existing APR
techniques may be overfitting to their evaluated subjects and we should
consider Mockito, or even more bugs from other projects, when evaluating newly
proposed APR techniques. We further find out a unique repair action required to
repair Mockito bugs named external package addition. Importing the external
packages from the test code associated with the source code is feasible for
enlarging the search space and this action can be augmented with existing
repair actions to advance existing techniques.