Title: Automated Refactoring: Can They Pass The Turing Test?,
Abstract: Refactoring is a maintenance activity that aims to improve design quality
while preserving the behavior of a system. Several (semi)automated approaches
have been proposed to support developers in this maintenance activity, based on
the correction of anti-patterns, which are "poor solutions" to recurring design
problems. However, little quantitative evidence exists about the impact of
automatically refactored code on program comprehension, and in which context
automated refactoring can be as effective as manual refactoring. We performed
an empirical study to investigate whether the use of automated refactoring
approaches affects the understandability of systems during comprehension tasks.
(1) We surveyed 80 developers, asking them to identify from a set of 20
refactoring changes if they were generated by developers or by machine, and to
rate the refactorings according to their design quality; (2) we asked 30
developers to complete code comprehension tasks on 10 systems that were
refactored by either a freelancer or an automated refactoring tool. We measured
developers' performance using the NASA task load index for their effort; the
time that they spent performing the tasks; and their percentages of correct
answers. Results show that for 3 out the 5 types of studied anti-patterns,
developers cannot recognize the origin of the refactoring (i.e., whether it was
performed by a human or an automatic tool). We also observe that developers do
not prefer human refactorings over automated refactorings, except when
refactoring Blob classes; and that there is no statistically significant
difference between the impact on code understandability of human refactorings
and automated refactorings. We conclude that automated refactorings can be as
effective as manual refactorings. However, for complex anti-patterns types like
the Blob, the perceived quality of human refactorings is slightly higher.