Title: Detecting Near Duplicates in Software Documentation,
Abstract: Contemporary software documentation is as complicated as the software itself.
During its lifecycle, the documentation accumulates a lot of near duplicate
fragments, i.e. chunks of text that were copied from a single source and were
later modified in different ways. Such near duplicates decrease documentation
quality and thus hamper its further utilization. At the same time, they are
hard to detect manually due to their fuzzy nature. In this paper we give a
formal definition of near duplicates and present an algorithm for their
detection in software documents. This algorithm is based on the exact software
clone detection approach: the software clone detection tool Clone Miner was
adapted to detect exact duplicates in documents. Then, our algorithm uses these
exact duplicates to construct near ones. We evaluate the proposed algorithm
using the documentation of 19 open source and commercial projects. Our
evaluation is very comprehensive - it covers various documentation types:
design and requirement specifications, programming guides and API
documentation, user manuals. Overall, the evaluation shows that all kinds of
software documentation contain a significant number of both exact and near
duplicates. Next, we report on the performed manual analysis of the detected
near duplicates for the Linux Kernel Documentation. We present both quantative
and qualitative results of this analysis, demonstrate algorithm strengths and
weaknesses, and discuss the benefits of duplicate management in software
documents.