Correction of "A Comparative Study to Benchmark Cross-project Defect Prediction Approaches"
Steffen Herbold Alexander Trautsch Jens Grabowski
cs.SE


 

thmTheorem
corollary[thm]Corollary
lemma[thm]Lemma
definition
defi[thm]Definition
remark[thm]Remark
beweis




























































IEEE Transactions on Software Engineering
Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals

























ANOVAANalysis Of VAriance
ASTAbstract Syntax Tree
AUCArea Under the ROC Curve
CaAfferent Coupling
CBOCoupling Between Objects
CCACanonical Correlation Analysis
CDCritical Distance
CeEfferent Coupling
CFSCorrelation-based Feature Subset
CLAClustering and LAbeling
CODEPCOmbined DEfect Predictor
CPDPCross-Project Defect Prediction
DBSCANDensity-Based Spatial Clustering
DCVDataset Characteristic Vector
DTBDouble Transfer Boosting
fnfalse negative
fpfalse positive
GBGigaByte
HLHosmer-Lemeshow
ITSIssue Tracking System
JITJust In Time
LCOMLack of COhession between Methods
LOCLines Of Code
MDPMetrics Data Program
MIMetric and Instances selection
MODEPMultiObjective DEfect Predictor
MPDPMixed-Project Defect Prediction
NNNearest Neighbor
PCAPrinciple Component Analysis
RAMRandom Access Memory
RFCResponse For a Class
SCMSourceCode Management system
SVMSupport Vector Machine
TCATransfer Component Analysis
tntrue negative
tptrue positive
RBFRadial Basis Function
ROCReceiver Operating Characteristic
UMRUnified Metric Representation
VCBValue-Cognitive Boosting
WPDPWithin-Project Defect Prediction







Unfortunately, the article “A Comparative Study to Benchmark Cross-project Defect
Prediction Approaches” has a problem in the statistical analysis which was
pointed out almost immediately after the pre-print of the article appeared
online. While the problem does not negate the contribution of the the article
and all key findings remain the same, it does alter some rankings of approaches
used in the study. Within this correction, we will explain the problem, how we
resolved it, and present the updated results. 











cross-project defect prediction, benchmark, comparison, replication, correction





Correction of “A Comparative Study to Benchmark Cross-project Defect
Prediction Approaches”
    Steffen Herbold, Alexander Trautsch, Jens Grabowski
S. Herbold, A. Trautsch, and J.
Grabowski are with the University of Goettingen, Institute of Computer Science,
Göttingen, Germany.



E-mail: {herbold,grabowski}@cs.uni-goettingen.de

alexander.trautsch@stud.uni-goettingen.de


    December 30, 2023
============================================================================================================================================================================================================================================================================================




























§ INTRODUCTION



Unfortunately, the article “A Comparative Study to Benchmark Cross-project
Defect Prediction Approaches” <cit.> has a problem in the
statistical analysis performed to rank CPDP approaches.
Prof. Yuming Zhou from Nanjing University
pointed out an inconsistency in Table 8 of the the article. He noted that in
some cases the rankscores are worse even if the mean values for
performance metrics are better. While this is possible in theory, with the Friedman
test <cit.> with post-hoc Nemenyi test <cit.>, such inconsistencies are unlikely.
Therefore, we immediately proceeded to check our results. These checks revealed that the inconsistencies are due to a problem
with our statistical analysis for the Research Question 1 (RQ1) “Which
CPDP approaches perform best in terms of F-measure, G-measure, AUC, and MCC?”. None
of the raw results of the benchmark, nor any of the other research questions are
affected by the problem.

We will describe the problem and how we solved in in Section <ref>.
Then, we will show the updated results regarding RQ1 and discuss the changes in
Section <ref>. Afterwards, we analyze the reasons for the changes in
Section <ref> to determine if all changes due to the correction are
plausible and the correction resolves the inconsistencies reported by Y. Zhou.
In Section <ref>, we describe how we updated our replication kit
as part of this correction. Finally, we will conclude in
Section <ref>.
Please note, that we assume that readers have read to the original article and are
familiar with the terminology used. We do not re-introduce any of the
terminology in this correction. 



§ PROBLEM WITH THE NEMENYI TEST IMPLEMENTATION



On July 15th 2017, Y. Zhou imformed us that he found an inconsistency between
the results of CV-NET and CamargoCruz09-DT for the RELINK data for the performance
metric AUC. He noted that the mean value for CV-NET was higher than for
CamargoCruz09-DT, but the rankscore was lower. He went to the raw data
provided as part of the replication kit <cit.> and confirmed that
the mean values were correct, and the AUC for CV-NET was higher for all three
products of the RELINK data. Based on this observervation, we re-checked our
statistical analysis of the results. We found the problem in our implementation
of the Nemenyi post-hoc test.



 §.§ Summary of the Friedman and Nemenyi tests


To understand the problem, we briefly recap how the Friedman test with
post-hoc Nemenyi test works. The Friedman test determines if there are
stastical significant differences between populations. This is done using
pair-wise comparisons between the rankings of populations. If the Friedman test
determines significant differences, the Nemenyi post-hoc test compares the
populations to each other to determine the statistically significantly
different ones. The analysis with the Nemenyi test is based on two parameters:
the CD and the average ranks of the populations in the pair-wise comparisons
between all populations on each data set.
Following the description by Demšar <cit.>, CD is defined as

    CD = q_α√(k(k+1)/6N)

where q_α = qtukey(α, N, inf)/√(2) is the studentized
range distribution with infinite degrees of freedom divided[For simplicity, we refer to the studentized range
distribution as qtukey(α, N), following the name of the related method in
R] by √(2), α the significance level, k the number of populations
compared and N the number of data sets. We can thus rewrite CD as

    CD    = qtukey(α, N, inf)/√(2)√(k(k+1)/6N)
       = qtukey(α, N, inf)√(k(k+1)/12N)


If we now assume that R_i, R_j are the average ranks of population 
i,j ∈{1,…,N}, the two populations are stastically significantly
different if

    |R_i-R_j| > CD.


In case a control population is available, it is possible to use a procedure
like Bonferroni correction <cit.>. In this case, all populations are compared
to the control classifier instead of to each other. This greatly reduces the
number of pair-wise comparisons and can make the test more powerful. In this
case, for each pair a z-value is computed as

    z = (R_i-R_j)/√(k(k+1)/6N).

The z-values are then used to rank classifiers. Since we do not have a control classifier and have to do pair-wise
comparisons with the CD and cannot make use of the z-values. However, the
z-values play an important role when it comes to the problem with our analysis. 



 §.§ z-values instead of Ranks


Now that the concepts of the statistical tests are introduced, we can discuss
the actual problem in our implementation. We used the
 function of the PMCMR
package <cit.> to implement the test. As part of the return values,
the function returns a matrix called . Without checking directly
in the source code, we assumed these were the average ranks for each population, based on the documention of the package.
However, these are actually the absolute z-values multiplied with √(2),
i.e., 

    z' = |R_i-R_j|/√(k(k+1)/6N·√(2)).

Thus, when we compared ranks, we did not actually compare the average ranks, but the
mean z'-values. This led to a wrong determination of ranks, which explain the
inconsistencies found by Y. Zhou.



 §.§ The solution

To resolve the problem, we adopted the code from the PMCMR package that
determines the average ranks. We cross-checked our code with another
implementation of the Nemenyi test <cit.>, to ensure that
the new code solves the problem.[Both implementations of the test do not return the raw
pair-wise comparisons and can, therefore, not be used directly.] We then used
the mean averages from that code, instead of the z-values that were returned
from the PMCMR package. As a result, the Nemenyi-test became much more
sensitive, because the scale of the average ranks is different than the scale of the z-values. Let us consider the scales for our experiments
with the JURECZKO data. Here, we have N=62 data sets and k=135 populations,
i.e., CPDP approach and classifier combinations. The best possible average
rank is 135 (always wins), the worst possible is 1 (always loses). Thus, the
average ranks are on a scale from 1 to 135. In comparison, the highest possible
z'-value is

    z = (135-1)/√(135·136/6·62)·√(2)≈ 26.97,

i.e., the scale is just from 0 (no difference in average ranks) and 26.97. Thus,
the scale of the z'-values has only about a fifth of the range the scale of
the average ranks has. Basically, with z'-values, 135
populations are fit into the scale 0 to 26.97, with rankings in the scale 1 to 135. This means that the
average distance between approaches is 0.2 with z'-values and 1 in case of
average ranks. Considering that we have a CD ≈ 1.26 for α=0.95
in this example, this makes a huge difference. With z'-values, it is unlikely
that two subsequently ranked approaches to have
a distance greater than CD, because CD was more than 6.3 times higher than
average distance expected on the scale. This changes if the real scale with ranks is used. If you
have 135 cases with an average distance of 1, it is quite likely that a few of
these distances will be greater than 1.26, i.e., the CD. 

We discuss this change in scales in this detail, because it requires a small
change in the ranking of approaches based on the Nemenyi test. Before, we considered three distinct ranks for
the calculation of the rankscore to achieve non-overlapping ranks:

  
  * The populations that are within the CD of the best average ranking
  population (top rank 1).
  
  * The populations that are within the CD of the worst average ranking
  population (bottom rank 3).
  
  * The populations that are neither (middle rank 2). 


This was the only way to deal with the small differences that resulted from
using the z'-values. However, this approach breaks on the larger scale,
because the distances now become larger, meaning fewer results are within the CD from
the best/worst ranking. For example, for the JURECZKO data and the performance
metric AUC, only two approaches would be on the first rank, i.e., only one
approach is within the CD of the best approach. Similarly, only six approaches
would be on the third rank, i.e., only five approaches are within the CD of the
worst approach. This would leave us with 127 approaches on the middle rank. This
ranking would be to coarse, and not show actual differences between approaches
anymore. To deal with this larger scale of ranks, we use a simple and more
fine-grained grouping strategy to create the ranks. We sort all approaches by their average ranking.
Each time that the difference between two subsequently ranked approaches is
larger than the CD, we increase the rank. Because the rank is only
increased if the difference is larger than the CD, we ensure that each
group only contains approaches that are statistically significantly different from the
other groups. Afterwards, we calculate the normalized rankscore as before.
Algorithm <ref> formalizes this strategy. This change in ranking
increases the sensitivity of the test and makes the results more fine-grained in
comparison to our original ranking procedure. 








§ RESULTS











We now show the corrected results for RQ1. We will directly
compare the changes in the results with the originally published results.
Figure <ref> shows the mean rankscore averaged over the four
performance metrics F-Measure, G-Measure, AUC, and MCC and the five data sets
JURECZKO, MDP, AEEEM, NETGENE, and RELINK. Table <ref> shows
detailed results including the mean values and rankscores for each performance
metrics and each data set. Figure <ref> is the correction of
Figure 3 and Table <ref> the correction of Table 8 from the original
publication. Table <ref> and Figure <ref> only report
the results for the best classifier for each approach. In case these changed between the original results
and our correction, you will not find the exact same rows. For example, for
CamargoCruz09, we reported DT as best classifier in the original analysis, and
now NB. This is because with the problem in the statistical analysis DT was
ranked best for CamargoCruz09, but in the corrected version NB performs better.
The reasons for these, and other changes are explained in
Section <ref>

The most important finding remains the same: the
approach CamargoCruz09 still provides the best-ranking classification model with
a mean rankscore of 0.917 for CamargoCruz09-NB. However, the
rankscore is not a perfect 1.0 anymore. We attribute this to the more
sensitive ranking due to the correction of the Nemenyi test.
The differences to the next-ranking approaches are still rather small, though the
group of approaches that is within 10% of the best ranking approach now only consists of CV-RF, Amasaki15, Peters15, and Ma12. The bottom of the ranking is nearly unaffected by the changes as well. The last
seven ranked approaches are still the same. Additionally, our findings regarding
the comparison of using ALL data, versus transfer learning approaches has not
changed: ALL is still in the upper mid-field of approaches. With the corrected
and more finegrained ranking, only six of the cross-project approaches actually
outperform this baseline, whereas seventeen are actually worse.

With respect to CV versus CPDP, we still observe that CPDP can
outperform CV in case multiple performance criteria are considered because
CV-RF is outperformed by CamargoCruz09-NB. Thus, we still note that this is
possible, but far less conclusively than before, where CV was actually only in
the mid-field of the approaches and not a close second.

Due to these small overall small differences, we change our answer to RQ1
slightly: 

Answer RQ1:
CamargoCruz09-NB performs best among the compared CPDP approaches and
even outperforms cross-validation. However, the differences to other
approaches are small. The baseline ALL-NB is ranked higher than seventeen of the
CPDP approaches.




§ REASONS FOR CHANGES



We checked our raw results for the reasons for all changes in rankings. The
problem with the statistical analysis actually led to two reasons for ranking
changes: first, the z-values already consider differences in ranks. Thus, if
the rank was very high, this could would lead to larger z-values, which would
negatively impact the ranking. Second, because differences were downscaled with
the z-values in comparison to differences in mean ranks, too many approaches
were grouped together as not statistically significantly different. For
approaches that are now better ranked than before, this means that they were
often among the best performing approaches within a group. For those that are
now ranking worse, they were at often near the bottom of their groups. For
example, CV was often among the best approaches on the middle rank. Now, it is
clearly distinguished from the others there, leading to the strong rise in the
ranking. Others that were affected the same way, though to a lesser extend are
Amasaki15, Peters15, YZhang15, and Herbold13. On the other hand Menzies11 and
Watanabe08 were often at the bottom of their groups, leading to the big losses
in rankings for both approaches.

Another change in our results is that NB is often the best
performing classifier, whereas before DT and LR were most often the best
performing classifiers. We previously already noted in our discussion that “for
many approaches the differences between the classifiers were rather
small” <cit.>. Together with the reasons for ranking
changes explained above, theses changes are not unexpected. 

Overall, all changes in the result are plausible. Moreover, our comparison
of the results of the statistical analysis with both mean values, as well as the
raw results of the benchmark did not reveal any inconsistencies of the type that
Y. Zhou reported to us. Therefore, we believe that the problem was
correctly resolved. 



§ UPDATE OF THE REPLICATION KIT



We updated the replication kit archived at Zenodo <cit.>. The
changes two the replication kit are two-fold.

  
  * We corrected the problem with the statistical analysis in the
  generate_results.R script.
  
  * We updated the provided CD diagrams due to the changes in the Nemenyi
  test.


The changes can be reviewed in detail in the commit to the GitHub archive of the
replication kit[https://goo.gl/AbvSRj]. 



§ CONCLUSION



A problem with the implementation of the Nemenyi post-hoc test led to incorrect
results being published in our benchmark paper on cross-project defect
prediction. The mistake only affected research question RQ1, the other three
research questions were not affected. Within this correction paper, we explained
the problem in the statistical test, how this problem affected our results,
presented the corrected, and explained the changes that occoured. The
major findings regarding RQ1 are not changed, including the best performing approach, the result that the naïve
baseline of using all data outperforms most proposed transfer learning
approaches, as well as that cross-validation can be outperformed by CPDP.
Thus, the contributions of the article are still valid. Still, the correction
leads to differences in the rankings which are properly corrected and
discussed here. We apologize for this mistake and hope that this timely
correction mitigates the potential negative impact the wrong results may have.



§ ACKNOWLEDGEMENTS


We want to thank Yuming Zhou  from Nanjing University for pointing out the
inconsistencies in the results to us so fast, as well as the editors of this
journals who helped to determine how we should communicate this problem to the
community within days.





















































IEEEtran
































































