A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms.
Twenty-two decision tree, nine statistical, and two neural network algorithms are compared on thirty-two datasets in terms of classification accuracy, training time, and (in the case of trees) number of leaves. Classification accuracy is measured by mean error rate and mean rank of error rate. Both criteria place a statistical, spline-based, algorithm called POLYCLSSS at the top, although it is not statistically significantly different from twenty other algorithms. Another statistical algorithm, logistic regression, is second with respect to the two accuracy criteria. The most accurate decision tree algorithm is QUEST with linear splits, which ranks fourth and fifth, respectively. Although spline-based statistical algorithms tend to have good accuracy, they also require relatively long training times. POLYCLASS, for example, is third last in terms of median training time. It often requires hours of training compared to seconds for other algorithms. The QUEST and logistic regression algorithms are substantially faster. Among decision tree algorithms with univariate splits, C4.5, IND-CART, and QUEST have the best combinations of error rate and speed. But C4.5 tends to produce trees with twice as many leaves as those from IND-CART and QUEST.
Introduction
There is much current research in the machine learning and statistics communities
on algorithms for decision tree classifiers. Often the emphasis is on the accuracy of
the algorithms. One study, called the StatLog Project (Michie, Spiegelhalter and
Taylor, 1994), compares the accuracy of several decision tree algorithms against
some non-decision tree algorithms on a large number of datasets. Other studies that
are smaller in scale include Brodley and Utgoff (1992), Brown, Corruble and Pittard
(1993), Curram and Mingers (1994), and Shavlik, Mooney and Towell (1991).
Recently, comprehensibility of the tree structures has received some attention.
Comprehensibility typically decreases with increase in tree size and complexity. If
two trees employ the same kind of tests and have the same prediction accuracy, the
one with fewer leaves is usually preferred. Breslow and Aha (1997) survey methods
of tree simplification to improve their comprehensibility.
A third criterion that has been largely ignored is the relative training time of
the algorithms. The StatLog Project finds that no algorithm is uniformly most
accurate over the datasets studied. Instead, many algorithms possess comparable
accuracy. For such algorithms, excessive training times may be undesirable (Hand,
1997).
The purpose of our paper is to extend the results of the StatLog Project in the
following ways:
1. In addition to classification accuracy and size of trees, we compare the training
times of the algorithms. Although training time depends somewhat on imple-
mentation, it turns out that there are such large differences in times (seconds
versus days) that the differences cannot be attributed to implementation alone.
2. We include some decision tree algorithms that are not included in the StatLog
Project, namely, S-Plus tree (Clark and Pregibon, 1993),
Holte and Maass, 1995), OC1 (Murthy, Kasif and Salzberg, 1994), LMDT
(Brodley and Utgoff, 1995), and QUEST (Loh and Shih, 1997).
3. We also include several of the newest spline-based statistical algorithms. Their
classification accuracy may be used as benchmarks for comparison with other
algorithms in the future.
4. We study the effect of adding independent noise attributes on the classification
accuracy and (where appropriate) tree size of each algorithm. It turns out that
except possibly for three algorithms, all the others adapt to noise quite well.
5. We examine the scalability of some of the more promising algorithms as the
sample size is increased.
Our experiment compares twenty-two decision tree algorithms, nine classical and
modern statistical algorithms, and two neural network algorithms. Many datasets
are taken from the University of California, Irvine (UCI), Repository of Machine
Learning Databases (Merz and Murphy, 1996). Fourteen of the datasets are from
real-life domains and two are artificially constructed. Five of the datasets were used
in the StatLog Project. To increase the probability of finding statistically significant
differences between algorithms, the number of datasets is doubled by the addition
of noise attributes. The resulting total of number of datasets is thirty-two.
Section 2 briefly describes the algorithms and Section 3 gives some background
to the datasets. Section 4 explains the experimental setup used in this study and
Section 5 analyzes the results. The issue of scalability is studied in Section 6 and
conclusions and recommendations are given in Section 7.
2. The algorithms
Only a short description of each algorithm is given. Details may be found in the
cited references. If an algorithm requires class prior probabilities, they are made
proportional to the training sample sizes.
2.1. Trees and rules
CART: We use the version of CART (Breiman, Friedman, Olshen and Stone,
1984) implemented in the cart style of the IND package (Buntine and Caru-
ana, 1992) with the Gini index of diversity as the splitting criterion. The trees
based on the 0-SE and 1-SE pruning rules are denoted by IC0 and IC1 respec-
tively. The software is obtained from
http://ic-www.arc.nasa.gov/ic/projects/bayes-group/ind/IND-program.html.
S-Plus tree: This is a variant of the CART algorithm written in the S language
(Becker, Chambers and Wilks, 1988). It is described in Clark and Pregibon
(1993). It employs deviance as the splitting criterion. The best tree is chosen
by ten-fold cross-validation. Pruning is performed with the p.tree() function
in the treefix library (Venables and Ripley, 1997) from the StatLib S Archive
at http://lib.stat.cmu.edu/S/. The 0-SE and 1-SE trees are denoted by
ST0 and ST1 respectively.
C4.5: We use Release 8 (Quinlan, 1993; Quinlan, 1996) with the default settings
including pruning (http://www.cse.unsw.edu.au/~quinlan/). After a tree is
constructed, the C4.5 rule induction program is used to produce a set of rules.
The trees are denoted by C4T and the rules by C4R.
FACT: This fast classification tree algorithm is described in Loh and Vanichse-
takul (1988). It employs statistical tests to select an attribute for splitting each
node and then uses discriminant analysis to find the split point. The size of
the tree is determined by a set of stopping rules. The trees based on univariate
splits (splits on a single attribute) are denoted by FTU and those based on linear
combination splits (splits on linear functions of attributes) are denoted by FTL.
The Fortran 77 program is obtained from http://www.stat.wisc.edu/~loh/.
QUEST: This new classification tree algorithm is described in Loh and Shih
(1997). QUEST can be used with univariate or linear combination splits. A
unique feature is that its attribute selection method has negligible bias. If all
the attributes are uninformative with respect to the class attribute, then each
has approximately the same chance of being selected to split a node. Ten-fold
cross-validation is used to prune the trees. The univariate 0-SE and 1-SE trees
are denoted by QU0 and QU1, respectively. The corresponding trees with linear
combination splits are denoted by QL0 and QL1, respectively. The results in
this paper are based on version 1.7.10 of the program. The software is obtained
from http://www.stat.wisc.edu/~loh/quest.html.
IND: This is due to Buntine (1992). We use version 2.1 with the default settings.
IND comes with several standard predefined styles. We compare four Bayesian
styles in this paper: bayes, bayes opt, mml, and mml opt (denoted by IB,
IBO, IM, and IMO, respectively). The opt methods extend the non-opt methods
by growing several different trees and storing them in a compact graph struc-
ture. Although more time and memory intensive, the opt styles can increase
classification accuracy.
OC1: This algorithm is described in Murthy et al. (1994). We use version 3
(http://www.cs.jhu.edu/~salzberg/announce-oc1.html) and compare three
styles. The first one (denoted by OCM) is the default that uses a mixture of univariate
and linear combination splits. The second one (option -a; denoted by
OCU) uses only univariate splits. The third one (option -o; denoted by OCL) uses
only linear combination splits. Other options are kept at their default values.
LMDT: The algorithm is described in Brodley and Utgoff (1995). It constructs
a decision tree based on multivariate tests that are linear combinations of the
attributes. The tree is denoted by LMT. We use the default values in the software
from http://yake.ecn.purdue.edu/~brodley/software/lmdt.html.
CAL5: This is from the Fraunhofer Society, Institute for Information and Data
Processing, Germany (M-uller and Wysotzki, 1994; M-uller and Wysotzki, 1997).
We use version 2. CAL5 is designed specifically for numerical-valued attributes.
However, it has a procedure to handle categorical attributes so that mixed attributes
(numerical and categorical) can be included. In this study we optimize
the two parameters which control tree construction. They are the predefined
threshold S and significance level ff. We randomly split the training set into
two parts, stratified by the classes: two-thirds are used to construct the tree and
one-third is used as a validation set to choose the optimal parameter configura-
tion. We employ the c-shell program that comes with the CAL5 package to
choose the best parameters by varying ff between 0.10 and 0.90 and S between
and 0.95 in steps of 0.05. The best combination of values that minimize
the error rate on the validation set is chosen. The tree is then constructed on all
the records in the training set using the chosen parameter values. It is denoted
by CAL.
T1: This is a one-level decision tree that classifies examples on the basis of only
one split on a single attribute (Holte, 1993). A split on a categorical attribute
5with b categories can produce up to b being reserved
for missing attribute values). On the other hand, a split on a continuous
attribute can yield up to J leaves, where J is the number of classes
(one leaf is again reserved for missing values). The software is obtained from
http://www.csi.uottawa.ca/~holte/Learning/other-sites.html.
2.2. Statistical algorithms
LDA: This is linear discriminant analysis, a classical statistical method. It models
the instances within each class as normally distributed with a common covariance
matrix. This yields linear discriminant functions.
QDA: This is quadratic discriminant analysis. It also models class distributions
as normal, but estimates each covariance matrix by the corresponding sample
covariance matrix. As a result, the discriminant functions are quadratic. Details
on LDA and QDA can be found in many statistics textbooks, e.g., Johnson and
(1992). We use the SAS PROC DISCRIM (SAS Institute, Inc., 1990)
implementation of LDA and QDA with the default settings.
NN: This is the SAS PROC DISCRIM implementation of the nearest neighbor
method. The pooled covariance matrix is used to compute Mahalanobis distances

LOG: This is logistic discriminant analysis. The results are obtained with a poly-
tomous logistic regression (see, e.g., Agresti (1990)) Fortran 90 routine written
by the first author (http://www.stat.wisc.edu/~limt/logdiscr/).
FDA: This is flexible discriminant analysis (Hastie, Tibshirani and Buja, 1994), a
generalization of linear discriminant analysis that casts the classification problem
as one involving regression. Only the MARS (Friedman, 1991) nonparametric
regression procedure is studied here. We use the S-Plus function fda
from the mda library of the StatLib S Archive. Two models are used: an additive
model (degree=1, denoted by FM1) and a model containing first-order
interactions (degree=2 with penalty=3, denoted by FM2).
PDA: This is a form of penalized LDA (Hastie, Buja and Tibshirani, 1995). It is
designed for situations in which there are many highly correlated attributes.
The classification problem is cast into a penalized regression framework via
optimal scoring. PDA is implemented in S-Plus using the function fda with
method=gen.ridge.
MDA: This stands for mixture discriminant analysis (Hastie and Tibshirani, 1996).
It fits Gaussian mixture density functions to each class to produce a classifier.
MDA is implemented in S-Plus using the library mda.
POL: This is the POLYCLASS algorithm (Kooperberg, Bose and Stone, 1997). It
fits a polytomous logistic regression model using linear splines and their tensor
products. It provides estimates for conditional class probabilities which can
then be used to predict class labels. POL is implemented in S-Plus using the
function poly.fit from the polyclass library of the StatLib S Archive. Model
selection is done with ten-fold cross-validation.
2.3. Neural networks
LVQ: We use the learning vector quantization algorithm in the S-Plus class library
(Venables and Ripley, 1997) at the StatLib S Archive. Details of the
algorithm may be found in Kohonen (1995). Ten percent of the training set are
used to initialize the algorithm, using the function lvqinit. Training is carried
out with the optimized learning rate function olvq1, a fast and robust LVQ
algorithm. Additional fine-tuning in learning is performed with the function
lvq1. The number of iterations is ten times the size of the training set in both
olvq1 and lvq1. We use the default values of 0.3 and 0.03 for ff, the learning
rate parameter, in olvq1 and lvq1, respectively.
RBF: This is the radial basis function network implemented in the SAS tnn3.sas
macro (Sarle, 1994) for feedforward neural networks (http://www.sas.com).
The network architecture is specified with the ARCH=RBF argument. In this
study, we construct a network with only one hidden layer. The number of
hidden units is chosen to be 20% of the total number of input and output units
[2.5% (5 hidden units) only for the dna and dna+ datasets and 10% (5 hidden
units) for the tae and tae+ datasets because of memory and storage limitations].
Although the macro can perform model selection to choose the optimal number
of hidden units, we did not utilize this capability because it would have taken too
long for some of the datasets (see Table 6 below). Therefore the results reported
here for this algorithm should be regarded as lower bounds on its performance.
The hidden layer is fully connected to the input and output layers but there is
no direct connection between the input and output layers. At the output layer,
each class is represented by one unit taking the value of 1 for that particular
category and 0 otherwise, except for the last one which is the reference category.
To avoid local optima, ten preliminary trainings were conducted and the best
estimates used for subsequent training. More details on the radial basis function
network can be found in Bishop (1995) and Ripley (1996).
3. The datasets
We briefly describe the sixteen datasets used in the study as well as any modifications
that are made for our experiment. Fourteen of them are from real domains
while two are artificially created. Thirteen are from UCI.
breast cancer (bcw). This is one of the breast cancer databases at
UCI, collected at the University of Wisconsin by W. H. Wolberg. The problem is
to predict whether a tissue sample taken from a patient's breast is malignant or
benign. There are two classes, nine numerical attributes, and 699 observations.
Sixteen instances contain a single missing attribute value and are removed from
the analysis. Our results are therefore based on 683 records. Error rates are
estimated using ten-fold cross-validation. A decision tree analysis of a subset
of the data using the FACT algorithm is reported in Wolberg, Tanner, Loh and
(1987), Wolberg, Tanner and Loh (1988), and Wolberg, Tanner
and Loh (1989). The dataset has also been analyzed with linear programming
methods (Mangasarian and Wolberg, 1990).
Contraceptive method choice (cmc). The data are taken from the 1987 National
Indonesia Contraceptive Prevalence Survey. The samples are married
women who were either not pregnant or did not know if they were pregnant
at the time of the interview. The problem is to predict the current contraceptive
method choice (no use, long-term methods, or short-term methods) of a
woman based on her demographic and socio-economic characteristics (Lerman,
Molyneaux, Pangemanan and Iswarati, 1991). There are three classes, two numerical
attributes, seven categorical attributes, and 1473 records. The error
rates are estimated using ten-fold cross-validation. The data are obtained from
http://www.stat.wisc.edu/p/stat/ftp/pub/loh/treeprogs/datasets/.
StatLog DNA (dna). This UCI dataset in molecular biology was used in the
Project. Splice junctions are points in a DNA sequence at which "su-
perfluous" DNA is removed during the process of protein creation in higher
organisms. The problem is to recognize, given a sequence of DNA, the boundaries
between exons (the parts of the DNA sequence retained after splicing)
and introns (the parts of the DNA sequence that are spliced out). There are
three classes and sixty categorical attributes each having four categories. The
sixty categorical attributes represent a window of sixty nucleotides, each having
one of four categories. The middle point in the window is classified as one of
exon/intron boundaries, intron/exon boundaries, or neither of these. The 3186
examples in the database were divided randomly into a training set of size 2000
and a test set of size 1186. The error rates are estimated from the test set.
StatLog heart disease (hea). This UCI dataset is from the Cleveland Clinic
Foundation, courtesy of R. Detrano. The problem concerns the prediction of
the presence or absence of heart disease given the results of various medical tests
carried out on a patient. There are two classes, seven numerical attributes, six
categorical attributes, and 270 records. The StatLog Project employed unequal
misclassification costs. We use equal costs here because some algorithms do
not allow unequal costs. The error rates are estimated using ten-fold cross-validation

Boston housing (bos). This UCI dataset gives housing values in Boston suburbs
(Harrison and Rubinfeld, 1978). There are three classes, twelve numerical
attributes, one binary attribute, and 506 records. Following Loh and Vanichse-
takul (1988), the classes are created from the attribute median value of owner-occupied
homes as follows: class
otherwise. The error rates are
estimated using ten-fold cross-validation.
LED display (led). This artificial domain is described in Breiman et al. (1984).
It contains seven Boolean attributes, representing seven light-emitting diodes,
and ten classes, the set of decimal digits. An attribute value is either zero or
one, according to whether the corresponding light is off or on for the digit. Each
attribute value has a ten percent probability of having its value inverted. The
class attribute is an integer between zero and nine, inclusive. A C program
from UCI is used to generate 2000 records for the training set and 4000 records
for the test set. The error rates are estimated from the test set.
BUPA liver disorders (bld). This UCI dataset was contributed by R. S. Forsyth.
The problem is to predict whether or not a male patient has a liver disorder
based on blood tests and alcohol consumption. There are two classes, six numerical
attributes, and 345 records. The error rates are estimated using ten-fold
cross-validation.
PIMA Indian diabetes (pid). This UCI dataset was contributed by V. Sigillito.
The patients in the dataset are females at least twenty-one years old of Pima
Indian heritage living near Phoenix, Arizona, USA. The problem is to predict
whether a patient would test positive for diabetes given a number of physiological
measurements and medical test results. There are two classes, seven numerical
attributes, and 532 records. The original dataset consists of 768 records with
eight numerical attributes. However, many of the attributes, notably serum in-
sulin, contain zero values which are physically impossible. We remove serum
insulin and records that have impossible values in other attributes. The error
rates are estimated using ten-fold cross validation.
StatLog satellite image (sat). This UCI dataset gives the multi-spectral values
of pixels within 3 \Theta 3 neighborhoods in a satellite image, and the classification
associated with the central pixel in each neighborhood. The aim is to predict
the classification given the multi-spectral values. There are six classes and
thirty-six numerical attributes. The training set consists of 4435 records while
the test set consists of 2000 records. The error rates are estimated from the test
set.
Image segmentation (seg). This UCI dataset was used in the StatLog Project.
The samples are from a database of seven outdoor images. The images are
hand-segmented to create a classification for every pixel as one of brickface,
sky, foliage, cement, window, path, or grass. There are seven classes, nineteen
numerical attributes and 2310 records in the dataset. The error rates are
estimated using ten-fold cross-validation.
The algorithm T1 could not handle this dataset without modification, because
the program requires a large amount of memory. Therefore for T1 (but not for
the other algorithms) we discretize each attribute except attributes 3, 4, and 5
into one hundred categories.
Attitude towards smoking restrictions (smo). This survey dataset (Bull, 1994)
is obtained from http://lib.stat.cmu.edu/datasets/csb/. The problem is
to predict attitude toward restrictions on smoking in the workplace (prohibited,
restricted, or unrestricted) based on bylaw-related, smoking-related, and sociodemographic
covariates. There are three classes, three numerical attributes,
and five categorical attributes. We divide the original dataset into a training
set of size 1855 and a test set of size 1000. The error rates are estimated from
the test set.
Thyroid disease (thy). This is the UCI ann-train.datacontributed by R. Werner.
The problem is to determine whether or not a patient is hyperthyroid. There
are three classes (normal, hyperfunction, and subnormal functioning), six numerical
attributes, and fifteen binary attributes. The training set consists of
3772 records and the test set has 3428 records. The error rates are estimated
from the test set.
StatLog vehicle silhouette (veh). This UCI dataset originated from the Turing
Institute, Glasgow, Scotland. The problem is to classify a given silhouette as
one of four types of vehicle, using a set of features extracted from the silhouette.
Each vehicle is viewed from many angles. The four model vehicles are double
decker bus, Chevrolet van, Saab 9000, and Opel Manta 400. There are four
classes, eighteen numerical attributes, and 846 records. The error rates are
estimated using ten-fold cross-validation.
Congressional voting records (vot). This UCI dataset gives the votes of each
member of the U. S. House of Representatives of the 98th Congress on sixteen
issues. The problem is to classify a Congressman as a Democrat or a
Republican based on the sixteen votes. There are two classes, sixteen categorical
attributes with three categories each ("yea", "nay", or neither), and 435 records.
rates are estimated by ten-fold cross-validation.
Waveform (wav). This is an artificial three-class problem based on three wave-
forms. Each class consists of a random convex combination of two waveforms
sampled at the integers with noise added. A description for generating the data
is given in Breiman et al. (1984) and a C program is available from UCI. There
are twenty-one numerical attributes, and 600 records in the training set. Error
rates are estimated from an independent test set of 3000 records.
evaluation (tae). The data consist of evaluations of teaching performance
over three regular semesters and two summer semesters of 151 teaching assistant
(TA) assignments at the Statistics Department of the University of
Wisconsin-Madison. The scores are grouped into three roughly equal-sized
categories ("low", "medium", and "high") to form the class attribute. The predictor
attributes are (i) whether or not the is a native English speaker (bi-
nary), (ii) course instructor (25 categories), (iii) course (26 categories), (iv) summer
or regular semester (binary), and (v) class size (numerical). This dataset
is first reported in Loh and Shih (1997). It differs from the other datasets
in that there are two categorical attributes with large numbers of categories.
As a result, decision tree algorithms such as CART that employ exhaustive
search usually take much longer to train than other algorithms. (CART has
to evaluate 2 splits for each categorical attribute with c values.) Error
rates are estimated using ten-fold cross-validation. The data are obtained from
http://www.stat.wisc.edu/p/stat/ftp/pub/loh/treeprogs/datasets/.
A summary of the attribute features of the datasets is given in Table 1.

Table

1. Characteristics of the datasets. The last three columns give the number and type of added noise attributes for
each dataset. The notation "N(0,1)" denotes the standard normal distribution, "UI(m,n)" denotes a uniform distribution
over the integers m through n inclusive, and "U(0,1)" denotes a uniform distribution over the unit interval.
Training No. of original attributes No. and type of noise attributes
Data sample No. of Num. Categorical Total Numerical Categorical Total
set size classes 2 3 4 5 25 26
dna 2000 3
led 2000
bld
pid
smo
thy
veh 846 4
vot 435 2
tae
4. Experimental setup
Some algorithms are not designed for categorical attributes. In these cases, each
categorical attribute is converted into a vector of 0-1 attributes. That is, if a
categorical attribute X takes k values fc g, it is replaced by a
1)-dimensional vector (d
otherwise, for , the vector consists of all zeros. The
affected algorithms are all the statistical and neural network algorithms as well as
the tree algorithms FTL, OCU, OCL, OCM, and LMT.
In order to increase the number of datasets and to study the effect of noise attributes
on each algorithm, we created sixteen new datasets by adding independent
noise attributes. The numbers and types of noise attributes added are given in the
right panel of Table 1. The name of each new dataset is the same as the original
dataset except for the addition of a '+' symbol. For example, the bcw dataset with
noise added is denoted by bcw+.
For each dataset, we use one of two different ways to estimate the error rate of
an algorithm. For large datasets (size much larger than 1000 and test set of size at
least 1000), we use a test set to estimate the error rate. The classifier is constructed
using the records in the training set and then it is tested on the test set. Twelve of
the thirty-two datasets are analyzed this way.
For the remaining twenty datasets, we use the following ten-fold cross-validation
procedure to estimate the error rate:
1. The dataset is randomly divided into ten disjoint subsets, with each containing
approximately the same number of records. Sampling is stratified by the class
labels to ensure that the subset class proportions are roughly the same as those
in the whole dataset.
2. For each subset, a classifier is constructed using the records not in it. The classifier
is then tested on the withheld subset to obtain a cross-validation estimate
of its error rate.
3. The ten cross-validation estimates are averaged to provide an estimate for the
classifier constructed from all the data.
Because the algorithms are implemented in different programming languages and
some languages are not available on all platforms, three types of UNIX workstations
are used in our study. The workstation type and implementation language for
each algorithm are given in Table 2. The relative performance of the workstations
according to SPEC marks is given in Table 3. The floating point SPEC marks
show that a task that takes one second on a DEC 3000 would take about 1.4 and
0.8 seconds on a SPARCstation 5 (SS5) and SPARCstation 20 (SS20), respectively.
Therefore, to enable comparisons, all training times are reported here in terms of
3000-equivalent seconds-the training times recorded on a SS5 and a SS20
are divided by 1.4 and 0.8, respectively.
5. Results
The error rates and training times for the algorithms are given in a separate table
for each dataset in the Appendix. The tables also report the error rates of the
'naive' plurality rule, which ignores the information in the covariates and classifies
every record to the majority class in the training sample.
5.1. Exploratory analysis of error rates
Before we present a formal statistical analysis of the results, it is helpful to study
the summary in Table 4. The mean error rate for each algorithm over the datasets
is given in the second row. The minimum and maximum error rates and that of

Table

2. Hardware and software platform for each algorithm. The workstations are DEC 3000 Alpha
Model 300 (DEC), Sun SPARCstation 20 Model 61 (SS20), and Sun SPARCstation 5 (SS5).
Algorithm Platform Algorithm Platform
Tree & Rules ST1 Splus tree, 1-SE DEC/S
QU0 QUEST, univariate 0-SE DEC/F90 LMT LMDT, linear DEC/C
QU1 QUEST, univariate 1-SE DEC/F90 CAL CAL5 SS5/C++
QL0 QUEST, linear 0-SE DEC/F90 single split DEC/C
QL1 QUEST, linear 1-SE DEC/F90
FTU FACT, univariate DEC/F77 Statistical
linear DEC/F77 LDA Linear discriminant anal. DEC/SAS
C4T C4.5 trees DEC/C QDA Quadratic discriminant anal. DEC/SAS
C4R C4.5 rules DEC/C NN Nearest-neighbor DEC/SAS
IB IND bayes style SS5/C LOG Linear logistic regression DEC/F90
IBO IND bayes opt style SS5/C FM1 FDA, degree 1 SS20/S
IM IND mml style SS5/C FM2 FDA, degree 2 SS20/S
IMO IND mml opt style SS5/C PDA Penalized LDA SS20/S
IC0 IND cart, 0-SE SS5/C MDA Mixture discriminant anal. SS20/S
IC1 IND cart, 1-SE SS5/C POL POLYCLASS SS20/S
OCU OC1, univariate SS5/C
OCL OC1, linear SS5/C Neural Network
OCM OC1, mixed SS5/C LVQ Learning vector quantization SS20/S
ST0 Splus tree, 0-SE DEC/S RBF Radial basis function network DEC/SAS

Table

3. SPEC benchmark summary
Workstation SPECfp92 SPECint92 Source
DEC DEC 3000 Model 300 91.5 66.2 SPEC Newsletter
(150MHz) Vol. 5, Issue 2, June 1993
SS20 Sun SPARCstation 20 102.8 88.9 SPEC Newsletter
Model 61 (60MHz) Vol. 6, Issue 2, June 1994
SS5 Sun SPARCstation 5 47.3 57.0 SPEC Newsletter
(70MHz) Vol. 6, Issue 2, June 1994
the plurality rule are given for each dataset in the last three columns. Let p denote
the smallest observed error rate in each row (i.e., dataset). If an algorithm has an
error rate within one standard error of p, we consider it to be close to the best and
indicate it by a p
in the table. The standard error is estimated as follows. If p is
from an independent test set, let n denote the size of the test set. Otherwise, if p is
a cross-validation estimate, let n denote the size of the training set. The standard
error of p is estimated by the formula
p)=n. The algorithm with the largest
error rate within a row is indicated by an X. The total numbers of p
and X-marks
for each algorithm are given in the third and fourth rows of the table.
The following conclusions may be drawn from the table:

Table

4. Minimum, maximum, and 'naive' plurality rule error rates for each dataset. A ` p '-mark indicates
that the algorithm has an error rate within one standard error of the minimum for the dataset. A 'X'-mark
indicates that the algorithm has the worst error rate for the dataset. The mean error rate for each algorithm
is given in the second row.
Decision trees and rules Statistical algorithms Nets Error rates
Naive
Mean
bcw
cmc
pp
pp
dna
dna+
hea
bos
led
bld
pid
seg
smo
thy
ppp ppp pp
ppp pp
vot
tae X
tae+
pp
1. Algorithm POL has the lowest mean error rate. An ordering of the other algorithms
in terms of mean error rate is given in the upper half of Table 5.
2. The algorithms can also be ranked in terms of total number of p
- and X-marks.
By this criterion, the most accurate algorithm is again POL, which has fifteen p

Table

5. Ordering of algorithms by mean error rate and mean rank of error rate
Mean POL LOG MDA QL0 LDA QL1 PDA IC0 FM2 IBO IMO
error .195 .204 .207 .207 .208 .211 .213 .215 .218 .219 .219
rate C4R IM LMT C4T QU0 QU1 OCU IC1 IB OCM ST0
Mean POL FM1 LOG FM2 QL0 LDA QU0 C4R IMO MDA PDA
rank 8.3 12.2 12.2 12.2 12.4 13.7 13.9 14.0 14.0 14.3 14.5
of C4T QL1 IBO IM IC0 FTL QU1 OCU IC1 ST0 ST1
error 14.5 14.6 14.7 14.9 15.0 15.4 16.6 16.6 16.9 17.0 17.7
rate LMT OCM IB RBF FTU QDA LVQ OCL CAL NN
marks and no X-marks. Eleven algorithms have one or more X-marks. Ranked
in increasing order of number of X-marks (in parentheses), they are:
FTL(1), OCM(1), ST1(1), FM2(1), MDA(1), FM1(2),
OCL(3), QDA(3), NN(4), LVQ(4), T1(11).
Excluding these, the remaining algorithms rank in order of decreasing number
of p
-marks (in parentheses) as:
POL(15), LOG(13), QL0(10), LDA(10), PDA(10), QL1(9), OCU(9), (1)
QU0(8), QU1(8), C4R(8), IBO(8), RBF(8), C4T(7), IMO(6),
IM(5), IC1(5), ST0(5), FTU(4), IC0(4), CAL(4), IB(3), LMT(1).
The top four algorithms in (1) also rank among the top five in the upper half
of

Table

5.
3. The last three columns of the table show that a few algorithms are sometimes
less accurate than the plurality rule. They are NN (at cmc, cmc+, smo+),
bld+), QDA (smo, thy, thy+), FTL (tae), and ST1 (tae+).
4. The easiest datasets to classify are bcw, bcw+, vot, and vot+; the error rates all
lie between 0.03 and 0.09.
5. The most difficult to classify are cmc, cmc+, and tae+, with minimum error
rates greater than 0.4.
6. Two other difficult datasets are smo and smo+. In the case of smo, only T1 has a
(marginally) lower error rate than that of the plurality rule. No algorithm has
a lower error rate than the plurality rule for smo+.
7. The datasets with the largest range of error rates are thy and thy+, where the
rates range from 0.005 to 0.890. However, the maximum of 0.890 is due to QDA.
If QDA is ignored, the maximum error rate drops to 0.096.
8. There are six datasets with only one p
-mark each. They are bld+ (POL), sat
(LVQ), sat+ (FM2), seg+ (IBO), veh and veh+ (QDA both times).
9. Overall, the addition of noise attributes does not appear to increase significantly
the error rates of the algorithms.
5.2. Statistical significance of error rates
5.2.1. Analysis of variance A statistical procedure called mixed effects analysis
of variance can be used to test the simultaneous statistical significance of differences
between mean error rates of the algorithms, while controlling for differences between
datasets (Neter, Wasserman and Kutner, 1990, p. 800). Although it makes the
assumption that the effects of the datasets act like a random sample from a normal
distribution, it is quite robust against violation of the assumption. For our data,
the procedure gives a significance probability less than 10 \Gamma4 . Hence the hypothesis
that the mean error rates are equal is strongly rejected.
Simultaneous confidence intervals for differences between mean error rates can be
obtained using the Tukey method (Miller, 1981, p. 71). According to this procedure,
a difference between the mean error rates of two algorithms is statistically significant
at the 10% level if they differ by more than 0.058.
To visualize this result, Figure 1(a) plots the mean error rate of each algorithm
versus its median training time in seconds. The solid vertical line in the plot is
units to the right of the mean error rate for POL. Therefore any algorithm
lying to the left of the line has a mean error rate that is not statistically significantly
different from that of POL.
The algorithms are seen to form four clusters with respect to training time. These
clusters are roughly delineated by the three horizontal dotted lines which correspond
to training times of one minute, ten minutes, and one hour. Figure 1(b) shows a
magnified plot of the eighteen algorithms with median training times less than ten
minutes and mean error rate not statistically significantly different from POL.
5.2.2. Analysis of ranks To avoid the normality assumption, we can instead
analyze the ranks of the algorithms within datasets. That is, for each dataset, the
algorithm with the lowest error rate is assigned rank one, the second lowest rank
two, etc., with average ranks assigned in the case of ties. The lower half of Table 5
gives an ordering of the algorithms in terms of mean rank of error rates. Again POL
is first and last. Note, however, that the mean rank of POL is 8.3. This shows
that it is far from being uniformly most accurate across datasets.
Comparing the two methods of ordering in Table 5, it is seen that POL, LOG,
QL0, and LDA are the only algorithms with consistently good performance. Three
algorithms that perform well by one criterion but not the other are MDA, FM1, and
FM2. In the case of MDA, its low mean error rate is due to its excellent performance
in four datasets (veh, veh+, wav, and wav+) where many other algorithms do poorly.
These domains concern shape identification and the datasets contain only numerical
Mean error rate
Median
sec.
FTU
C4R
IB
IBO
IM
IMO
OCU
OCL
OCM ST0 ST1
LDA
QDA
PDA
MDA
POL
RBF
1hr
10min
1min
(a) All thirty-three methods
Mean error rate
Median
sec.
FTU
C4R
IB
IM
OCU
LDA
PDA
MDA
1min
5min
(b) Less than 10min., accuracy not sig. different from POL

Figure

1. Plots of median training time versus mean error rate. The vertical axis is in log-scale.
The solid vertical line in plot (a) divides the algorithms into two groups: the mean error rates of
the algorithms in the left group do not differ significantly (at the 10% simultaneous significance
level) from that of POL, which has the minimum mean error rate. Plot (b) shows the algorithms
that are not statistically significantly different from POL in terms of mean error rate and that have
median training time less than ten minutes.
attributes. MDA is generally unspectacular in the rest of the datasets and this is the
reason for its tenth place ranking in terms of mean rank.
The situation for FM1 and FM2 is quite different. As its low mean rank indicates,
FM1 is usually a good performer. However, it fails miserably in the seg and seg+
datasets, reporting error rates of more than fifty percent when most of the other
algorithms have error rates less than ten percent. Thus FM1 seems to be less robust
than the other algorithms. FM2 also appears to lack robustness, although to a
lesser extent. Its worst performance is in the bos+ dataset, where it has an error
rate of forty-two percent, compared to less than thirty-five percent for the other
algorithms. The number of X-marks against an algorithm in Table 4 is a good
predictor of erratic if not poor performance. MDA, FM1, and FM2 all have at least
one X-mark.
The Friedman (1937) test is a standard procedure for testing statistical significance
in differences of mean ranks. For our experiment, it gives a significance
probability less than 10 \Gamma4 . Therefore the null hypothesis that the algorithms are
equally accurate on average is again rejected. Further, a difference in mean ranks
greater than 8.7 is statistically significant at the 10% level (Hollander and Wolfe,
1999, p. 296). Thus POL is not statistically significantly different from the twenty
other algorithms that have mean rank less than or equal to 17.0. Figure 2(a) shows
a plot of the median training time versus the mean ranks of the algorithms. Those
algorithms that lie to the left of the vertical line are not statistically significantly
different from POL. A magnified plot of the subset of algorithms that are not significantly
different from POL and that have median training time less than ten minutes
is given in Figure 2(b).
The algorithms that differ statistically significantly from POL in terms of mean
error rate form a subset of those that differ from POL in terms of mean ranks. Thus
the rank test appears to be more powerful than the analysis of variance test for this
experiment. The fifteen algorithms in Figure 2(b) may be recommended for use in
applications where good accuracy and short training time are desired.
5.3. Training time

Table

6 gives the median DEC 3000-equivalent training time for each algorithm
and the relative training time within datasets. Owing to the large range of training
times, only the order relative to the fastest algorithm for each dataset is reported.
The fastest algorithm is indicated by a '0'. An algorithm that is between 10 x\Gamma1 to
times as slow is indicated by the value of x. For example, in the case of the
dna+ dataset, the fastest algorithms are C4T and T1, each requiring two seconds.
The slowest algorithm is FM2, which takes more than three million seconds (almost
forty days) and hence is between 10 6 to 10 7 times as slow. The last two columns
of the table give the fastest and slowest times for each dataset.

Table

7 gives an ordering of the algorithms from fastest to slowest according to
median training time. Overall, the fastest algorithm is C4T, followed closely by FTU,
FTL, and LDA. There are two reasons for the superior speed of C4T compared to the
other decision tree algorithms. First, it splits each categorical attribute into as
Mean rank
Median
sec.
FTU
C4R
IB
IBO
IM
IMO
OCU
OCL
OCM
LDA
QDA
PDA
MDA
POL
RBF
1hr
10min
1min
(a) All thirty-three methods
Mean rank
Median
sec.
C4R
IM
OCU
LDA
PDA
MDA
1min
5min
(b) Less than 10min., accuracy not sig. different from POL

Figure

2. Plots of median training time versus mean rank of error rates. The vertical axis is in
log-scale. The solid vertical line in plot (a) divides the algorithms into two groups: the mean ranks
of the algorithms in the left group do not differ significantly (at the 10% simultaneous significance
level) from that of POL. Plot (b) shows the algorithms that are not statistically significantly
different from POL in terms of mean rank and that have median training time less than ten
minutes.
Table

6. DEC 3000-equivalent training times and relative times of the algorithms. The second and third rows
give the median training time and rank for each algorithm. An entry of 'x' in the each of the subsequent rows
indicates that an algorithm is between times slower than the fastest algorithm for the dataset. The
fastest algorithm is denoted by an entry of '0'. The minimum and maximum training times are given in the
last two columns. 's', `m', 'h', `d' denote seconds, minutes, hours, and days, respectively.
Decision trees and rules Statistical algorithms Nets CPU time
Median CPU 3.2m 3.2m 5.9m 5.9m 7s 8s 5s 20s 34s 27.5m 34s 33.9m 52s 47s 46s 14.9m 13.7m 15.1m 14.4m 5.7m 1.3h 36s 10s 15s 20s 4m 15.6m 3.8h 56s 3m 3.2h 1.1m 11.3h 5s
Rank

Table

7. Ordering of algorithms by median training time
5s 7s 8s 10s 15s 20s 20s 34s 34s 36s 46s
47s 52s 56s 1.1m 3m 3.2m 3.2m 4m 5.7m 5.9m 5.9m
OCM
13.7m 14.4m 14.9m 15.1m 15.6m 27.5m 33.9m 1.3h 3.2h 3.8h 11.3h
many subnodes as the number of categories. Therefore it wastes no time in forming
subsets of categories. Second, its pruning method does not require cross-validation,
which can increase training time several fold.
The classical statistical algorithms QDA and NN are also quite fast. As expected,
decision tree algorithms that employ univariate splits are faster than those that use
linear combination splits. The slowest algorithms are POL, FM2, and RBF; two are
spline-based and one is a neural network.
Although IC0, IC1, ST0 and ST1 all claim to implement the CART algorithm, the
IND versions are faster than the S-Plus versions. One reason is that IC0 and IC1
are written in C whereas ST0 and ST1 are written in the S language. Another reason
is that the IND versions use heuristics (Buntine, personal communication) instead
of greedy search when the number of categories in a categorical attribute is large.
This is most apparent in the tae+ dataset where there are categorical attributes
with up to twenty-six categories. In this case IC0 and IC1 take around forty seconds
versus two and a half hours for ST0 and ST1. The results in Table 4 indicate that
IND's classification accuracy is not adversely affected by such heuristics; see Aronis
and Provost (1997) for another possible heuristic.
Since T1 is a one-level tree, it may appear surprising that it is not faster than
algorithms such as C4T that produce multi-level trees. The reason is that splits
each continuous attribute into J is the number of classes.
On the other hand, C4T always splits a continuous attribute into two intervals only.
Therefore when J ? 2, T1 has to spend a lot more time to search for the intervals.
5.4. Size of trees

Table

8 gives the number of leaves for each tree algorithm and dataset before noise
attributes are added. In the case that an error rate is obtained by ten-fold cross-
validation, the entry is the mean number of leaves over the ten cross-validation
trees.

Table

9 shows how much the number of leaves changes after addition of noise
attributes. The mean and median of the number of leaves for each classifier are
given in the last columns of the two tables. IBO and IMO clearly yield the largest
trees by far. Apart from T1, which is necessarily short by design, the algorithm with
the shortest trees on average is QL1, followed closely by FTL and OCL. A ranking
of the algorithms with univariate splits (in increasing median number of leaves) is:
T1, IC1, ST1, QU1, FTU, IC0, ST0, OCU, QU0, and C4T. Algorithm C4T tends
to produce trees with many more leaves than the other algorithms. One reason
may be due to under-pruning (although its error rates are quite good). Another is
that, unlike the binary-tree algorithms, C4T splits each categorical attribute into as
many nodes as the number of categories.
Addition of noise attributes typically decreases the size of the trees, except for C4T
and CAL which tend to grow larger trees, and IMO which seems to fluctuate rather
wildly. These results complement those of Oates and Jensen (1997) who looked
at the effect of sample size on the number of leaves of decision tree algorithms
and found a significant relationship between tree size and training sample size for
C4T. They observed that tree algorithms which employ cost-complexity pruning are
better able to control tree growth.
6. Scalability of algorithms
Although differences in mean error rates between POL and many other algorithms
are not statistically significant, it is clear that if error rate is the sole criterion, POL
would be the method of choice. Unfortunately, POL is one of the most compute-intensive
1algorithms. To see how training times increase with sample size, a small
scalability study was carried out with the algorithms QU0, QL0, FTL, C4T, C4R, IC0,
LOG, FM1, and POL.
Training times are measured for these algorithms for training sets of size
Four datasets are used to generate the samples-sat, smo+,
tae+, and a new, very large UCI dataset called adult which has two classes and six
continuous and seven categorical attributes. Since the first three datasets are not
large enough for the experiment, bootstrap re-sampling is employed to generate the
training sets. That is, N samples are randomly drawn with replacement from each
dataset. To avoid getting many replicate records, the value of the class attribute
for each sampled case is randomly changed to another value with probability 0.1.
(The new value is selected from the pool of alternatives with equal probability.)
Bootstrap sampling is not carried out for the adult dataset because it has more
than 32,000 records. Instead, the nested training sets are obtained by random
sampling without replacement.
The times required to train the algorithms are plotted (in log-log scale) in Figure
3. With the exception of POL, FM1 and LOG, the logarithms of the training
times seem to increase linearly with log(N ). The non-monotonic behavior of POL
and FM1 is puzzling and might be due to randomness in their use of cross-validation
for model selection. The erratic behavior of LOG in the adult dataset is caused by
convergence problems during model fitting.
Many of the lines in Figure 3 are roughly parallel. This suggests that the relative
computational speed of the algorithms is fairly constant over the range of sample
sizes considered. QL0 and C4R are two exceptions. Cohen (1995) had observed that
C4R does not scale well.
7. Conclusions
Our results show that the mean error rates of many algorithms are sufficiently
similar that their differences are statistically insignificant. The differences are also
probably insignificant in practical terms. For example, the mean error rates of
the top ranked algorithms POL, LOG, and QL0 differ by less than 0.012. If such a
small difference is not important in real applications, the user may wish to select
an algorithm based on other criteria such as training time or interpretability of the
classifier.
Unlike error rates, there are huge differences between the training times of the
algorithms. POL, the algorithm with the lowest mean error rate, takes about fifty
times as long to train as the next most accurate algorithm. The ratio of times is
roughly equivalent to hours versus minutes, and Figure 3 shows that it is maintained
over a wide range of sample sizes. For large applications where time is a factor, it
may be advantageous to use one of the quicker algorithms.
It is interesting that the old statistical algorithm LDA has a mean error rate
close to the best. This is surprising because (i) it is not designed for binary-valued
attributes (all categorical attributes are transformed to 0-1 vectors prior to
application of LDA), and (ii) it is not expected to be effective when class densities
CPU
time
CPU
time
tae+
CPU
time
adult
CPU
time

Figure

3. Plots of training time versus sample size in log-log scale for selected algorithms.
are multi-modal. Because it is fast, easy to implement, and readily available in
statistical packages, it provides a convenient benchmark for comparison against
future algorithms.
The low error rates of LOG and LDA probably account for much of the performance
of the better algorithms. For example, POL is basically a modern version of LOG. It
enhances the flexibility of LOG by employing spline-based functions and automatic
model selection. Although this strategy is computationally costly, it does produce
a slight reduction in the mean error rate-enough to bring it to the top of the pack.
The good performance of QL0 may be similarly attributable to LDA. The QUEST
linear-split algorithm is designed to overcome the difficulties encountered by LDA
in multi-modal situations. It does this by applying a modified form of LDA to
partitions of the data, where each partition is represented by a leaf of the decision
tree. This strategy alone, however, is not enough, as the higher mean error rate
of FTL shows. The latter is based on the FACT algorithm which is a precursor
to QUEST. One major difference between the QUEST and FACT algorithms is
that the former employs the cost-complexity pruning method of CART whereas
the latter does not. Our results suggest that some form of bottom-up pruning may
be essential for low error rates.
If the purpose of constructing an algorithm is for data interpretation, then perhaps
only decision rules or trees with univariate splits will suffice. With the exception
of CAL and T1, the differences in mean error rates of the decision rule and tree
algorithms are not statistically significant from that of POL. IC0 has the lowest
mean error rate and QU0 is best in terms of mean ranks. C4R and C4T are not far
behind. Any of these four algorithms should provide good classification accuracy.
C4T is the fastest by far, although it tends to yield trees with twice as many leaves as
IC0 and QU0. C4R is the next fastest, but Figure 3 shows that it does not scale well.
IC0 is slightly faster and its trees have slightly fewer leaves than QU0. However,
Loh and Shih (1997) show that CART-based algorithms such as IC0 are prone to
produce spurious splits in some situations.

Acknowledgments

We are indebted to P. Auer, C. E. Brodley, W. Buntine, T. Hastie, R. C. Holte,
C. Kooperberg, S. K. Murthy, J. R. Quinlan, W. Sarle, B. Schulmeister, and W.
Taylor for help and advice on the installation of the computer programs. We
are also grateful to J. W. Molyneaux for providing the 1987 National Indonesia
Contraceptive Prevalence Survey data. Finally, we thank W. Cohen, F. Provost,
and the reviewers for many helpful comments and suggestions.



--R

Categorical Data Analysis
Increasing the efficiency of data mining algorithms with breadth-first marker propagation

The New S Language
Neural Networks for Pattern Recognition
Classification and Regression Trees
Simplifying decision trees: A survey
Multivariate versus univariate decision trees
Multivariate decision trees
A comparison of decision tree classifiers with backpropagation neural networks for multimodal classification problems
Analysis of attitudes toward workplace smoking restrictions
Learning classification trees
Introduction to IND Version 2.1 and Recursive Partitioning

Fast effective rule induction
Neural networks
Multivariate adaptive regression splines (with discussion)
The use of ranks to avoid the assumption of normality implicit in the analysis of variance
Construction and Assessment of Classification Rules
Hedonic prices and the demand for clean air
Discriminant analysis by Gaussian mixtures
Penalized discriminant analysis
Flexible discriminant analysis by optimal scoring
Nonparametric Statistical Methods
Very simple classification rules perform well on most commonly used datasets
Applied Multivariate Statistical Analysis

Polychotomous regression



Cancer diagnosis via linear programming
UCI Repository of Machine Learning Databases

Machine Learning

Automatic construction of decision trees for classification
The decision-tree algorithm CAL5 based on a statistical approach to its splitting algorithm
A system for induction of oblique decision trees
Applied Linear Statistical Models
The effects of training set size on decision tree complexity

Improved use of continuous attributes in C4.
Pattern Recognition and Neural Networks
Neural networks and statistical models

SAS Institute
Symbolic and neural learning algorithms: an empirical comparison
Modern Applied Statistics with S-Plus
Diagnostic schemes for fine needle aspirates of breast masses
Fine needle aspiration for breast mass diagnosis
Statistical approach to fine needle aspiration diagnosis of breast masses
--TR
Applied multivariate statistical analysis
Symbolic and Neural Learning Algorithms
C4.5: programs for machine learning
Very Simple Classification Rules Perform Well on Most Commonly Used Datasets
Multivariate Decision Trees
Self-organizing maps
SAS/ETS User''s Guide, Version 6
Neural Networks for Pattern Recognition
The Effects of Training Set Size on Decision Tree Complexity
Multivariate Versus Univariate Decision Trees
Simplifying decision trees: A survey

--CTR
Ganesan Velayathan , Seiji Yamada, Behavior-based web page evaluation, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
Ganesan Velayathan , Seiji Yamada, Behavior-Based Web Page Evaluation, Proceedings of the 2006 IEEE/WIC/ACM international conference on Web Intelligence and Intelligent Agent Technology, p.409-412, December 18-22, 2006
Samuel E. Buttrey , Ciril Karo, Using k-nearest-neighbor classification in the leaves of a tree, Computational Statistics & Data Analysis, v.40 n.1, p.27-37, 28 July 2002
Kweku-Muata Osei-Bryson, Evaluation of decision trees: a multi-criteria approach, Computers and Operations Research, v.31 n.11, p.1933-1945, September 2004
Richi Nayak , Laurie Buys , Jan Lovie-Kitchin, Data mining in conceptualising active ageing, Proceedings of the fifth Australasian conference on Data mining and analystics, p.39-45, November 29-30, 2006, Sydney, Australia
Xiangyang Li , Nong Ye, A supervised clustering algorithm for computer intrusion detection, Knowledge and Information Systems, v.8 n.4, p.498-509, November 2005
Laura Elena Raileanu , Kilian Stoffel, Theoretical Comparison between the Gini Index and Information Gain Criteria, Annals of Mathematics and Artificial Intelligence, v.41 n.1, p.77-93, May 2004
Nong Ye , Xiangyang Li, A scalable, incremental learning algorithm for classification problems, Computers and Industrial Engineering, v.43 n.4, p.677-692, September 2002
Jonathan Eckstein , Peter L. Hammer , Ying Liu , Mikhail Nediak , Bruno Simeone, The Maximum Box Problem and its Application to Data Analysis, Computational Optimization and Applications, v.23 n.3, p.285-298, December 2002
Sattar Hashemi , Mohammad R. Kangavari, Parallel learning using decision trees: a novel approach, Proceedings of the 4th WSEAS International Conference on Applied Mathematics and Computer Science, p.1-8, April 25-27, 2005, Rio de Janeiro, Brazil
Khaled M. S. Badran , Peter I. Rockett, The roles of diversity preservation and mutation in preventing population collapse in multiobjective genetic programming, Proceedings of the 9th annual conference on Genetic and evolutionary computation, July 07-11, 2007, London, England
Nigel Williams , Sebastian Zander , Grenville Armitage, A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification, ACM SIGCOMM Computer Communication Review, v.36 n.5, October 2006
Sorin Alexe , Peter L. Hammer, Accelerated algorithm for pattern detection in logical analysis of data, Discrete Applied Mathematics, v.154 n.7, p.1050-1063, 1 May 2006
S. Ruggieri, Efficient C4.5, IEEE Transactions on Knowledge and Data Engineering, v.14 n.2, p.438-444, March 2002
Md. Zahidul Islam , Ljiljana Brankovic, A framework for privacy preserving classification in data mining, Proceedings of the second workshop on Australasian information security, Data Mining and Web Intelligence, and Software Internationalisation, p.163-168, January 01, 2004, Dunedin, New Zealand
Efstathios Stamatatos , Gerhard Widmer, Automatic identification of music performers with learning ensembles, Artificial Intelligence, v.165 n.1, p.37-56, June 2005
Niels Landwehr , Mark Hall , Eibe Frank, Logistic Model Trees, Machine Learning, v.59 n.1-2, p.161-205, May       2005
Kar-Ann Toh , Quoc-Long Tran , Dipti Srinivasan, Benchmarking a Reduced Multivariate Polynomial Pattern Classifier, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.6, p.740-755, June 2004
Abraham Bernstein , Foster Provost , Shawndra Hill, Toward Intelligent Assistance for a Data Mining Process: An Ontology-Based Approach for Cost-Sensitive Classification, IEEE Transactions on Knowledge and Data Engineering, v.17 n.4, p.503-518, April 2005
Rich Caruana , Alexandru Niculescu-Mizil, An empirical comparison of supervised learning algorithms, Proceedings of the 23rd international conference on Machine learning, p.161-168, June 25-29, 2006, Pittsburgh, Pennsylvania
Nicolas Baskiotis , Michle Sebag, C4.5 competence map: a phase transition-inspired approach, Proceedings of the twenty-first international conference on Machine learning, p.10, July 04-08, 2004, Banff, Alberta, Canada
Gabriela Alexe , Peter L. Hammer, Spanned patterns for the logical analysis of data, Discrete Applied Mathematics, v.154 n.7, p.1039-1049, 1 May 2006
Ingolf Geist, A framework for data mining and KDD, Proceedings of the 2002 ACM symposium on Applied computing, March 11-14, 2002, Madrid, Spain
Anthony J. T. Lee , Yao-Te Wang, Efficient data mining for calling path patterns in GSM networks, Information Systems, v.28 n.8, p.929-948, December
Kweku-Muata Osei-Bryson, Post-pruning in decision tree induction using multiple performance measures, Computers and Operations Research, v.34 n.11, p.3331-3345, November, 2007
Friedhelm Schwenker , Hans A. Kestler , Gther Palm, Unsupervised and supervised learning in radial-basis-function networks, Self-Organizing neural networks: recent advances and applications, Springer-Verlag New York, Inc., New York, NY, 2001
Chen-Fu Chien , Wen-Chih Wang , Jen-Chieh Cheng, Data mining for yield enhancement in semiconductor manufacturing and an empirical study, Expert Systems with Applications: An International Journal, v.33 n.1, p.192-198, July, 2007
Irma Becerra-Fernandez , Stelios H. Zanakis , Steven Walczak, Knowledge discovery techniques for predicting country investment risk, Computers and Industrial Engineering, v.43 n.4, p.787-800, September 2002
Zhiwei Fu , Bruce L. Golden , Shreevardhan Lele , S. Raghavan , Edward Wasil, Diversification for better classification trees, Computers and Operations Research, v.33 n.11, p.3185-3202, November 2006
Ruey-Shiang Guh, A hybrid learning-based model for on-line detection and analysis of control chart patterns, Computers and Industrial Engineering, v.49 n.1, p.35-62, August 2005
Krzysztof Krawiec, Genetic Programming-based Construction of Features for Machine Learning and Knowledge Discovery Tasks, Genetic Programming and Evolvable Machines, v.3 n.4, p.329-343, December 2002
Huimin Zhao , Sudha Ram, Combining schema and instance information for integrating heterogeneous data sources, Data & Knowledge Engineering, v.61 n.2, p.281-303, May, 2007
Kar-Ann Toh, Training a reciprocal-sigmoid classifier by feature scaling-space, Machine Learning, v.65 n.1, p.273-308, October   2006
O. Asparoukhov , W. J. Krzanowski, Non-parametric smoothing of the location model in mixed variable discrimination, Statistics and Computing, v.10 n.4, p.289-297, October 2000
R. Chandrasekaran , Young U. Ryu , Varghese S. Jacob , Sungchul Hong, Isotonic Separation, INFORMS Journal on Computing, v.17 n.4, p.462-474, October 2005
Ching-Pao Chang , Chih-Ping Chu, Defect prevention in software processes: An action-based approach, Journal of Systems and Software, v.80 n.4, p.559-570, April, 2007
Tony Van Gestel , Johan A. K. Suykens , Bart Baesens , Stijn Viaene , Jan Vanthienen , Guido Dedene , Bart De Moor , Joos Vandewalle, Benchmarking Least Squares Support Vector Machine Classifiers, Machine Learning, v.54 n.1, p.5-32, January 2004
Elena Baralis , Silvia Chiusano, Essential classification rule sets, ACM Transactions on Database Systems (TODS), v.29 n.4, p.635-674, December 2004
Siddharth Pal , David J. Miller, An Extension of Iterative Scaling for Decision and Data Aggregation in Ensemble Classification, Journal of VLSI Signal Processing Systems, v.48 n.1-2, p.21-37, August    2007
Man Cheang , Kwong Sak Leung , Kin Hong Lee, Genetic parallel programming: design and implementation, Evolutionary Computation, v.14 n.2, p.129-156, June 2006
Foster Provost , Pedro Domingos, Tree Induction for Probability-Based Ranking, Machine Learning, v.52 n.3, p.199-215, September
Johannes Gehrke , Wie-Yin Loh , Raghu Ramakrishnan, Classification and regression: money *can* grow on trees, Tutorial notes of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.1-73, August 15-18, 1999, San Diego, California, United States
Vasant Dhar , Dashin Chou , Foster Provost, Discovering Interesting Patterns for Investment Decision Making with GLOWER &xcirc;-A Genetic Learner Overlaid with Entropy Reduction, Data Mining and Knowledge Discovery, v.4 n.4, p.251-280, October 2000
Perlich , Foster Provost , Jeffrey S. Simonoff, Tree induction vs. logistic regression: a learning-curve analysis, The Journal of Machine Learning Research, 4, p.211-255, 12/1/2003
Foster Provost , Venkateswarlu Kolluri, Data mining tasks and methods: scalability, Handbook of data mining and knowledge discovery, Oxford University Press, Inc., New York, NY, 2002
Gabriela Alexe , Sorin Alexe , Tibrius O. Bonates , Alexander Kogan, Logical analysis of data --- the vision of Peter L. Hammer, Annals of Mathematics and Artificial Intelligence, v.49 n.1-4, p.265-312, April     2007
Krzysztof J. Cios , Lukasz A. Kurgan, CLIP4: hybrid inductive machine learning algorithm that generates inequality rules, Information Sciences: an International Journal, v.163 n.1-3, p.37-83, 14 June 2004
Foster Provost , Venkateswarlu Kolluri, A Survey of Methods for Scaling Up Inductive Algorithms, Data Mining and Knowledge Discovery, v.3 n.2, p.131-169, June 1999
S. B. Kotsiantis , I. D. Zaharakis , P. E. Pintelas, Machine learning: a review of classification and combining techniques, Artificial Intelligence Review, v.26 n.3, p.159-190, November  2006
decision tree;classification tree;statistical classifier;neural net