An Evaluation of Statistical Approaches to Text Categorization.
This paper focuses on a comparative evaluation of a wide-range of  text categorization methods, including previously published results on the  Reuters corpus and new results of additional experiments. A  controlled study using three classifiers, kNN, LLSF and WORD, was  conducted to examine the impact of configuration variations in five  versions of Reuters on the observed performance of classifiers.  Analysis and empirical evidence suggest that the evaluation results on  some versions of Reuters were significantly affected by the inclusion  of a large portion of unlabelled documents, mading those results  difficult to interpret and leading to considerable confusions in the  literature. Using the results evaluated on the other versions of  Reuters which exclude the unlabelled documents, the performance of  twelve methods are compared directly or indirectly. For indirect  compararions, kNN, LLSF and WORD were used as baselines, since they  were evaluated on all versions of Reuters that exclude the unlabelled  documents. As a global observation, kNN, LLSF and a neural network  method had the best performance&semi; except for a Naive Bayes approach,  the other learning algorithms also performed relatively well.
Introduction
Text categorization is the problem of assigning predefined categories to free text documents. A growing number of
statistical learning methods have been applied to this problem in recent years, including regression models[5, 18],
nearest neighbor classifiers[3, 19], Bayes belief networks [14, 9], decision trees[5, 9, 11], rule learning algorithms[1,
15, 12], neural networks[15] and inductive learning techniques[2, 8]. With more and more methods available, cross-
method evaluation becomes increasingly important. However, without an unified methodology of empirical validation,
an objective comparison is difficult.
The most serious problem is the lack of standard data collections. Even when a shared collection is chosen, there
are still many ways to introduce inconsistency. For example, the commonly used Reuters newswire corpus[6] has at
least four different versions, depending on how the training/test sets were divided, and what categories are included
or excluded in the evaluation. Lewis and Ringuette used this corpus to evaluate a decision tree approach and a naive
Bayes classifier, where they included a large portion of unlabelled documents (47% in the training set, and 58% in
the test set) [9]. It is not clear whether these unlabelled documents are all negative instances of the categories in
consideration, or that they are unlabelled simply as an oversight. Apte et al. run a rule learning algorithm, SWAP-1,
on the same set of documents after removing the unlabelled documents[1]. They observed an 12-14% improvement
of SWAP-1 over the results in Lewis&Ringuette's experiments, and concluded that SWAP-1 can often substantially
improve results over decision trees, and that "text classification has a number of characteristics that make optimized
rule induction particularly suitable." This would be a significant finding if the same data were used in the two
experiments. However, given that 58% of the test documents were removed from the original set, it is questionable
whether the observed difference came from the change in the data, or from the difference in the methods. An analysis
later in Sections 3 and 5 will further clarify the point: the inclusion or exclusion of unlabelled documents could have
a significant impact to the results; ignoring this issue makes an evaluation problematic.
It would be ideal if a universal test collection were shared by all the text categorization researchers, or if
a controlled evaluation of a wide range of categorization methods were conducted, similar to the Text Retrieval
Conference for document retrieval[4]. The reality, however, is still far from the ideal. Cross-method comparisons
have often been attempted but only for two or three methods. The small scale of these experiments could lead to overly
general statements based on insufficient observations at one extreme, or the inability to state significant differences
at the other extreme. A solution for these problems is to integrate the available results of categorization methods
into a global evaluation, by carefully analyzing the test conditions in different experiments, and by establishing a
common basis for cross-collection and cross-experiment integration. This paper reports on an effort in this direction.
Section 2 outlines the fourteen methods being investigated. Section 3 analyzes the collection differences in
commonly used corpora, using three classifiers to examine to what degree a difference in conditions effects the
evaluation of a classifier. Section 4 defines a variety of performance measures in use and addresses the equivalence
and comparability between them. Section 5 reports on new evaluations, and compares them with previously published
results. The performance of a baseline classifier on multiple data collections is used as a reference point for a cross-
collection observation. Section 6 concludes the findings.
Categorization Methods
The intention here is to integrate available results from individual experiments into a global evaluation. Two commonly
used corpora, the Reuters news story collection[9] and the OHSUMED bibliographical document collection[7]
are chosen for this purpose. Fourteen categorization methods are investigated, including eleven methods which were
previously evaluated using these corpora, and three methods which were newly evaluated by this author. Not all of
the results are directly comparable because different versions or subsets of these corpora were used. These methods
are outlined below; the data sets and the result comparability will be analyzed in the next section.
1. CONSTRUE, an expert system consisting of manuallydeveloped categorization rules for Reuters news stories[6].
2. Decision tree (DTree) algorithms for classification[9, 11].
3. A naive Bayes model (NaiveBayes) for classification where word independence is assumed in category prediction[9,

Table

1. Data collections examination using WORD, kNN and LLSF in category ranking
Corpus Set UniqCate TrainDoc TestDos (labelled) WORD kNN LLSF
CONSTRUE*
CONSTRUE.2
Reuters Lewis* 113 14,704 6,746 (42%) .10 .84 -
Apte 93 7,789 3,309 (100%) .21 .93 .92
full range 14,321 183,229 50,216 (100%) .16 .52 -
OHSUMED HD big*
HD small* 28 183,229 50,216 (100%) -
* Unlabelled documents are included.
* Heart Diseases (a sub-domain) Categories only, with a training-set category frequency of at least 75.
* Heart Diseases (a sub-domain) Categories only, with a training-set category frequency between 15 to 74.
4. SWAP-1, an inductive learning algorithm for classification using rules in Disjunctive Normal Form (DNF)[1].
5. A neural network approach (NNets) to classification[15].
6. CHARADE, a DNF rule learning system for classification by I. Moulinier[12].
7. RIPPER, a DNF rule learning system for classification by W. Cohen[2].
8. Rocchio, a vector space model for classification where a training set of documents are used to construct a
prototype vector for each category, and category ranking given a document is based on a similarity comparison
between the document vector and the category vectors [8].
9. An exponentiated gradient (EG) inductive learning algorithm which approximates a least squares fit [8].
10. The Widrow-Hoff (WH) inductive learning algorithm which approximates a least squares fit[8].
11. Sleeping Experts (EXPERTS), an inductive learning system using n-gram phrases in classification [2].
12. LLSF, a linear least squares fit (LLSF) approach to classification [18]. A single regression model is used for
ranking multiple categories given a test document. The input variables in the model are unique terms (words or
phrases) in the training documents, and the output variables are unique categories of the training documents.
13. kNN, a k-nearest neighbor classifier[16]. Given an arbitrary input document, the system ranks its nearest
neighbors among training documents, and uses the categories of the k top-ranking neighbors to predict the
categories of the input document. The similarity score of each neighbor document is used as the weight of its
categories, and the sum of category weights over the k nearest neighbors are used for category ranking.
14. A simple, non-learning method which ranks categories for a document based on word matching (WORD)
between the document and category names. The conventional Vector Space Model is used for representing
documents and category names (each name is treated as a bag of words), and the SMART system [13] is used
as the search engine.
3 Collection Analysis
3.1 Two corpora
The Reuters corpus, a collection of newswire stories from 1987 to 1991, is commonly used for text categorization
research, starting from an early evaluation of the CONSTRUE expert system [6, 9, 1, 15, 12, 2] 1 . This collection is
newly refined version named Reuters-21578 is available through Lewis' home page http://www.research.att.com/ ~ lewis.
split into training and test sets when used to evaluate various learning systems. However, the split is not the same in
different studies. Also, various choices were made for the inclusion and exclusion of some categories in an evaluation,
as described in the next section.
The OHSUMED corpus, developed by William Hersh and colleagues at the Oregon Health Sciences University,
is a subset of the documents in the MEDLINE database 2 . It consists of 348,566 references from 270 medical journals
from the years 1987 to 1991. All of the references have titles, but only 233,445 of them have abstracts. We refer to the
title plus abstract as a document. The documents were manually indexed using subject categories (Medical Subject
Headings, or MeSH; about 18,000 categories defined) in the National Library of Medicine. The OHSUMED collection
has been used with the full range of categories (14,321 MeSH categories actually occurred) in some experiments[17],
or with a subset of categories in the heart disease sub-domain (HD, 119 categories) in other experiments[8].
3.2 Different versions

Table

1 lists the different versions or subsets of Reuters and OHSUMED. Each is referred as a "set" or "collection",
and labelled for reference. To examine the collection differences from a text categorization point of view, three
classifiers (WORD, kNN and LLSF) were applied to these collections. The assumption is that if two collections are
statistically homogeneous, then the results of a classifier on these collections should not differ too much. Inversely, if
a dramatic performance change is observed between collections, then this would indicate a need for further analysis.
Since the behavior of a single classifier may lead to biased conclusions, the multiple and fundamentally different
classifiers were used instead. All the systems produces a ranked list of candidate categories given a document. The
conventional 11-point average precision[13] was used to measure the goodness of category ranking. WORD and
kNN were tested on all the collections, while LLSF was only tested on the smaller collections due to computational
limitations. The HD sets were examined together with the OHSUMED superset instead of being examined separately.
Several observations emerge from Table 1:
Homogeneous collections. The Apte set, the PARC set and the Lewis.2 of the Reuters documents are relatively
homogeneous, evident from the similar performance of WORD, kNN and LLSF on these sets. The Lewis.2 is derived
(by this author) from the original Lewis set by removing the unlabelled documents. The Apte set is obtained by
further restricting the categories to have a training set frequency of at least two. In both sets, a continuous chunk
of documents (the early ones) are used for training, and the remaining chunk of documents (the later ones) are used
for testing. The PARC set is drawn from the CONSTRUE set by eliminating the unlabelled documents and some
rare categories[15]. Instead of taking continuous chunks of documents for training and testing, it uses a different
partition. The collection is sliced into many subsets using non-overlapping time windows. The odd subsets are used
for training, and the even subsets are used for testing. The differences between the PARC set, the Apte set and the
Lewis.2 set do not seem to have a significant impact on the performance of the classifiers.
An outlier collection. The CONSTRUE collection has an unusual test set. The training set contains all the
documents in the Lewis set, Apte set or PARC set, and therefore should be statistically similar. The test set contains
only 723 documents which are not included in the other sets. The performance of WORD and kNN on this set are
clearly in favor of word matching over statistical learning. Comparing the Apte set to the CONSTRUE set, the
relative improvement in WORD is 33% (changing from 21% to 28% in average precision), while the performance
change in kNN is \Gamma13% (from 92% to 80%). Although we do not know what criteria were used in selecting the test
documents, it is clear that using this set for evaluation would lead to inconsistent results, compared to using the
other sets. The small size of this test set also makes its results statistically less reliable for evaluation.
collection. The categorization task in OHSUMED seems to be more difficult than in Reuters, as
evidenced from the significant performance decrease in both WORD and kNN. The category space is two magnitudes
larger than Reuters. The number of categories per document is also larger, about 12 to 13 categories on average in
OHSUMED while about 1.2 categories in Reuters. This means that the word/category correspondences are more
"fuzzy" in OHSUMED. Consequently, the categorization is more difficult to learn. The collections named "HD big"
(containing common categories) or "HD small" (containing 28 secondarily common categories) are sub-domains of
the heart diseases sub-domain. Since they contains only about 0.2-.3% of the full range of the categories, performance
of a classifier on these sets may not be sufficiently representative of its performance over the full domain. This does
2 OHSUMED is anonymously ftp-able from medir.ohsu.edu in the directory /pub/ohsumed
not invalidate the use of the HD data sets, but it should be taken into consideration in a cross-collection comparison
of categorization methods.
collection. The Lewis set of the Reuters corpus seems to be problematic given the large portion
of suspiciously unlabelled documents. Note that 58% of the test documents are unlabelled. According to D. Lewis,
"it may (or may not) have been a deliberate decision by the indexer" 3 . It is observed by this author that on randomly
selected test documents, the categories assigned by kNN appeared to be correct in many cases, but they were counted
as failures because these documents were given as unlabelled. This raises a serious question as to whether or not
these unlabelled documents should be included in the test set, and treated as negative instances of all categories, as
they were handled in the previous experiments[9, 2]. The following analysis addresses this question.
Assume the test set has 58% unlabelled documents, and suppose that all of the unlabelled documents should be
assigned categories but are erroneously unlabelled. Let us further assume A to be a perfect classier which assigns a
category to a document if and only if they match, and B a trivial classifier which never assigns any category to a
document. Now if we use the errorful test set as the gold standard to evaluate the two systems, system A will have
an assessed error rate of 58% instead of the true rate of zero percent. System B will have an assessed error rate of
42% instead of the true rate of 100%. Clearly, conclusions based on such a test set can be extremely misleading.
In other words, it can make a better method look worse, and a worse method look better. Of course we do not
know precisely how many documents in the Lewis set should be labelled with categories, so the argument above is
only indicative. Nevertheless, to avoid unnecessary confusion, it would be more sensible to remove the unlabelled
documents, or use the Apte set or PARC set instead. This point will be further addressed in Section 5, with a
discussion on the problems with the experimental results on the Lewis set.
Performance Measures
Classifiers either produce scores, and hence ranked lists of potential category labels, or make binary decisions to
assign categories. A classifier that produces a score can be made into a binary classifier by thresholding the score.
The inverse process is considerably more difficult. An evaluation method applicable to a scoring classifier may not
apply to a binary method. We present evaluations suitable to the two cases and indicate in the following which are
used for comparison.
4.1 Evaluation of category ranking
The recall and precision of a category ranking is similar to the corresponding measures used in text retrieval. Given
a document as the input to a classifier, and a ranked list of categories as the output, the recall and precision at a
particular threshold on this ranked list are defined to be:
categories found and correct
total categories correct
categories found and correct
total categories found
where "categories found" means that the categories are above the threshold. For a collection of test documents, the
category ranking for each document is evaluated first, then the performance scores are averaged across documents.
The conventional 11-point average precision is used to measure the performance of a classifier on a collection of
documents[13].
4.2 Evaluation of binary classification
Performance measures in binary classification can be defined using a two-way contingency table (Table 2). The table
contains four cells:
ffl a counts the assigned and correct cases,
3 Refer to the documentation of the newly refined Reuters-21578 collection.
counts the assigned and incorrect cases,
ffl c counts the not assigned but incorrect cases, and
ffl d counts the not assigned and correct cases.

Table

2. A contingency table
YES is correct No is correct
Assigned YES a b
Assigned NO c d
The recall (r), precision (p), error (e) and fallout (f) are defined to be:
c) if a
Given a classifier, the values of often depend on internal parameter tuning; there is a trade-off
between recall and precision in general. A commonly used measure in method comparison [9, 1, 15, 12] is the
break-even point (BrkEvn) of recall and precision, i.e., when r and p are tuned to be equal. Another common
is called the F -measure, defined to be:
where fi is the parameter allowing differential weighting of p and r. When the value of fi is set to one (denoted as
precision is weighted equally:
When the value of F 1 (r; p) is equivalent to the break-even point. Often the break-even point is close to the
optimal score of F 1 (r; p), but they are not necessarily equivalent. In other words, the optimal score of F 1 (r; p) given a
system can be higher-valued than the break-even point of this system. Therefore, the break-even point of one system
should not be compared directly with the optimal F 1 value of another system.
4.3 Global averaging
There are two ways to measure the average performance of a binary classifier over multiple categories, namely, the
macro-average and the the micro-average. In macro-averaging, one contingency table per category is used, and the
local measures are computed first and then averaged over categories. In micro-averaging, the contingency tables of
individual categories are merged into a single table where each cell of a, b, c and d is the sum of the corresponding
cells in the local tables. The global performance then is computed using the merged table. Macro-averaging gives
an equal weight to the performance on every category, regardless how rare or how common a category is. Micro-
averaging, on the other hand, gives an equal weight to the performance on every document (category instance), thus
favoring the performance on common categories. The micro-average is used in the following evaluation section.

Table

3 summarizes the results of all the categorization methods investigated in this study. The results of
kNN, LLSF and WORD are newly obtained. The results of the other methods are either directly from previous
publications.

Table

3. Results of different methods in category assignments
Reuters Reuters OHSUMED OHSUMED Reuters Reuters
Apte PARC full range HD big Lewis CONSTRUE
BrkEvn BrkEvn F
NNets (N) - .82* -
DTree
NaiveBayes (L) .71 (\Gamma16%) - .65 -
"L" indicates a linear model, and "N" indicates a non-linear model;
"*" marks the local optimal on a fixed collection;
"(.)" includes the performance improvement relative to kNN;
"[.]" includes a F(1) score; the corresponding break-even point should be the same or slightly lower.
5.1 The new experiments
The KNN, LLSF and WORD experiments used the SMART system for unified preprocessing, including stop word
removal, stemming and word weighting. A phrasing option is also available in SMART but not used in these
experiments. Several term weighting options (labelled as "ltc", "atc", "lnc" , "bnn" etc. in SMART's notation) were
tried, which combine the term frequency (TF) measure and the Inverted Document Frequency (IDF) measure in a
variety of ways. The best results (with "ltc" in most cases) are reported in the Table 3.
In kNN and LLSF, aggressive vocabulary reduction based on corpus statistics was also applied as another step
of the preprocessing. This is necessary for LLSF which would otherwise be too computationally expensive to apply
to large training collections. Computational tractability is not an issue for kNN but vocabulary reduction is still
desirable since it improves categorization accuracy. About 1-2% improvements in average precision and break-even
point were observed in both kNN and LLSF when an 85% vocabulary reduction was applied. Several word selection
criteria were tested, including information gain, mutual information, a - 2 statistic and document frequency[20]. The
best results (using the - 2 statistic) were included in Table 3. Aggressive vocabulary reduction was not used in
WORD because it would reduce the chance of word-based matching between documents and category names.
KNN, LLSF and WORD produces a ranked list of categories first when a test document is given. A threshold
on category scores then is applied to obtain binary category assignments to the document. The thresholding on
category scores was optimized on training sets (for individual categories) first, and then applied to the test sets.
Other parameters in these systems include:
ffl k in kNN indicates the number of nearest neighbors used for category prediction, and
ffl p in LLSF indicates the number of principal components (or singular vectors) used in computing the linear
regression.
The performance of kNN is relatively stable for a large range of k, so three values (30, 45 and 65) were tried, and the
best results are included in the result table. A satisfactory performance of LLSF depends on whether p is sufficiently
large. In the experiments of LLSF on the Reuters sets, the optimal or nearly optimal results were obtained when
using about 800 to 1000 singular vectors. A Sun SPARC Ultra-2 Server was used for the experiments. LLSF has not
yet applied to the full set of OHSUMED training documents due to computational limitations.
5.2 Cross-experiment comparison
A row-wise comparison in Table 3 allows observation of the performance variance of a method across collections.
Unfortunately, most of the rows are sparse except for kNN and WORD. A column-wise comparison allows observation
of different methods on a fixed collection. A star marks the best result for each collection.
KNN is chosen to provide the baseline performance on each collection. Several characteristics of this method
make it preferable, i.e., efficient to test, easy to scale up, and relatively robust as a learning method. LLSF is equally
effective, based on the empirical results obtained so far; however, its training is computationally intensive, and thus
has not yet been applied to the full range of the OHSUMED collection. WORD is chosen to provide an secondary
reference point in addition to kNN, to enable a quantitative comparison between learning approaches to a simple
method that requires no knowledge or training.
The Reuters Apte set has the densest column where the results of eight systems are available. Although the
document counts reported by different researchers are somewhat inconsistent[1, 2] 4 , the differences are relatively
small compared to the size of the corpus (i.e., at most 21 miscounted out of over ten thousands training documents,
and at most 7 miscounted out of over three thousands of test documents), so the impact of such differences on the
evaluation results for this set maybe be considered negligible.
The results on the Lewis set, on the other hand, are more problematic. That is, the inclusion of the 58%
"mysteriously" unlabelled documents in the test set makes the results difficult to interpret. For example, most of the
methods (kNN, RIPPER, Rocchio and WORD) which were evaluated on both the Apte set and the Lewis set show a
significant decrease in their performance scores on the Lewis set, but the scores of EXPERTS are almost insensitive
to the inclusion or exclusion of the large amounts of unlabelled documents in the test set. Moreover, EXPERTS has
a score near the lower end among all the learning methods evaluated on the Apte set, but the highest score on the
Lewis set. Cohen concluded EXPERTS the best performer ever reported on the Lewis set without an explanation
on its mysterious insensitivity to the large change in test documents[2]. This is suspicious because the inclusion of a
large amounts of incorrectly labelled documents in the test set should decrease the performance of a good classifier,
as analyzed in Section 3.
Another example of potential difficulties is the misleading comparison by Apte et al. between SWAP-1 (or rule
learning), NaiveBayes and DTree methods (Section 1). They claim an advantage for SWAP-1 based on a score on
the Apte set versus scores for the other methods on the Lewis set. To see the perils in such an inference, kNN has a
score of 85% on the Apte set, versus the SWAP-1 score of 79% on the same set. On the Lewis set, however, the kNN
score is 69%, i.e., 10% lower than Apte SWAP-1 score. Should we then conclude that SWAP-1 is better than kNN,
or the opposite? More interestingly, a recent result using a DTree algorithm (via C4.5) due to Moulinier scores 79%
on the Apte set[11], which is exactly the same as the SWAP-1 result. How should this be interpreted? To make the
point clear, the Lewis set should not be used for text categorization evaluation unless the status of the unlabelled
documents is resolved. Results obtained on this set can be seriously misleading, and therefore should not be used
for a comparison or to draw any conclusions. Inferences based on the CONSTRUE set should also be questioned
because the test set is much smaller than the other sets, contains 20% mysteriously unlabelled documents, and may
possibly be a biased selection (Section 3).
Finally, it may worth mentioning that the cross-method comparisons here are not necessarily precise, because
some experimental parameters might contribute to a difference in the results but are not available. For instance,
different choices could be made in stemming, term selection, term weighting, sampling strategies for training data,
thresholding for binary decisions, and so on. Without detailed information, we cannot be sure that a one or two
percent difference in break-even point or F-measure is an indication of the theoretical strength or weakness of a
learning method. It is also unclear how a significance test should be designed, given that the performance of a
method is compressed into a single number, e.g., to the break-even point of averaged recall and precision. A variance
analysis would be difficult given that the necessary input data is not generally published. Further research is needed
on this issue. Nonetheless, missing detailed information should not prohibit the good use of available information. As
long as the related issues are carefully addressed, as shown above, an integrated view across methods and experiments
is possible, especially for significant variations in results on a fully-labelled common test set.
4 Inconsistent numbers about the documents in the Apte set were found in previous papers and the corpus documentation, presumably
due to counting errors or processing errors by the individuals. The numbers included in Table 1 are those agreed by at least two research
sites. Details are available through yiming@cs.cmu.edu.
6 Discussions
Despite the imperfectness of the comparison across collections and experiments, the integrated results are clearly
informative, enabling a global observation which is not possible otherwise. Several points in the results appear to be
interesting regarding the analysis of classification models.
The impressive performance of kNN is rather surprising given that the method is quite simple and computationally
efficient. It has the best performance, together with LLSF, on the Apte set, and is equally effective as
NNets on the PARC set. On the OHSUMED set, it is the only learning method evaluated on the full domain, i.e.,
a category space which is more than one hundred times larger than those used in the evaluations of most learning
algorithms. When extending the target space from the sub-domain of 49 "HD big" categories to the full domain
of 14,321 categories, the performance decline of kNN is only 5% in absolute value, or a 9% decrease relative. In
contrast, the performance of WORD declined from 44% to 27%, or a 39% decrease relatively. This suggests that
kNN is more powerful than WORD in making fine distinctions between categories. Or, it "failed" more gracefully
when the category space grows by several orders of magnitude.
The good performance of WH on "HD big" calls for deeper analysis. WH is an incremental learning algorithm
trained based on an least squares fit criterion. Its optimal performance therefore should be bounded by or close to a
least squares fit solution obtained in a batch-mode training, such as LLSF. It would be interesting in future research
to compare the empirical results of LLSF with WH. It is also worth asking whether there is something else, beyond
the core theory, which contributed to the good performance. In the WH experiment on "HD big", Lewis used a
"pocketing" strategy to select a subset of training instances from a large pool[8]. This is similar or equivalent to a
sampling strategy which divides available training instances into small chunks, examines one chunk at a time using
a validation set, and adds a new chunk to the selected ones only if it improves the performance on the validation set.
This strategy would be particularly effective when the training data are highly noisy, such as OHSUMED documents.
Nevertheless, the sampling strategy is not a part of the WH algorithm, and can be used in any other classifiers. It
would be interesting to examine the effect of the pocketing strategy in kNN on OHSUMED in feature research, for
example.
Rocchio has a relatively poor performance compared to the other learning methods, and is almost as poor as
WORD on the "HD big" subset, surprisingly. This suggests that Rocchio may not be a good choice (although
commonly used) for the baseline in evaluating learning methods, because it is inferior to most methods and thus
would be not very informative especially when the comparison includes only one or two other learning methods. In
other words, Rocchio is a straw man rather than a challenging standard. KNN would be a better alternative, for
instance.
The mixture of the linear (L) and non-linear (N) classifiers among the top-ranking performers (WH, NNets,
kNN and LLSF) suggests that no general conclusion can be fetched regarding reliable improvement of non-linear
approaches over linear approaches, or vice versa. It is also hard to draw a conclusion about the advantage of a
multiple-category classification model (kNN or LLSF) over unary classification models (WH, NNets, EG, RIPPER
etc.) Either the category independence assumption in the latter type of methods is reasonable, or an improvement
in kNN and LLSF is needed in the handling of the dependence or mutual exclusiveness among categories. Resolving
this issue requires future research.
The rule induction algorithms (SWAP-1, RIPPER and CHARADE) have a similar performance, but below the
local optimum of kNN on the Apte set, and also below some other classifiers (WH, NNets) based on an indirect
comparison across collections via kNN as the baseline. This observation raises a question with respect to a claim
about the particular advantage of rule learning in text categorization. The claim was based on context-sensitivity,
i.e., the power in capturing term combinations[1, 2]. It seems that the methods which do not explicitly identify term
combinations but use the context implicitly (such as in WH, NNets, kNN and LLSF) performed at least as well.
It may be worth mentioning that a classifier can have a degree of context-sensitivity without explicitly identifying
term combinations or phrases. The classification function in LLSF, for instance, is sensitive to weighted linear
combinations of words that co-occur in training documents. This does not makes it equivalent to a non-linear model,
but makes a fundamental distinction from the methods based on a term independence assumption, such as naive
Bayes models. This may be a reason for the impressive performance of kNN and LLSF. It would be interesting to
compare them with NaiveBayes if the latter were tested on the Apte set, for example.
Conclusions
The following conclusions are reached from this study:
1. The performance of a classifier depends strongly on the choice of data used for evaluation. Using a seriously
problematic collection[8], comparing categorization methods without analyzing collection differences[1], and
drawing conclusion based on the results of flawed experiments[2] raise questions about the validity of some
published evaluations. These problems need to be addressed to clarify of the confusions among researchers,
and to prevent the repetition of similar mistakes. Providing information and analysis on these problems is a
major effort in this study.
2. Integrating results from different evaluations into a global comparison across methods is possible, as shown in
this paper, by evaluating one or more baseline classifiers on multiple collections, by normalizing the performance
of other classifiers using a common baseline classifier, and by analyzing collection biases based on performance
variations of several baseline classifiers. Such an integration allows insights on methods and collections which
are rarely apparent in comparisons involving two or three classifiers. It also shows an evaluation methodology
which is complementary to the effort to standardize collections and unify evaluations.
3. WH, kNN, NNets and LLSF are the top performers among the learning methods whose results were empirically
validated in this study. Rocchio had a relatively poor performance, on the other hand. All the learning methods
outperformed WORD, the non-learning method. However, the differences between some learning methods are
not as large as previously claimed[1, 2]. It is not evident in the collected results that non-linear models are
better than linear models, or that more sophisticated methods outperform simpler ones. Conclusive statements
on the strengths and weaknesses of different models requires further research.
4. Scalability of a classifier when the problem size grows by several magnitudes, or when the category space
becomes a hundred times denser, has been rarely examined in text categorization evaluations. KNN is the
only learning method evaluated on the full set of the OHSUMED categories. Its robustness in scaling up and
dealing with harder problems, and its computational efficiency make it the method of choice for approaching
very large and noisy categorization problems.

Acknowledgement

I would like to thank Jan Pedersen at Verity, David Lewis and William Cohen at AT&T, and Isabelle Moulinier at
University of Paris VI for providing the information of their experiments. I would also like to thank Jaime Carbonell
at Carnegie Mellon University for suggesting an improvement in binary decision making, Yibing Geng and Danny
Lee for the programming support, and Chris Buckley at Cornell for making the SMART system available.



--R

Towards language independent automated learning of text categorization models.

Trading mips and memory for knowledge engineering: classifying census returns on the connection machine.
Harman.

Construe/tis: a system for content-based indexing of a database of new stories
Ohsumed: an interactive retrieval evaluation and new large text collection for research.
Training algorithms for linear text classifiers.
Comparison of two learning algorithms for text categorization.
Une approche de la cat'egorisation de textes par l'apprentissage symbolique.
Is learning bias an issue on the text categorization problem?
Text categorization: a symbolic approach.
Automatic Text Processing: The Transformation
Automatic indexing based on bayesian inference networks.
A neural network approach to topic spotting.
Expert network: Effective and efficient learning from human decisions in text categorization and retrieval.
An evaluation of a statistical approaches to medline indexing.
A linear least squares fit mapping method for information retrieval from natural language texts.
An example-based mapping method for text categorization and retrieval
Feature selection in statistical learning of text categorization.
--TR
Automatic text processing: the transformation, analysis, and retrieval of information by computer
Trading MIPS and memory for knowledge engineering
Automatic indexing based on Bayesian inference networks
An example-based mapping method for text categorization and retrieval
Expert network
Towards language independent automated learning of text categorization models
OHSUMED
Document filtering for fast ranking
Noise reduction in a statistical approach to text categorization
Cluster-based text categorization
The design of a high performance information filtering system
Training algorithms for linear text classifiers
Context-sensitive learning methods for text categorization
Feature selection, perception learning, and a usability case study for text categorization
Information Retrieval
Introduction to Modern Information Retrieval
Induction of Decision Trees
CONSTRUE/TIS
A Comparative Study on Feature Selection in Text Categorization
A Linear Least Squares Fit mapping method for information retrieval from natural language texts

--CTR
Verayuth Lertnattee , Thanaruk Theeramunkong, Effect of term distributions on centroid-based text categorization, Information SciencesInformatics and Computer Science: An International Journal, v.158 n.1, p.89-115, January 2004
J. Scott Olsson, An analysis of the coupling between training set and neighborhood sizes for the kNN classifier, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
Sofus A. Macskassy , Haym Hirsh , Arunava Banerjee , Aynur A. Dayanik, Converting numerical classification into text classification, Artificial Intelligence, v.143 n.1, p.51-77, January
Tan , Xueqi Cheng, Using hypothesis margin to boost centroid text classifier, Proceedings of the 2007 ACM symposium on Applied computing, March 11-15, 2007, Seoul, Korea
Anne R. Diekema , Jiangping Chen, Experimenting with the automatic assignment of educational standards to digital library content, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA
Pui Y. Lee , Siu C. Hui , Alvis Cheuk M. Fong, Neural Networks for Web Content Filtering, IEEE Intelligent Systems, v.17 n.5, p.48-57, September 2002
Anne R. Diekema , Ozgur Yilmazel , Jennifer Bailey , Sarah C. Harwell , Elizabeth D. Liddy, Standards alignment for metadata assignment, Proceedings of the 2007 conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada
Deendayal Dinakarpandian , Vijay Kumar, BlOMIND-protein property prediction by property proximity profiles, Proceedings of the 2002 ACM symposium on Applied computing, March 11-14, 2002, Madrid, Spain
Xiaobing Wu, Knowledge Representation and Inductive Learning with XML, Proceedings of the 2004 IEEE/WIC/ACM International Conference on Web Intelligence, p.491-494, September 20-24, 2004
Hang Su , Qiaozhu Mei, Template extraction from candidate template set generation: a structure and content approach, Proceedings of the 43rd annual southeast regional conference, March 18-20, 2005, Kennesaw, Georgia
Fred J. Damerau , Tong Zhang , Sholom M. Weiss , Nitin Indurkhya, Text categorization for a comprehensive time-dependent benchmark, Information Processing and Management: an International Journal, v.40 n.2, p.209-221, March 2004
Parisut Jitpakdee , Worapoj Kreesuradej, Dimensionality reduction of features for text categorization, Proceedings of the third conference on IASTED International Conference: Advances in Computer Science and Technology, p.506-509, April 02-04, 2007, Phuket, Thailand
Minoru Yoshida , Hiroshi Nakagawa, Reformatting web documents via header trees, Proceedings of the ACL 2005 on Interactive poster and demonstration sessions, p.121-124, June 25-30, 2005, Ann Arbor, Michigan
Yiming Yang , Jian Zhang , Jaime Carbonell , Chun Jin, Topic-conditioned novelty detection, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada
Makoto Suzuki, Text classification based on the bias of word frequency over categories, Proceedings of the 24th IASTED international conference on Artificial intelligence and applications, p.400-405, February 13-16, 2006, Innsbruck, Austria
Jos M. Gmez Hidalgo , Manuel Maa Lpez , Enrique Puertas Sanz, Combining text and heuristics for cost-sensitive spam filtering, Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning, September 13-14, 2000, Lisbon, Portugal
Sholom M. Weiss , Chidanand Apte , Fred J. Damerau , David E. Johnson , Frank J. Oles , Thilo Goetz , Thomas Hampp, Maximizing Text-Mining Performance, IEEE Intelligent Systems, v.14 n.4, p.63-69, July 1999
Sholom M. Weiss , Brian F. White , Chidanand V. Apte , Fredrick J. Damerau, Lightweight Document Matching for Help-Desk Applications, IEEE Intelligent Systems, v.15 n.2, p.57-61, March 2000
Marti A. Hearst, Support Vector Machines, IEEE Intelligent Systems, v.13 n.4, p.18-28, July 1998
Ronen Feldman , Benjamin Rosenfeld , Ron Lazar , Joshua Livnat , Benjamin Segal, Computerized retrieval and classification: An application to reasons for late filings with the securities and exchange commission, Intelligent Data Analysis, v.10 n.2, p.183-195, March 2006
Norbert Gvert , Mounia Lalmas , Norbert Fuhr, A probabilistic description-oriented approach for categorizing web documents, Proceedings of the eighth international conference on Information and knowledge management, p.475-482, November 02-06, 1999, Kansas City, Missouri, United States
Youngjoong Ko , Jungyun Seo, Automatic text categorization by unsupervised learning, Proceedings of the 18th conference on Computational linguistics, p.453-459, July 31-August
Taeho Jo, Index based approach for categorizing online news articles, Proceedings of the 2nd WSEAS International Conference on Computer Engineering and Applications, p.125-130, January 25-27, 2008, Acapulco, Mexico
Daniel Billsus , Michael J. Pazzani, A personal news agent that talks, learns and explains, Proceedings of the third annual conference on Autonomous Agents, p.268-275, April 1999, Seattle, Washington, United States
Son Doan , Susumu Horiguchi, An efficient feature selection using multi-criteria in text categorization for nave Bayes classifier, Proceedings of the 4th WSEAS International Conference on Artificial Intelligence, Knowledge Engineering Data Bases, p.1-6, February 13-15, 2005, Salzburg, Austria
Daniel Billsus , Clifford A. Brunk , Craig Evans , Brian Gladish , Michael Pazzani, Adaptive interfaces for ubiquitous web access, Communications of the ACM, v.45 n.5, May 2002
Fuchun Peng , Xiangji Huang , Dale Schuurmans , Shaojun Wang, Text classification in Asian languages without word segmentation, Proceedings of the sixth international workshop on Information retrieval with Asian languages, p.41-48, July 07-07, 2003, Sappro, Japan
D. Ferrucci , A. Lally, Building an example application with the unstructured information management architecture, IBM Systems Journal, v.43 n.3, p.455-475, July 2004
Tong Zhang , Alexandrin Popescul , Byron Dom, Linear prediction models with graph regularization for web-page categorization, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
Francois Paradis , Jian-Yun Nie, Contextual feature selection for text classification, Information Processing and Management: an International Journal, v.43 n.2, p.344-352, March 2007
Jeonghee Yi , Neel Sundaresan, A classifier for semi-structured documents, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.340-344, August 20-23, 2000, Boston, Massachusetts, United States
Wenqian Shang , Houkuan Huang , Haibin Zhu , Yongmin Lin , Youli Qu , Zhihai Wang, A novel feature selection algorithm for text categorization, Expert Systems with Applications: An International Journal, v.33 n.1, p.1-5, July, 2007
Ruihua Song , Haifeng Liu , Ji-Rong Wen , Wei-Ying Ma, Learning block importance models for web pages, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA
Zhaohui Zheng , Xiaoyun Wu , Rohini Srihari, Feature selection for text categorization on imbalanced data, ACM SIGKDD Explorations Newsletter, v.6 n.1, June 2004
Ji He , Ah-Hwee Tan , Chew-Lim Tan, Machine learning methods for Chinese web page categorization, Proceedings of the second workshop on Chinese language processing: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, October 08-08, 2000, Hong Kong
Ruihua Song , Haifeng Liu , Ji-Rong Wen , Wei-Ying Ma, Learning important models for web page blocks based on layout and content analysis, ACM SIGKDD Explorations Newsletter, v.6 n.2, p.14-23, December 2004
Rui Fang , Alexander Mikroyannidis , Babis Theodoulidis, A Voting Method for the Classification of Web Pages, Proceedings of the 2006 IEEE/WIC/ACM international conference on Web Intelligence and Intelligent Agent Technology, p.610-613, December 18-22, 2006
Jhy-Jong Tsay , Jing-Doo Wang, Improving automatic Chinese text categorization by error correction, Proceedings of the fifth international workshop on on Information retrieval with Asian languages, p.1-8, September 30-October 01, 2000, Hong Kong, China
Chih-Chin Lai, An empirical study of three machine learning methods for spam filtering, Knowledge-Based Systems, v.20 n.3, p.249-254, April, 2007
adaptive k-nearest neighbor text categorization strategy, ACM Transactions on Asian Language Information Processing (TALIP), v.3 n.4, p.215-226, December 2004
Thomas Galen Ault , Yiming Yang, Information Filtering in TREC-9 and TDT-3: A Comparative Analysis, Information Retrieval, v.5 n.2-3, p.159-187, April-July 2002
Yiming Yang, A study of thresholding strategies for text categorization, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.137-145, September 2001, New Orleans, Louisiana, United States
Ganesh Ramakrishnan , Krishna Prasad Chitrapura , Raghu Krishnapuram , Pushpak Bhattacharyya, A model for handling approximate, noisy or incomplete labeling in text classification, Proceedings of the 22nd international conference on Machine learning, p.681-688, August 07-11, 2005, Bonn, Germany
Aixin Sun , Ee-Peng Lim , Wee-Keong Ng, Web classification using support vector machine, Proceedings of the 4th international workshop on Web information and data management, November 08-08, 2002, McLean, Virginia, USA
Tanveer Syeda-Mahmood , Dulce Ponceleon, Learning video browsing behavior and its application in the generation of video previews, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada
Jyh-Jong Tsay , Jing-Doo Wang, Improving linear classifier for Chinese text categorization, Information Processing and Management: an International Journal, v.40 n.2, p.223-237, March 2004
Nadia Ghamrawi , Andrew McCallum, Collective multi-label classification, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany
Jinsuk Kim , Myoung Ho Kim, An Evaluation of Passage-Based Text Categorization, Journal of Intelligent Information Systems, v.23 n.1, p.47-65, July 2004
Oh-Woog Kwon , Jong-Hyeok Lee, Web page classification based on k-nearest neighbor approach, Proceedings of the fifth international workshop on on Information retrieval with Asian languages, p.9-15, September 30-October 01, 2000, Hong Kong, China
Massimiliano Ciaramita, Boosting automatic lexical acquisition with morphological information, Proceedings of the ACL-02 workshop on Unsupervised lexical acquisition, p.17-25, July 12-12, 2002, Philadelphia, Pennsylvania
Yiming Yang , Jian Zhang , Bryan Kisiel, A scalability analysis of classifiers in text categorization, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada
Jae-Ho Kim , Key-Sun Choi, Patent document categorization based on semantic structural information, Information Processing and Management: an International Journal, v.43 n.5, p.1200-1215, September, 2007
Jos Mara Gmez Hidalgo , Guillermo Cajigas Bringas , Enrique Puertas Snz , Francisco Carrero Garca, Content based SMS spam filtering, Proceedings of the 2006 ACM symposium on Document engineering, October 10-13, 2006, Amsterdam, The Netherlands
Jos Mara Gmez Hidalgo, Evaluating cost-sensitive Unsolicited Bulk Email categorization, Proceedings of the 2002 ACM symposium on Applied computing, March 11-14, 2002, Madrid, Spain
Manabu Sassano, Virtual examples for text classification with Support Vector Machines, Proceedings of the conference on Empirical methods in natural language processing, p.208-215, July 11,
Sholom M. Weiss , Chidanand Apte , Fred J. Damerau , David E. Johnson , Frank J. Oles , Thilo Goetz , Thomas Hampp, Maximizing Text-Mining Performance, IEEE Intelligent Systems, v.14 n.4, p.63-69, July 1999
Arnulfo P. Azcarraga , Teddy N. Yap, Jr., Extracting meaningful labels for WEBSOM text archives, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA
Fabrizio Sebastiani , Alessandro Sperduti , Nicola Valdambrini, An improved boosting algorithm and its application to text categorization, Proceedings of the ninth international conference on Information and knowledge management, p.78-85, November 06-11, 2000, McLean, Virginia, United States
Wahyu Wibowo , Hugh E. Williams, Strategies for minimising errors in hierarchical web categorisation, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA
Henrik Nottelmann , Norbert Fuhr, Learning probabilistic datalog rules for information classification and transformation, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA
Stephan Busemann , Sven Schmeier , Roman G. Arens, Message classification in the call center, Proceedings of the sixth conference on Applied natural language processing, p.158-165, April 29-May
Hana Kopackova, Text categorization: potential tool for managerial decision-making, Proceedings of the 5th WSEAS International Conference on Applied Informatics and Communications, p.209-214, September 15-17, 2005, Malta
Youngjoong Ko , Jungyun Seo, Using the feature projection technique based on a normalized voting method for text classification, Information Processing and Management: an International Journal, v.40 n.2, p.191-208, March 2004
Fuchun Peng , Dale Schuurmans , Shaojun Wang, Language and task independent text categorization with simple language models, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.110-117, May 27-June 01, 2003, Edmonton, Canada
Yiming Yang , Tom Ault , Thomas Pierce , Charles W. Lattimer, Improving text categorization methods for event tracking, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.65-72, July 24-28, 2000, Athens, Greece
Hyoungdong Han , Youngjoong Ko , Jungyun Seo, Using the revised EM algorithm to remove noisy data for improving the one-against-the-rest method in binary text classification, Information Processing and Management: an International Journal, v.43 n.5, p.1281-1293, September, 2007
Parry Husbands , Horst Simon , Chris Ding, Term norm distribution and its effects on latent semantic indexing, Information Processing and Management: an International Journal, v.41 n.4, p.777-787, July 2005
Vikramjit Mitra , Chia-Jiu Wang , Satarupa Banerjee, Text classification: A least square support vector machine approach, Applied Soft Computing, v.7 n.3, p.908-914, June, 2007
Quan Wang , Yiu-Kai Ng, An Ontology-Based Binary-Categorization Approach for Recognizing Multiple-Record Web Documents Using a Probabilistic Retrieval Model, Information Retrieval, v.6 n.3-4, p.295-332, September-December
Mark Steyvers , Padhraic Smyth , Michal Rosen-Zvi , Thomas Griffiths, Probabilistic author-topic models for information discovery, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA
Moshe Koppel , Jonathan Schler, Authorship verification as a one-class classification problem, Proceedings of the twenty-first international conference on Machine learning, p.62, July 04-08, 2004, Banff, Alberta, Canada
Youngjoong Ko , Jinwoo Park , Jungyun Seo, Improving text categorization using the importance of sentences, Information Processing and Management: an International Journal, v.40 n.1, p.65-79, January 2004
Mei-Ling Shyu , Choochart Haruechaiyasak , Shu-Ching Chen, Category cluster discovery from distributed WWW directories, Information SciencesInformatics and Computer Science: An International Journal, v.155 n.3-4, p.181-197, 15 October
Daqing He , Ayse Gker , David J. Harper, Combining evidence for automatic web session identification, Information Processing and Management: an International Journal, v.38 n.5, p.727-742, September 2002
Chun-Yan Liang , Li Guo , Zhao-Jie Xia , Feng-Guang Nie , Xiao-Xia Li , Liang Su , Zhang-Yuan Yang, Dictionary-based text categorization of chemical web pages, Information Processing and Management: an International Journal, v.42 n.4, p.1017-1029, July 2006
Rey-Long Liu, Dynamic category profiling for text filtering and classification, Information Processing and Management: an International Journal, v.43 n.1, p.154-168, January 2007
Yiming Yang , Jaime G. Carbonell , Ralf D. Brown , Thomas Pierce , Brian T. Archibald , Xin Liu, Learning Approaches for Detecting and Tracking News Events, IEEE Intelligent Systems, v.14 n.4, p.32-43, July 1999
Sun Lee Bang , Jae Dong Yang , Hyung Jeong Yang, Hierarchical document categorization with k-NN and concept-based thesauri, Information Processing and Management: an International Journal, v.42 n.2, p.387-406, March 2006
Hang Cui , Min-Yen Kan , Tat-Seng Chua, Unsupervised learning of soft patterns for generating definitions from online news, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA
Sue J. Ker , Jen-Nan Chen, A text categorization based on summarization technique, Proceedings of the ACL-2000 workshop on Recent advances in natural language processing and information retrieval: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, October 08-08, 2000, Hong Kong
Yiming Yang , Jaime G. Carbonell , Ralf D. Brown , Thomas Pierce , Brian T. Archibald , Xin Liu, Learning Approaches for Detecting and Tracking News Events, IEEE Intelligent Systems, v.14 n.4, p.32-43, July 1999
Dimitris Fragoudis , Dimitris Meretakis , Spiridon Likothanassis_aff1n3, Best terms: an efficient feature-selection algorithm for text categorization, Knowledge and Information Systems, v.8 n.1, p.16-33, July 2005
Tong Zhang , Frank J. Oles, Text Categorization Based on Regularized Linear Classification Methods, Information Retrieval, v.4 n.1, p.5-31, April 2001
Tong Zhang , Vijay S. Iyengar, Recommender systems using linear classifiers, The Journal of Machine Learning Research, 2, p.313-334, 3/1/2002
Patrick Ruch, Query translation by text categorization, Proceedings of the 20th international conference on Computational Linguistics, p.686-es, August 23-27, 2004, Geneva, Switzerland
Ying Liu , Han Tong Loh , Shu Beng Tor, Comparison of extreme learning machine with support vector machine for text classification, Proceedings of the 18th international conference on Innovations in Applied Artificial Intelligence, p.390-399, June 22-24, 2005, Bari, Italy
Filippo Menczer , Gautam Pant , Padmini Srinivasan , Miguel E. Ruiz, Evaluating topic-driven web crawlers, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.241-249, September 2001, New Orleans, Louisiana, United States
Goran Nenadi , Simon Rice , Irena Spasi , Sophia Ananiadou , Benjamin Stapley, Selecting text features for gene name classification: from documents to terms, Proceedings of the ACL workshop on Natural language processing in biomedicine, p.121-128, July 11-11, 2003, Sapporo, Japan
Yiming Yang , Sen Slattery , Rayid Ghani, A Study of Approaches to Hypertext Categorization, Journal of Intelligent Information Systems, v.18 n.2-3, p.219-241, March-May 2002
Oh-Woog Kwon , Jong-Hyeok Lee, Text categorization based on k-nearest neighbor approach for web site classification, Information Processing and Management: an International Journal, v.39 n.1, p.25-44, January
Yan Liu , Yiming Yang , Jaime Carbonell, Boosting to correct inductive bias in text classification, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA
Natalie Glance , Matthew Hurst , Kamal Nigam , Matthew Siegler , Robert Stockton , Takashi Tomokiyo, Deriving marketing intelligence from online discussion, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA
Fernando Ruiz-Rico , Jose Luis Vicedo , Mara-Consuelo Rubio-Snchez, NEWPAR: an automatic feature selection and weighting schema for category ranking, Proceedings of the 2006 ACM symposium on Document engineering, October 10-13, 2006, Amsterdam, The Netherlands
Shen , Jian-Tao Sun , Qiang Yang , Zheng Chen, Text classification improved through multigram models, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
H. Cenk Ozmutlu , Fatih avdur, Application of automatic topic identification on excite web search engine data logs, Information Processing and Management: an International Journal, v.41 n.5, p.1243-1262, September 2005
Yaxin Bi , Sally Mcclean , Terry Anderson, Combining rough decisions for intelligent text mining using Dempster's rule, Artificial Intelligence Review, v.26 n.3, p.191-209, November  2006
Athanasios Kehagias , Vassilios Petridis , Vassilis G. Kaburlasos , Pavlina Fragkou, A Comparison of Word- and Sense-Based Text Categorization Using Several Classification Algorithms, Journal of Intelligent Information Systems, v.21 n.3, p.227-247, November
George Karypis , Eui-Hong (Sam) Han, Fast supervised dimensionality reduction algorithm with applications to document categorization & retrieval, Proceedings of the ninth international conference on Information and knowledge management, p.12-19, November 06-11, 2000, McLean, Virginia, United States
Hana Kopackova , Ludek Kopacek , Renata Bilkova , Karel Naiman, New methods for text categorization, Proceedings of the 5th WSEAS International Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, p.240-245, November 20-22, 2006, Venice, Italy
Yiming Yang , Xin Liu, A re-examination of text categorization methods, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.42-49, August 15-19, 1999, Berkeley, California, United States
Mohammed Benkhalifa , Abdelhak Mouradi , Houssaine Bouyakhf, Integrating External Knowledge to Supplement Training Data in Semi-Supervised Learning for Text Categorization, Information Retrieval, v.4 n.2, p.91-113, July 2001
Xin Li , Hsinchun Chen , Zhu Zhang , Jiexun Li, Automatic patent classification using citation network information: an experimental study in nanotechnology, Proceedings of the 2007 conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada
Pei-Yi Hao , Jung-Hsien Chiang , Yi-Kun Tu, Hierarchically SVM classification based on support vector clustering method and its application to document categorization, Expert Systems with Applications: An International Journal, v.33 n.3, p.627-635, October, 2007
Jrg Ontrup , Helge Ritter, Large-scale data exploration with the hierarchically growing hyperbolic SOM, Neural Networks, v.19 n.6, p.751-761, July 2006
Daniel Billsus , Michael J. Pazzani, User Modeling for Adaptive News Access, User Modeling and User-Adapted Interaction, v.10 n.2-3, p.147-180, 2000
Dharmendra S. Modha , W. Scott Spangler, Feature Weighting in k-Means Clustering, Machine Learning, v.52 n.3, p.217-237, September
Wendy W. Chapman , Lee M. Christensen , Michael M. Wagner , Peter J. Haug , Oleg Ivanov , John N. Dowling , Robert T. Olszewski, Classifying free-text triage chief complaints into syndromic categories with natural languages processing, Arificial Intelligence in Medicine, v.33 n.1, p.31-40, 1 January 2005
Steven C. H. Hoi , Rong Jin , Michael R. Lyu, Large-scale text categorization by batch mode active learning, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
Efstathios Stamatatos , George Kokkinakis , Nikos Fakotakis, Automatic text categorization in terms of genre and author, Computational Linguistics, v.26 n.4, p.471-495, December 2000
Theodore Dalamagas , Tao Cheng , Klaas-Jan Winkel , Timos Sellis, A methodology for clustering XML documents by structure, Information Systems, v.31 n.3, p.187-228, May 2006
Saddys Segrera , Mara N. Moreno, An experimental comparative study of web mining methods for recommender systems, Proceedings of the 6th WSEAS International Conference on Distance Learning and Web Engineering, p.56-61, September 22-24, 2006, Lisbon, Portugal
Tom M. Mitchell , Rebecca Hutchinson , Radu S. Niculescu , Francisco Pereira , Xuerui Wang , Marcel Just , Sharlene Newman, Learning to Decode Cognitive States from Brain Images, Machine Learning, v.57 n.1-2, p.145-175, October-November 2004
Nayer M. Wanas , Dina A. Said , Nadia H. Hegazy , Nevin M. Darwish, A study of local and global thresholding techniques in text categorization, Proceedings of the fifth Australasian conference on Data mining and analystics, p.91-101, November 29-30, 2006, Sydney, Australia
Andreas S. Weigend , Erik D. Wiener , Jan O. Pedersen, Exploiting Hierarchy in Text Categorization, Information Retrieval, v.1 n.3, p.193-216, October 1999
Rudy Prabowo , Mike Thelwall, A comparison of feature selection methods for an evolving RSS feed corpus, Information Processing and Management: an International Journal, v.42 n.6, p.1491-1512, December 2006
Shen , Rong Pan , Jian-Tao Sun , Jeffrey Junfeng Pan , Kangheng Wu , Jie Yin , Qiang Yang, Query enrichment for web-query classification, ACM Transactions on Information Systems (TOIS), v.24 n.3, p.320-352, July 2006
S. Jaillet , A. Laurent , M. Teisseire, Sequential patterns for text categorization, Intelligent Data Analysis, v.10 n.3, p.199-214, January 2006
Bill B. Wang , R. I. Bob Mckay , Hussein A. Abbass , Michael Barlow, A comparative study for domain ontology guided feature extraction, Proceedings of the twenty-sixth Australasian conference on Computer science: research and practice in information technology, p.69-78, February 01, 2003, Adelaide, Australia
Robert E. Schapire , Yoram Singer, BoosTexter: A Boosting-based Systemfor Text Categorization, Machine Learning, v.39 n.2-3, p.135-168, May-June 2000
David D. Lewis , Yiming Yang , Tony G. Rose , Fan Li, RCV1: A New Benchmark Collection for Text Categorization Research, The Journal of Machine Learning Research, 5, p.361-397, 12/1/2004
Mieczysaw A. Kopotek, Very large Bayesian multinets for text classification, Future Generation Computer Systems, v.21 n.7, p.1068-1082, July 2005
Haoran Wu , Tong Heng Phang , Bing Liu , Xiaoli Li, A refinement approach to handling model misfit in text categorization, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada
O. de Vel , A. Anderson , M. Corney , G. Mohay, Mining e-mail content for author identification forensics, ACM SIGMOD Record, v.30 n.4, December 2001
Miguel E. Ruiz , Padmini Srinivasan, Hierarchical Text Categorization Using Neural Networks, Information Retrieval, v.5 n.1, p.87-118, January 2002
B. Barla Cambazoglu , Evren Karaca , Tayfun Kucukyilmaz , Ata Turk , Cevdet Aykanat, Architecture of a grid-enabled Web search engine, Information Processing and Management: an International Journal, v.43 n.3, p.609-623, May, 2007
I. E. Kuralenok , I. S. Nekrestyanov, Evaluation of Text Retrieval Systems, Programming and Computing Software, v.28 n.4, p.226-242, July-August 2002
Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000
Tina Eliassi-Rad , Jude Shavlik, Intelligent Web agents that learn to retrieve and extract information, Intelligent exploration of the web, Physica-Verlag GmbH, Heidelberg, Germany,
Tai-Yue Wang , Huei-Min Chiang, Fuzzy support vector machine for multi-class text categorization, Information Processing and Management: an International Journal, v.43 n.4, p.914-929, July, 2007
Raymond J. Mooney , Loriene Roy, Content-based book recommending using learning for text categorization, Proceedings of the fifth ACM conference on Digital libraries, p.195-204, June 02-07, 2000, San Antonio, Texas, United States
Amir Ahmad , Lipika Dey, A k-mean clustering algorithm for mixed numeric and categorical data, Data & Knowledge Engineering, v.63 n.2, p.503-527, November, 2007
Michelangelo Ceci , Donato Malerba, Classifying web documents in a hierarchy of categories: a comprehensive study, Journal of Intelligent Information Systems, v.28 n.1, p.37-78, February  2007
Fuchun Peng , Dale Schuurmans , Shaojun Wang, Augmenting Naive Bayes Classifiers with Statistical Language Models, Information Retrieval, v.7 n.3-4, p.317-345, September-December 2004
Tsvi Kuflik , Zvi Boger , Peretz Shoval, Filtering search results using an optimal set of terms identified by an artificial neural network, Information Processing and Management: an International Journal, v.42 n.2, p.469-483, March 2006
Goran Nenadi , Sophia Ananiadou, Mining semantically related terms from biomedical literature, ACM Transactions on Asian Language Information Processing (TALIP), v.5 n.1, p.22-43, March 2006
Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002
comparative study;evaluation;text categorization;statistical learning algorithms