Evaluating the novelty of text-mined rules using lexical knowledge.
In this paper, we present a new method of estimating the novelty of rules discovered by data-mining methods using WordNet, a lexical knowledge-base of English words. We assess the novelty of a rule by the average semantic distance in a knowledge hierarchy between the words in the antecedent and the consequent of the rule - the more the average distance, more is the novelty of the rule. The novelty of rules extracted by the DiscoTEX text-mining system on Amazon.com book descriptions were evaluated by both human subjects and by our algorithm. By computing correlation coefficients between pairs of human ratings and between human and automatic ratings, we found that the automatic scoring of rules based on our novelty measure correlates with human judgments about as well as human judgments correlate with one another. @Text mining
Introduction
A data-mining system may discover a large body of rules; however, relatively few of these may
convey useful new knowledge to the user. Several metrics for evaluating the \interestingness" of
mined rules have been proposed [BA99, HK01]. These metrics can be used to lter out a large
percentage of the less interesting rules, thus yielding a more manageable number of higher quality
rules to be presented to the user. However, most of these measure simplicity (e.g. rule size),
certainty (e.g. condence), or utility (e.g. support). Another important aspect of interestingness
is novelty: does the rule represent an association that is currently unknown. For example, a
text-mining system we developed that discovers rules from computer-science job announcements
posted to a local newsgroup [NM00] induced the rule: \SQL ! database". A knowledgeable
computer scientist may nd this rule uninteresting because it conveys a known association.
Evaluating the novelty of a rule requires comparing it to an existing body of knowledge the user
is assumed to already possess.
For text mining [Hea99, Fel99, Mla00], in which rules consist of words in natural language, a
relevant body of common knowledge is basic lexical semantics, i.e. the meanings of words and the
semantic relationships between them. A number of lexical knowledge bases are now available.
WordNet [Fel98] is a semantic network of about 130,000 English words linked to about 100,000
lexical senses (synsets) that are interconnected by relations such as antonym, generalization
(hypernym), and part-of (holonym). We present and evaluate a method for measuring the
novelty of text-mined rules using such lexical knowledge.
We dene a measure of the semantic distance, d(w words based on the
length of the shortest path connecting w i and w j in WordNet. The novelty of a rule is then
dened as the average value of d(w all pairs of words (w is in the
antecedent and w j is in the consequent of the rule. Intuitively, the semantic dissimilarity of
the terms in a rule's antecedent and in its consequent is an indication of the rule's novelty. For
example, \beer ! diapers" would be considered more novel than \beer ! pretzels" since beer
and pretzels are both food products and therefore closer in WordNet.
We present an experimental evaluation of this novelty metric by applying it to rules mined
from book descriptions extracted from Amazon.com. Since novelty is fundamentally subjective,
we compared the metric to human judgments. We have developed a web-based tool that allows
human subjects to enter estimates of the novelty of rules. We asked multiple human subjects to
score random selections of mined rules and compared the results to those obtained by applying
our metric to the same rules. We found that the average correlation between the scoring of our
algorithm and that of the human users, using both raw score correlation (Pearson's metric) and
rank correlation (Spearman's metric), was comparable to the average score correlation between
the human users. This suggests that the algorithm has a rule scoring judgment similar to that
of human users.
Background
2.1 Text Mining
Traditional data mining algorithms are generally applied to structured databases, but text mining
algorithms try to discover knowledge from unstructured or semi-structured textual data, e.g.
web-pages. Text mining is a relatively new research area at the intersection of natural language
processing, machine learning and information retrieval. Various new useful techniques are being
developed by researchers for discovering knowledge from large text corpora, by appropriately
integrating methods from these dierent disciplines. DiscoTEX [NM00] is one such system, that
discovers prediction rules from natural language corpora using a combination of information
extraction and data mining. It learns an information extraction system to transform text into
more structured data, and this structured data is then mined for interesting relationships.
For our experiments, we have used rules mined by DiscoTEX from book descriptions extracted
from Amazon.com, in the \science", \romance" and \literature" categories. DiscoTEX
rst extracts a structured template from the Amazon.com book description web-pages. It constructs
a template for each book description, with pre-dened slots (e.g. title, author, subject,
etc.) that are lled with words extracted from the text. DiscoTEX then uses a rule mining
technique to extract prediction rules from this template database. An example extracted rule
is shown in Figure 1, where the   slot is predicted from the other slots. For our
purpose, we only use the ller words in the slot, ignoring the slotnames | in our algorithm,
the rule in Figure 1 would be used in the form \daring love woman romance historical ction
story read wonderful".
daring, love
woman
romance, historical, fiction
->
story, read, wonderful

Figure

1: DiscoTEX rule mined from Amazon.com \romance" book descriptions
2.2 WordNet
WordNet [Fel98] is an online lexical knowledge-base of 130,000 English words, developed at
Princeton University. In WordNet, English nouns, adjectives, verbs and adverbs are organized
into synonym sets or synsets, each representing an underlying lexical concept. A synset contains
words of similar meaning pertaining to a common semantic concept. But since a word can have
dierent meanings in dierent contexts, a word can be present in multiple synsets. A synset
contains associated pointers representing its relation to other synsets. WordNet supports many
pointer types e.g. antonyms, synonyms, etc. The pointer types we used in our algorithm are
explained below:
1. Synonym: This pointer is implicit. Since words in the same synset are synonymous, e.g.
life and existence, the synonym of a synset is itself.
2. Antonym: This pointer type refers to another synset that is quite opposite in meaning to
the given synset, e.g. front is the antonym of back.
3. Attribute: This pointer type refers to another synset that is implicated by this synset, e.g.
benevolence is an attribute of good.
4. Pertainym: This pointer refers to a relation from a noun to an adjective, an adjective to a
noun, or an adverb to an adjective, indicating morphological relation, e.g. alphabetical is
a pertainym of alphabet.
5. Similar: This pointers refers to another adjective that is very close in terms of meaning to
the current adjective, although not enough to be part of the same synset, e.g. unquestioning
is similar to absolute.
6. Cause: This pointer type refers to a cause and eect relation, e.g. kill is cause to die.
7. Entailment: This pointer refers to the implication of another action e.g. breathe is an
entailment of inhale.
8. Holonym: This pointer refers to a part in a part-whole relation, e.g. chapter is a holonym
of text. There are three kinds of holonyms | by member, by substance and by part.
9. Meronym: This pointer refers to a whole in a part-whole relation, e.g. computer is a
meronym of cpu. There are three kinds of meronyms | by member, by substance and by
part.
10. Hyponym: This pointer refers to a specication of the concept, e.g. fungus is a hyponym
of plant.
11. Hypernym: This pointer refers to a generalization of the concept, e.g. fruit is a hypernym
of apple.
2.3 Semantic Similarity of Words
Several measures of semantic similarity based on distance between words in WordNet have been
used by dierent researchers. Leacock and Chodorow [LC98] have used the negative logarithm
of the normalized shortest path length as a measure of similarity between two words, where
the path length is measured as the number of nodes in the path between the two words and
the normalizing factor is the maximum depth in the taxonomy. In this metric, the greater
the semantic distance between two words in the WordNet hierarchy, the less is their semantic
similarity. Lee et al. [LKY93] and Rada et al. [RMBB89] have used conceptual distance, based
on an edge counting metric, to measure similarity of a query to documents. Resnick [Res92]
observed that two words deep in the WordNet are more closely related than two words higher
up in the tree, both pairs having the same path length (number of nodes) between them.
Sussna [Sus93] took this into account in his semantic distance measure that uses depth-relative
scaling. Hirst et al. [HSO98] classied the relations of WordNet into the three broad directional
categories and used a distance measure where they took into account not only the path length but
also the number of direction changes in the semantic relations along the path. Resnick [Res95]
has used an information-based measure instead of path length to measure the similarity, where
the similarity of two words is estimated from the information content of the least probable class
to which both words belong.
3 Scoring the Novelty of Rules
3.1 Semantic Distance Measure
We have dened the semantic distance between two words w i and w j as:
where is the distance along path p according to
our weighting scheme, Dir(p) is the number of direction changes of relations along path p, and
K is a suitably chosen constant.
The second component of the formula is derived from the denition of Hirst et al. [HSO98],
where the relations of WordNet are divided into three direction classes | \up", \down" and
\horizontal", depending on how the two words in the relation are lexically related. Table 1
summarizes the direction information for the relation types we use. The more direction changes
in the path from one word to another, the greater the semantic distance between the words,
since changes of direction along the path re
ect large changes in semantic context.
The path distance component of the above formula is based on the semantic distance de-
nition of Sussna [Sus93]. It is dened as the shortest weighted path between w i and w j , where
every edge in the path is weighted according to the WordNet relation corresponding to that
edge, and is normalized by the depth in the WordNet tree where the edge occurs. We have
used 15 dierent WordNet relations in our framework, and we have assigned dierent weights
to dierent link types, e.g. hypernym represents a larger semantic change than synonym, so
hypernym has a higher weight than synonym. The weight chosen for the dierent relations are
given in Table 1.
One point to note here is that Sussna's denition of semantic distance calculated the weight
of an edge between two nouns w i and w j as the average of the two relations w
corresponding to the edge, relation r 0 being the inverse of relation r. This made
the semantic distance between two words a symmetric measure. He had considered the noun
hierarchy, where every relation between nouns has an inverse relation. But in our framework,
where we have considered all the four types of words in WordNet (nouns, adverbs, adjectives
and verbs) and 15 dierent relation types between these words, all of these relations do not have
inverses, e.g. the entailment relation has no direct inverse. So, we have used only the weight of
the relation w as a measure of the weight of the edge between w i and w j . This gives a
directionality to our semantic measure, which is also conceptually compatible with the fact that
w i is a word in the antecedent of the rule and w j is a word in the consequent of the rule.
3.2 Rule Scoring Algorithm
The scoring algorithm of rules according to novelty is outlined in Figure 2. The algorithm
calculates the semantic distance d(w is in the
antecedent and w j is in the consequent of the rule, based on the length of the shortest path
Relation Direction Weight
Synonym, Attribute, Pertainym, Similar Horizontal 0.5
Antonym Horizontal 2.5
Hypernym, (MemberjPartjSubstance) Meronym Up 1.5
Hyponym, (MemberjPartjSubstance) Holonym, Down 1.5
Cause, Entailment

Table

1: Direction and weight information for the 15 WordNet relations used
connecting w i and w j in WordNet. The novelty of a rule is then calculated as the average value
of all pairs of words (w
The noun hierarchy of the WordNet is disconnected | there are 11 trees with distinct root
nodes. The verb hierarchy is also disconnected, with 15 distinct root nodes. For our purpose,
following the method of Leacock and Chodorow [LC98], we have connected the 11 root nodes of
the noun hierarchy to a single root node R noun so that a path can always be found between two
nouns. Similarly, we have connected the verb root nodes by a single root node R verb . R noun
and R verb are further connected to a top-level root node, R top . This connects all the verbs
and nouns in the WordNet database. Adjectives and adverbs are not hierarchically arranged
in WordNet, but they are related to their corresponding nouns. In this composite connected
hierarchy derived from the WordNet hierarchy, we nd the shortest weighted path between two
words by performing a branch and bound search.
In this composite word hierarchy, any two words are connected by a path. However, we
have used 15 dierent WordNet relations while searching for the path between two words |
this creates a combinatorial explosion while performing the branch and bound search on the
composite hierarchy. So, for e-cient implementation, we have a user-specied time-limit (set
to 3 seconds in our experiments) within which we try to nd the shortest path between the
words w i and w j . If the shortest path cannot be found within the time-limit, the algorithm
nds a default path between w i and w j by going up the hierarchy from both w i and w j , using
hypernym links, till a common root node is reached.
The function PathViaRoot in Figure 2 computes the distance of the default path. For nouns
and verbs, the PathViaRoot function calculates the distance of the path between the two words
as the sum of the path distances of each word to its root. If the R noun or the R verb node are
For each rule in a rule le
set of antecedent words,
set of consequent words
For each word w
and w j
are not a valid words in WordNet
Score (w
Elseif w j is not a valid word in WordNet
Score (w
is not a valid word in WordNet
Score (w
Elseif path not found between w i and w j (in
user-specied time-limit)
Score (w
Else
Score (w
Score of rule = Average of all (w
Sort scored rules in descending order

Figure

2: Rule Scoring Algorithm
a part of this path, it adds a penalty term POSRootPenalty = 3.0 to the path distance. If the
R top node is a part of this path, it adds a larger penalty TopRootPenalty = 4.0 to the path
distance. These penalty terms re
ect the large semantic jumps in paths which go through the
root nodes R noun , R verb and R top .
If one of the words is an adjective or an adverb, and the shortest path method does not
terminate within the specied time-limit, then the algorithm nds the path from the adjective
or adverb to the nearest noun, through relations like \pertainym", \attribute", etc. It then nds
the default path up the noun hierarchy, and the PathViaRoot function incorporates the distance
of the path from the adjective or adverb to the noun form into the path distance measurement.
Some of the words extracted from the rules are not valid words in WordNet e.g. abbrevi-
ations, names like Philip, domain specic terms like booknews, etc. We assigned such words
the average depth of a word (d avg in Figure 2) in the WordNet hierarchy, which was estimated
by sampling techniques to be about 6, and then estimated its path distance to the root of the
combined hierarchy by using the PathViaRoot function.
4 Experimental Results
We performed experiments to compare the novelty judgment of human users to the automatic
ratings of our algorithm. The objective here is that if the automatic ratings correlate with human
High score (9.5):
romance love heart -> midnight
Medium score (5.8):
author romance -> characters love
Low
astronomy science -> space

Figure

3: Examples of rules scored by our novelty measure
judgments about as well as human judgments correlate with each other, then the novelty metric
can be considered successful.
4.1 Methodology
For the purpose of our experiments, we took rules generated by DiscoTEX from 9000 Ama-
zon.com book descriptions: 2000 in the \literature" category, 3000 in the \science" category
and 4000 in the \romance" category. From the total set of rules, we selected a subset of rules
that had less than a total of 10 words in the antecedent and consequent of the rule | this
was done so that the rules were not too large for human users to rank. Further pruning was
performed to remove duplicate words from the rules. For the Amazon.com book description do-
main, we also created a stoplist of commonly occurring words, e.g. book, table, index, content,
etc., and removed them from the rules. There were 1258 rules in the nal pruned rule-set.
We sampled this pruned rule-set to create 4 sets of random rules, each containing 25 rules.
We created a web-interface, which the subjects used to rank these rules with scores in the range
from (least interesting) to 10.0 (most interesting), according to their judgment. The 48
subjects were randomly divided into 4 groups and each group scored one of the rule-sets.
For each of the rule-sets, two types of average correlation were calculated. The rst average
correlation was measured between the human subjects, to nd the correlation in the judgment of
novelty between human users. The second average correlation measure was measured between
the algorithm and the users in each group, to nd the correlation between the novelty scoring
of the algorithm and that of the human subjects. We used both Pearson's raw score correlation
metric and Spearman's rank correlation metric to compute the correlation measures.
One of the rule-sets was used as a training set, to tune the parameters of the algorithm. The
results on the 3 other rule-sets, used as test sets for our experiment, are summarized in Table 2.
Human - Human Algorithm - Human
Correlation Correlation
Raw Rank Raw Rank
Group2
Average

Table

2: Summary of experimental results
4.2 Results and Discussion
Some of the rules scorings generated by our algorithm are shown in Figure 3. The high-scoring
rule and the low-scoring rule were rated by the human subjects, on the average, as high scoring
and low-scoring too.
From the results, considering both the raw and the rank correlation measures, we see that
the correlation between the human subjects and the algorithm is comparable to that between
the human subjects, averaging over the three random rule-sets considered. The average raw
correlation values among the human subjects and between the human subjects and the algorithm
are both not very high. This is because for some rules, the human subjects diered a lot in
their novelty assessment. This is also due to the fact that these are initial experiments, and
we are working on improving the methodology. In later experiments, we intend to apply our
method to domains where we can expect human users to agree more in their novelty judgment
of rules. However, it is important to note that it is very unlikely that these correlations are
due to random chance, since both the average raw correlation values are above the minimum
signicant r at the p < 0:1 level of signicance determined by a t-test.
The correlation between the human subjects and the algorithm was low for the rst rule-
set. For the second and the third rule-sets, the algorithm-human correlation is better than
the human-human correlation. On closer analysis of the results of Group1, we noticed that
this rule-set contained many rules involving proper names. Our algorithm currently uses only
semantic information from WordNet, so it's scoring on these rules diered from that of human
subjects. For example, one rule many users scored as uninteresting was \ieee society ! science
mathematics", but since WordNet does not have an entry for \ieee", our algorithm gave the
overall rule a high score. Another rule to which some users gave a low score was \physics science
nature ! john wiley publisher sons", presumably based on their background knowledge about
publishing houses. In this case, our algorithm found the name John in the WordNet hierarchy
(synset lemma: disciple of Jesus), but there was no short path between John and the words in
the antecedent of the rule. As a result, the algorithm gave this rule a high score. A point to
note here is that some names like Jesus, John, James, etc. have entries in WordNet, but others
like Sandra, Robert, etc. do not | this makes it di-cult to use any kind of consistent handling
of names using lters like name lists.
In the training rule-set, we had also noticed that the rule \sea ! oceanography" had been
given a large score by our algorithm, while most subjects in that group had rated that rule as
uninteresting. This happened because there is no short path between sea and oceanography in
WordNet | these two words are related thematically, and WordNet does not have thematic
connections, an issue which is discussed in detail in Section 6.
5 Related Work
Soon after the Apriori algorithm for extracting association rules was proposed, researchers in
the data mining area realized that even modest settings for support and condence typically
resulted in a large number of rules. So much eort has gone into reducing such rule-sets by
applying both objective and subjective criteria. Klemettinen et al. [KMR proposed the
use of rule templates to describe the structure of relevant rules and constrain the search space.
Another notable attempt in using objective measures was by Bayardo and Agrawal [BA99], who
dened a partial order, in terms of both support and condence, to identify a smaller set of
rules that were more interesting than the rest. Sahar [Sah99] proposed an iterative elimination
of uninteresting rules, limiting user interaction to a few simple classication questions. Hussain
et al. [HLSL00] developed a method for identifying exception rules, with the interestingness of
a rule being estimated relative to common sense rules and reference rules. In a series of papers,
Tuzhilin and his co-researchers [ST96, PT98, AT99] argued the need for subjective measures for
the interestingness of rules. Rules that were not only actionable but also unexpected in that they
con
icted with the existing system of beliefs of the user, were preferred. Liu et al. [LHMH99]
have further built on this theme, implementing it as an interactive, post-processing routine.
They have also analyzed classication rules, such as those extracted from C4.5, dening a
measure of rule interestingness in terms of the syntactic distance between a rule and a belief. A
rule and a belief are \dierent" if either the consequents of the rule and the belief are \similar"
but the antecedents are far apart, or vice versa.
In contrast, in this paper we have analyzed information extracted from unstructured or
semi-structured data such as web-pages, and extracted rules depicting important relations and
regularities in such data. The nature of rules as well as of prior domain knowledge is quite
dierent from those extracted, say, from market baskets. We have proposed an innovative use of
WordNet to estimate the semantic distance between the antecedents and consequents of a rule,
which is used as an indication of the novelty of the rule. Domain-specic concept hierarchies have
previously been used to lter redundant mined rules [HF95, FD95]; however, to our knowledge
they have not been used to evaluate novelty quantitatively, or applied to rules extracted from
text data.
6 Future Work
An important issue that we want to address in future is the selection of the parameters of
the algorithm, e.g. the weights of the relations, and values of K, POSRootPenalty and Top-
RootPenalty. These constants are now chosen experimentally. We would like to learn these
parameters automatically from training data, using a machine learning technique. The novelty
score could then be adaptively learnt for a particular user and tailored to suit the user's
expectation.
We are using the average of the pairwise word similarity measures as the novelty score of a
rule. The average measure smoothes out the skewing eect due to large distances between any
two pairs of word in a rule. This is ne for most rules, except for some special cases, e.g. if we
have a rule \science ! scientic home", then the distance between \science" and \scientic"
is small, but that between \science" and \home" is large. Using average here gives the whole
rule a medium novelty score, which does not re
ect the fact that a part of the rule involving
the words \science" and \home" is highly interesting, while the other part involving the words
\science" and \scientic" is uninteresting. In this case, a combination method like maximum
might be more useful. A suitable combination of the average and the maximum metrics would
hopefully give a better novelty scoring.
Unfortunately, WordNet fails to capture all semantic relationships between words, such
as general thematic connections like that between \pencil" and \paper". However, other approaches
to lexical semantic similarity, such as statistical methods based on word co-occurrence
[MS99], can capture such relationships. In these methods, a word is typically represented by a
vector in which each component is the number of times the word co-occurs with another specied
word within a particular corpus. Co-occurrence can be based on appearing within a xed-size
window of words, or in the same sentence, paragraph, or document. The similarity of two words
is then determined by a vector-space metric such as the cosine of the angle between their corresponding
vectors [MS99]. In techniques such as Latent Semantic Analysis
the dimensionality of word vectors is rst reduced using singular value decomposition (SVD)
in order to produce lexical representations with a small number of highly-relevant dimensions.
Such methods have been shown to accurately model human lexical-similarity judgments [LD97].
By utilizing a co-occurrence-based metric for d(w rules could be ranked by novelty using
statistical lexical knowledge. In the end, some mathematical combination of WordNet and
co-occurrence based metrics may be the best approach to measuring lexical semantic distance.
To the extent that the names of relations, attributes, and values in a traditional database are
natural-language words (or can be segmented into words), our approach could also be applied
to traditional data mining as well as text mining. The algorithm can be easily generalized for
scoring the novelty of other types of rules, e.g. association rules derived from market-basket data.
In that case, we would require a knowledge-base for the corresponding domain, e.g. a concept
hierarchy of the company products. The domain-specic concept hierarchies and knowledge-bases
could be used to nd semantic connections between rule antecedents and consequents and
thereby contribute to evaluating novelty.
Finally, the overall interestingness of a rule might be best computed as a suitable mathematical
combination of novelty and more traditional metrics such as condence and support.
7 Conclusion
This paper proposes a methodology for extracting, analyzing and ltering rules extracted from
unstructured or semi-structured data such as web pages. These rules can underscore novel
and useful relations and regularities in textual sources of information such as web pages, email
and usenet postings. Note that the nature of rules as well as of prior domain knowledge is
quite dierent from those extracted, say, from market baskets. A salient contribution of this
paper is a new approach for measuring the novelty of rules mined from text data, based on the
lexical knowledge in WordNet. This algorithm can also be extended to rules in other domains,
where a domain-specic knowledge hierarchy is available. We have also introduced a systematic
method of empirically evaluating interestingness measures for rules, based on average correlation
statistics, and have successfully shown that the automatic scoring of rules based on our novelty
measure correlates with human judgments about as well as human judgments correlate with
each other.

Acknowledgments

We would like to thank Un Yong Nahm for giving us the DiscoTEX rules sets on which we ran
our experiments. We are grateful to John Didion for providing the JWNL Java interface to
WordNet, which we used to develop the software, and for giving us useful feedback about the
package. We are also grateful to all the people who volunteered to take part in our experiments.
The rst author was supported by the Microelectronics and Computer Development (MCD)
Fellowship, awarded by the University of Texas at Austin, while doing this research.



--R

User pro
Bayardo Jr.
Indexing by latent semantic analysis.
Knowledge discovery in textual databases (KDT).
An Electronic Lexical Database.

Untangling text data mining.
Discovery of multiple-level association rules from large databases
Data Mining: Concepts and Techniques.
Exception rule mining with a relative interestingness measure.
Lexical chains as representations of context for the detection and correction of malapropims.
Finding interesting rules from large sets of discovered association rules.
Combining local context and WordNet similarity for word sense identi

Finding interesting patterns using user expectations.
Information retrieval based on a conceptual distance in IS-A heirarchy
Dunja Mladeni

A mutually bene
A belief-driven method for discovering unexpected patterns
WordNet and distribution analysis: A class-based approach to lexical discovery
Using information content to evaluate semantic similarity in a taxon- omy
Development and application of a metric on semantic nets.
Interestingness via what is not interesting.
What makes patterns interesting in knowledge discovery systems.
Word sense disambiguation for free-text indexing using a massive semantic network
--TR
Word sense disambiguation for free-text indexing using a massive semantic network
Finding interesting rules from large sets of discovered association rules
Foundations of statistical natural language processing
Mining the most interesting rules
Interestingness via what is not interesting
Data mining
What Makes Patterns Interesting in Knowledge Discovery Systems
Finding Interesting Patterns Using User Expectations
Discovery of Multiple-Level Association Rules from Large Databases
Exception Rule Mining with a Relative Interestingness Measure
A Mutually Beneficial Integration of Data Mining and Information Extraction

--CTR
Xin Chen , Yi-fang Brook Wu, Web mining from competitors' websites, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA
Raz Tamir , Yehuda Singer, On a confidence gain measure for association rule discovery and scoring, The VLDB Journal  The International Journal on Very Large Data Bases, v.15 n.1, p.40-52, January 2006
B. Shekar , Rajesh Natarajan, A Framework for Evaluating Knowledge-Based Interestingness of Association Rules, Fuzzy Optimization and Decision Making, v.3 n.2, p.157-185, June 2004
Combining Information Extraction with Genetic Algorithms for Text Mining, IEEE Intelligent Systems, v.19 n.3, p.22-30, May 2004
wordnet;interesting rules;knowledge hierarchy;novelty;semantic distance