Title: Prior matters: simple and general methods for evaluating and improving topic quality in topic modeling,
Abstract: Latent Dirichlet Allocation (LDA) models trained without stopword removal
often produce topics with high posterior probabilities on uninformative words,
obscuring the underlying corpus content. Even when canonical stopwords are
manually removed, uninformative words common in that corpus will still dominate
the most probable words in a topic. In this work, we first show how the
standard topic quality measures of coherence and pointwise mutual information
act counter-intuitively in the presence of common but irrelevant words, making
it difficult to even quantitatively identify situations in which topics may be
dominated by stopwords. We propose an additional topic quality metric that
targets the stopword problem, and show that it, unlike the standard measures,
correctly correlates with human judgements of quality. We also propose a
simple-to-implement strategy for generating topics that are evaluated to be of
much higher quality by both human assessment and our new metric. This approach,
a collection of informative priors easily introduced into most LDA-style
inference methods, automatically promotes terms with domain relevance and
demotes domain-specific stop words. We demonstrate this approach's
effectiveness in three very different domains: Department of Labor accident
reports, online health forum posts, and NIPS abstracts. Overall we find that
current practices thought to solve this problem do not do so adequately, and
that our proposal offers a substantial improvement for those interested in
interpreting their topics as objects in their own right.