In many applications of natural language processing (NLP) it is necessary to determine the likelihood of a given word combination. For example, a speech recognizer may need to determine which of the two word combinations eat a peach'' and eat a beach'' is more likely. Statistical NLP methods determine the likelihood of a word combination from its frequency in a training corpus. However, the nature of language is such that many word combinations are infrequent and do not occur in any given corpus. In this work we propose a method for estimating the probability of such previously unseen word combinations using available information on most similar'' words. We describe probabilistic word association models based on distributional word similarity, and apply them to two tasks, language modeling and pseudo-word disambiguation. In the language modeling task, a similarity-based model is used to improve probability estimates for unseen bigrams in a back-off language model. The similarity-based method yields a 20 perplexity improvement in the prediction of unseen bigrams and statistically significant reductions in speech-recognition error. We also compare four similarity-based estimation methods against back-off and maximum-likelihood estimation methods on a pseudo-word sense disambiguation task in which we controlled for both unigram and bigram frequency to avoid giving too much weight to easy-to-disambiguate high-frequency configurations. The similarity-based methods perform up to 40 better on this particular task.
@cite_28: Selectional constraints specify, for a particular domain, the combinations of semantic classes acceptable in subject-verb-object relationships and other syntactic structures. These constraints are important in blocking incorrect analyses in natural language processing systems. However, these constraints are domain-specific and hence must be developed anew when a system is ported to a new domain. A discovery procedure for selectional constraints is therefore essential in enhancing the portability of such systems.This paper describes a semi-automated procedure for collecting the co-occurrence patterns from a sample of texts in a domain, and then using these patterns as the basis for selectional constraints in analyzing further texts. We discuss some of the difficulties in automating the collection process, and describe two experiments that measure the completeness of these patterns and their effectiveness compared with manually-prepared patterns. We then describe and evaluate a procedure for selectional constraint relaxation, intended to compensate for gaps in the set of patterns. Finally, we suggest how these procedures could be combined with a system that queries a domain expert, in order to produce a more efficient discovery procedure.
An early attempt to automatically classify words into semantic classes was carried out in the Linguistic String Project @cite_28 . Semantic classes were derived from similar cooccurrence patterns of words within syntactic relations. Cooccurrence statistics were then considered at the class level and used to alleviate data sparseness in syntactic disambiguation.