Abstract
In this paper, a method of domain adaptation for clustered language models is developed. It is based on a previously developed clustering algorithm, but with a modified optimisation criterion. The results are shown to be slightly superior to the previously published 'Fillup' method, which can be used to adapt standard n-gram models. However, the improvement both methods give compared to models built from scratch on the adaptation data is quite small (less than 11% relative improvement in word error rate). This suggests that both methods are still unsatisfactory from a practical point of view.
INTRODUCTION
Current large vocabulary speech recognition systems can achieve good performance on domains for which large quantities (e.g. millions of words) of textual data are available to train a language model. In real world applications, however, this is quite often not the case. The issue of language model domain adaptation is therefore of great practical importance.
INTRODUCTION
One approach to tackle this problem is to try to learn from an analogy to the speaker dependence issue: current systems perform well by training speaker independent models, which can then be adapted with relatively little data from a given speaker (see [SI). Can the same approach be applied to language model adaptation?
INTRODUCTION
In section 2., previous work in this area is reviewed and a rough working definition of domain is given. A method to perform domain adaptation with clustered language models is then developed (Section 3.). Experimental results to evaluate the method are given in Section 4., followed by conclusions in Section 5..
BACKGROUND
In order to make the description of domain adaptation more precise, a definition of domain is needed. One might be tempted to define domain in the sense of semantic topic. However, texts might differ in other aspects (e.g. style), which could still require language model adaptation. A more general definition of domain, more in line with the term sublanguage, is therefore required. According to [lo] , there are many different definitions of the term, but most of them seem to agree on the following characteristics of a sublanguage: [13] ). Since the 'Fillup' method presented in [l] gives better performance than linear interpolation, the 'Fillup' method is used i s method of comparison for the adaptive clustering, which will be developed in the next section.
ADAPTIVE CLUSTERING
The task of a language model is to calculate p(wl(ct), the probability of the next word being w1 given the current context ct. Language models differ in the way this probability is modelled and how the context ct is defined. A quite general model proposed in [12] The basic building block in the derivation of equation 2 is the likelihood of one event in the training corpus, as estimated from the training corpus in which this one event has been removed (leaving-one-out likelihood). The main idea behind the adaptive clustering is to use as basic building block the likelihood of one event in S A d a p t , as estimated from a linear interpolation of counts from S B~~A . and S A d a p t from which this one event has been removed. The motivation for this is that the clustering can thus optimise the perplexity on SA&#, while having access to a linear combination of counts from S B~~~ and S A d a p t .
ADAPTIVE CLUSTERING
Let N A (~) (N,(e) ) denote the number of times event e appeared in S A d a p t ( S B a c k where Roand(z) returns the integer nearest to z. The only events that can contribute to the optimisation function are events that occur at least once in '!$Adapt (because, as explained above, the likelihood of ,?Adapt is taken as optimisation function). However, their probability is calculated based on the combined counts. Therefore, the smoothing has to apply to the combined counts. Define n & , O r n b t , l , n b t , + as the number of pairs ( s , g ) that have a combined count N c ( s , g ) of 0, of 1, and larger than 0. In order to introduce absolute discounting for the unigram estimates as well, also define ns,o,nS,l, n,,+ as the number of states s that have a combined count N c ( s ) of 0, of 1, and larger than 0 (similarly, define n9,o etc. for the unigram estimates involving 9 ) . Changing equation 2 according to the basic idea outlined above, this leads to By using the same clustering algorithm as before, but with F A d a p t instead of F as optimisation criterion, language model domain adaptation can be performed.
RESULTS
In order to test different adaptation methods, two textual samples S B a c k and '!?Adapt and acoustic testing data from D A d a p t are required. Since the WSJ domain has the associated acoustic data, it is used as D A d a p t . As D B a c k , the patent domain (PAT) was chosen, for which a large sample (about 3.5 million words are used) is also available from the LDC as part of the TIPSTER database.
RESULTS
The recognition system is a state-of-the-art HMM based system (continuous densities, mixtures, triphones). All experiments are based on bigram language models, either clustered (500 clusters) or backoff (singleton bigrams were ignored). The different methods evaluated were e ClustAdapt: a model built with the adaptive clustering presented in the previous section; the initial starting point for the clustering is taken to be the clustering produced by Backcl; one global X parameter was used and optimised iteratively at the end of each iteration;
RESULTS
For all methods except BackBo and Backcl , the vocabulary was defined to be all the words that appeared in S A d a p t , plus additional words from S B~~~ until 20K words were reached. For BackB, and B a c k c l , the vocabulary consisted of the 20K most frequent words in S s a c k . Because of this difference, the perplexities of Backgo and Backcl are not directly comparable to those of the other models. For each method and a given amount of adaptation material, the perplexity of the resulting model was calculated on a held-out section of S A d a p t and a recognition run was performed on the acoustic data. Table 1 gives the results of the two baseline methods, which do not use any of the adaptation material. The high perplexities show that the PAT and WSJ domains are considerably different. The rate of out-of-vocabulary words is about 15%, which is one reason for the very high error rate.
RESULTS
Tables 2, 3, 4 and 5 give the results for the different methods and different amounts of adaptation material.
RESULTS
Comparing Table 3 to Table 2 , one can seen that Adaptcl is more robust than A d a p t s , and it leads to better recognition results for almost the entire range of adaptation material. This is consistent with previous results (see [15] ), which showed that clustered models are more robust in terms of perplexity.
RESULTS
Comparing Table 4 to 
. C O N C L U S I O N
Compared to the success of some methods for acoustic adaptation, the results obtained here are somewhat disappointing. In particular, they seem to suggest that the improvements from the adaptation techniques compared to starting from scratch on the adaptation data become quite small when several tens of thousands of words are available '. One reason for this could be the fact that the acoustic space has an underlying distance metric and thus allows the comparison of two elements. Moreover, one can specify the kind of transformations one would want the adaptation to be able to perform. Both of these points seem more difficult in the case of language model adaptation.
. C O N C L U S I O N
Even though the adaptation method for clustered language models developed in this paper gives slightly better results than the 'Fillup' method, the accuracies obtained with the adaptive clustering and the 'Fillup' method are still very low compared to the about 80% or more the system can achieve with a backoff bigram trained on about 40 million words of the WSJ corpus. Both adaptation methods are therefore still unsatisfactory from a practical point of view.
R E F E R E N C E S
[l] Besling and Meier. Language model speaker adaptation. In European Conference on Speech Communication and Technology, pages pp.1755-1758. Detroit, Michigan, USA, 1995.
R E F E R E N C E S
[2] S. della Pietra, V. della Pietra, L.R. Mercer, and S. Roukos. Adaptive language modeling using minimum discriminant estimation. In Proceedings of Inter-'However, it is important to note that this threshold will depend on how dissimilar the two domains are. Moreover, a more fine grained analysis for different amounts of adaptation data would be beneficial, especially since the results for 25,000 words seem to be falling somewhat outside the trend.

{'paper_id': '6333340', 'title': 'Domain adaptation with clustered language models', 's2_url': 'https://api.semanticscholar.org/CorpusID:6333340', 'source': 'S2ORC'}