We formulate sequence to sequence transduction as a noisy channel decoding problem and use recurrent neural networks to parameterise the source and channel models. Unlike direct models which can suffer from explaining-away effects during training, noisy channel models must produce outputs that explain their inputs, and their component models can be trained with not only paired training samples but also unpaired samples from the marginal output distribution. Using a latent variable to control how much of the conditioning sequence the channel model needs to read in order to generate a subsequent symbol, we obtain a tractable and effective beam search decoder. Experimental results on abstractive sentence summarisation, morphological inflection, and machine translation show that noisy channel models outperform direct models, and that they significantly benefit from increased amounts of unpaired output data that direct models cannot easily use.
@cite_4: In this paper, we describe a Dynamic Programming (DP) based search algorithm for statistical translation and present experimental results. The statistical translation uses two sources of information: a translation model and a language model. The language model used is a standard bigram model. For the translation model, the alignment probabilities are made dependent on the differences in the alignment positions rather than on the absolute positions. Thus, the approach amounts to a first-order Hidden Markov model (HMM) as they are used successfully in speech recognition for the time alignment problem. Under the assumption that the alignment is monotone with respect to the word order in both languages, an efficient search strategy for translation can be formulated. The details of the search algorithm are described. Experiments on the EuTrans corpus produced a word error rate of 5.1 .
@cite_3: 
@cite_2: We describe a supervised approach to predicting the set of all inflected forms of a lexical item. Our system automatically acquires the orthographic transformation rules of morphological paradigms from labeled examples, and then learns the contexts in which those transformations apply using a discriminative sequence model. Because our approach is completely data-driven and the model is trained on examples extracted from Wiktionary, our method can extend to new languages without change. Our end-to-end system is able to predict complete paradigms with 86.1 accuracy and individual inflected forms with 94.9 accuracy, averaged across three languages and two parts of speech.
Noisy channel decompositions have been successfully used in a variety of problems, including speech recognition , machine translation , spelling correction , and question answering . The idea of adding language models and monolingual data in machine translation has been explored in earlier work. @cite_4 propose two strategies of combining a language model with a neural sequence to sequence model. In shallow fusion, during decoding the sequence to sequence model (direct model) proposes candidate outputs and these candidates are reranked based on the scores calculated by a weighted sum of the probability of the translation model and that of the language model. In deep fusion, the language model is integrated into the decoder of the sequence to sequence model by concatenating their hidden state at each time step. @cite_3 incorporate target language unpaired training data by doing back-translation to create synthetic parallel training data. While this technique is quite effective, its practicality seems limited to problems where the inputs and outputs contain roughly the same information (such as translation). @cite_2 leverages the abundant monolingual data by doing multitask learning with an autoencoding objective.