Word alignment is an important natural language processing task that indicates the correspondence between natural languages. Recently, unsupervised learning of log-linear models for word alignment has received considerable attention as it combines the merits of generative and discriminative approaches. However, a major challenge still remains: it is intractable to calculate the expectations of non-local features that are critical for capturing the divergence between natural languages. We propose a contrastive approach that aims to differentiate observed training examples from noises. It not only introduces prior knowledge to guide unsupervised learning but also cancels out partition functions. Based on the observation that the probability mass of log-linear models for word alignment is usually highly concentrated, we propose to use top-n alignments to approximate the expectations with respect to posterior distributions. This allows for efficient and accurate calculation of expectations of non-local features. Experiments show that our approach achieves significant improvements over state-of-the-art unsupervised word alignment methods.
@cite_0: It is possible to combine multiple latent-variable models of the same data by multiplying their probability distributions together and then renormalizing. This way of combining individual "expert" models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert, because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data. A product of experts (PoE) is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary. Training a PoE by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule. Fortunately, a PoE can be trained using a different objective function called "contrastive divergence" whose derivatives with regard to the parameters can be approximated accurately and efficiently. Examples are presented of contrastive divergence learning using several types of expert on several types of data.
Contrastive learning has received increasing attention in a variety of fields. @cite_0 proposes contrastive divergence (CD) that compares the data distribution with reconstructions of the data vector generated by a limited number of full Gibbs sampling steps. It is possible to apply CD to unsupervised learning of latent-variable log-linear models and use top- @math sampling to approximate the expectation on posterior distributions within each full Gibbs sampling step. The noise-contrastive estimation (NCE) method casts density estimation, which is a typical unsupervised learning problem, as supervised classification by introducing noisy data. However, a key limitation of NCE is that it cannot be used for models with latent variables that cannot be integrated out analytically. There are also many other efforts in developing contrastive objectives to avoid computing partition functions . Their focus is on choosing assignments to be compared with the observed data and developing sub-objectives that allow for dynamic programming for tractable sub-structures. In this work, we simply remove the partition functions by comparing pairs of observed and noisy examples. Using noisy examples to guide unsupervised learning has also been pursued in deep learning .