Grammatical Error Correction with Neural Reinforcement Learning

We propose a neural encoder-decoder model with reinforcement learning (NRL) for grammatical error correction (GEC). Unlike conventional maximum likelihood estimation (MLE), the model directly optimizes towards an objective that considers a sentence-level, task-specific evaluation metric, avoiding the exposure bias issue in MLE. We demonstrate that NRL outperforms MLE both in human and automated evaluation metrics, achieving the state-of-the-art on a fluency-oriented GEC corpus.


Introduction
Research in automated Grammatical Error Correction (GEC) has expanded from token-level, closed class corrections (e.g., determiners, prepositions, verb forms) to phrase-level, open class issues that consider fluency (e.g., content word choice, idiomatic collocation, word order, etc.).
The expanded goals of GEC have led to new proposed models deriving from techniques in data-driven machine translation, including phrasebased MT (PBMT) (Felice et al., 2014;Chollampatt et al., 2016;Junczys-Dowmunt and Grundkiewicz, 2016) and neural encoder-decoder models (Yuan and Briscoe, 2016). Napoles et al. (2017) recently showed that a neural encoderdecoder can outperform PBMT on a fluencyoriented GEC data and metric.
We investigate training methodologies in the neural encoder-decoder for GEC. To train the neural encoder-decoder models, maximum likelihood estimation (MLE) has been used, where the objective is to maximize the (log) likelihood of the parameters for a given training data.
As Ranzato et al. (2015) indicates, however, MLE has drawbacks. The MLE objective is based on word-level accuracy against the reference, and the model is not exposed to the predicted output during training (exposure bias). This becomes problematic, because once the model fails to predict a correct word, it falls off the right track and does not come back to it easily. To address the issues, we employ a neural encoder-decoder GEC model with a reinforcement learning approach in which we directly optimize the model toward our final objective (i.e., evaluation metric). The objective of the neural reinforcement learning model (NRL) is to maximize the expected reward on the training data. The model updates the parameters through back-propagation according to the reward from predicted outputs. The high-level description of the training procedure is shown in Algorithm 1, and more details are elaborated in §2. To our knowledge, this is the first attempt to employ reinforcement learning for directly optimizing the encoder-decoder model for GEC task.
We run GEC experiments on a fluency-oriented GEC corpus ( §3), demonstrating that NRL outperforms the MLE baseline both in human and automated evaluation metrics.

Model and Optimization
We use the attentional neural encoder-decoder model (Bahdanau et al., 2014) as a basis for both NRL and MLE. The model takes (possibly ungrammatical) source sentences x ∈ X as an input, and predicts grammatical and fluent output sentences y ∈ Y according to the model parameter θ. The model consists of two sub-modules, encoder and decoder. The encoder transforms x into a sequence of vector representations (hidden states) using a bidirectional gated recurrent neural network (GRU) (Chung et al., 2014). The decoder predicts a word y t at a time, using previous token y t−1 and linear combination of encoder information as attention.

Maximum Likelihood Estimation
Maximum Likelihood Estimation training (MLE) is a standard optimization method for encoderdecoder models. In MLE, the objective is to maximize the log likelihood of the correct sequence for a given sequence for the entire training data.
The gradient of L(θ) is as follows: One drawback of MLE is the exposure bias (Ranzato et al., 2015). The decoder predicts a word conditioned on the correct word sequence (y t−1 1 ) during training, whereas it does with the predicted word sequence (ŷ t−1 1 ) at test time. Namely, the model is not exposed to the predicted words in training time. This is problematic, because once the model fails to predict a correct word at test time, it falls off the right track and does not come back to it easily. Furthermore, in most sentence generation tasks, the MLE objective does not necessarily correlate with our final evaluation metrics, such as BLEU (Papineni et al., 2002) in machine translation and ROUGE (Lin, 2004) in summarization. This is because MLE optimizes word level predictions at each time step instead of evaluating sentences as a whole.
GEC is no exception. It depends on sentencelevel evaluation that considers grammaticality and fluency. For this purpose, it is natural to use GLEU (Napoles et al., 2015), which has been used as a fluency-oriented GEC metric. We explain more details of this metric in §2.3.

Neural Reinforcement Learning
To address the issues in MLE, we directly optimize the neural encoder-decoder model toward our final objective for GEC using reinforcement learning. In reinforcement learning, agents aim to maximize expected rewards by taking actions and updating the policy under a given state. In the neural encoder-decoder model, we treat the encoderdecoder as an agent which predicts a word from a fixed vocabulary at each time step (the action), given the hidden states of the neural encoderdecoder representation. The key difference from MLE is that the reward is not restricted to tokenlevel accuracy. Namely, any arbitrary metric is applicable as the reward. 1 Since we use GLEU as the final evaluation metric, the objective of NRL is to maximize the expected GLEU by learning the model parameter.
where S(x) is a sampling function that produces k samplesŷ 1 , ...ŷ k , p(ŷ|x; θ) is a probability of the output sentence, and r(ŷ, y) is the reward forŷ k given a reference set y. As described in Algorithm 1, given a pair of source sentence and the reference (x, y), NRL takes k sample outputsŷ 1 , ...ŷ k and their probabilities p(ŷ 1 ), ... p(ŷ k ) (line 3). 2 Then, the expected reward is computed by multiplying the probability and metric score for each samplê y i (line 7).
In the encoder-decoder model, the parameters θ are updated through back-propagation and the number of parameter updates is determined by the partial derivative of J(θ), called the policy gradient (Williams, 1992;Sutton et al., 1999) in reinforcement learning: where α is a learning rate and b is an arbitrary baseline reward to reduce the variance. The sample mean reward is often used for b (Williams, 1992), and we follow it in NRL.
It is reasonable to compare NRL to minimum risk training (MRT) (Shen et al., 2016). In fact, NRL with a negative expected reward can be regarded as MRT. The gradient of MRT objective is a special case of policy gradient in NRL. We show mathematical details about the relevance between NRL and MRT in the supplemental material (Appendix A).

Reward in Grammatical Error Correction
To capture fluency as well as grammaticality in evaluation on such references, we use GLEU as the reward. We have shown GLEU to be more strongly preferred than other GEC metrics by native speakers (Sakaguchi et al., 2016). Similar to BLEU in machine translation, GLEU computes n-gram precision between the system hypothesis (H) and the reference (R). In GLEU, however, ngrams in source (S) are also considered. The precision is penalized when the n-gram in H overlaps with the source and not with the reference.
is the number of overlapped n-grams among the sets, and BP brevity penalty is compute based on token length in the system hypothesis (h) and the reference (r).

Experiments
Data For training the models (MLE and NRL), we use the following corpora: the NUS Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013), the Cambridge Learner Corpus First Certificate English (FCE) (Yannakoudakis et al., 2011), and the Lang-8 Corpus of learner English (Tajiri et al., 2012). The basic statistics are shown in Table 1. 3 We exclude some unreasonable edits (comments by editors, incomplete sentences such  as URLs, etc.) using regular expressions and setting a maximum token edit distance within 50% of the original length. We also ignore sentences that are longer than 50 tokens or sentences where more than 5% of tokens are out-of-vocabulary (the vocabulary size is 35k). In total, we use 720k pairs of sentences for training (21k from NUCLE, 32k from FCE, and 667k from Lang-8 Hyperparameters For both MLE and NRL, we set the vocabulary size to be 35k for both source and target. Words are represented by a vector with 512 dimensions. Maximum output token length is 50. The size of hidden layer units is 1,000. Gradients are clipped at 1, and beam size during decoding is 5. We regularize the GRU layer with a dropout probability of 0.2. For MLE we use mini-batches of size 40, and the ADAM optimizer with a learning rate of 10 −4 . We train the encoder-decoder with MLE for 900k updates, selecting the best model according to the development set evaluation. For NRL we set the sample size to be 20. We use the SGD optimizer with a learning rate of 10 −4 . For the baseline reward, we use average of sampled reward following Williams (1992). The sentence GLEU score is used as the reward r(ŷ, y). Following a similar (but not the same) strategy of the Mixed Incremental Cross-Entropy Reinforce (MIXER) algorithm (Ranzato et al., 2015), we initialize the model by MLE for 600k updates, followed by another 600k updates using NRL, and select the best model according to the development set evaluation. Our NRL is implemented by extending the Nematus toolkit (Sennrich et al., 2017)  In addition to the automated metric (GLEU), we run a human evaluation using Amazon Mechanical Turk (MTurk). We randomly select 200 sentences each from the dev and test set. For each sentence, two turkers are repeatedly asked to rank five systems randomly selected from all eight: the four baseline models, MLE, NRL, one randomly selected human correction, and the original sentence. We infer the evaluation scores by comparing pairwise rankings with the TrueSkill algorithm (Herbrich et al., 2006;Sakaguchi et al., 2014). 6 The four baselines are not tuned toward the same dev set as MLE and NRL. Also, they use different training set (Table  2). We compare them just for reference.    Results Table 3 shows the human evaluation by TrueSkill and automated metric (GLEU). In both dev and test set, NRL outperforms MLE and other baselines in both the human and automatic evaluations. Human evaluation and GLEU scores correlate highly, corroborating the reliability of GLEU. With respect to inter-annotator agreement, Spearman's rank correlation between Turkers is 55.6 for the dev set and 49.2 for the test set. The correlations are sufficiently high to show the agreement between Turkers, considering the low chance level (i.e., ranking five randomly selected systems consistently between two Turkers). Table 4 and 5 show the M2 (F 0.5 ) scores (Dahlmeier and Ng, 2012), which compute phrase-level edits between the system hypothesis and source and compare them with the oracle edits. Although this metric has several drawbacks such as underestimation of system performance and indiscrimination between "no change" and "wrong edits" (Felice et al., 2014), we see that the correlation between the M2 scores and human evaluation is still high in the result.
Finally, Table 6 shows the percentages of preference in the pairwise comparisons between NRL and MLE. In both the dev and test sets, around 30% of NRL corrections are preferred over MLE and approximately 50% are tied.
Orig. but found that successful people use the people money and use there idea for a way to success . Ref.
But it was found that successful people use other people 's money and use their ideas as a way to success . MLE But found that successful people use the people money and use it for a way to success . NRL But found that successful people use the people 's money and use their idea for a way to success .
Orig. Fish firming uses the lots of special products such as fish meal . Ref.
Fish firming uses a lot of special products such as fish meal . MLE Fish contains a lot of special products such as fish meals . NRL Fish shops use the lots of special products such as fish meal .  Table 7 presents example outputs from MLE and NRL. In the first example, both MLE and NRL successfully corrected the homophone error (there vs. their), but MLE changed the meaning of the original sentence by replacing their idea to it. Meanwhile, NRL made the sentence more grammatical by adding a possessive 's. The second example demonstrates challenging issues for future work in GEC. The correction by MLE looks fairly fluent as well as grammatical, but it is semantically nonsense. The correction by NRL is also fairly fluent and makes sense, but the meaning has been changed too much. For further improvement, better GEC models that are aware of the context or possess world knowledge are needed.

Conclusions
We have presented a neural encoder-decoder model with reinforcement learning for GEC. To alleviate the MLE issues (exposure bias and tokenlevel optimization), NRL learns the policy (model parameters) by directly optimizing toward the final objective by treating the final objective as the reward for the encoder-decoder agent. Using a GEC-specific metric, GLEU, we have demonstrated that NRL outperforms the MLE baseline on the fluency-oriented GEC corpus both in human and automated evaluation metrics. As a supplement, we have explained the relevance between minimum risk training (MRT) and NRL, claiming that MRT is a special case of NRL.

A Minimum Risk Training and Policy Gradient in Reinforcement Learning
We explain the relevance between minimum risk training (MRT) (Shen et al., 2016) and neural reinforcement learning (NRL) for training neural encoder-decoder models. We describe the detailed derivation of gradient in MRT, and show that MRT is a special case of NRL. As introduced in §2, the model takes ungrammatical source sentences x ∈ X as an input, and predicts grammatical and fluent output sentences y ∈ Y . The objective function in NRL and MRT are written as follows.
where r(ŷ, y) is the reward and ∆(ŷ, y) is the risk for an output (ŷ). For the sake of simplicity, we consider expected loss in MRT for a single training pair: q(ŷ|x; θ, α)∆(ŷ, y) where q(ŷ|x; θ, α) = p(ŷ|x; θ) α ŷ ∈S(x) p(ŷ |x; θ) α S(x) is a sampling function that produces k samplesŷ 1 , ...ŷ k , and α is a smoothing parameter for the samples (Och, 2003). Although the direction to optimize (i.e., minimizing or maximizing) is different, we see the similarity between J(θ) and R(θ) in the sense that they both optimize models directly towards evaluation metrics. The partial derivative ofR(θ) with respect to the model parameter θ is derived as follows.