TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation
Shaojie Jiang Thomas Wolf Christof Monz Maarten de Rijke
cs.CL

A Tutorial  on the Basic  Special Functions of Fractional Calculus
     
[t]2.3in 12.5pt
     FRANCESCO MAINARDI
       University Bologna and INFN
    Department of Physics & Astronomy


    francesco.mainardi@bo.infn.it
 0in







 
 
   
[b]6.9in 12.5pt Abstract:


In this tutorial survey we recall  the basic properties of the 
special function of the Mittag-Leffler and Wright type that are known  to be relevant
in processes dealt with the fractional calculus.
We outline the major applications of these functions.
For the Mittag-Leffler functions 
we analyze  the Abel integral equation of the second kind and 
the fractional relaxation and oscillation phenomena.
For the Wright functions we distinguish them in two kinds.
We mainly stress the relevance of the Wright functions of the second kind  in probability theory with particular regard to the so-called 
M-Wright functions that generalizes the Gaussian  
and is related  with the time-fractional diffusion equation. 



 [4mm] Key–Words:  
Mittag-Leffler functions,
Wright functions, Fractional Calculus,
Laplace, Fourier  and Mellin transforms,  
 Probability theory,   Stable distributions.
 



    
   Paper published in
WSEAS TRANSACTIONS on MATHEMATICS, Vol 19 (2020), pp. 74–98.

 DOI: 10.37394/23206.2020.19.8
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================


  NLG models are prone to generating repetitive utterances.
  In this work, we study the repetition problem for encoder-decoder models, using both RNN and transformer architectures.
  To this end, we consider the chit-chat task, where the problem is more prominent than in other tasks that need encoder-decoder architectures.

  We first study the influence of model architectures.
  By using pre-attention and highway connections for RNN, we manage to achieve lower repetition rates.
  However, this method does not generalize to other models such as transformers.
  
  We hypothesize that the deeper reason is that in the training corpora, there are hard tokens that are more difficult for a generative model to learn than others and, once learning has finished, hard tokens are still under-learned, so that repetitive generations are more likely to happen.
  Based on this hypothesis, we propose TLDRTLDR that applies differentiable weights to individual token losses.
  By using higher weights for hard tokens and lower weights for easy tokens, NLG models are able to learn individual tokens at different paces.
  Experiments on chit-chat benchmark datasets show that  is more effective in repetition reduction for both RNN and transformer architectures than baselines using different weighting functions.






§ INTRODUCTION


It has been widely reported that NLG models are prone to generating repetitive utterances, which is a cross-task and cross-architecture problem.
E.g., the repetition problem has been observed in QA <cit.>, LM <cit.>, abstractive summarization <cit.>, MT <cit.>, and image captioning <cit.>, etc.
The problem arises with most popular neural architectures, including Seq2Seq modeling using RNN <cit.> and transformers <cit.>.
Previous approaches to the repetition problem have mostly been developed for decoder-only models <cit.>.
Whenever it is possible to use them, encoder-decoder models usually outperform decoder-only models <cit.>; because of this, we study the repetition problem for encoder-decoder models in this work.
We base our experiments on the chit-chat task,[Throughout this paper, we use `chit-chat,' `open-domain dialogue,' and `response generation' to refer to the same task interchangeably, and use `chatbot' to refer to the model trained for this task.] where the repetition problem is more prominent than other tasks that need encoder-decoder architectures.
Figure <ref> is an illustration of a chit-chat scenario.









The repetition problem on which we focus in this paper is not to be confused with the problem where responses of different turns share a great deal of similarity, which is sometimes also referred to as repetitive responses <cit.>.


We first surmise that the repetition problem is caused by model architectures.
By studying existing approaches to repetition reduction involving changing model architectures, we identify that a certain model design for RNN can help reduce repetition.
More specifically, by using pre-attention (<ref>) together with RNN instead of the conventional usage of post-attention, the model generates less repetitive utterances.
The usage of highway connection <cit.> after pre-attention can further reduce repetition.
Though it is effective for RNN, this architectural design unfortunately does not generalize to transformers.
Therefore, we believe that there are deeper reasons to the repetition problem.



Inspired by the well-studied class imbalance problem <cit.>, we hypothesize that in training corpora there are hard tokens and easy tokens; if the training stops when hard tokens are still under-learned, repetitive generations are more likely to happen.
We empirically validate this hypothesis by using the FL method proposed by <cit.>.


Based on this finding, we propose TLDR, which is inspired by FL (<ref>) and which assigns differentiable weights to training losses according to the training difficulty of tokens.

By dynamically using higher weights for hard tokens and lower weights for easy tokens, the model is able to learn tokens at different paces.

Experiments on benchmark chit-chat datasets show that  is more effective in reducing repetitions for RNN and transformer architectures than the baselines.

The main contributions of this work include:


  * We discover that RNN with pre-attention and highway connections can help reduce repetition.

  * We hypothesize that the deeper reason for repetitions is due to hard tokens, and empirically validate this by using FL.

  * We propose a more effective TLDR method for reducing repetitive generations that is architecture insensitive.

  * We share our source code at https://github.com/ShaojieJiang/tldrhttps://github.com/ ShaojieJiang/tldr.











§ RELATED WORK





 §.§ Repetition reduction


There are generally three types of approach to the repetition problem, namely architecture-based methods, auxiliary loss, and sampling-based methods.

Architecture-based methods try to address the repetition problem by exerting inductive biases on model architectures, such as introducing coverage vectors to attention mechanisms <cit.>.

However, these solutions are based on the desirable property of source-target correspondence of NMT tasks, which rarely holds for other tasks like open-domain dialogue generation and abstractive summarization.
Instead, we observe that by using pre-attention (<ref>), RNN models can generate less repetitive responses.
Adding a highway connection to attention layers can further reduce repetition.
However, it is not yet clear which modifications to transformers can help reduce repetitive generations, which we surmise to be mainly due to their lack of temporal inner states.

Auxiliary loss adds an unlikelihood item of repeated tokens to the conventional CE loss <cit.> to discourage repetitive generations.
However, besides the need for extra data types <cit.> or extra processing on model generations <cit.>, it is also worth noting that the repetition problem seldom exists in the training data (<ref>), and therefore, the models are not trained to be repetitive in the first place.
Our experiments show that the repetition problem is mainly due to the fact that trained models have different levels of proficiency toward easy and hard tokens.
Based on this, we propose to weight the token losses w.r.t. their training difficulties to address the repetition problem.
Though, normally, hard tokens already result in larger gradients, apparently this is not enough to learn them well as their effects may be dissolved by large amount of easy tokens.

Sampling based approaches discourage repetition at the sampling stage, e.g., by creating an n-gram blacklist <cit.>.
Some more sophisticated methods include discounting the scores of previously generated tokens <cit.>, top-k random sampling <cit.>, and top-p sampling with a dynamic nucleus <cit.>.
These methods can only be used at inference time to remedy the repetition problem.
In this work, we tackle the problem at a deeper level and show that by dynamically reweighting token losses at training time, the problem can be largely alleviated.



 §.§ Class imbalance


Class imbalance is a common cause of poor performance of trained models.
To mitigate the low efficiency of heuristic sampling approaches to balancing classes, <cit.> try to balance the CE loss by weighting easy and hard examples differently for visual object detection.
Similarly, FACE <cit.> tries to balance the CE loss for frequent and rare tokens through heuristics such as token frequency for improving generation diversity.
In our case, however, the frequency of a token does not necessarily reflect its training difficulty, as the difficulty is also influenced by the context of this token.

We hypothesize that the repetition problem is also caused by different training difficulties of tokens.
This hypothesis is in line with <cit.>, where the authors improve the learning of hard tokens by using back-translation and by sampling examples with difficult words in similar contexts.
Considering that at the token level in the chit-chat task, balancing easy and hard tokens through sampling is infeasible, loss reweighting such as Focal Loss <cit.> is a preferable solution.
Focal Loss down-weights[Models trained without weighting can be thought of as using a uniform weight of 1.] both easy and hard examples, albeit at different ratios, which is likely to slow down training.
Instead, we propose a different weighting function that simultaneously up-weights hard tokens and down-weights easy tokens.










§ RNN WITH PRE-ATTENTION AND HIGHWAY CONNECTION
 


In this section, we introduce some architectural modifications that alleviate the repetition problem.



 §.§ Notation and response generation



We write x = (x_1, x_2, …, x_|x|), y = (y_1, y_2, …, y_|y|) for message and response utterances, respectively.
Given x, the objective of a chatbot model parameterized by θ is to assign a higher conditional probability p(y|x; θ) to the ground truth response y than to other responses, which usually follows a sequential decomposition:

    p(y|x; θ) = ∏_t=1^|y| p(y_t|y_<t, x; θ).

Here, y_<t = (y_1, y_2, …, y_t-1), and p(y_t|y_<t, x; θ) represents the model-predicted probability for token y_t.

For the sake of clarity, we further denote
p_t = p(y_t|y_<t, x; θ).
Training the above chatbot model is usually achieved by minimizing the CE loss:


    (p_t) = -log(p_t).


For estimating p_t using an RNN model, we largely follow the notation in <cit.> by denoting the decoder output and attention output at decoding step t as s_t and c_t, respectively.
Readers are referred to <cit.> for more details.



 §.§ Pre-attention


Inspired by IF attention <cit.>, we find that pre-attention performs better w.r.t. repetition rates.
Figure <ref> is an illustration of the difference between pre-attention and IF attention.



Formally, the input of RNN using IF attention at each decoding step t, i.e., y_t-1, is modified:

    ỹ'_t-1 = [y_t-1; s̃_t-1],

with [· ; ·] denoting vector concatenation.
Besides, at step t the decoder output s_t is also transformed with attention output c_t:

    s̃_t = g'(c_t, s_t).

The g'(·) function used in IF attention is a vector concatenation followed by a linear projection, to resize s̃_t to the dimension of s_t.

We find that only transforming the input y_t-1 is more helpful for reducing repetition, which leads us to pre-attention:

    ỹ_t-1 = g(c_t-1, y_t-1),

with g(·) being a transformation function, and ỹ_t-1 has the same dimensionality as y_t-1.
The g(·) function can take the form of g'(·), but in this paper, we observe that by using a highway connection for g(·), the repetition rate can be further reduced.
The highway connection is formulated as follows:

    ỹ_t-1   = Highway(c_t-1, y_t-1) 
       = z ∘ c_t-1 + (1 - z) ∘ y_t-1,

where ∘ represents element-wise product; z is a learnable gating vector, through which the model learns how to effectively combine attentional information c_t-1 and input information y_t-1.
Readers are referred to <cit.> for details of the highway connection.



 §.§ Discussion



The attention mechanism does not have internal states to keep track of attentions previously paid, therefore when using post-attention, similar attentions may be paid repeatedly, increasing the chance of repetition.
On the other hand, when using pre-attention as part of the input to RNN, it enables RNN states to keep track of previous attention status.
Furthermore, by using highway connections, the model learns to more effectively combine context and step-wise input than simple concatenation and linear projection.

However, noticing that the transformer model <cit.> already has an analogous architecture as our pre-attention, we tried to apply highway connections to the transformer attention outputs, but have not yet been successful with reducing repetitive generations in this manner.
We suspect that this is due to the lack of temporal inner states in transformers, and more work needs to be done to mitigate this drawback in order to reduce repetition.
Meanwhile, we believe there are more fundamental reasons for the repetition problem.










§ TOKEN LOSS DYNAMIC REWEIGHTING










Inspired by <cit.>, we hypothesize that the difficulty of some training examples is the main reason for repetition.
Based on this hypothesis, example reweighting methods such as FL <cit.> can be used to alleviate the repetition problem.
In this section, we first adapt FL to our chit-chat task, and later we present a more effective weighting method than FL.



 §.§ Example-level loss reweighting

Using the notation introduced in <ref> and writing p^y for p(y|x; θ), a direct application of FL to the response generation task can be formulated as follows:

    (p^y) = w · L(y).

Here, w = (1 - p^y)^γ is the weighting coefficient, which is scaled by an exponential focusing parameter γ≥ 0, with (1 - p^y) reflecting the training difficulty of an example, that is, a pair (x, y).
L(y) denotes the loss of predicting y that is usually calculated as an average over the CE losses of all tokens that make up y:

    L(y) = 1/|y|∑_t=1^|y|(p_t).

Since p_t ∈ [0, 1], in practice p^y is usually extremely small according to the sequential decomposition Eq. (<ref>), turning almost all training examples into hard examples.
Therefore, we propose to approximate the training difficulty of (x, y) w.r.t. the average of p_t as follows:


    p̃^y = 1 - 1/|y|∑_t=1^|y| p_t,


which is more suitable for our task.

However, there is another unwanted feature of FL.
Although w in Eq. (<ref>) is higher for hard examples than for easy ones, to make the model focus on examples that are hard to train, which is essentially what the method is named after, this method is likely to slow down the training as w is always smaller than 1.
To address this, we propose to use a different weighting function:

    (p̃^y) = cos(p̃^y * π) + 1,

which projects all probabilities to the weight domain of [0, 2].
With p̃^y ≤ 0.5 indicating a hard example, which results in (p̃^y) > 1, we up-weight the loss of hard examples, and otherwise we down-weight easy examples with p̃^y > 0.5.
To distinguish our method (using (<ref>)) from the adapted FL, we refer to it method as LDRLDR:

    (p^y) = (p̃^y) · L(y).




 §.§ Token-level loss reweighting


Noticing from Eq. (<ref>) that in our task, CE losses are calculated for each token, we can alternatively infer the difficulty at the token-level and apply FL or LDR methods thereafter.
To achieve this, we use p_t instead of p^y for Eq. (<ref>), resulting in TFL:

    (p_t) = w_t ·(p_t),

where w_t = (1 - p_t)^γ.
Using p_t instead of p̃^y and p^y for Eq. (<ref>) results in TLDR:

    (p_t) = (p_t) ·(p_t).


Although treating each token as an example is somewhat counterintuitive, it has been shown in <cit.> that such a token-level formulation can have a more direct impact on the generation of each token.
We confirm in <ref> that TLDR results in less repetition than LDR.



 §.§ Gradient analysis


We now compare the gradients of our loss function TLDR to those of CE and TFL w.r.t. p_t.
The gradient function of CE is:

    ∂(p_t)/∂ p_t = -1/p_t.

The gradient function of TFL is:

    ∂(p_t)/∂ p_t = 
       γ (1 - p_t)^γ - 1·log(p_t) - (1 - p_t)^γ/p_t.

Finally, the gradient function of TLDR is:

    ∂(p_t)/∂ p_t = 
       -πsin(p_t ·π) log(p_t) - cos(p_t ·π) + 1/p_t.




To see the differences between the three loss functions more clearly, we visualize their curves in Figure <ref>.
Compared to CE and TFL, the gradients for easy tokens derived by TLDR are suppressed and gradients for hard tokens are amplified.










§ EXPERIMENTAL SETUP



For our experiments, we use 3 multi-turn chit-chat datasets: Dailydialog <cit.>, Cornell Movie Dialogs <cit.> and Ubuntu Dialogue <cit.>, in increasing order of sizes.
As a quantitative measure of system generated responses, we use BLEU-4 scores <cit.>.
For repetition measurement, unlike existing work utilizing repeated n-gram counts <cit.> or n-gram diversity <cit.> that reflect only one order of n-gram repetition (usually 4-gram), we propose to use a DIMEN, which is inspired by BLEU by weighted-averaging the n-gram diversities at multiple granularities, thus being more comprehensive.
To introduce DIMEN, we first introduce the definition of n-gram diversity proposed by <cit.>:

    (text, k) = 
         |{k(text) }|/max(|1(text)| - k, 1).

Here, k(text) means getting the n-gram list of text for n=k;
{·} maps a list to a set, and |·| denotes the cardinality of a set/list, respectively.
For text with less than k unigrams, we clip the denominator to be 1 such that (text, k) = 1.
With n-gram diversity defined, the DIMEN score of a given text is calculated as

    (text, n) =∑_k=1^n α_k ·(text, k),

where n is the highest order of n-grams and α_k is the weight coefficient corresponding to k with ∑_1^n α_k = 1.

The text argument can be either a response utterance, in which case we refer to the score as , or a list of response utterances for a whole validation set, where we refer to as .
 and  scores are compensating.
With (·) ∈ [0, 1], a high  score represents a non-repetitive utterance, and a high  score represents a diverse list of utterances.

To further measure the repetition performance on a validation set, we can consider averaging the  scores of each utterance.
However, since the majority of system generated utterances are non-repetitive, the averaged  score can be very close to 1 and thus indistinguishable from model to model.
Instead, we first bin the  scores into m buckets to create a histogram vector ∈ℤ^m, and then calculate the weighted L2-norm 
of the count of each bucket to emphasize the overall repetition performance of a model.
Formally,

    () = √(∑_i = 1^mβ_i · (_i)^2),

where β∈ℝ^m is a weighting vector.
When we assign higher weights to bins with lower  scores, the  score emphasizes highly repetitive utterances.

With all the datasets and metrics introduced, we seek to answer the following research questions through our experiments:



  * Are pre-attention and highway connection helpful for reducing repetition of RNN Seq2Seq?

  * Is our hypothesis on hard tokens correct?

  * How effectively can LDR and TLDR reduce repetition for both RNN and transformer models?

  * Is the improvement of LDR and TLDR merely due to up-weighting?


To answer RQ1, we use an RNN Seq2Seq with post-attention as the baseline.
To answer RQ2, we obtain results of using TFL on both RNN and transformer models.
For answering RQ3, we compare the results of LDR and TLDR to those of baseline models trained without weighting and with TFL weighting.
For RQ4, we also add a baseline using uniform weights, with w = 2 for all training examples.

We conduct our experiments on the ParlAI framework <cit.>.
Below we introduce the model parameters.
Unless stated otherwise, we use the same settings for different models on each dataset whenever possible.
For RNN Seq2Seq, we use 2-layer LSTMs <cit.> with a hidden size of 512 and separate embedding matrices with an embedding size of 200 for both the encoder and the decoder.
We use the `general' attention variant from <cit.>.
For transformer Seq2Seq, we use 6-layer transformer blocks with 8 attention heads <cit.> and an embedding size of 256 with separate matrices for both the encoder and the decoder.
We use 800 as the feed-forward layer size with `relu' activation.
We train both RNN and transformer Seq2Seq models using Adam optimizer <cit.> with fixed learning rate of 0.001 and β_1 = 0.9, β_2 = 0.999.
The gradients of RNN are clipped with L2-norm <cit.>, no larger than 5 during training, while for transformers we clip to 1.
For all the intermediate outputs of both models, we use dropout rate of 0.1 <cit.>.

We train the models on the Dailydialog and Movie Dialogs datasets for 100 epochs and check the performances on the validation sets every 0.5 epochs.
On the Ubuntu Dialogue dataset, we train the models for 30 epochs with a validation interval of 0.1 epochs.
We save checkpoints after every validation and select the best checkpoint according to the lowest repetition rate.
Since the repetition rate can sometimes be very low, this checkpoint-saving strategy may fail occasionally, in which case we use the last checkpoint during training.
For all three datasets, we tokenize the utterances using the NLTK <cit.> tokenizer and keep the most frequent 30,000 tokens according to their training sets, respectively.
We use up to 3 turns of history as the input message, and truncate the message to 128 tokens and response to 32 tokens.
A batch size of 256 is used.

For FL, we use a focusing factor of γ = 2 as recommended in <cit.>.
For the hyper-parameters introduced in this work, we use n = 4 for calculating the DIMEN scores, with α = [0.25, 0.25, 0.25, 0.25].
We group the  scores into m = 10 bins, with the weighting vector β = [0.9, 0.8, …, 0.0].










§ RESULTS





The results of all models on the three datasets are shown in Table <ref>.
Since the validation repetition ( scores) of models can be very low in the early stage of training when the  score is still very low, we mainly use  to make sure that the results are shown for checkpoints at similar stages.

Table <ref> is divided into three main blocks row-wise.
The first block ranging from rows (a)–(c) is for answering RQ1.
The second (rows (d)–(g)) and third (rows (h)–(l)) blocks are for answering RQ2 and RQ3, with RNN and transformer models, respectively.
For reference, we also include the Human performance calculated using the ground truth of each dataset in the bottom row of Table <ref>.



Answer to RQ1:
Row (a) of Table <ref> are the results for RNN models with post-attention.
From row (b), we can see that pre-attention is effective in reducing repetition on all three datasets.
On top of pre-attention, row (c) shows that highway connections are helpful with repetition reduction on two datasets, namely Dailydialog and Movie Dialogs.
BLEU scores show that the quality of the utterances generated by our models (b) and (c) are on a par with those generated by the baseline (a).
These results indicate that proper architectural modifications can indeed reduce repetition, and more work needs to be done to find such modifications for transformers.

Answer to RQ2:
From rows (e) and (j), we can see that in most cases  helps to reduce repetition, compared to rows (b) and (h), respectively.
This empirically shows that our hypothesis on hard tokens is correct.
However, due to monotonically down-weighting all training examples, the improvement brought by  is limited.


Answer to RQ3:
Next, we proceed to our proposed weighting methods.
Giving that the LDR and TLDR methods are about model learning, thus should be architecture insensitive, we use the RNN variant (b) for the following experiments, as (c) needs more computation due to the utilization of a gating (<ref>).
From the second and third blocks of Table <ref>, we can see that both RNN (row (g)) and transformer (row (l)) models trained using TLDR are consistently better than their corresponding baselines without weighting or with TFL weighting.
Though TLDR occasionally performs worse than TFL, e.g., row (g) vs row (e) on the Movie Dialogs dataset, we note that this is probably due to sample bias of the test sets.
We plot the result curves for the Movie Dialogs validation set at each validation point in Figure <ref>, where we can see that TLDR is no worse, if not consistently better, than the baselines.
These comparisons indicate that dynamically weighting tokens according to their difficulty is an effective solution to reducing repetition.

The results in rows (d) and (i) show that, on the smaller datasets Dailydialog and Movie Dialogs, LDR has a limited effect on reducing repetition.
However, on the large Ubuntu Dialogue dataset, LDR hurts the repetition performance.
Besides, LDR consistently results in lower BLEU scores, which implies that more work needs to be done to estimate the training difficulty at the example level.



Answer to RQ4:
To understand whether the improvement of TLDR is brought by up-weighting, let us compare the performance of models trained using uniform weight w = 2 at rows (f) and (k) to rows (g) and (l), respectively.
We can see that w.r.t. repetition, TLDR wins the majority of the times, which supports our claim that TLDR has more effect than simply up-weighting, as can also be seen from the gradient visualization in Figure <ref>.
Multiplying a constant factor (w = 2 in our experiments) to the loss function also multiplies the gradients with the same factor, while TLDR changes the gradient function in a more complex way so that the gradients resulting from hard examples are amplified while those resulting from easy examples are suppressed, which is important for reducing repetition.
It is also worth noting that using a uniform weight of 2 is effectively doubling the base learning rate, which seems to be helpful with improving BLEU and .














§ CONCLUSION AND DISCUSSION



We have studied the repetition problem of encoder-decoder architectures, using both RNN and transformer models.
We have discovered that by using pre-attention instead of post-attention for RNN, the repetition problem can be alleviated.
Together with highway connections, repetitive generations can be further reduced.


We have hypothesized that the repetition problem is caused by hard tokens and find empirical support for this claim using FL.
We then propose a more effective weighting function than FL, namely TLDR.
With a differentiable cosine weighting function, TLDR amplifies the gradients resulting from hard tokens while it suppresses those from easy tokens.
Through experiments we show that TLDR outperforms strong baselines like TFL and uniform weighting.

Our hard token hypothesis and the TLDR weighting function both operate mainly on target side tokens, while hard source side tokens can also have a detrimental effect on the decoder generations.
Future work can be done by applying TLDR to source side language representation learning, possibly by training the encoder independently on a language representation task.
It might also be worth exploring the effect on decoder-only models on tasks like LM.

We also observe that in our experiments, before the repetition rate converges during training, the repetition rate on the Ubuntu Dialogues validation set fluctuates with a regular pattern, which suggests that there can be certain training examples that can harm the repetition rate.
If this is true, then the work by <cit.> can be used to target such harmful examples and hence reduce repetition.
We leave this for future work.









§ ACKNOWLEDGMENTS


This research was supported by the China Scholarship Council, Ahold Delhaize, the Association of Universities in the Netherlands (VSNU), and the Innovation Center for Artificial Intelligence (ICAI).
All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

acl_natbib




