abstract: Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation. Arguably, one of the major factors behind this success has been the availability of high quality parallel corpora. In this work, we investigate how to leverage abundant monolingual corpora for neural machine translation. Compared to a phrase-based and hierarchical baseline, we obtain up to $1.96$ BLEU improvement on the low-resource language pair Turkish-English, and $1.59$ BLEU on the focused domain task of Chinese-English chat messages. While our method was initially targeted toward such tasks with less parallel data, we show that it also extends to high resource languages such as Cs-En and De-En where we obtain an improvement of $0.39$ and $0.47$ BLEU scores over the neural machine translation baselines, respectively.

# Introduction

*Neural machine translation* (NMT) is a novel approach to machine translation that has shown promising results . Until recently, the application of neural networks to machine translation was restricted to extending standard machine translation tools for rescoring translation hypotheses or re-ranking n-best lists (see, e.g., . In contrast, it has been shown that, it is possible to build a competitive translation system for English-French and English-German using an end-to-end neural network architecture  (also see Sec. ).

Arguably, a large part of the recent success of these methods has been due to the availability of large amounts of high quality, sentence aligned corpora. In the case of low resource language pairs or in a task with heavy domain restrictions, there can be a lack of such sentence aligned corpora. In contrast, monolingual corpora is almost always universally available. Despite being "unlabeled", monolingual corpora still exhibit rich linguistic structure that may be useful for translation tasks. This presents an opportunity to leverage such corpora to give hints to an NMT system.

In this work, we present a way to effectively integrate a language model (LM) trained only on monolingual data (target language) into an NMT system. We provide experimental results that incorporating monolingual corpora can improve a translation system on a low-resource language pair (Turkish-English) and a domain restricted translation problem (Chinese-English SMS chat). In addition, we show that these methods improve the performance on the relatively high-resource German-English (De-En) and Czech-English (Cs-En) translation tasks.

In the following section (Sec. ), we review recent work in neural machine translation. We present our basic model architecture in Sec.   and describe our shallow and deep fusion approaches in Sec. . Next, we describe our datasets in Sec. . Finally, we describe our main experimental results in Sec. .

# Background: Neural Machine Translation

Statistical machine translation (SMT) systems maximize the conditional probability $p(\vy \mid \vx)$ of a correct target translation $\vy$ given a source sentence $\vx$. This is done by maximizing separately a language model $p(\vy)$ and the (inverse) translation model $p(\vx \mid \vy)$ component by using Bayes' rule: $$\begin{aligned}
    p(\vy \mid \vx) \propto p(\vx \mid \vy) p(\vy).
\end{aligned}$$

This decomposition into a language model and translation model is meant to make full use of available corpora: monolingual corpora for fitting the language model and parallel corpora for the translation model. In reality, however, SMT systems tend to model $\log p(\vy\mid\vx)$ directly by linearly combining multiple features by using a so-called log-linear model: $$\begin{aligned}
    \label{eq:trad_smt}
    \log p(\vy \mid \vx) = \sum_{j} f_j(\vx, \vy) + C,
\end{aligned}$$ where $f_j$ is the $j$-th feature based on both or either of the source and target sentences, and $C$ is a normalization constant which is often ignored. These features include, for instance, pair-wise statistics between two sentences/phrases. The log-linear model is fitted to data, in most cases, by maximizing an automatic evaluation metric other than an actual conditional probability, such as BLEU.

Neural machine translation, on the other hand, aims at directly optimizing $\log p(\vy \mid \vx)$ including the feature extraction as well as the normalization constant by a single neural network. This is typically done under the encoder-decoder framework  consisting of neural networks. The first network encodes the source sentence $\vx$ into a continuous-space representation from which the decoder produces the target translation sentence. By using RNN architectures equipped to learn long term dependencies such as Gated Recurrent Units (GRU) or Long Short-Term Memory (LSTM), the whole system can be trained end-to-end .

Once the model learns the conditional distribution or translation model, given a source sentence we can find a translation that approximately maximizes the conditional probability using, for instance, a beam search algorithm.

# Model Description

We use the model recently proposed by that learns to jointly (soft-)align and translate as the baseline neural machine translation system in this paper. Here we describe in detail this model to which we refer as "NMT".

The encoder of the NMT is a bidirectional RNN which consists of forward and backward RNNs . The forward RNN reads the input sequence/sentence $\vx=(x_1, \dots, x_T)$ in a forward direction, resulting in a sequence of hidden states $(\ora{\vh}_1, \dots, \ora{\vh}_T)$. The backward RNN reads $\vx$ in an opposite direction and outputs $(\ola{\vh}_1, \dots,
\ola{\vh}_T)$. We concatenate a pair of hidden states at each time step to build a sequence of *annotation* vectors $(\vh_1, \dots, \vh_T)$, where $$\vh_j^\top = \left[
            \ola{\vh}_j^\top ; \ora{\vh}_j^\top
    \right].$$ Each annotation vector $\vh_j$ encodes information about the $j$-th word with respect to all the other surrounding words in the sentence.

In our decoder, which we construct with a single layer RNN, at each timestep $t$ a soft-alignment mechanism first decides on which annotation vectors are most relevant. The relevance weight $\alpha_{tj}$ of the $j$-th annotation vector for the $t$-th target word is computed by a feedforward neural network $\f$ that takes as input $\vh_j$, the previous decoder's hidden state $\vs_{t-1}$ and the previous output $\vy_{t-1}$: $$\begin{aligned}
    e_{tj} = \f(\vs_{t-1}, \vh_j, \vy_{t-1}).
\end{aligned}$$ The outputs $e_{tj}$ are normalized over the sequence of the annotation vectors so that the they sum to $1$: $$\label{eq:dec:alpha}
    \alpha_{tj} = \frac{\text{exp}(e_{tj})}{\sum_{k=1}^T\text{exp}(e_{tk})},$$ and we call $\alpha_{tj}$ a relevance score, or an alignment weight, of the $j$-th annotation vector.

The relevance scores are used to get the *context vector* $\vc_t$ of the $t$-th word in the translation: $$\begin{aligned}
    \vc_t = \sum_{j=1}^T \alpha_{tj} \vh_j ~,
\end{aligned}$$

Then, the decoder's hidden state $\vs_t^{\text{TM}}$ at time $t$ is computed based on the previous hidden state $\vs_{t-1}^{\text{TM}}$, the context vector $\vc_t$ and the previously translated word $\vy_{t-1}$: $$\label{eq:dec:hidden}
    \vs_t^{\text{TM}} = \f_r(\vs_{t-1}^{\text{TM}}, \vy_{t-1}, \vc_t),$$ where $\f_r$ is the gated recurrent unit .

We use a deep output layer  to compute the conditional distribution over words: $$\begin{split}
    \label{eq:dec:output}
    p(&\vy_t | \vy_{<t}, \vx) \propto \\
    & \exp(\vy_t^{\top}(\mW_o
    \f_o(\vs_t^{\text{TM}}, \vy_{t-1}, \vc_t) + \vb_o)),
    \end{split}$$ where $\vy_t$ is a one-hot encoded vector indicating one of the words in the target vocabulary. $\mW_o$ is a learned weight matrix and $\vb_o$ is a bias. $\f_o$ is a single-layer feedforward neural network with a two-way maxout non-linearity .

The whole model, including both the encoder and decoder, is jointly trained to maximize the (conditional) log-likelihood of the bilingual training corpus: $$\begin{aligned}
    \max_{\TT} \frac{1}{N} \sum_{n=1}^N \log p_{\theta}(\vy^{(n)} | \vx^{(n)}),
\end{aligned}$$ where the training corpus is a set of $(\vx^{(n)}, \vy^{(n)})$'s, and $\TT$ denotes a set of all the tunable parameters.

# Integrating Language Model into the Decoder

In this paper, we propose two alternatives to integrating a language model into a neural machine translation system which we refer as *shallow fusion* (Sec. ) and *deep fusion* (Sec. ). Without loss of generality, we use a language model based on recurrent neural networks (RNNLM, ) which is equivalent to the decoder described in the previous section except that it is not biased by a context vector (i.e., $\vc_t = 0$ in Eqs. –).

In the sections that follow, we assume that both an NMT model (on parallel corpora) as well as a recurrent neural network language model (RNNLM, on larger monolingual corpora) have been pre-trained separately before being integrated. We denoted the hidden state at time $t$ of the RNNLM with $\vs_t^{\text{LM}}$.

## Shallow Fusion

Shallow fusion is analogous to how language models are used in the decoder of a usual SMT system . At each time step, the translation model proposes a set of candidate words. The candidates are then scored according to the weighted sum of the scores given by the translation model and the language model.

More specifically, at each time step $t$, the translation model (in this case, the NMT) computes the score of every possible next word for each hypothesis all of hypotheses $\left\{ \vy_{\leq t-1}^{(i)}\right\}$. Each score is the summation of the score of the hypothesis and the score given by the NMT to the next word. All these new hypotheses (a hypothesis from the previous timestep with a next word appended at the end) are then sorted according to their respective scores, and the top $K$ ones are selected as candidates $\left\{ \hat{\vy}^{(i)}_{\leq t}
\right\}_{i=1,\dots,K}$.

We then rescore these hypotheses with the weighted sum of the scores by the NMT and RNNLM, where we only need to recompute the score of the "new word" at the end of each candidate hypothesis. The score of the new word is computed by $$\label{eq:dec:output_shallowfusion}
    \begin{split}
        \log p (\vy_t=k)& = \log p_{\text{TM}} (\vy_t = k) \\
        & +
        \beta \log p_{\text{LM}} (\vy_t = k),
    \end{split}$$ where $\beta$ is a hyper-parameter that needs to be tuned to maximize the translation performance on a development set.

See Fig.  (a) for illustration.

## Deep Fusion

In deep fusion, we integrate the RNNLM and the decoder of the NMT by concatenating their hidden states next to each other (see Fig.  (b)). The model is then finetuned to use the hidden states from both of these models when computing the output probability of the next word (see Eq. ). Unlike the vanilla NMT (without any language model component), the hidden layer of the deep output takes as input the hidden state of the RNNLM in addition to that of the NMT, the previous word and the context such that

$$\begin{split}
    \label{eq:dec:output_deepfusion}
    p(&\vy_t | \vy_{<t}, \vx) \propto \\
    &\exp(\vy_t^{\top}(\mW_o
    \f_o(\vs_t^{\text{LM}}, \vs_t^{\text{TM}}, \vy_{t-1}, \vc_t) + \vb_o)),
    \end{split}$$

where again we use the superscripts $^{\text{LM}}$ and $^{\text{TM}}$ to denote the hidden states of the RNNLM and NMT respectively.

During the finetuning of the model, we tune only the parameters that were used to parameterize the output . This is to ensure that the structure learned by the LM from monolingual corpora is not overwritten. It is possible to use monolingual corpora as well while finetuning all the parameters, but in this paper, we alter only the output parameters in the stage of finetuning.

### Balancing the LM and TM

In order for the decoder to flexibly balance the input from the LM and TM, we augment the decoder with a "controller" mechanism. The need to flexibly balance the signals arises depending on the work being translated. For instance, in the case of Zh-En, there are no Chinese words that correspond to articles in English, in which case the LM may be more informative. On the other hand, if a noun is to be translated, it may be better to ignore any signal from the LM, as it may prevent the decoder from choosing the correct translation. Intuitively, this mechanism helps the model dynamically weight the different models depending on the word being translated.

The controller mechanism is implemented as a function taking the hidden state of the LM as input and computing $$\begin{aligned}
    \label{eq:controller}
    g_t = \sigma\left( \vv^{\top}_g \vs^{\text{LM}}_{t} + b_g \right),
\end{aligned}$$ where $\sigma$ is a logistic sigmoid function. $\vv_g$ and $b_g$ are learned parameters.

The output of the controller is then multiplied with the hidden state of the LM. This lets the decoder use the signal from the TM fully, while the controller controls the magnitude of the LM signal.

In our experiments, we empirically found that it was better to initialize the bias $\vb_g$ to a small, negative number. This allows the decoder to decide the importance of the LM only when it is deemed necessary. [^1]

# Datasets

We evaluate the proposed approaches on four diverse tasks: Chinese to English (Zh-En), Turkish to English (Tr-En), German to English (De-En) and Czech to English (Cs-En). We describe each of these datasets in more detail below.

## Parallel Corpora

### Zh-En: OpenMT'15

We use the parallel corpora made available as a part of the NIST OpenMT'15 Challenge. Sentence-aligned pairs from three domains are combined to form a training set: (1) SMS/CHAT and (2) conversational telephone speech (CTS) from DARPA BOLT Project, and (3) newsgroups/weblogs from DARPA GALE Project. In total, the training set consists of 430K sentence pairs (see Table  for the detailed statistics). We train models with this training set and the development set (the concatenation of the provided development and tune sets from the challenge), and evaluate them on the test set. The domain of the development and test sets is restricted to CTS.

```latex
\begin{table*}[t]\centering

    \begin{minipage}{0.48\textwidth}
        \centering
    \begin{tabular}{c | c c}
        & Chinese & English \\
        \hline
        \hline
        \# of Sentences & \multicolumn{2}{c}{436K} \\
        \hline
        \# of Unique Words & 21K & 150K \\
        \hline
        \# of Total Words & 8.4M & 5.9M \\
        \hline
        Avg. Length & 19.3 & 13.5 \\
    \end{tabular}

    (a) Zh-En
    \end{minipage}
    \hspace{2mm}
    \begin{minipage}{0.48\textwidth}
        \centering
    \begin{tabular}{c | c c}
        & Turkish & English \\
        \hline
        \hline
        \# of Sentences & \multicolumn{2}{c}{160K} \\
        \hline
        \# of Unique Words & 96K$^\star$ & 95K \\
        \hline
        \# of Total Words & 11.4M$^\star$ & 8.1M \\
        \hline
        Avg. Length & 31.6 & 22.6 \\
    \end{tabular}

    (b) Tr-En
    \end{minipage}
    \vspace{3mm}
    \begin{minipage}{0.48\textwidth}
        \centering
    \begin{tabular}{c | c c}
        & Czech & English \\
        \hline
        \hline
        \# of Sentences & \multicolumn{2}{c}{12.1M} \\
        \hline
        \# of Unique Words & 1.5M & 911K \\
        \hline
        \# of Total Words & 151M & 172M \\
        \hline
        Avg. Length & 12.5 & 14.2 \\
    \end{tabular}

    (c) Cs-En
    \end{minipage}
    \hspace{1mm}
    \begin{minipage}{0.48\textwidth}
        \centering
    \begin{tabular}{c | c c}
        & German &English \\
        \hline
        \hline
        \# of Sentences & \multicolumn{2}{c}{4.1M} \\
        \hline
        \# of Unique Words & 1.16M$^\dagger$ & 742K \\
        \hline
        \# of Total Words & 11.4M$^\dagger$ & 8.1M \\
        \hline
        Avg. Length & 24.2 & 25.1 \\
    \end{tabular}
    (d) De-En
    \end{minipage}
    \caption{Statistics of the Parallel Corpora. $\star$: After segmentation,
    $\dagger$: After compound splitting.}
\label{table:stats}
\end{table*}
```

#### Preprocessing

Importantly, we did "not segment" the Chinese sentences and considered each character as a symbol, unlike other approaches which use a separate segmentation tool to segment the Chinese characters into words . Any consecutive non-Chinese characters such as Latin alphabets were, however, considered as an individual word. Lastly, we removed any HTML/XML tags from the corpus, chose only the intended meaning word if both intended and literal translations are available, and ignored any indicator of, e.g., typos. The only preprocessing we did on the English side of the corpus was a simple tokenization using the tokenizer from Moses. [^2].

### Tr-En: IWSLT'14

We used the WIT parallel corpus  and SETimes parallel corpus made available as a part of IWSLT'14 (machine translation track). The corpus consists of the sentence-aligned subtitles of TED and TEDx talks, and we concatenated dev2010 and tst2010 to form a development set, and tst2011, tst2012, tst2013 and tst2014 to form a test set. See Table  for the detailed statistics of the parallel corpora.

#### Preprocessing

As done with the case of Zh-En, initially we removed all special symbols from the corpora and tokenized the Turkish side with the tokenizer provided by Moses. To overcome the exploding vocabulary due to the rich inflections and derivations in Turkish, we segmented each Turkish sentence into a sequence of sub-word units using Zemberek[^3] followed by morphological disambiguation on the morphological analysis . We removed any non-surface morphemes corresponding to, for instance, part-of-speech tags.

## Cs-En and De-En: WMT'15

For the training of our models, we used all the available training data provided for Cs-En and De-En in the WMT'15 competition. We used newstest2013 as a development set and newstest2014 for a test set. The detailed statistics of the parallel corpora is provided in Table .

#### Preprocessing

We tokenized the datasets with Moses tokenizer first. Sentences longer than eighty words and those that have large mismatch between lengths of the source and target sentences were removed from the training set. Then, we filtered the training data by removing sentence pairs in which one sentence (or both) was written in the wrong language by using a language detection toolkit , unless the sentence had 5 words or less. For De-En, we also split the compounds in the German side by using Moses. Finally we shuffled the training corpora seven times and concatenated its outputs.

## Monolingual Corpora

The English Gigaword corpus by the Linguistic Data Consortium (LDC), which mainly consists of newswire documents, was allowed in both OpenMT'15 and IWSLT-15 challenges for language modelling. We used the tokenized Gigaword corpus without any other preprocessing step to train three different RNNLM's to fuse into NMT for Zh-En, Tr-En and the WMT'15 translation tasks (De-En and Cs-En.)

# Settings

## Training Procedure

### Neural Machine Translation

The input and output of the network were sequences of one-hot vectors whose dimensionality correspond to the sizes of the source and target vocabularies, respectively. We constructed the vocabularies with the most common words in the parallel corpora. The sizes of the vocabularies for Chinese, Turkish and English were 10K, 30K and 40K, respectively, for the Tr-En and Zh-En tasks. Each word was projected into the continuous space of $620$-dimensional Euclidean space first to reduce the dimensionality, on both the encoder and the decoder. We chose the size of the recurrent units for Zh-En and Tr-En to be $1,200$ and $1,000$ respectively.

In Cs-En and De-En experiments, we were able to use larger vocabularies. We trained our NMT model for Cs-En and De-En with large vocabularies using the importance sampling based technique introduced in and with this technique we were able to use large vocabulary of size $200k$.

Each model was optimized using Adadelta  with minibatches of $80$ sentence pairs. At each update, we normalized the gradient such that if the $L_2$ norm of the gradient exceeds $5$, gradient is renormalized back to $5$ . For the non-recurrent layers (see Eq. ), we used dropout and additive Gaussian noise (mean $0$ and std. dev. $0.001$) on each parameter to prevent overfitting . Training was early-stopped to maximize the performance on the development set measured by BLEU. [^4] We initialized all recurrent weight matrices as random orthonormal matrices.

### Language Model

We trained three RNNLM's with $2,400$ long short-term memory (LSTM) units on English Gigaword Corpus using respectively the vocabularies constructed separately from the English sides of Zh-En and Tr-En corpora. The third language model was trained using $2,000$ LSTM units on the English Gigaword Corpus again but with a vocabulary constructed from the intersection the English sides of Cs-En and De-En. The parameters of the former two language models were optimized using RMSProp , and Adam optimizer  was used for the latter one. Any sentence with more than ten percent of its words out of vocabulary was discarded from the training set. We did early-stopping using the perplexity of development set.

## Shallow and Deep Fusion

### Shallow Fusion

The hyperparameter $\beta$ in Eq.  was selected to maximize the translation performance on the development set, from the range $0.001$ and $0.1$. In preliminary experiments, we found it important to renormalize the softmax of the LM without the end-of-sequence and out-of-vocabulary symbols ($\beta=0$ in Eq.  ). This may be due to the difference in the domains of TM and LM.

### Deep Fusion

We finetuned the parameters of the deep output layer (Eq. ) as well as the controller (see Eq.  using the Adam optimizer for Zh-En, and RMSProp with momentum for Tr-En. During the finetuning, the dropout probability and the standard deviation of the weight noise were set to $0.56$ and $0.005$, respectively. Based on our preliminary experiments, we reduced the level of regularization after the first $10K$ updates. In Cs-En and De-En tasks with large vocabularies, the model parameters were finetuned using Adadelta while scaling down the magnitude of the update steps by $0.01$.

### Handling Rare Words

On the De-En and Cs-En translation tasks, we replaced the unknown words generated by the NMT with the words the NMT assigned to which the highest score in the source sentence (Eq. ). We copied the selected source word in the place of the corresponding unknown token in the target sentence. This method is similar to the technique proposed by for addressing rare words. But instead of relying on an external alignment tool, we used the attention mechanism of the NMT model to extract alignments. This method consistently improved the results by approximately $1.0$ BLEU score.

# Results and Analysis

## Zh-En: OpenMT'15

In addition to NMT-based systems, we also trained a phrase-based as well as hierarchical phrase-based SMT systems with/without re-scoring by an external neural language model (CSLM) . We present the results in Table .

We observed that integrating an additional LM by deep fusion (see Sec. ) helped the models achieving better performance in general, except in the case of the CTS task. We noticed that the NMT-based models, regardless of whether the LM was integrated or not, outperformed the more traditional phrase-based SMT systems.

|         |           |           |           |           |
|:--------|:---------:|:---------:|:---------:|:---------:|
|         | SMS/CHAT  |           |    CTS    |           |
|         |    Dev    |   Test    |    Dev    |   Test    |
| PB      |   15.5    |   14.73   |   21.94   |   21.68   |
| \+ CSLM |   16.02   |   15.25   |   23.05   |   22.79   |
| HPB     |   15.33   |   14.71   |   21.45   |   21.43   |
| \+ CSLM |   15.93   |   15.8    |   22.61   |   22.17   |
| NMT     |   17.32   |   17.36   |   23.4    | **23.59** |
| Shallow |   16.59   |   16.42   |   22.7    |   22.83   |
| Deep    | **17.58** | **17.64** | **23.78** |   23.5    |

Results on the task of Zh-En. PB and HPB stand for the phrase-based and hierarchical phrase-based SMT systems, respectively.

```latex
\begin{table*}[t]\centering
\begin{tabular}{l|| cc |cccc}
& \multicolumn{2}{c|}{Development Set} & \multicolumn{4}{c}{Test Set}  \\
 & \textbf{dev2010} & \textbf{tst2010} & \textbf{tst2011} & \textbf{tst2012} & \textbf{tst2013} & \textbf{Test 2014}  \\
 \hline \hline
 \textbf{Previous Best {\scriptsize (Single)}} & 15.33 & 17.14 & 18.77 & 18.62 & 18.88 & - \\
 \textbf{Previous Best {\scriptsize (Combination)}} & - & 17.34 & 18.83 & 18.93 & 18.70 & - \\
 \hline
\textbf{NMT} & 14.50 & 18.01 & 18.40 & 18.77 & 19.86 & 18.64 \\
\textbf{NMT+LM (Shallow)} & 14.44 & 17.99 & 18.48 & 18.80 & 19.87 & 18.66 \\
\textbf{NMT+LM (Deep)} & \textbf{15.69} & \textbf{19.34} & \textbf{20.17} & \textbf{20.23} & \textbf{21.34} & \textbf{20.56} \\
\end{tabular}
\caption{Results on Tr-En. We show for each set separately to make it easier to
compare to previously reported scores.}
\label{tab:tr_en_iwslt2014}
\end{table*}
```

|                |           |           |           |           |
|:---------------|:---------:|:---------:|:---------:|:---------:|
|                |   De-En   |           |   Cs-En   |           |
|                |    Dev    |   Test    |    Dev    |   Test    |
| NMT Baseline   |   25.51   |   23.61   |   21.47   |   21.89   |
| Shallow Fusion |   25.53   |   23.69   |   21.95   |   22.18   |
| Deep Fusion    | **25.88** | **24.00** | **22.49** | **22.36** |

Results for De-En and Cs-En translation tasks on WMT'15 dataset.

## Tr-En: IWSLT'14

In Table , we present our results on Tr-En. Compared to Zh-En, we saw a greater performance improvement up to +1.19 BLEU points from the basic NMT to the NMT integrated with the LM under the proposed method of deep fusion. Furthermore, by incorporating the LM using deep fusion, the NMT systems were able to outperform the best previously reported result by up to $+1.96$ BLEU points on all of the separate test sets.

## Cs-En and De-En: WMT-15

We provide the results of Cs-En and De-En on Table . Shallow fusion achieved $0.09$ and $0.29$ BLEU score improvements respectively on De-En and Cs-En over the baseline NMT model. With deep fusion the improvements of $0.39$ and $0.47$ BLEU score were observed again over the NMT baseline.

## Analysis: Effect of Language Model

The performance improvements we report in this paper reflect a heavy dependency on the degree of similarity between the domain of monolingual corpora and the target domain of translation.

In the case of Zh-En, intuitively, we can tell that the style of writing in both SMS/CHAT as well as the conversational speech will be different from that of news articles (which constitutes the majority of the English Gigaword corpus). Empirically, this is supported by the high perplexity on the development set with our LM (see the column Zh-En of Table ). This explains the marginal improvement we observed in Sec. .

On the other hand, in the case of Tr-En, the similarity between the domains of the monolingual corpus and parallel corpora is higher (see the column Tr-En of Table ). This led to a significantly larger improvement in translation performance by integrating the external language model than the case of Zh-En. Similarly, we observed the improvement by both shallow and deep fusion in the case of De-En and Cs-En, where the perplexity on the development set was much lower.

Unlike shallow fusion, deep fusion allows a model to selectively incorporate the information from the additional LM by the controller mechanism from Sec. . Although this controller mechanism works on per-word basis, it can be expected that if the additional LM models the target domain better, the controller mechanism will be more frequently active on average, i.e., $\E[g_t] \gg 0$. From Table , we can see that, on average, the controller mechanism is most active with De-En and Cs-En, where the additional LM was able to model the target sentences best. This effectively means that deep fusion allows the model to be more robust to the domain mismatch between the TM and LM, thus suggests why deep fusion was more successful than shallow fusion in the experiments.

|             | Zh-En  | Tr-En  | De-En | Cs-En |     |
|:------------|:------:|:------:|:-----:|:-----:|:---:|
| Perplexity  | 223.68 | 163.73 | 78.20 | 78.20 |     |
| Average $g$ |  0.23  |  0.12  | 0.28  | 0.31  |     |
| Std Dev $g$ | 0.0009 |  0.02  | 0.003 | 0.008 |     |

Perplexity of RNNLM's on development sets and the statistics of the controller gating mechanism $g$.

# Conclusion and Future Work

In this paper, we propose and compare two methods for incorporating monolingual corpora into an existing NMT system. We empirically evaluate these approaches (shallow fusion and deep fusion) on low-resource En-Tr (TED/TEDx Subtitles), focused domain for En-Zh (SMS/Chat and conversational speech) and two high-resource language pairs: Cs-En and De-En. We show that with our approach on the Tr-En and Zh-En language pairs, the NMT models trained with deep fusion were able to achieve better results than the existing phrase-based statistical machine translation systems (up to a $+1.96$ BLEU points on En-Tr). We also observed up to a $0.47$ BLEU score improvement for high resource language pairs such as De-En and Cs-En on the datasets provided in WMT'15 competition over our NMT baseline. This provides an evidence that our method can also improve the translation performance regardless of the amount of available parallel corpora.

Our analysis also revealed that the performance improvement from incorporating an external LM was highly dependent on the domain similarity between the monolingual corpus and the target task. In the case where the domain of the bilingual and monolingual corpora were similar (De-En, Cs-En), we observed improvement with both deep and shallow fusion methods. In the case where they were dissimilar (Zh-En), the improvement using shallow fusion were much smaller. This trend might also explain why deep fusion, which implements an adaptive mechanism for modulating information from the integrated LM, works better than shallow fusion. This analysis also suggests that future work on domain adaption of the language model may further improve translations.

[^1]: In all our experiments, we set $\vb_g = -1$ to ensure that $\vg_t$ is initially $0.26$ on average.

[^2]: <https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl>

[^3]: <https://github.com/ahmetaa/zemberek-nlp>

[^4]: We compute the BLEU score using the multi-blue.perl script from Moses on tokenized sentence pairs.


abstract: In this paper, we propose a single end-to-end trainable deep neural network, convolutional gated recurrent mask-rcnn, for tackling the semi-supervised video object segmentation (VOS) task. As mask-rcnn has the ability to simultaneously detect, classify and segment multiple objects, it is naturally applicable for multiple objects detection, id classification and segmentation in a video object segmentation task. To extend the model with temporal inference ability for long sequence video data, we insert a visual memory module into mask-rcnn, namely the convolutional gated recurrent unit (Conv-GRU). The network learns to selectively propagate information for multiple instances simultaneously to predict object segmentation in future. After offline and online training under purely instance segmentation losses, our model is able to achieve good result without any post-processing or synthetic video data augmentation. Experimental results on DAVIS 2016 dataset and DAVIS 2017 dataset have demonstrated the effectiveness of our method for video object segmentation.
author: Ye Lyu
University of Twente
`firstname.lastname@example.com`; George Vosselman
University of Twente
`email@example.com`; Gui-Song Xia
Wuhan University
`firstname.lastname@example.com`; Michael Ying Yang
University of Twente
`email@example.com`
bibliography: egbib.bib
title: LIP: Learning Instance Propagation for Video Object Segmentation by Convolutional Gated Recurrent Mask-RCNN

# Introduction

Video object segmentation aims at segmenting foreground objects from background in a video with coherent object identities. Such visual object tracking task serves for many applications including video analysis and editing, robotics and autonomous cars. Compared to the video object tracking task in bounding box level, this task is more challenging as pixel level segmentation is more detailed description of an object.

The video object segmentation task is defined as a semi-supervised problem if ground truth annotations are given for the first several frames. It is otherwise an unsupervised problem if no annotation is provided. The ground truth annotations are masks that mark the objects that need to be tracked through the whole video. In our work, we focus on semi-supervised video object segmentation task, where the ground truth annotations are provided only for the first frame.

There are several challenges that make video object segmentation a difficult task. First, both the appearance of the target objects and the background surroundings may change significantly over time. Second, there could be a large pose and scale variation over time. Third, there could be occlusions between different objects, which hinder the performance of tracking. Examples of the above three challenges are shown in Fig..

A notable and challenging dataset for video object segmentation task is the DAVIS 2016 dataset , which is designed for single-object video segmentation. Later the DAVIS 2017 is brought out focusing on segmentation of multiple video objects. Both of the datasets are provided with mask annotations of extremely high accuracy.

In this paper, we tackle the video object segmentation task by utilizing a single end-to-end trainable network, namely the convolutional gated recurrent mask-rcnn. As video object segmentation focuses on segmentation of foreground object instances, which has much in common with instance segmentation, we extend instance segmentation network to tackle video object segmentation task. The whole network is a combination of instance segmentation network (Mask-RCNN) and visual memory module (Conv-GRU). Such design is to bridge the instance segmentation task to the video object segmentation task. The model structure is shown in Fig. .

**Contributions** of this paper are three-fold.

-   We have revised the powerful instance segmentation network (Mask-RCNN ) for video object segmentation task by combining it with Conv-GRU module. Our model simultaneously segments all the target objects in the images.

-   We have shown the effectiveness of the inserted convolutional gated recurrent  unit for mask propagation in the long term. In addition, bottom-up path augmentation is easily achieved in our Conv-GRU module.

-   A training strategy to successfully train the model for video object segmentation task has been brought out. All the training processes are guided by the instance segmentation losses only. The results of our model are reported for both DAVIS 2016 and DAVIS 2017 datasets.

# Related work

In this section, we will discuss the researches that are relevant to our work.

**Object detectors**. There has been a great improvement for object detection technique in recent years. The task starts with box level object detection. Single-stage detectors  have faster running speed while two-stage networks  are more accurate in general. The brilliant design of Mask-RCNN  merges object detection with semantic segmentation by combining Faster-RCNN  and FCN , which form a conceptually simple, flexible yet effective network for instance segmentation task. Mask-RCNN network is suitable for instance segmentation on static images, but lacks the ability for temporal inference. Our work is to further extend Mask-RCNN with conv-gru module to solve video object segmentation task.

**Recurrent neural networks (RNNs)**. RNNs  are widely used for tasks with sequential data, such as image captioning , image generation  and speech recognition . The key for RNNs is the hidden state, which selectively accumulate information from current input and the previous hidden state overtime. However, RNN has its limitation as it fails to propagate information for long sequence due to the problem of gradient vanishing or explosion in training . Two RNN variants, LSTM  and GRU  are more effective for long term prediction by taking advantage of gating mechanism. To further encode spatial information, they are extended to Conv-LSTM  and Conv-GRU  respectively and have been used for video prediction  and action recognition .

**Methods for VOS**. Conv-GRU has already been used for video object segmentation. It serves as visual memory in  and has been proved to boost the performance for VOS task. However, their model performs binary semantic segmentation only, which is not suitable for video object segmentation task with multiple objects.

VPN , MSK  and RGMP  learns to propagate mask for VOS task. VPN utilizes learnable bilateral filters to achieve video-adaptive information propagation across frames. MSK learns to utilize both current frame and mask estimation from previous frame for mask prediction. RGMP utilizes the the first frame and mask as reference for instant information propagation besides the usage of current frame and previous mask estimation. Both MSK and RGMP achieve good results, but they can only propagate information for instances one by one.

Specially, OSVOS , OSVOS+  and OnAVOS  tackle video object segmentation from static images, achieving temporal consistency as a by-product. They learn a general object segmentation model from image segmentation datasets and transfer the knowledge for video object segmentation. They all rely on additional post processing for better segmentation result. OnAVOS further applies online adaptation to continuously fine-tune the model, which is very time consuming.

  explores the benefits from in-domain training data synthesis with the labelled frames of the test sequences.   synthesizes video training data from static image dataset to add to limited video training samples.   explore fast prediction without online training through matching based method. CINM  achieves good prediction by spatial-temporal post-processing based on results from OSVOS . To handle the problem of long term occlusion,   apply re-identification network to retrieve the missing objects, which complements their mask propagation methods. Recently, there are still many researches focusing on single-object video segmentation , which are not easily transformed for video segmentation of multiple objects.

The best results are achieved by ensemble of multiple specialized networks. PReMVOS  wins the 1st place of DAVIS2018 semi-supervised vos task by utilizing complex pipeline with multiple specialized networks trained on multiple datasets.

# Method

In this section, we first introduce different parts of our network structure. Our convolutional gated recurrent mask-rcnn is comprised of mainly three parts. They are the feature extraction backbone, the visual memory and the prediction heads. The backbone network extracts features that are forwarded to visual memory module (Conv-GRU). The Conv-GRU module then selectively remembers the new input features and forgets the old hidden states. On top of Conv-GRU, region proposal network (RPN), bounding box regression head, id classification head and mask segmentation head are constructed to solve the VOS task. The whole network is end-to-end trainable under the guidance of instance segmentation losses.

## Mask-RCNN

Mask-RCNN  is one of the most popular framework designed for instance segmentation task. It is used for instance-wise object detection, classification and mask segmentation, which makes it naturally suitable for multiple video objects segmentation. Roles of different components in mask-rcnn directly shift to fit VOS task as illustrated below.

**Backbone:** The backbone network still serves to extract features from images, but more focused on generating useful features adaptively for gates of Conv-GRU module. ResNet101-FPN  with group normalization  is used as our backbone network. Detailed structure is shown in Fig..

**RPN:** Mask-RCNN is known as a two stage instance segmentation network. Bounding boxes of general objects are proposed in the first stage, while classes and masks are predicted instance-wisely in the second stage. Such two stage framework adopts the same philosophy as the training stages of OSVOS . For OSVOS, the network first learns to segment binary mask for general objects in a class-agnostic manner. Then it learns to segment specific objects during online training. In mask-rcnn, RPN learns to reject background objects and to propose foreground objects in the first stage, which is also class-agnostic. It is in the second stage that classes and masks of different objects are determined.

**Bounding box regression head:** This branch is used to refine the bounding box proposals. Each predicted box contains one object. The boxes serve to separate different objects in an image.

**Classification head:** This branch is used to assign the object a correct class label. However, class type is unknown for VOS task. Instead, different objects are associated with different ids, which need to be predicted coherently in a video sequence. Classification branch is naturally transformed into an id classification branch.

**Mask segmentation head:** This branch is used to extract mask for each foreground object in the image, which is the main target of VOS task.

Clearly, for the components in mask-rcnn, there is a direct responsibility mapping from instance segmentation task to VOS task.

## Convolutional gated recurrent unit

One difficulty for video object segmentation is the problem of long term dependency. The ground truth is provided only for the first frame, but the objects still need to be predicted after tens or hundreds of frames based on the ground truth from the first frame. The appearance of different objects in the videos may vary greatly and the objects sometimes get partially or even completely occluded, which makes coherent prediction more difficult.

In order to handle the above problem, we utilize the convolutional gated recurrent unit, serving as a visual memory to handle appearance morphing and occlusion. The memory module learns to selectively propagate the memorized features and to merge them with the newly observed ones. The key role for conv-gru module is to maintain a good feature over time for prediction of region proposal, bounding box regression, id classification and mask segmentation. The mathematical formulation of Conv-GRU is shown as follows. $$\label{eq_gru}
        \begin{align}
        z_t & = \sigma(W_{hz}*h_{t-1}+W_{xz}*x_t+b_z)\\
        r_t & = \sigma(W_{hr}*h_{t-1}+W_{xr}*x_t+b_r)\\
        \hat{h_{t}} & = \Phi(W_h*(r_t \odot h_{t-1})+W_x*x_t+b)\\
        h_t & = (1-z_t) \odot h_{t-1} + z_t \odot \hat{h_t}
        \end{align}$$ $x_t$ is the input feature of time t, $h_t$ is the hidden state of time $t$. $z_t, r_t$ are update gate and reset gate respectively. W and b are convolutional filters and bias parameters. $\sigma$ and $\Phi$ are sigmoid function and tanh function respectively.$*$ and $\odot$ denote the convolution operation and element-wise multiplication respectively.

Compared to instance segmentation task, where each training batch is comprised of multiple randomly sampled images, the batch in temporal training has less variation due to that consecutive images from one sequence are highly correlated. This is similar to the problem of small batch size. To relieve such effect, we further replace the bias term in Conv-GRU with the group normalization (GN) layer, which are proved to give consistent performance across different batch sizes. $$\label{eq_gru_gn}
        \begin{align}
        z_t & = \sigma(GN(W_{hz}*h_{t-1}+W_{xz}*x_t))\\
        r_t & = \sigma(GN(W_{hr}*h_{t-1}+W_{xr}*x_t))\\
        \hat{h_{t}} & = \Phi(GN(W_h*(r_t \odot h_{t-1})+W_x*x_t))\\
        h_t & = (1-z_t) \odot h_{t-1} + z_t \odot \hat{h_t}
        \end{align}$$

For each level of FPN network, we create a corresponding Conv-GRU layer. The layers at different levels learn different transition functions for the hidden states. As bottom up path augmentation has been proved to be useful for instance segmentation, we easily achieve it by up-sampling and addition operation with output features from multi-level conv-gru module. The structure is shown in Fig. . The output features after path augmentation are used for RPN and prediction heads. Conv-GRU module is deliberately directly inserted after backbone network. In this way, information for both region proposal and instance prediction can be propagated through time.

# Training the network

In this section, we will describe our training strategy in detail. The training modality for video object segmentation can be divided into offline training and online training . During offline training, the model is trained with ground truth of training set only. During online training, the model is fine-tuned with the ground truth of the first frame from test set. As the class types of test set are not known and objects may never be seen during offline training, online fine-tuning is necessary to help the model to generalize better for test set.

Our network needs both offline training and online training. During offline training stage, our network learns the features to differentiate all the object instances and learns to predict class-agnostic boxes and masks. During online training stage, our network is fine-tuned to differentiate objects for each test sequence and to predict boxes and masks in a class-specific manner.

## Class-agnostic offline training

To provide our model with as much generality as possible, we apply class-agnostic training for bounding box and mask through the whole offline training process. Offline training for our model can be divided into two steps. First, our model is trained with instance segmentation dataset. This step is to provide our model with general object detection ability. Then, we train the model with video dataset to learn to propagate information over time for video object segmentation.

### Pre-train on instance segmentation dataset

Pre-training on additional dataset is a common practice . We initialize our model by pre-training on Microsoft COCO dataset . Ms-COCO dataset has been widely used for object detection task. It targets common objects in context with annotations including boxes, classes and masks. By first training on Ms-COCO dataset, our model learns to extract useful features for general object detection. As the training is on static images, we set hidden states to be zeros without update for Conv-GRU module.

After this step, our model gains general region proposal ability, general bounding box prediction ability and general object segmentation ability. Our model also learns to differentiate general objects by classes defined on Ms-COCO dataset.

### Fine-tune on video object segmentation dataset

In this stage, we train all the modules except the backbone network. By fine-tuning our model on video object segmentation dataset, the Conv-GRU module learns to tune its gates to best propagate information. It should be noted that the class number has changed as the video object segmentation dataset does not share the class definition with instance segmentation dataset. Instead, we replace the last linear layer right before softmax layer in the class prediction head with a new one, which now predicts the id index in the dataset. The class prediction head turns into an id prediction head.

The network is trained purely with instance segmentation losses. The different losses guide our model to have different abilities. The mask loss helps our model to propagate mask segmentation. The losses from id head and bbox head help our model to propagate information differently for each instance. Although the mask head and bbox head are trained in a class-agnostic manner, the id head and bbox head provide a chance to learn to propagate class-specific information.

To facilitate the information propagation, we further add a shortcut connection between the ROI aligned feature and the head logits as shown in Fig. .

## Class-specific online fine-tuning

As the instances in test sequences are not the same as in training sequences, the last linear layer in id head needs to be re-initialized and trained to differentiate instances in the current sequence. We replace the last linear layer in the same way as in section . We also adopt focal loss  for id head to balance the training for multiple instances.

During online fine-tuning, the parameters in backbone network and Conv-GRU module are frozen to keep the learned propagation property. All other parts are fine-tuned for the new objects in test image. The class-agnostic prediction in mask head and bbox head are altered to be class-specific in order to have less competition for different instances. The process is illustrated in Fig. .

# Online inference

As our model predicts mask for each unique instance, there naturally exist constraints for prediction.

**One maximum constraint**. For each instance, there should be at most one object detected. This constraint is achieved by selecting highest id prediction score.

**Location continuity constraint**. If an instance is detected with high id prediction score in the last frame, the location of current detection should not be far from its previous location. To achieve this constraint, we suppress the prediction for the instance, whose boxes iou between consecutive frames is low.

As probability for id prediction decays over time, we further apply a very light weighted fine-tuning process for the last linear layer of id head during online prediction. If there exists target object detected with high enough id prediction score, its predicted bounding box is set as ground truth for fine-tuning id head only. By saving and reusing intermediate tensors, the speed for fine-tuning is fast.

# Experiments

To test how our model learns to propagate instance information in a long term sequence, we evaluate our model on both DAVIS 2016  and DAVIS 2017  datasets, which provide video sequences of high quality and accurate mask annotations of objects. DAVIS 2016 dataset focuses on single-object video segmentation. It has 30 training and 20 validation videos. As an extension to DAVIS 2016 dataset, DAVIS 2017 dataset brings 30 more video sequences for training set and 10 more for validation set. It also provides another 30 sequences for testing. As DAVIS 2017 dataset focuses on multiple object segmentation, it has been re-annotated for each individual target object.

## Implementation Details

Our model is implemented with PyTorch  library. A Nvidia Titan X (Pascal) GPU with 12GB memory is used for experiments. Details of convolutional gated recurrent mask-rcnn are shown below.

**Model structure.** Our backbone network is a ResNet101-FPN  with group normalization . Imagenet  pre-trained weights are used to initialize resnet101. In Conv-GRU module, the channel number of each hidden state is 256. Kernels of all convolutions in Conv-GRU are of size $3\times3$ with 256 filters. The ROI aligned feature resolution is $28\times28$ for mask head, and $7\times7$ for bbox head and id head. In all cases, we adopt image centric training .

**Pretrain on Ms-COCO dataset.** For each image, we randomly scale it to have its shorter side equal 1 of 11 different length: 640, 608, 576, 544, 512, 480, 448, 416, 384, 352, 320 and its longer size to be maximumly 1333. We sample 512 ROIs with foreground-to-background ratio 1:3. RPN adopts 5 aspect ratios (0.2, 0.5, 1, 2, 5) and 5 scales (28, 56 ,112 ,224, 448). The model is trained with SGD for 270K iterations. We fix input hidden states to be zeros for Conv-GRU module, weight decay 0.0001, momentum 0.9. The initial learning rate is 0.02 and dropped by a factor of 10 at 210K and 250K. In the following cases, the configuration is kept the same unless otherwise stated.

**Fine-tune on DAVIS dataset.** We generate ground truth (GT) bounding boxes from GT masks of DAVIS dataset. The width and height of the boxes are expanded by $10\%$, which is to prevent incomplete mask prediction caused by inaccurate box prediction. We randomly shuffle the video sequences. For each sequence, we draw consecutive images and randomly resize them with the same scale. As there is no causal reasoning in the task, we draw images inversely to create more training data. The parameters of backbone are kept fixed during training to prevent over-fitting for DAVIS dataset. 128 ROIs are sampled from each image. The model is trained with SGD for 12K iterations. The initial learning rate is 0.002 and dropped by a factor of 10 at 8K and 10K. Due to the GPU memory limitation, it only allows to train our model with maximum recurrence of 4. We extend the length to 8 by stop gradient back propagation between 4th and 5th frames. We apply focal loss  for id head to balance the training between different classes.

**Online fine-tuning.** The network is fine-tuned with the GT mask of the first image for maximally 1000 iterations. We apply early stopping to prevent network from over-fitting. If the loss for a prediction head is smaller than an empirically chosen threshold, the loss is ignored. If all the losses are ignored, we stop the training. The thresholds for bbox regression loss, id cross-entropy loss and mask binary cross-entropy loss are 0.015, 0.010 and 0.15 respectively. We also stop the loss back-propagation of id head at its last fully connected layer, so the features to distinguish ids will not be affected by the newly initialized head. Focal loss  is used to balance id training.

**Online inference.** For Location continuity constraint, we suppress object instance that has IOU lower than 0.3 with detection from previous frame if previous id score is higher than 0.4. To relieve the id score from decaying over time, we apply fine-tuning for id head for maximally 500 iterations. The early stopping threshold for id cross-entropy loss is set to 0.015.

## Compare with other methods

We compare our method with some state-of-the-art methods on both the DAVIS 2016 benchmark [^1] and the DAVIS 2017 benchmark [^2] by using standard evaluation metrics J and F . It should be noted that our method does not apply any post-processing, but pre-trained with Ms-COCO dataset . Among the several top methods, we remove CINM  and RGMP  to avoid unfair comparison. CINM  is built upon OSVOS  and further adopts a refinement CNN and MRF for post-processing. It is a method depends on the initial prediction, in this case OSVOS. The better initial prediction, the better its result. RGMP  cannot be successfully trained with DAVIS dataset alone for mask propagation. Besides being trained with static images, it has created a large number of synthetic video training data from Pascal VOC , ECSSD  and MSRA10K  datasets. It is not fair to compare with RGMP as the quality of video training data are not the same and cannot be controlled. For DAVIS 2017 benchmark, we avoid PReMVOS  and OSVOS+  as they both use multiple specialized networks in multiple processes to refine their results.

For DAVIS 2016, we compare with OnAVOS , FAVOS , OSVOS , MSK , PML , SFL , OSMN , CTN  and VPN . We detect multiple objects and evaluate in the way for single-object.

Our method ranks the 4th among the compared methods as shown in Table . It should be noted that our results are better than FAVOS and OSVOS if they are without post-processing. FAVOS achieves J mean and F mean of 0.779 and 0.760 respectively without tracker and CRF . OSVOS achieves J mean and F mean of 77.4 and 78.1 respectively without boundary snapping post-processing . OnAVOS achieves J mean of 82.8 without CRF post-processing .

For DAVIS 2017, we compare with OnAVOS , OSVOS , FAVOS  and OSMN  as shown in Table . Our network achieves comparably good results without any post-processing. Qualitative results on DAVIS 2016 and DAVIS 2017 are shown in Fig.  and Fig.  respectively. More qualitative examples are shown in supplementary material.

In addition, we compare our method with another visual memory (Conv-GRU) based video object segmentation method  on DAVIS 2016 dataset. Both of the methods are trained with additional image dataset, but we achieve better result without optical flow and CRF post-processing.

```latex
\begin{table*}[htp]\begin{center}
            \begin{tabular}{r|c|c|c|c|c|c|c|c|c|c}
                \hline
                Method             & OnAVOS  & FAVOS & OSVOS & LIP(Ours) & MSK & PML & SFL & OSMN & CTN & VPN\\
                \hline
                J\&F Mean$\uparrow$& 85.5    & 81.0  & 80.2  & 78.5      & 77.6& 77.4& 76.1& 73.5 & 71.4& 67.9\\
                \hline
                J Mean$\uparrow$   & 86.1    & 82.4  & 79.8  & 78.0      & 79.7& 75.5& 76.1& 74.0 & 73.5& 70.2\\
                J Recall$\uparrow$ & 96.1    & 96.5  & 93.6  & 88.6      & 93.1& 89.6& 90.6& 87.6 & 87.4& 82.3\\
                J Decay$\downarrow$& 5.2     & 4.5   & 14.9  & 0.05      & 8.9 & 8.5 & 12.1& 9.0  & 15.6& 12.4\\
                \hline
                F Mean$\uparrow$   & 84.9    & 79.5  & 80.6  & 79.0      & 75.4& 79.3& 76.0& 72.9 & 69.3& 65.5\\
                F Recall$\uparrow$ & 89.7    & 89.4  & 92.6  & 86.8      & 87.1& 93.4& 85.5& 84.0 & 79.6& 69.0\\
                F Decay$\downarrow$& 5.8     & 5.5   & 15.0  & 0.06      & 9.0 & 7.8 & 10.4& 10.6 & 12.9& 14.4\\
                \hline
            \end{tabular}
        \end{center}
        \caption{Results on DAVIS 2016. Left column shows different metrics. Up-arrow$\uparrow$ means the higher the better. Down-arrow$\downarrow$ means the lower the better. Methods are in descent order according to J\&F mean from left to right.}
        \label{tab:davis16}
    \end{table*}
```
```latex
\begin{table*}[htp]
        \begin{center}
            \begin{tabular}{r|c|c|c|c|c}
                \hline
                Method              & OnAVOS & LIP(Ours)& OSVOS & FAVOS & OSMN\\
                \hline
                J\&F Mean$\uparrow$ & 65.4   & 61.1     & 60.3  & 58.2  & 54.8\\
                \hline
                J Mean$\uparrow$    & 61.6   & 59.0     & 56.6  & 54.6  & 52.5\\
                J Recall$\uparrow$  & 67.4   & 69.0     & 63.8  & 61.1  & 60.9\\
                J Decay$\downarrow$ & 27.9   & 16.0     & 26.1  & 14.1  & 21.5\\
                \hline
                F Mean$\uparrow$    & 69.1   & 63.2     & 63.9  & 61.8  & 57.1\\
                F Recall$\uparrow$  & 75.4   & 72.6     & 73.8  & 72.3  & 66.1\\
                F Decay$\downarrow$ & 26.6   & 20.1     & 27.0  & 18.0  & 24.3\\
                \hline
            \end{tabular}
        \end{center}
        %\end{adjustbox}
        \caption{Results on DAVIS 2017. Left column shows different metrics. Up-arrow$\uparrow$ means the higher the better. Down-arrow$\downarrow$ means the lower the better. Methods are in descent order according to J\&F mean from left to right.}
        \label{tab:davis17}
    \end{table*}
```

|             Method |    VisualMem     | LIP(Ours) |
|-------------------:|:----------------:|:---------:|
| Additional dataset |    PASCAL VOC    |  Ms COCO  |
|     Additional aid | CRF&Optical Flow |    \-     |
| J&F Mean$\uparrow$ |       74.0       |   78.5    |
|   J Mean$\uparrow$ |       75.9       |   78.0    |
|   F Mean$\uparrow$ |       72.1       |   79.0    |

Comparison with another visual memory based method . Results reported on DAVIS 2016.

## Ablation study

We perform ablation study on DAVIS 2017 validation dataset by gradually adding up key components into the network structure as shown in Table . We first evaluate the static model by fixing input hidden states ($h_{t-1}$) to zeros for Conv-GRU module. This is Mask-RCNN with static Conv-GRU module and bottom up path augmentation. Fine-tuning on video dataset is done by training with static images only. The J&F mean score is 59.2, which is close to the performance of OSVOS  with post-processing. The full version of the model reaches the best J&F mean score of 61.1.

```latex
\begin{adjustbox}{width=\columnwidth,center}
            %\begin{center}
            \begin{tabular}{c|c|c|c|c}
                \hline
                Mask-RCNN &Conv-GRU  &J Mean&F Mean&J\&F Mean\\
                \hline
                \checkmark&input zero&56.9  &61.5  &59.2    \\
                &hidden states&&&\\
                \hline
                \checkmark&\checkmark&59.0  &63.2  &61.1    \\
                \hline
            \end{tabular}
            %\end{center}
        \end{adjustbox}
```

## Failed cases and limitation

During online inference, we find two failed cases that are most relevant to our method. Although long-term visual memory is included in our model, it still fails to handle some occlusions as shown in Fig. . Our model also finds it difficult to propagate information for multiple instances with large overlaps as shown in Fig. . Another limitation is that the probability of id prediction decays over time. As the id classifier is learned from the first image, it does not generalize for all images, which requires online fine-tuning for id head.

# Conclusion

We have presented a single end-to-end trainable neural network for video segmentation of multiple objects. We extend the powerful instance segmentation network with visual memory for inference ability across time. Such design serves as an instance segmentation based baseline for VOS task. The newly designed convolutional gated recurrent mask-rcnn learns to extract and propagate information for multiple instances simultaneously and achieves results that are comparable to some of the state of the art methods.

[^1]: <https://davischallenge.org/davis2016/soa_compare.html>

[^2]: <https://davischallenge.org/davis2017/soa_compare.html>


bibliography: /home/sacha/Documents/svn/library.bib

(Keywords: Phylogeny, Diversification, Mass Extinctions)

**Background:** Recent methodological advances allow better examination of speciation and extinction processes and patterns. A major open question is the origin of large discrepancies in species number between groups of the same age. Existing frameworks to model this diversity either focus on changes between lineages, neglecting global effects such as mass extinctions, or focus on changes over time which would affect all lineages. Yet it seems probable that both lineages differences and mass extinctions affect the same groups.
**Results:** Here we used simulations to test the performance of two widely used methods under complex scenarios of diversification. We report good performances, although with a tendency to over-predict events with increasing complexity of the scenario.
**Conclusion:** Overall, we find that lineage shifts are better detected than mass extinctions. This work has significance to assess the methods currently used to estimate changes in diversification using phylogenetic trees. Our results also point toward the need to develop new models of diversification to expand our capabilities to analyse realistic and complex evolutionary scenarios.

The estimation of the rates of speciation and extinction provides important information on the macro-evolutionary processes shaping biodiversity through time . Since the seminal paper by Nee $et$ $al.$ , much work has been done to extend the applicability of the birth-death process, which now allows us to test a wide range of hypotheses on the dynamics of the diversification process.
Several approaches have been developed to identify the changes in rates of diversification occurring along a phylogenetic tree. Among them, we can distinguish between lineage-dependent, trait-dependent, time-dependent and diversity-dependent changes. Lineage specific methods identify changes in macro-evolutionary rates  — speciation and extinction rates, denoted as $\lambda$ and $\mu$, respectively —  at inner nodes of a phylogenetic tree . We can also identify trait-dependence in speciation and extinction rates if the states of the particular trait of interest are known for the species under study . It is also possible to look for concerted changes in rates on independent branches of the phylogenetic tree by dividing it into time slices . Finally, diversity-dependent effects can be detected when changes of diversification are correlated with overall species number . Most methods can correct for incomplete taxon sampling, by assigning species numbers at tips of the phylogeny , or by introducing a sampling parameter . By taking into account this sampling parameter at time points in the past, it is also possible to look for events of mass extinction .
These methods provide insights into the dynamics of species diversification and it is now well accepted that differences in lineage-specific rates exist . However, it seems unlikely that both lineage specific shifts and mass extinction events would not have occurred, especially when studying large phylogenetic trees covering hundreds of million years of evolution. For example, several global crises, which caused the extinction of a high proportion of species , have occurred since the appearance of the last common ancestor of vertebrates. Among them, the Cretaceaous-Paleogene (K-Pg) boundary and the Permian-Triassic events, which happened $65$ million years ago (Mya) and $251$ Mya, respectively, induced the most dramatic losses of biodiversity . Moreover, other less extensive events have also occurred in the past hundred million years .
Alternative models have been proposed for mass extinctions. They could be represented as a high number of species disappearing at the same time (single-pulse model), or as an increase of the background rate of extinction during an extended period of time (time-slice model) . They could also impact biodiversity in different ways. Three main hypotheses, corresponding to different patterns of extinction, have been proposed . First, the event could affect all lineages equally and terminate any extant lineage with the same probability. This "field of bullets" scenario is often used as a null model . Second, in the "fair game" scenario, some form of lineage selection would occur, where the most successful species  — in our case, the most diversifying species —  before the event would be the most likely to survive. This could, for instance, happen if the probability of survival depends on a specific trait varying across the lineages of the phylogeny . Finally, in the "wanton destruction" scenario , the event could induce such changes in the environmental conditions that the probability of extinction of the species and their post-event diversification rate would be uncorrelated to their initial speciation and extinction rates.
Although lineage-dependent differences in macro-evolutionary rates and mass extinctions are known to happen, the performances of the existing methods to identify both lineage-specific rate shifts when mass extinctions have occurred, and mass extinctions when lineage-specific rate shifts have occurred has not, to our knowledge, been investigated. The aim of this study was thus to assess the performance of current methods to estimate the rates of diversification using complex scenarios involving both mass extinctions and lineage shifts. We used simulations to assess the impact of varying number and magnitude of rate shifts and mass extinction events.

Figure gives an overview of the simulation design. We used a backward algorithm to simulate phylogenetic trees as implemented in the function $sim.rateshift.taxa$ from the R package TreeSim . Direct forward approaches to simulate trees using a birth-death process are also available. They can be used by conditioning either on the number of tips or on the total amount of time of the process. The former approach can lead to bias , while the latter could be less practical in our specific context as the procedure would result in trees with highly variable numbers of taxa, in particular when adding mass extinction events. A backward simulation procedure is therefore the best solution to simulate the different diversification scenarios of interest for our study. This procedure enables both single-pulse or time-slice modeling of mass extinctions, but we chose to represent them only using the single-pulse model because paleontological data indicates very high species loss at major mass extinction events in a limited amount of time. For instance, a 52% decrease in marine families was observed at the Permian-Triassic boundary .

Our algorithm takes as input the number of extant species, the evolutionary rates $\lambda$ and $\mu$, and the time of occurrence and survival rate $\rho$ for mass extinction events. We assumed in the first part of our simulations that these events happened according to the field of bullet scenario (step $1$). We randomly grafted different trees having experienced the same mass extinction events but different evolutionary rates to account for rate shifts in diversification (step $2$; see Table ). First, we ran as many backward simulations as the number of lineages shifts in our tree. We defined the number of species in each backward simulation by drawing samples from a Dirichlet distribution to keep the total sum equal to the overall number of species. We then ranked the trees by decreasing order of their total age, which included the stem branch length provided by TreeSim. We selected from the oldest tree (referred to as acceptor tree) the branches that overlapped in time with the age of the stem branch of the second oldest tree (referred to as donor tree). Thus, the branches considered for possible grafting were the ones that included the age of the donor tree between the timing of the two speciation events defining them in the acceptor tree. We randomly chose one of those branches to graft the donor tree onto the acceptor. This ensures ultrametricity of the newly created tree and leaves the branch lengths of each separate tree unmodified once the lineage having experienced the diversification shift is removed. We iterated over this protocol until all donor trees, whose number varied in our simulations between 0 and 5 (Table ), were grafted. Finally, we ran Medusa and TreePar analyses on each simulated tree to investigate our capacity to recover the signal of mass extinctions and diversification shifts (Fig. ). We simulated trees with different numbers of lineages and extinction events to assess the influence of these factors. Table summarizes the parameter space explored for the $16,371$ trees that we simulated. For the values of $\lambda$ and $\mu$, we targeted distributions similar to the estimates calculated on a mammalian phylogeny .

```latex
\begin{tabu}{lc}

\bf{Parameter} & \bf{Possible values}\\
\hline
$\lambda$ & $Unif(0.05, 0.25)$ \\
$\mu$ & $Unif(0, 0.05)$\\
$\rho$& $Unif(0.2, 0.9)$\\
Number of tips & $200, 500, 1000, 2500, 5000$\\
Mass extinction event number & $0$ to $5$ \\
Rateshift event number & $0$ to $5$\\
Mass extinction event time& $ Unif(0, min(\frac{Log(N_i)}{\lambda_i-\mu_i}))$\\
\hline
\end{tabu}
```

Despite the issues to use existing forward algorithm, we nevertheless compared our backward algorithm with a "forward-like" algorithm based on the R package TESS . We simulated trees with different values of $\lambda$, $\mu$ and species number to model the lineage shifts in diversification rates. We carry a similar grafting process as described in our backward algorithm. However, we removed all daughter species of the sister clade of the donnor tree in the acceptor tree. This step has the consequence of removing the instance of the artificially created speciation event that was present in our former algorithm and effectively mimic a forward algorithm with a change in diversification rate possible anywhere between two speciation events. As the first conditioning is made on the number of species, and as we subsequently remove species, the total number of species at the end of the process in not constant but varies slightly below the number used for the conditioning. We simulated trees according to both our backward and forward algorithms and compared them using two different measurements: the distribution of branching times and the outcomes of Medusa on both our trees (Additional file 1). These two measures resulted in very similar outcomes and we present here only the results obtained by the backward algorithm.
Medusa is a maximum likelihood-based framework to detect shifts in diversification by iteratively adding breakpoints on inner branches of the tree with different rates of speciation and extinction. It uses $\Delta$AIC to discriminate between models with an increasing number of parameters . Rabosky also recently presented a new method (BAMM) to estimate the number of possible rate changes along a phylogenetic tree and to fit exponential responses in macroevolutionary rates to time or to species number . Unlike Medusa, BAMM uses a Bayesian framework, with reversible jump Markov chain Monte Carlo to estimate the number of shifts in diversification in the phylogeny. In our design, we chose not to simulate varying speciation and extinction rates except at speciation nodes, thus using higher complexity models is not necessary. Comparisons between BAMM and Medusa have been performed, but only on simulations involving either time-dependent or diversity-dependent rates . This framework led to a clear bias in favor of BAMM as Medusa can not evaluate such models, and resulted in Medusa estimating a lower number of events than what was actually simulated . The numbers of estimated shifts obtained with Medusa can therefore be considered as conservative. Finally, we do not expect a different behavior for Medusa and BAMM regarding the identification of mass extinction events, as neither method incorporates them in their model. Those reasons, as well as the large computational burden to run Bayesian analyses on over $16,000$ trees, led us to favor the simpler Medusa framework for the rest of the study. Medusa was run until a more complex model was not supported by the $\Delta$AIC. We did not extract the macro evolutionary rate estimations from Medusa as we were only interested in testing the ability of the method to detect the events, and not the accuracy of the parameter estimation.
TreePar uses the birth-death process to identify changes in $\lambda$ and $\mu$ through time. This is done by estimating the probability of a change in parameter values within small time intervals, which can be extended to test for the occurrence of mass extinction events . The parameters of the rate shifts might be correlated with those related to mass extinction , which will be a problem for our simulations. We therefore restricted our analysis to the identification of mass extinction events to avoid this issue. The number of iterations of TreePar was set to the simulated number of mass extinction events plus one to test for the appearance of false positive events. A standard Likelihood Ratio Test (LRT) is used to extract the most likely models from TreePar and more complex models were favored when their p-value was less than $0.01$, following the standard approach for this framework . Similarly to what was done with Medusa, we did not analyze estimations of survival rates at mass extinctions events given by this framework.
To verify that our simulation design had no effects on the methods evaluated, we tested the influence of the subtree grafting approach with a constant rate of diversification. We simulated trees with 200 species using both the standard procedures implemented in TreeSim and by grafting two subtrees of 150 and 50 species having evolved under the same $\lambda$ and $\mu$ values. We then compared the results obtained by TreePar and Medusa. We ran 250 pairs of simulations and we observed no significant differences in the number of false positive found between the groups with and without artificial grafting (7 and 13 for Medusa respectively, and none in both cases for TreePar), showing that our simulation design does not bias the estimation of the rate shifts by the two methods used.
We used a slightly different framework to study the impact of the different types of mass extinction events. We simulated a scenario that aimed at testing for the presence of the K-Pg mass extinction event using high order phylogenetic trees. We therefore simulated trees with a large number of extant species ($5,000$ tips, similar to the number of mammalian species) and a large number of lineage shifts ($5$), but only one event of mass extinction. The other parameters were still drawn at random from the ranges specified in Table , except for the survival rate $\rho$ that was modified according to the models of mass extinction. For the fair game hypothesis, we randomly drew $\lambda$ and $\mu$ for the $5$ different lineage shifts, but the survival rate $\rho$ was modified for each lineage based on its diversification rate ($r$, $\lambda-\mu$). We thus considered that the trait influencing the probability of extinction for each species was its diversification rate. For the wanton destruction hypothesis, the mass extinction event induced a change in rates for each lineage, again drawn according to the distribution stated in Table , and their survival rate $\rho$ was then based on their new diversification value. For the wanton destruction, our simulations included both a global rate shift and a mass extinction and we ran TreePar twice in order to detect both events. For the two latter cases, we chose to linearly parametrize $\rho$ with regards to diversification. As diversification could range between $0$ and $0.25$ and $\rho$ between $0$ and $1$, we applied a factor four to the diversification to obtain the survival rates of the lineages. We also ran Medusa on the three sets of simulations to assess the potential impact of the three extinction hypotheses on the detection of lineage shifts. For this second part, we generated over $700$ trees for each model of mass extinction event, for a total of $2289$ simulations.

The backward and "forward-like" algorithms gave very similar results (Additional file 1) and we only present here the results obtained with the backward algorithm. To estimate the baseline behavior of both frameworks, we first tested the performance of the methods on the simplest scenarios. We thus selected simulations that included a single rate shift for Medusa, or a single mass extinction for TreePar. Figure represents the fraction of shifts detected by Medusa relative to the absolute difference between the new and the old diversification values (Figure A) and to the number of species in the lineage (Figure B). More than $80\%$ of the changes in diversification larger than $0.05$ are detected by Medusa, which shows a good performance in assessing strong shifts. Further, Figure B shows that the overall tree size has no influence on the detection, since lineages of the same size are as likely to be detected in small or larger trees.

We then checked the ability of TreePar to detect mass extinction as a function of the survival rate, $\rho$, as well as of the number of ancestral species predating this event in the reconstructed tree. We also used first the simplest simulation to limit the effect of other parameters. Figure A shows that the signal of mass extinction in the phylogenetic tree is very weak when less than $100$ ancestral species are present before the event. This has implications for our ability to find evidence for the K-Pg boundary using phylogenetic trees of vertebrates, for example. We can only reach more than a hundred ancestral species older than $65$ My by considering phylogenetic trees encompassing distantly related lineages of tetrapods (see or ). Besides, as detection drops with increasing survival rate (Fig. B), the signal is even less likely to be picked as the ancestors of the extant species might have experienced the mildest extinction rates.

In a second stage, we analyzed simulations with more events and a mix of different types of events. We evaluated the performance of rate shift detection by Medusa, or of mass extinction events by TreePar, by comparing the events detected to the relevant simulated events. To perform the assignment between detected and simulated events (see Fig. ), we chose to minimize the sum of the distances between each potential pairing of events ($\sum \limits_i \delta_i^{Medusa}$ and $\sum \limits_i \delta_i^{TreePar}$). The distance metric used for Medusa was the sum of the branch lengths along the shortest path separating the two nodes, whereas we used the time between the estimated and simulated pairs of mass extinction events for TreePar (see caption of Figure for details).
The simulations incorporated several factors and we tested the effect on the framework of three categorical parameters: total number of tips, number of mass extinctions and number of shifts in diversification rate (see Table for their possible values). To ensure that the effects observed were related to the parameter of interest, we designed a reshuffling scheme for each parameter. First, we randomly selected an equal number of simulations for each combination of every possible value of the other two parameters. As an example, to study the outputs for trees of $200$ tips, we randomly drew an equal number of simulations with (i) no lineage shift, no mass extinction and $200$ tips; (ii) one lineage shift, no mass extinction and $200$ tips; (iii) one lineage shift, one mass extinction and $200$ tips; etc. This draw was repeated a hundred times and we determined, for each bin created, the proportion of simulations for which each method favored the model with the correct number of relevant events it was looking for, and the proportion of simulations for which they favored a model with too many events. Finally, we report the median and $95\%$ intervals of those proportions based on our hundred bins.
*Tree size influence.*— Both Medusa and TreePar perform better in assessing the correct number of events they are set to detect with an increasing number of tips (Fig. ). The median proportion of simulations correctly assessed reaches $60\%$ for Medusa and $32\%$ for TreePar with $5,000$ tips. The increase in the number of tips also leads to an increased acceptance by TreePar of models with too many mass extinctions ($28\%$ for $5,000$ tips). However, the number of tips in the tree has no effect on the error of the estimated time of mass extinction (Fig. ), even though more events are predicted. We only see a slight effect of tree size for Medusa, which is probably due to the fact that the method only detects lineage related events and does not depend on the total number of tips. We also investigated the effect of lineage size on the outputs of Medusa. We first compared the variance of lineage sizes relative to the overall tree size, contrasting the simulations with false positives to those with the correct number of rate shifts found. To remove the effect of lineage number, we compared groups of trees with the same number of diversification shifts. To account for a potential effect of tree imbalance, we compared the variance in lineage sizes inside trees, with or without false positives. There is no effect in most cases, except in the simulations with 4 or 5 rate shifts (p-values: $0.01$ and $3.6 \cdot 10^{-3}$, respectively, Mann-Whitney test). Thus, simulations with lineages of similar size are more likely to yield false positives only when they include more than 4 rate shifts. We also compared the variance in lineage sizes between simulations for which we recovered the correct number of events against those for which we recovered too few events. For every possible number of lineages, we find significantly lower variance for simulations that were correctly assessed. Thus, we only see a slight effect of the lineage size on the occurrence of false positives, whereas high variance in lineage size significantly increases false negatives. This indicates on the one hand, a tendency to overestimate the number of shifts when lineages are comparable in size, and on the other hand, problems with Medusa for identifying diversification shifts specific to a low number of species, as showed in the first part.

*Impact of events violating the model.*— We tested the robustness of the methods by studying the behavior of (1) Medusa to detect rate shifts with an increasing number of mass extinctions, and (2) TreePar to detect mass extinction events with an increasing number of lineages shifts. The results of Medusa are unaffected by the number of mass extinctions in the simulations (Fig. ). In contrast, an increase in the number of lineage shifts results in an increase of the proportion of false positives for TreePar ($2\%$ with no lineage shift vs. $20\%$ with five; Fig. ). However, the accuracy of the estimate of the timing of the event is not affected (Fig. ). The number of lineage shifts has almost no impact on the probability of detecting a true mass extinction event, $i.e.$ on false negatives.

We note that false positive rates remain very low throughout all cases for Medusa, less than $10\%$ overall and around $5\%$ when dealing with simulations without mass extinctions (Fig. A). Recently, May $et$ $al.$ have also studied the performances of Medusa but with a different focus. Medusa also enables the characterization of diversification changes on incomplete phylogenies by letting the user assign species diversity at each tips of the tree. Two different equations are then used to calculate the likelihood function. One of them incorporates the likelihood of getting a specific number of species given a pair of $\lambda$ and $\mu$ after a certain amount of time, and is now used to account for the terminally input species numbers. May $et$ $al.$ simulated complete phylogenies before introducing uncertainties by sequentially collapsing some of the tips, and tested the different flavors of the three different Medusa algorithms ever made available. They found high Type I errors in every algorithm and biased parameter estimates. We note that in our study, we did not consider the estimation of the macro evolutionary parameters, and did not use unresolved trees, that can be used in Medusa to account uncertainties in the phylogeny. Interestingly, May $et$ $al.$ also tested the algorithm that we used in this study ($turboMedusa$, defined as $tMEDUSA$ in their study) on completely resolved trees, and found about the same rate of Type I errors as we did in the comparable trees (Figure S.20 of their study). Thus even though the focus of the two studies differs, they are in agreement in the few common analysis.
*\*.*—Impact of patterns of extinction The effect of different scenarios of mass extinction on the results of Medusa and TreePar are presented in Figure . First, as expected, no effect of the extinction scenarios is observed on the detection of lineage rate shifts detected by Medusa (Fig. A). In contrast, the fair game and wanton destruction scenarios impact the estimation made by TreePar. They produce, for comparable levels of detection, more false positives than the field of bullets which was used in the previous simulations ($73\%$ and $74\%$ for fair and wanton against $58\%$ for field of bullets, Fig. B). Irrespective of the type of mass extinction simulated, there are very few false negatives, $i.e.$ at least one extinction was detected in almost every tree. The error on the timing of this event was kept under $5\%$ of the root age. We also performed a search for global rate shifts in the case of wanton destruction (Fig. B, dashed background). Regarding this scenario, we also compared simulations where all lineages undergo an increase of diversification after the mass extinction event against those who undergo a decrease and observe no difference between the outcomes of the two frameworks. Even though the shifts are different between lineages ($i.e.$, increase of diversification in some lineages, decrease in others), TreePar detects the period of this shift with more power than for the detection of the associated mass extinction ($34\%$ and $21\%$ correctly assessed simulations, respectively). Overall, these results show that departure from the simplest model of mass extinction should not affect our ability to detect these events in phylogenetic trees ($i.e.$ no increase in false negatives rate). But it should lead to an increase of false positive detections.

Previous studies involving mass extinctions and changes in macro-evolutionary rates have only focused on their effect on lineage through time plots . This lead to the identification of a possible mass extinction event in some plants lineages around 32 Mya, which was further suggested to be linked with changes in climate. Recently, Hohna developed a new algorithm to perform simulations with varying macro-evolutionary rates, allowing for mass extinction events. Other ongoing work aims at studying and simulating increasingly complex scenarios of diversification , but we would like to emphasize that no method allows the simultaneous discovery of both time-specific or lineage-specific rate changes and mass extinction events.
The study of diversification rates has become a standard part of the analysis of large phylogenetic trees , and recent efforts have also assessed the methods used when their assumptions are violated . We have shown that departure from the assumption of consistency in rates across lineages causes a large increase in false positives when looking for mass extinction events. This can be problematic as we know that rate consistency rarely holds , and casts doubts on our ability to reliably find such events using only phylogenetic trees. Nevertheless, an increasing number of disparities between lineages caused neither a decrease in the probability of detecting an event nor an increase in the error on its timing. As we observed the same pattern under more complex scenarios of extinction, the difficulty in detecting the K-Pg event in mammals is therefore probably not due to biases in the methods used. We might be limited by the power of TreePar to detect mass extinction events, although in simulations we reach 60% of true events detected for a tree size similar to that of mammals.
Recent efforts aim to reach a better agreement between paleontological and molecular data , including looking for mass extinctions in molecular phylogenies. For instance, there is much debate on whether the K-Pg extinction event triggered the mammalian diversification . The fossil record also indicates higher extinction rates of mammalians species around $65$ Mya . In this work, we have shown that for phylogenetic trees similar in size to that of mammals (i.e. $ca.$ $5000$ species), the signal for mass extinctions was usually recovered in the tree, even though lineage discrepancies in macro-evolutionary rates had a tendency to yield more false positives. Thus, if the ancestor lineages of the extant mammal families did experience a mass extinction at the K-Pg boundary, we should theoretically be able to identify it using phylogenetic trees. The underlying assumption about the mass extinction made when using TreePar is that lineages are terminated randomly with a fixed $\rho$ value everywhere in the tree, $i.e.$ a field of bullets type of mass extinction. But other models of extinction seem to increase false positives but not false negatives, not explaining difficulties in finding a K-Pg signal in real phylogenetic trees.
Recent studies have used Markov processes to account for the effect of specific traits upon the probability of extinction of a species, thus extending models of mass extinction beyond the field of bullets . Such models can be used for instance to estimate the loss of phylogenetic diversity after a mass extinction event . Our simulations can be seen as a special case of such models, where the trait influencing survival probabilities is the diversification value of the species. We have shown that more complex models of mass extinction cause more false positive detection than the simple field of bullets, as well as a decrease in the error for the fair game scenario. Choosing a specific model of extinction (field of bullets, wanton destruction, fair game) might require the incorporation of fossil information into the phylogenetic tree, and thus the further development of methods capable of dealing with both molecular and fossil data.

The authors declare that they have no competing interests.

SL, MRR and NS designed the study, SL performed the simulations, SL, MRR and NS analyzed the results and wrote the manuscript.

This work was supported by the ProDoc grant number 134931 of the Swiss National Science Fondation; and État de Vaud. The computations were performed at the Vital-IT (http://www.vital-it.ch) Center for high-performance computing of the SIB Swiss Institute of Bioinformatics. We thank Tanja Stadler, Daniele Silvestro and four anonymous reviewers for helpful discussions.