abstract: Recently, unsupervised pre-training is gaining increasing popularity in the realm of computational linguistics, thanks to its surprising success in advancing natural language understanding (NLU) and the potential to effectively exploit large-scale unlabelled corpus. However, regardless of the success in NLU, the power of unsupervised pre-training is only partially excavated when it comes to natural language generation (NLG). The major obstacle stems from an idiosyncratic nature of NLG: Texts are usually generated based on certain context, which may vary with the target applications. As a result, it is intractable to design a universal architecture for pre-training as in NLU scenarios. Moreover, retaining the knowledge learned from pre-training when learning on the target task is also a non-trivial problem. This review summarizes the recent efforts to enhance NLG systems with unsupervised pre-training, with a special focus on the methods to catalyse the integration of pre-trained models into downstream tasks. They are classified into architecture-based methods and strategy-based methods, based on their way of handling the above obstacle. Discussions are also provided to give further insights into the relationship between these two lines of work, some informative empirical phenomenons, as well as some possible directions where future work can be devoted to.
author: Yuanxin Liu^1,2^, Zheng Lin^1^
^1^Institute of Information Engineering, Chinese Academy of Sciences
^2^School of Cyber Security, University of Chinese Academy of Sciences
bibliography: neurips_2019.bib
title: Unsupervised Pre-training for Natural Language Generation: A Literature Review

# Introduction

Unsupervised pre-training has sparked a sensational research interest in the natural language processing (NLP) community. This technology provides a promising way to exploit linguistic information from large-scale unlabelled textual data, which can serve as an auxiliary prior knowledge to benefit a wide range of NLP applications. In the literature, language modeling (LM) is a prevalent task for pre-training, where the target words are predicted conditioned on a given context. Therefore, it is intuitive to employ the pre-trained LMs for natural language generation, as the pre-training objective naturally accords with the goal of NLG. However, revolutionary improvements are only observed in the field of NLU.

The primary factor that impedes the progress of unsupervised pre-training in NLG is an idiosyncratic nature of text generation: Basically, we do not write words from scratch, but instead based on particular context, e.g., the source language sentences for translation, the dialog histories for response generation, and the visual scenes for image captioning, among others. In unsupervised pre-training, the task-specific context is not available, which leads to a discrepancy between pre-training and training in the target task. More precisely, the challenges posed by the discrepancy can be reflected in two aspects: First, the diverse context makes it intractable to design a universal representation extractor as in the case of NLU, and the pre-trained language generators may have to modify their inner structures to deal with the task-specific context. Second, the mismatch in data distribution and objective between the two training stages might result in the performance on the pre-training tasks being compromised during fine-tuning, which is dubbed as the *catastrophic forgetting* problem .

In response to the above challenges, two lines of work are proposed by resorting to architecture-based and strategy-based solutions, respectively. Architecture-based methods either try to induce task-specific architecture during pre-training (task-specific methods), or aim at building a general pre-training architecture to fit all downstream tasks (task-agnostic methods). Strategy-based methods depart from the pre-training stage, seeking to take advantage of the pre-trained models during the process of target task learning. The approaches include fine-tuning schedules that elaborately design the control of learning rates for optimization, proxy tasks that leverage labeled data to help the pre-trained model better fit the target data distribution, and knowledge distillation approaches that ditch the paradigm of initialization with pre-trained parameters by adopting the pre-trained model as a teacher network.

The remainder of this review is organized as follows: In Section , we will introduce the background knowledge about unsupervised pre-training for NLU, followed by a sketch of how the pre-trained models are employed through parameter initialization for NLG in Section . In Section , we will describe the architecture-based methods, and the strategy-based methods are presented in Section . Section provides some in-depth discussions, and Section concludes this review.

# Background: Unsupervised Pre-training for NLU

Learning fine-grained language representations is a perennial topic in natural language understanding. In restrospect, compelling evidences suggest that good representations can be learned through unsupervised pre-training.

Early work focused on word-level representations , which encodes each word independently. For sentence-level representations, there are roughly two kinds of pre-training objectives, namely discriminative pre-training and generative pre-training. Discriminative pre-training distinguishes context sentence(s) for a given sentence from non-context sentence(s) , with the aim to capture inter-sentence relationships. Generative pre-training follows the language model paradigm: $$\max_{\theta} \sum_{t=1}^{T} \log P\left(x_{t} | C; \theta\right)$$ where $x_{t}$ is the $t^{th}$ word in the textual sequence to generate, $T$ indicates sequence length, $\theta$ stands for learnable parameters, and $C$ is the context information, which is defined by the pre-training objective. ELMo and GPT (short for Generative Pre-training) adopt uni-directional Transformer and bi-directional LSTM language models, respectively. In this case, the context is defined as $x_{1:t}$ or $x_{t+1:T}$. BERT is trained with a novel masked language model (MLM), which is a non-autoregressive way of generation. Specifically, MLM randomly replaces a fixed proportion of tokens in each sentence with a special \[MASK\] token or a random token, which results in a corrupted sentence $X_{\text{mask}}$, and predicts each replaced token based on the same context $X_{\text{mask}}$. To alleviate the inconsistency with target tasks caused by the introduction of \[MASK\] token, XLNet introduces permutation-based language model, which conducts autoregressive language modeling over all possible permutations of the original word sequence. This gives rise to a context $C=X_{\mathbf{z}_{1:t-1}}$, where $\mathbf{z}$ is a certain permutation of $[1,2, \ldots, T]$, according to the definitions in . and pre-trained an encoder-decoder framework to reconstruct the input sentence and the surrounding sentence, respectively, and the encoded input sentence thereby is included in the context $C$.

The sentence representations learned by LMs [^1] can be used to perform many NLU tasks by adding a simple linear classifier. Despite the objective of language modeling, the pre-trained representations and have successfuly pushed the state-of-the-art on multiple benchmarks .

# Unsupervised Pre-training and Parameter Initialization for NLG

NLG systems are usually built with an encoder-decoder framework, where the encoder reads the context information and the decoder generates the target text from the encoded vectorial representations. A direct way to utilize the pre-trained models is to initialize part of the encoder (when dealing with textual context) and/or the decoder with pre-trained parameters. For the encoder, pre-training is expected to provide better sentence representations, as we discussed in Section . For the decoder, the intuition is to endow the model with some rudimentary ability for text generation.

employed BERT as the encoder for abstractive text summarization, with some additional techniques to help integrate the BERT-initialized encoder with the randomly initialized decoder, which we will explicate in Section . GPT-2 inherited the left-to-right LM pre-training objective from GPT and extended the application to NLG, where the pre-trained LM directly serves as the language generator, with some special symbols to identify task-specific contexts. In the case of zero-shot task transfer, preliminary experiments showed that straightforward adaption of GPT-2 compares unfavorably with other unsupervised baselines.

is among the first attempts to investigate unsupervised pre-training for sequence to sequence (Seq2Seq) learning. They used pre-trained LSTM-based LMs to initialize the first layer of the encoder and the decoder, which act as representation extractors. An additional LSTM layer, which is randomly initialized, is then added on top of the pre-trained LMs to build the Seq2Seq framework. To make use of the text generation ability of LMs, the output softmax layer of the decoder LM is also retained. Some recent endeavours explored multiple combinations of GPT- and BERT-based models to initialize the encoder and the decoder, respectively. Although remarkable results are observed, the separately pre-trained LMs are still inconsistent with the Seq2Seq framework.

# Architecture-based Methods

## Inducing Task-Specific Architecture in Pre-training

Separately initializing the encoder and the decoder with LMs neglects the interaction between the two modules at the pre-training stage, which is sub-optimal. For NLG tasks that can be modeled as Seq2Seq learning, it is feasible to jointly pre-train the encoder and the decoder. Existing approaches for this sake can be categorized into three variants: Denoising autoencoders (DAEs), conditional masked language models (CMLMs) and sequence to sequence language models (Seq2Seq LMs).

### Denoising Autoencoder

An intuitive way to conduct unsupervised Seq2Seq learning is to train an autoencoder (AE) based on encoder-decoder framework. Different from AEs, DAEs take a corrupted sentence as input and reconstruct the original sentence. The advantage is that the corrupted input will force the decoder to extract relevant information from the source side for text generation. To obtain the corrupted sentence, designed three noising functions: *shuffle*, *delete* and *replace* (the left plot of Figure gives an illustration), each of which is controlled by a pre-defined probability distribution. To be more specific, each token in the raw sequence is assigned with a new index based on a gaussion distribution $N(0, \sigma)$; the delete and replace operations of a token are determined by a Bernoulli distribution $B(p)$ with Beta distribution as prior. The three functions are applied to the raw sequences in random order.

### Conditional Masked Language Model

CMLM extends the single model MLM proposed by to the encoder-decoder setting, where the masked text sequence is read by the encoder, and the decoder only reconstructs the masked tokens, in construct to the entire sequence in DAEs. As the middle plot of Figure shows, CMLM masks consecutive tokens [^2], and the unmasked tokens in the encoder side are masked when being feed to the decoder. Following the notations in , let us assume that the tokens with index from $u$ to $v$ are masked from the raw sentence $X$, which results in $X^{\backslash u: v}$, and $X^{u: v}$ denotes the decoder input. Then, when predicting each masked token $x_{t}$ ($u \leq t \leq v$), the context is $X^{u: v}_{<t}$ and $X^{\backslash u: v}$. The underlying motivation, as argued, is to force the encoder to understand the meaning of the unmasked tokens, which is achieved by encoder side masks, and encourage the decoder to refer to the source information rather than the leftward target tokens, which is achieved by decoder side masks.

### Sequence to Sequence Language Model

Seq2Seq LM performs Seq2Seq modeling using a single Transformer model, with the concatenation of source sentence and target sentence as input. To simulate Seq2Seq learning with encoder-decoder frameworks, the attention span of each target token is constrained to the source tokens and the leftward target tokens, which is achieved by self-attention masks (see the right plot of Figure ). In this way, the ability to extract language representation and generate texts are integrated into a single model. It is worth mentioning that Seq2Seq LM does not auto-regressively generate the target sentence, but instead predicting masked tokens based on the contexts controlled by self-attention masks. In other words, Seq2Seq LM still belongs into the family of MLMs. Apart from Seq2Seq LM, also explored uni-directional LM and bi-directional LM structures to perform the MLM-based cloze task, and incorporated the three kinds of LMs to build the final pre-training objective.

## Encoder-Agnostic Architectures for Adaptation

Although the Seq2Seq-based pre-training methods exhibit strong performance, they are confined to text-to-text generation. In order to encompass more diverse contexts, some researches began to investigate encoder-agnostic pre-training architectures . *Context Attention* and *Pseudo Self-Attention* are two typical variants presented by , which differ in the way that the task-specific context is injected (see Figure ). Context Attention takes the form of a standard Transformer decoder, with the layer that attends to the encoder outputs being randomly initialized. Pseudo Self-Attention considers the context vectors and the previous layer decoder outputs as an integral input, and the attended results are computed as follows: $$\operatorname{PSA}(C, Y)=\operatorname{softmax}\left(\left(Y W_{q}\right)\left[\begin{array}{c}{C W^{c}_{k}} \\ {Y W^{y}_{k}}\end{array}\right]^{\top}\right)\left[\begin{array}{c}{C W^{c}_{v}} \\ {Y W^{y}_{v}}\end{array}\right]$$ where $C \in \mathbb{R}^{|C| \times d_{c}}$ and $Y \in \mathbb{R}^{|Y| \times d_{y}}$ are the context vectors and representations of the target textual sequence, respectively. The linear transformation matrices $W^{c}_{k}, W^{c}_{v} \in \mathbb{R}^{|C| \times d_{model}}$ with respect to $C$ are added to project the context to the self-attention space, and $W_{q}, W^{y}_{k}, W^{y}_{v} \in \mathbb{R}^{|Y| \times d_{model}}$ are part of the pre-trained model.

Except for the performance on target tasks, an alternative metric to gauge the quality of encoder-agnostic architectures is the degree to which the pre-trained parameters have to change, in order to inject the task-specific context. compared the parameter changes of Context Attention and Pseudo Self-Attention in the feed forward layer, and discovered that Pseudo Self-Attention is more robust under this evaluation.

# Strategy-based Methods

## Fine-tuning Schedules for Adaption

When the pre-trained model is only a part of the target task system, fine-tuning requires joint learning of the components initialized in different fashion, which can make the training process unstable. The pre-trained model may also suffer from aggravated catastrophic forgetting problem as it has to coordinate with other components during fine-tuning . From the perspective of optimization, it is unreasonable to schedule the pre-trained components and the newly-introduced components with the same learning rate, considering that the former have already possessed some unique knowledge. A common assumption is that the pre-trained parameters should be updated at a slower learning rate and with smoother decay . The rationale behind such setting is that fine-tuning with more accurate gradient can prevent the pre-trained parameters from deviating too faraway from the original point, and the newly-introduced components need to quickly converge to the target parameter space. To this end, adopted two Adam optimizers with different learning rates for the pre-trained encoder and the randomly initialized decoder. The learning rates are scheduled as in with different warming up steps: $$\begin{array}{l}{l r_{\operatorname{Enc}}=\tilde{l} r_{\operatorname{Enc}} \cdot \min \left(\operatorname{step}^{-0.5}, { step } \cdot { warmup }_{\operatorname{Enc}}^{-1.5}\right)} \\ {l r_{\operatorname{Dec}}=\tilde{l} r_{\operatorname{Dec}} \cdot \min \left({step}^{-0.5},  { step } \cdot  { warmup }_{\operatorname{Dec}}^{-1.5}\right)}\end{array}$$ where ${warmup}_{\operatorname{Enc/Dec}}$ and $\tilde{l}r_{\operatorname{Enc/Dec}}$ determine the speed of learning rate changes and the max learning rates, respectively.

## Proxy Tasks for Adaption

Large-scale unlabelled data provides generic linguistic knowledge, but the target tasks have unique data distribution and objectives. An effective way to bridge this gap is to introduce proxy tasks with moderate changes to the pre-training objectives, but at the same time take the labeled data into account . *Translation Language Modeling* (TLM) is a special generalization of MLM in the cross-lingual situation. It leverages the paralleled machine translation corpus for further training of the LMs that are pre-trained on monolingual corpora. Specifically, the source language sentence and the corresponding target language sentence are fed to the model in parallel, with random tokens from each language being masked to perform the cloze-style prediction as in MLM. Different from monolingual MLM, TLM encourages word predictions to rely on the interdependence from two languages, therefore the sentence representations learned from separate languages can be well aligned.

For some particular NLG tasks, existing proxy tasks designed under the supervised setup can also work with unsupervised pre-training models. For instance, in neural text summarization, the combination of extractive and abstractive[^3] objectives can generate better summaries . Inspired by this, introduced extractive summarization as a proxy task to fine-tune the pre-trained BERT, before adopting it as the abstractive summarization encoder. Compared with the original BERT features, the representations learned from extractive summarization contain more task-specific information, therefore conveying the meaning of source texts better.

## Knowledge Distillation for Adaption

The aforementioned methods are diverse in implementation, but share the common idea of employing the pre-trained models through parameter initialization. An alternative way to exploit the pre-trained models is using the knowledge distillation technique . Knowledge distillation is a special form of training, where a *student* network learns from the supervision signals produced by a *teacher* network.

Taking BERT as an example, the pre-trained MLM contains global information, which can teach the autoregressive Seq2Seq models to "see from the future" . In practice, the probability distribution predicted by BERT is regarded as a soft label to compute the cross-entropy loss function : $$\mathcal{L}^{cross}_{kd}(\theta)=-\sum_{t=1}^{|Y|}\sum_{w \in \mathcal{V}}\left[P(y_{t}=w | Y^{masked}, X ; \phi ) \cdot \log P\left(y_{t}=w | Y_{1: t-1}, X ; \theta \right)\right]$$ where $X$, $Y$ and $Y^{masked}$ are the source sequence, the raw target sequence and the masked target sequence, respectively. $\mathcal{V}$ denotes the output vocabulary. $\theta$ indicates the parameters of the student network (Seq2Seq), which are learnable, and $\phi$ indicates the BERT parameters, which are fixed. In this way, the knowledge from unsupervised pre-training can be flexibly transferred to the target tasks, dispensing with the size and architecture limitations.

The supervision can also be derived from the hidden representations , with a mean-squared-error (MSE) distillation loss: $$\mathcal{L}^{mse}_{kd}=-\left\|{h}^{bert}_{m}-h^{seq2seq}_{n}\right\|_{2}^{2}$$ where $m$ and $n$ are hyper-parameters denoting the layer subscripts. Compared with the probability soft labels, the representation distillation method requires the Seq2Seq model to have the same hidden size with BERT, which is a more strict constrain.

Combining the knowledge distillation loss and the standard generative loss for Seq2Seq learning gives rise to the final objective to optimize: $$\mathcal{L}(\theta)=\alpha \mathcal{L}_{kd}(\theta)+(1-\alpha) \mathcal{L}_{seq2seq}(\theta)$$ where $\alpha$ is the weighting term that balances the contribution of the two kinds of loss functions.

# Discussions

## The Relationship between Architecture- and Strategy-based Methods

We have analysed two major challenges faced by the application of unsupervised pre-training to NLG (see Section ). On this basis, we introduced existing methodologies from the architecture and strategy considerations. The architecture-based methods are mainly proposed in response to the first challenge. Since the architecture of pre-trained model has a significant effect on the downstream task (when the pre-trained parameters are used for initialization), architecture designings have to plan in advance to narrow the discrepancy between pre-training and training on target tasks. This motivation has shown great effectiveness on the Seq2Seq framework . The strategy-based methods focus on the second challenge. They take a postprocessing point of view, with the aim to make the best of the pre-trained model at the target task training stage. It is noteworthy that the challenges are not independent inherently, and the two types of methods can actually work as complement to each other. For example, the fine-tuning schedules can alleviate the negative effects caused by the modification of pre-trained structures, and the catastrophic forgetting problem can also seek solution by devising a general task-agnostic architecture.

## Experimental Phenomenons

Existing researches on unsupervised pre-training for NLG are conducted on various tasks for different purposes. Probing into the assorted empirical results may help us discover some interesting phenomenons:

-   The advantage of pre-training gradually diminishes with the increase of labeled data .

-   Fixed representations yield better results than fine-tuning in some cases .

-   Overall, pre-training the Seq2Seq encoder outperforms pre-training the decoder .

The first two phenomenons attest to the catastrophic forgetting theory. Thanks to the access to large-scale unlabeled corpora, unsupervised pre-training is able to excel at zero/low-shot settings, while the pre-trained models can only achieve few gains when abundant labeled data is available. This can be explained by the high quality of the dataset and the capacity of the task-specific models, which leave little space for improvement. Nonetheless, the increased supervision from labeled data can also influence the performance on pre-training tasks. By fixing the pre-trained parameters, the learned representations will not be affected by the numerous iterations of training on the target task, which makes them work better without fine-tuning.

The third phenomenon is kind of counter-intuitive, as the generative pre-training objectives are more similar to the decoder's function. There is no unanimous theory to explain why the encoder is a more important element to pre-train. But this discovery suggests that the pre-trained LMs are more robust when acting as representation extractors, while they are more sensitive the the change of context when acting as conditional language generators.

## Future Directions

The diversity of NLG applications poses challenges on the employment of unsupervised pre-training, yet it also raises more scientific questions for us to explore. In terms of the future development of this technology, we emphasize the importance of answering four questions: 1) How to introduce unsupervised pre-training into NLG tasks with cross-modal context? 2) How to design a generic pre-training algorithm to fit a wide range of NLG tasks? 3) How to reduce the computing resources required for large-scale pre-training? 4) What aspect of knowledge do the pre-trained models provide for better language generation?

NLG tasks can be defined by the context features and mapping functions. The introduction of cross-lingual textual features and task-specific Seq2Seq architectures in the pre-training stage has successfully boosted the performance on text-to-text generation. For NLG tasks concerning multiple modalities, it is conceivable that pre-training methods could also benefit from the joint consideration of cross-modal features. For example, in the vision-and-language field, the learning of cross-modal representations has proven to be highly effective , but such representations can not yet be extracted from unpaired images and texts for image-grounded text generation, to the best of our knowledge.

In NLU, it is possible to pre-train one model to obtain language representations once and for all. As for NLG, a task-agnostic pre-training algorithm should transcend the purpose of representation learning, and consider the general ability for language generation. The notion of "encoder-agnostic adaption" makes a preliminary step towards this direction, but still remains far from approaching the equivalent performance as its NLU counterparts .

Due to the colossal scale of the pre-training corpora, including a large number of parameters is essential to achieve favorable performance. As a result, the model size usually costs at least 8 GPU cards in the pre-training for NLG systems, and it also hinders real-world applications. To reduce the memory consumption problem, existing work resorted to knowledge distillation to transfer the knowledge from a large teacher network to a small student network , or parameter reduction techniques to prune the model size in a more direct way . However, the research context is limited to the NLU scenarios, and same endeavours are necessary to NLG applications.

Another important branch of researches on unsupervised pre-training in NLP try to explain what kind of knowledge can be learned from pre-training. Related work has been done on the basis of both language understanding and generation . Specially, analysed the characters of texts generated from a pre-trained GPT-2 by evaluating them over a wide spectrum of metrics. We argue that deeper understanding the way in which unsupervised pre-training contributes to better text generation, and the intrinsic mechanisms of the pre-trained models are also crucial to future work.

# Conclusion

Unsupervised pre-training has defined the state-of-the-arts on a variety NLP tasks. However, in the field of NLG, the diversity of context information is still impeding the the application of unsupervised pre-training. The major challenges exist in designing model architectures to cater for the assorted context, and retaining the general knowledge learned from pre-training. In this review, we survey the recent unsupervised methods to utilize large-scale corpora for NLG purposes, with a highlight on those aiming at facilitating the integration of pre-trained models with downstream tasks. We propose to classify them into architecture- and strategy-based methods, followed with detailed introductions and discussions of their pros and cons. Based on the comparison of these methods and analyses of some informative experimental results from previous publications, we summarize some scientific questions that has not yet been well understood, and suggest attention being paid to these questions by future work.

[^1]: In the rest of this review, we refer to unsupervised pre-training as the generative LMs, unless otherwise specified.

[^2]: CMLM can also be implemented in other manners , where discrete tokens in the textual sequence are masked.

[^3]: Extractive text summarization selects sub-sequences from the source texts, while abstractive text summarization treats the task as a Seq2Seq learning problem.


author: Miguel A. Fortuna[^1] , Raúl Ortega, and Jordi Bascompte
date:  Integrative Ecology Group
  Estación Biológica de Doñana (EBD-CSIC)
  Seville, Spain
title: The Web of Life

mm

**The Web of Life (www.web-of-life.es) provides a graphical user interface, based on Google Maps, for easily visualizing and downloading data on ecological networks of species interactions. It is designed and implemented in a relational database management system, allowing sophisticated and user-friendly searching across networks. Users can access the database by any web browser using a variety of operating systems. Data can be downloaded in several common formats, and a data transmission webservice in JavaScript Object Notation is also provided.**

# Introduction

In nature, species do not live in isolation but form large networks of interdependences that are often depicted as a set of nodes (species) connected by links (species interactions). This entangled web of life is increasingly threated by several drivers of global change (Tylianakis et al., 2008, 2010), such as climate warming (e.g., Memmott *et al.*, 2007), habitat loss and fragmentation (e.g., Tylianakis *et al.*, 2007), and invasive species (e.g., Aizen *et al.*, 2008). In this network context, species interactions are a component of biodiversity as important as species themselves (Thompson, 2009). We know we are losing species interactions (Aizen *et al.*, 2012), which may drive ecological communities towards tipping points (Lever *et al.*, 2014). We need, hence, a network thinking to predict future community-wide scenarios (Tylianakis *et al.*, 2008).

Since the 1980s, data on who interacts-with-whom in ecological communities have been compiled, focused mainly on food webs and plant-animal mutualistic networks. Data, however, can be found mainly as appendices or supplementary material in the papers where the authors originally published their work. The Interaction Web DataBase (www.nceas.ucsb.edu/interactionweb/) created in 2003 and hosted by the National Center for Ecological Analysis and Synthesis (NCEAS) at the University of California (USA) is, to our knowledge, the only initiative with the aim of centralizing the available information on ecological networks. However, as far as we know, it is currently outdated. In addition, because it only provides datasheets, searching options are not implemented. This limits the potential use of the database as a working tool to tackle scientific questions across networks. In addition, if we aim to reach stackeholders and policy makers, an easy way of visualizing the network dataset compiled throughout the world is of paramount importance to capture their attention.

# Features

We provide the most comprenhensive dataset of plant-animal mutualistic networks to date. In contrast to antagonistic interactions in which one species obtains a benefit at expenses of the other, in plant-animal mutualistic interactions, like those between a plant and a pollinator or between a plant and a seed disperser, both species obtain mutual benefit. Food webs and other ecological interactions such as host-parasite, host-parasitoid, plant-ant, plant-epiphyte, plant-herbivore, and anemone-fish interactions, will be available soon.

## Location map

The user interface is based on Google Maps. Over the map, colored circles indicate where the compiled networks are located. Colors depict the type of ecological interaction. The user can zoom in or out and drag the map.

## Network datasheet

Some basic information about the network can be obtained when the mouse pointer is over one of the colored circles. By left clicking on it, a detailed information about the network is dynamically generated: number of species, number of interactions, network connectance, locality, geographical coordinates, original reference, and a unique network identifier. The network identifier is intended to be adopted by the community of researchers as a unique tag to identify a given network across future studies. The network of interactions is also displayed along the species names (when they are identified). Depending on the data compiled by the researchers, a matrix of ones and zeros (presence/absence of the interaction, respectively) or a matrix of natural numbers indicating the number of visits performed by a pollinator species on a plant species, is displayed. A java script graphical representation using the D3js library is available for a quick visualization of the network.

## Data filtering criteria

Network selection can be filtered directly from the menu bar by selecting the type of interaction (up to now pollination or seed dispersal), type of data (binary: presence/absence of the interactions, or weighted: number of visits), number of species, and number of interactions. The list of networks selected as resulting from the filtering criteria applied is immediately ready for visualization or download. Searchers across networks by species names can be performed from the list of networks selected, which allows a meta-analysis never accomplished before.

## Data download

Network data can be downloaded from the location map and from the filtering criteria. Species names can also be included. The following file formats are available: comma-separated values (.csv), Excel spreadsheet format (.xls), Pajek format (.net) that can be imported in Gephi as well for visualization, and as Java Script Object Notation (.json). Downloading a large dataset could take some time because a zip file containing each network as a single file, a file for the references, and a readme file, is dynamically generated. A log file tracking the history of changes for the downloaded network dataset (if exists) is also included in the zip file.

# Implementation

The Web of Life has been designed and implemented in an open-source relational database management system (MySQL). This allows sophisticated and user-friendly searching across networks. It also provides an easy way of incorporating new network data available in the future. In order to minimize spelling mistakes when introducing new data, we do not provide an online interface for data entry, so that users cannot enter and edit their data directly. Users can access the database through any web browser using a variety of operating systems.

# Conclusion

Biodiversity is much more than a list of species. Interactions among species are the *glue* of biodiversity. If we aim at predicting future community-wide scenarios and anticipating planetary critical transitions, we have to consider the entangled web of interactions among species. Here, we introduce The Web of Life, a database for visualizing and downloading data of species interaction networks. This repository allows scientists to do research within and between networks compiled at different places all over the world.

# Acknowledgements

We would like to thank Jordi Bascompte's lab members for their suggestions during the design of the web, as well as the authors of the compiled networks for their invaluable effort during the fieldwork. This web service is supported by the European Research Council under the European Community's Seventh Framework Programme (FP7/2007-2013) through an Advanced Grant to Jordi Bascompte (grant agreement 268543). M.A.F. holds a postdoctoral fellowship (JAE-Doc) from the Program "Junta para la Ampliacion de Estudios" co-funded by the Fondo Social Europeo (FSE).

# References

Aizen, M. A., Morales, C., and Morales, J. (2008). Invasive mutualists erode native pollination webs. *PLoS Biology*, 6:e31.

Aizen, M. A., Sabatino, M., and Tylianakis, J. M. (2012). Specialization and rarity predict nonrandom loss of interactions from mutualist networks. *Science*, 335:1486-1489.

Lever, J., van Nes, E. H., Scheffer, M., and Bascompte, J. (2014). The sudden collapse of pollinator communities. *Ecol. Lett.*, 17:350-359.

Memmott, J., Craze, P. G., Waser, N. M., and Price, M. V. (2007). Global warming and the disruption of plant-pollinator interactions. *Ecol. Lett.*, 10:710-717.

Thompson, J. N. (2009). The coevolving web of life. *Am. Nat.*, 173:125-140.

Tylianakis, J. M., Tscharntke, T., and Lewis, O. T. (2007). Habitat modification alters the structure of tropical host-parasitoid food webs. *Nature*, 445:202-205.

Tylianakis, J. M., Didham, R. K., Bascompte, J., and Wardle, D. A. (2008). Global change and species interactions in terrestrial ecosystems. *Ecol. Lett.*, 11:1351-1363.

Tylianakis, J. M., Laliberté, E., Nielsen, A., and Bascompte, J. (2010). Conservation of species interaction networks. *Biol. Cons.*, 143:2270-2279.

# Figures

[^1]: To whom correspondence should be addressed. e-mail: firstname.lastname@example.com


abstract: Intraductal papillary mucinous neoplasm (IPMN) is a precursor to pancreatic ductal adenocarcinoma. While over half of patients are diagnosed with pancreatic cancer at a distant stage, patients who are diagnosed early enjoy a much higher 5-year survival rate of $34$% compared to $3$% in the former; hence, early diagnosis is key. Unique challenges in the medical imaging domain such as extremely limited annotated data sets and typically large 3D volumetric data have made it difficult for deep learning to secure a strong foothold. In this work, we construct two novel "inflated" deep network architectures, *InceptINN* and *DenseINN*, for the task of diagnosing IPMN from multisequence (T1 and T2) MRI. These networks inflate their 2D layers to 3D and bootstrap weights from their 2D counterparts (Inceptionv3 and DenseNet121 respectively) trained on ImageNet to the new 3D kernels. We also extend the inflation process by further expanding the pre-trained kernels to handle any number of input modalities and different fusion strategies. This is one of the first studies to train an end-to-end deep network on multisequence MRI for IPMN diagnosis, and shows that our proposed novel inflated network architectures are able to handle the extremely limited training data (139 MRI scans), while providing an absolute improvement of $\bm{8.76}$% in accuracy for diagnosing IPMN over the current state-of-the-art. Code is publicly available at <https://github.com/lalonderodney/INN-Inflated-Neural-Nets>.
author: Rodney LaLonde; Irene Tanner; Katerina Nikiforaki; Georgios Z. Papadakis; Pujan Kandel; Candice W. Bolan; Michael B. Wallace; Ulas Bagci
title: INN: Inflated Neural Networks
  for IPMN Diagnosis

# Introduction

Pancreatic cancer is currently the fourth leading cause of cancer-related death in the United States in both men and women, behind Lung & Bronchus, Breast (women)/Prostate (men), and Colon & Rectum. These latter four forms of cancer have all seen massive declines in mortality over the past decades; however, the same cannot be said for pancreatic cancer, which has seen an increase in both occurrence and mortality from 2006 to 2017. Worse still, pancreatic cancer continues to carry one of the poorest prognoses of any form of cancer at a grim $9\%$ 5-year survival rate . Because the cancer has usually spread beyond the pancreas by the time it is diagnosed, less than $20\%$ of patients are candidates for surgery. However, for cases in which diagnosis occurs while the disease is still local, which is true for approximately $10\%$ of patients, the 5-year survival rate has been steadily increasing from $29\%$ to $32\%$ to $34\%$ from 2017 to 2019 . For these reasons, early diagnosis will be vital to increase the five-year survival rate of pancreatic cancer and make surgery a viable option for more patients.

**Diagnosing Intraductal Papillary Mucinous Neoplasm**
Intraductal papillary mucinous neoplasm (IPMN) is a radiographically detectable neoplasm that is found in the main and branch pancreatic ducts and is often a precursor to pancreatic ductal adenocarcinoma. IPMN has the potential to progress into invasive carcinoma, as a large proportion of main duct-IPMN exhibits malignant progression. Example magnetic resonance imaging (MRI) scans with the associated grade of IPMN are shown in Figure , where the scans are cropped to a regions of interest (ROI) surrounding the pancreas. In this study, the cases of IPMN were graded in a pathology report after surgery: 0) normal, 1) low-grade IPMN, 2) high-grade IPMN, and 3) invasive carcinoma. Being able to accurately detect IPMN early may help diagnose pancreatic cancer sooner and, in turn, increase the abysmally low 5-year survival rates. While progress has been made in diagnosing and managing IPMN pre-operatively using the radiographic criteria and international consensus guidelines, automated computer-aided diagnosis (CAD) systems are a strong candidate for helping with this task. CAD systems could be used to distinguish IPMN in the pancreas and differentiate between grades potentially more efficiently and effectively than current standards.

**Literature on Classification of Pancreatic Cysts**
There are very few studies attempting to diagnose IPMN from MRI scans, and the presented work in this study is one of the first in literature to diagnose IPMN in MRI scans using deep learning. Sarfaraz et al. (2018) introduced an architecture for automatic IPMN classification using a pre-trained 3D convolutional neural network (CNN) to perform feature extraction with canonical correlation analysis and feature fusion. Unlike , our proposed approach allows for novel 3D network structures, intermediate fusion strategies, and end-to-end training.

Although the goal is not related to IPMN diagnosis, Chen et al. (2018) proposed PCN-Net, a study which can be considered a relevant work to ours because PCN-Net aims to classify pancreatic cystic neoplasms (PCNs). Briefly, the proposed network structure uses the 2D Inceptionv3 as a backbone, pre-trained on ImageNet and fine-tuned on optical coherence tomography images. MRI sequences are aligned with a Z-Continuity Filter and combined using a late fusion strategy. Unlike , our proposed approach utilizes 3D networks, again allows for intermediate fusion strategies, and does not require additional costly pre-training on external medical datasets. Furthermore, we focus on a relatively more challenging problem of IPMN diagnosis, not PCNs. Other studies in literature have focused on the use of pre-deep learning strategies such as conventional radiomics measurements (texture, intensity, and component enhancing features) to classify IPMNs. Not only is the success of these features fundamentally limited compared to deep learning-based approaches, but also these studies have focused on using CT images and require a priori segmentation of the pancreas , which itself is a challenging process.

**Proposed Solution: Multimodal Inflated Neural Networks**
While newer and more powerful deep network architectures continue to emerge, a difficulty arises when the new architecture differs in structure, preventing efficient weight transfer from pre-trained models. This is especially an issue in the medical imaging domain where annotated data is extremely limited and utilizing pre-trained deep networks often leads to dramatic performance increases. Additionally, due to the 3D nature of most medical imaging modalities, 3D networks tend to outperform their 2D counterparts, but most pre-trained networks exist only in 2D versions. There currently exists two solutions to this challenging problem. First, researchers can take their novel architecture and train on existing large datasets (e.g. ImageNet). This solution, however, is impractical and prohibitively expensive for many researchers. While some recent works in the literature have been proposed for shortening the typically multi-week process of training on ImageNet , these approaches typically require large clusters of GPUs for processing.

The other solution is to cannibalize existing pre-trained networks and transfer weights in a process often referred to as "network surgery". Carreira and Zisserman used this strategy to great effect in a process they called "network inflation" to inflate the Inceptionv1 network for the task of action recognition. The network inflation process swaps out all 2D convolutional kernels and pooling operations for their 3D counterparts and replicates the kernels along the third dimension, while dividing the value of the weights by the number of replications to preserve relatively similar activation values. Liu et al. (2018) proposed AH-Net, which inflated part of its architecture from 2D for lesion segmentation. In this work, we follow a similar strategy of network inflation to inflate deeper, more advanced, and more complicatedly-connected networks while also tackling the issue of transferring weights when we have multiple imaging modalities.

Several different fusion strategies exist for combining imaging modalities: early (pixel-level) fusion, intermediate fusion, and late fusion. In early fusion, images from different modalities are simply concatenated at the pixel level before being input to the network. In late fusion, inputs are fed to the network separately and the final embeddings are concatenated and fed to the final classification layers. Intermediate fusion is between early and late fusion, where information from multiple modalities are combined somewhere in the network. Which of these strategies is best depends on how similar the information is in each modality. Working with multiple modalities as input to a neural network adds an additional challenge for transferring weights from pre-trained networks, which typically accept only three-channel images as input. Transferring the weights of these earliest layers is critical for transfer learning, as these lowest-level layers have been found to share filters across virtually any imaging data, while higher-level layers become more specialized to the specific training data.

**Summary of Our Contributions**
**(1)** In this work, we construct two novel inflated deep network architectures, *InceptINN* and *DenseINN*, transferring weights from their 2D counterparts (Inceptionv3 and DenseNet121 respectively) trained on ImageNet to the new 3D kernels. **(2)** We extend the inflation process by further expanding the pre-trained kernels to handle any number of input modalities and different fusion strategies. In general, this modification is the key component for any image-based diagnosis system accepting multiple image modalities as input. Particularly for our study, since T1- and T2-weighted MRI scans are the modality of choice for IPMN diagnosis, there is a strong need to combine the complementary information of both imaging modalities. The proposed network architecture provides the needed flexibility both at the input and fusion levels. **(3)** We investigate both early fusion as well as a version of intermediate fusion in greater detail and perform image-based diagnosis experiments at both the whole-MRI and pancreas-ROI levels for comprehensive evaluations, which have never been done in the literature before. **(4)** To our best of knowledge, our study is also the first one to train an end-to-end deep network on multisequence MRI for IPMN diagnosis. **(5)** Finally, we demonstrate that our newly designed inflated network architectures have a major advantage over the state-of-the-art by utilizing the extremely limited training data (139 MRI scans) while still obtaining an absolute improvement in accuracy for diagnosing IPMN of $\bm{8.76}$%.

# Methods

The overview of our proposed framework is shown in Fig. . During training and testing, $k$ slices from either whole MRI scans or cropped pancreas-ROIs are used as input to one of our deep networks. Prior to entering the network, MRI scans are preprocessed by first aligning the T1 scans to the T2 scans using b-spline registration, then performing N4 bias field correction and applying a curvature anisotropic image filter. Following this, the $k$ slices of each modality are either concatenated along the channel axis ($height \times width \times slices \times channels$) and input to the network or are input separately, depending on if the early or intermediate fusion strategy is being followed. If intermediate fusion is chosen, each modality is passed through its own convolutional layer before being concatenated and passed through the remainder or the network. At the end of the network, either *InceptINN* or *DenseINN*, a softmax output over three values is obtained, representing the probability of our three possible grades of IPMN.

**INN: Inflating Neural Networks**
INN is a relatively straightforward two-step process. A 2D network is chosen for which pre-trained weights are available without needing major modifications to the overall network architecture that cannot be corrected with basic network surgery. The first step simply requires changing all 2D layers to their corresponding 3D counterparts (e.g. 2D convolution becomes 3D convolution, 2D pooling becomes 3D pooling). The choice of convolutional kernel size, pooling size, or stride length in this new third dimension is somewhat application dependant. For example, if training on a 30 fps video, one can afford larger kernels, strides, and pooling in the third dimension. For the application of IPMN diagnosis in MRI, since the pancreas typically only occupies a relatively small number of slices, we chose to favor strides of one in this new third dimension. For square kernels, we chose to extend these to cubes; however, for the $h \times 1$ or $1 \times w$ kernels, we chose to extend the third dimension to be $h \times 1 \times 1$ or $1 \times w \times 1$.

The second step is to transfer the weights from the 2D kernels to the 3D kernels by bootstrapping them along the third dimension. This is accomplished by tiling the weights along the new dimension, then dividing all of the values of the kernel by the new depth of the kernel. This division is important to keep the network activations approximately the same from layer to layer. It may be straightforward when working with RGB images, but it can quickly become complicated with non-three-channel images and multiple imaging modalities.

**Handling Multiple Imaging Modalities**
There is a critical limitation of not having the proper number of kernels in the first convolutional layer when using a 2D network pre-trained on ImageNet with non-RGB images. This becomes even more worrisome when you want to extend to multiple imaging modalities. In this work, we introduce the following strategies for handling multiple modalities while still allowing for the transfer of pre-trained weights. First, since the majority of medical imaging modalities (e.g. MRI, computed tomography) are single-channel, we tile the images to create three-channel images. This would be sufficient for a single modality input, but not for multiple modalities. If we are following the early fusion strategy, these three-channel images are concatenated prior to input into the network. Therefore, for each modality $M$, we create a copy of the first layer's kernels not only along the third dimension during the inflation process, but also along the input channel axis for the number of modalities given ($kern_h \times kern_w \times kern_d * 1 \times |M| * channels)$. This also increases the amount we divide the kernel values by a factor of $|M|$.

If we are following the intermediate fusion strategy, the original first convolutional layer simply transfers copies of its weights (now also tiled along the new third dimension) to each modalities' individual convolution layer. These initial layer kernels are only divided by the length of the new third dimension. After these layers, the results are all concatenated and fed into the remainder of the network. The first layer after concatenation is now the one which must have $M$ copies made of its kernels along the input channel dimension and its values divided by $|M|$. *DenseINN* adds some further complications due to the addition of concatenation layers starting from the second layer, but careful bookkeeping avoids any confusion.

| Method |  |  | Pre (SEM)% |  | Rec (SEM)% |  | Acc (SEM)% |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| Hussein et al. |  |  | \- |  | \- |  | $64.67$ ($0.83$) |
| Baseline *InceptINN* Whole-MRI |  |  | $61.59$ ($5.83$) |  | $58.18$ ($4.05$) |  | $61.87$ ($2.83$) |
| *InceptINN* Whole-MRI |  |  | $74.51$ ($4.70$) |  | $\bm{71.24}$ ($4.39$) |  | $73.38$ ($3.52$) |
| Baseline *InceptINN* Pancreas-ROI |  |  | $68.49$ ($2.63$) |  | $69.18$ ($3.40$) |  | $69.37$ ($3.11$) |
| *InceptINN* Pancreas-ROI |  |  | $71.68$ ($2.25$) |  | $70.11$ ($1.33$) |  | $72.32$ ($0.97$) |
| Baseline *DenseINN* Pancreas-ROI |  |  | $66.81$ ($4.51$) |  | $67.05$ ($3.06$) |  | $67.16$ ($2.93$) |
| *DenseINN* Pancreas-ROI |  |  | $\bm{78.20}$ ($4.17$) |  | $69.09$ ($2.97$) |  | $\bm{73.43}$ ($2.26$) |

Experimental results for IPMN diagnosis. Precision, recall, and accuracy are shown with the standard error across the ten-fold cross-validation splits. The proposed *INN* networks outperforms their 3D trained from scratch versions (baselines) and the previous state-of-the-art at both the pancreas-ROI and whole-MRI levels.

|  |  |  |  |  |  |  |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| Method | Early Fusion |  |  | Intermediate Fusion |  |  |
|  | Pre % | Rec % | Acc % | Pre % | Rec % | Acc % |
| *InceptINN* Whole-MRI | $69.44$ | $66.35$ | $73.33$ | $66.67$ | $59.68$ | $66.67$ |
| *InceptINN* Pancreas-ROI | $79.29$ | $\bm{80.48}$ | $78.57$ | $70.77$ | $69.66$ | $75.00$ |
| *DenseINN* Pancreas-ROI | $73.08$ | $73.08$ | $75.00$ | $\bm{88.10}$ | $75.21$ | $\bm{82.14}$ |

Examining the role of different fusion strategies. Precision, recall, and accuracy are shown comparing early versus intermediate fusion on a single training fold.

# Experiments and Results

Images were split across three categories: normal (i.e. no IPMN was present), low-grade IPMN, and high-grade IPMN/invasive carcinoma. Whole-MRI experiments used 139 scans, of which 29 were normal, 45 were low-grade, and 65 were high-grade. Due to the challenging nature of cropping the pancreas-ROI, two sets of crops were extracted by two experts. In cases where an expert could not confidently extract a pancreas-ROI, this scan was skipped, yielding 271 pancreas-ROIs. Experiments were carried out corresponding to each of the '-or-'s in our framework overview shown in Fig. . The results are summarized in Table  for the intermediate fusion strategy. For all experiments, slices is set to $k = 5$, excluding the experiments specifically examining the effect of $k$ shown in Table . Unless otherwise specified, all experiments were conducted with stratified 10-fold cross-validation. Although the main focus of this work is on inflated networks and not modality fusion strategies, a set of experiments was conducted across one training fold to determine the relative performance of the two fusion strategies, with results shown in Table . Note, experiments were not performed with *DenseINN* for whole-MRI due to memory constraints. Whole-MRI images are resized in-plane to $256 \times 256$ and pancreas-ROIs are resized to $128 \times 128$.

All training and testing was performed using Keras with TensorFlow on a single Titan-X GPU with 12G memory. The Adam optimizer was used with its default parameters, early stopping, and learning rate reduction by $0.05$ on loss plateau. At training, input batches are formed by first sliding through each set of $k$ slices containing the pancreas in a given scan, before moving on to the next scan. At testing, $k$ slices are chosen around a central slice, where this slice is determined as the one in which the pancreas appears the largest. For pancreas-ROI images, a batch size of 32 was used for both networks. Due to memory limitations, *InceptINN* used a batch size of 16 when using the whole-MRI. The average test time for InceptINN per patient on a single Titan X GPU is 0.2 sec for whole-slide MRI and 0.05 seconds for pancreas-ROI of MRI.

|  |  |  |  |  |  |  |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| $k$ slices | Whole-MRI |  |  | Pancreas-ROI |  |  |
|  | Pre% | Rec% | Acc% | Pre% | Rec% | Acc% |
| $k = 3$ | $57.22$ | $56.83$ | $60.00$ | $82.83$ | $\bm{83.76}$ | $\bm{85.71}$ |
| $k = 5$ | $\bm{66.67}$ | $\bm{59.68}$ | $\bm{66.67}$ | $70.77$ | $69.66$ | $75.00$ |
| $k = 7$ | \- | \- | \- | $\bm{85.45}$ | $81.91$ | $82.14$ |

Examining the role of 3D context information. Precision, recall, and accuracy are shown for *InceptINN* on a single training fold on both whole-MRI and pancreas-ROI inputs. Note: the whole-MRI level at $k = 7$ slices is unable to fit into GPU memory, even at a reduced batch size, and thus was not included in experiments.

# Discussion and Concluding Remarks

In this work, we demonstrate the effectiveness of inflating deep CNNs for the task of IPMN diagnosis, and train two different end-to-end, novel network architectures on the extremely limited multisequence MRI data set. Our proposed networks, *InceptINN* and *DenseINN*, outperform the previous state-of-the-art by over $8\%$ when operating either on a cropped bounding box around the pancreas or when using the entire MRI. One fundamental advantage of the proposed approach is that it is flexible enough to adapt almost any network structures to volumetric data while any number of input modalities can be handled. We have also showed that while *InceptINN* tended to favor the early fusion approach, *DenseINN* favored the intermediate fusion approach instead. Although our observations were empirical, full cross validation experiments can be performed as an extension of this study to determine if this is a data driven or network structure driven preference.