Question: Represent this text!

Question: We propose Adversarial Inductive Transfer Learning (AITL), a method for addressing discrepancies in input and output spaces between source and target domains. AITL utilizes adversarial domain adaptation and multi-task learning to address these discrepancies. Our motivating application is pharmacogenomics where the goal is to predict drug response in patients using their genomic information. The challenge is that clinical data (i.e. patients) with drug response outcome is very limited, creating a need for transfer learning to bridge the gap between large pre-clinical pharmacogenomics datasets (e.g. cancer cell lines) and clinical datasets. Discrepancies exist between 1) the genomic data of pre-clinical and clinical datasets (the input space), and 2) the different measures of the drug response (the output space). To the best of our knowledge, AITL is the first adversarial inductive transfer learning method to address both input and output discrepancies. Experimental indicate that AITL outperforms state-of-the-art pharmacogenomics and transfer learning baselines and may guide precision oncology more accurately. Deep neural networks have demonstrated the state-of-the-art performance in different problems, ranging from computer vision and natural language processing to genomics and medicine . However, these networks often require a large number of samples for training, which is challenging and sometimes impossible to obtain in the real world applications. Transfer learning attempts to solve this challenge by leveraging the knowledge in a source domain, a large data-rich dataset, to improve the generalization performance on a small target domain. Training a model on the source domain and testing it on the target domain violates the i.i.d assumption that the train and test data are from the same distribution. The discrepancy in the input space decreases the prediction accuracy on the test data, which leads to poor generalization . Many methods have been proposed to minimize the discrepancy between the source and the target domains using different metrics such as Jensen Shannon Divergence , Maximum Mean Discrepancy , and Margin Disparity Discrepancy . While transductive transfer learning (e.g. domain adaptation) uses a labeled source domain to improve generalization on an unlabeled target domain, inductive transfer learning (e.g. few-shot learning) uses a labeled source domain to improve the generalization on a labeled target domain where label spaces are different in the source and the target domains . Adversarial domain adaptation has shown great performance in addressing the discrepancy in the input space for different applications (; ; ; ; ; ;, however, adversarial adaptation to address the discrepancies in both the input and output spaces has not yet been explored. Our motivating application is pharmacogenomics where the goal is to predict response to a cancer drug given the genomic data (e.g. gene expression). Since clinical datasets in pharmacogenomics (patients) are small and hard to obtain, many studies have focused on large pre-clinical pharmacogenomics datasets such as cancer cell lines as a proxy to patients . A majority of the current methods are trained on cell line datasets and then tested on other cell line or patient datasets ). However, cell lines and patients data, even with the same set of genes, do not have identical distributions due to the lack of an immune system and the tumor microenvironment in cell lines . Moreover, in cell lines, the response is often measured by the drug concentration that reduces viability by 50% (IC50), whereas in patients, it is often based on changes in the size of the tumor and measured by metrics such as response evaluation criteria in solid tumors (RECIST) . This means that drug response prediction is a regression problem in cell lines but a classification problem in patients. Therefore, discrepancies exist in both the input and output spaces in pharmacogenomics datasets. Table A1 provides the definition of these biological terms. In this paper, we propose Adversarial Inductive Transfer Learning (AITL), the first adversarial method of inductive transfer learning. Different from existing methods for transfer learning, AITL adapts not only the input space but also the output space. Our motivating application is transfer learning for pharmacogenomics datasets. In our driving application, the source domain is the gene expression data obtained from the cell lines and the target domain is the gene expression data obtained from patients. Both domains have the same set of genes (i.e., raw feature representation). Discrepancies exist between the gene expression data in the input space, and the measure of the drug response in the output space. AITL learns features for the source and target samples and uses these features as input for a multi-task subnetwork to predict drug response for both the source and the target samples. The output space discrepancy is addressed by the multi-task subnetwork, which has one shared layer and separate classification and regression towers, and assigns binary labels (called cross-domain labels) to the source samples. The multi-task subnetwork also alleviates the problem of small sample size in the target domain by sharing the first layer with the source domain. To address the discrepancy in the input space, AITL performs adversarial domain adaptation. The goal is that features learned for the source samples should be domain-invariant and similar enough to the features learned for the target samples to fool a global discriminator that receives samples from both domains. Moreover, with the cross-domain binary labels available for the source samples, AITL further regularizes the learned features by class-wise discriminators. A class-wise discriminator receives source and target samples from the same class label and should not be able to predict the domain accurately. We evaluated the performance of AITL and state-of-the-art inductive and adversarial transductive transfer learning baselines on pharmacogenimcs datasets in terms of the Area Under the Receiver Operating Characteristic curve (AUROC) and the Area Under the Precision-Recall curve (AUPR). In our experiments, AITL achieved a substantial improvement compared to the baselines, demonstrating the potential of transfer learning for drug response prediction, a crucial task of precision oncology. Following the notation of , a domain like DM is defined by a raw input feature space 1 X and a probability distribution p(X), where X = {x 1, x 2, ..., x n} and x i is the i-th raw feature vector of X. A task T is associated with DM = {X, p(X)}, where T = {Y, F } is defined by a label space Y and a predictive function F which is learned from training data of the form (X, Y), where X ∈ X and Y ∈ Y. A source domain is defined as DM S = {(x s1, y s1), (x s2, y s2),..., (x sn S, y sn S)} and a target domain is defined as DM T = {(x t1, y t1), (x t2, y t2),..., (x tn T, y tn T)}, where x s ∈ X S, x t ∈ X T, y s ∈ Y S, and y t ∈ Y T. Since n T << n S and it is challenging to train a model only on the target domain, transfer learning aims to improve the generalization on a target task T T using the knowledge in DM S and DM T and their corresponding tasks T S and T T. Transfer learning can be categorized into three categories: 1) unsupervised transfer learning, 2) transductive transfer learning, and 3) inductive transfer learning. In unsupervised transfer learning, there is no label in the source and target domains. In transductive transfer learning, source domain is labeled but target domain is unlabeled, domains can be either the same or different (domain adaptation), but source and target tasks are the same. In inductive transfer learning, target domain is labeled and source domain can be either labeled or unlabeled, and domains can be the same or different, but in this category tasks are always different . There are three approaches to inductive transfer learning: 1) deep metric learning, 2) few-shot learning, and 3) weight transfer . Deep metric learning methods are independent of the number of samples in each class of the target domain, denoted by k, meaning that they work for small and large k values. Few-shot learning methods focus on small k (k ≤ 20). Finally, weight transfer methods require a large k (k ≥ 100 or k ≥ 1000) . Figure A1 (in Appendix) presents this taxonomy. In drug response prediction, the target domain is small, which means a limited number of samples for each class is available, therefore, few-shot learning is more suitable for such a problem. Fewshot learning involves training a classifier to recognize new classes, provided only a small number of examples from each of these new classes in the training data. Various methods have been proposed for few-shot learning (; ;). For example, Prototypical Networks (ProtoNet) constructs prototypical representatives (class means) from source domain learned features and compares the Euclidean distance between the target domain learned features and these class representatives to assign labels to the target samples. Recent advances in adversarial learning leverage deep neural networks to learn transferable representation that disentangles domain-invariant and class-invariant features from different domains and matches them properly (; ;). In this section, we first introduce the Generative Adversarial Networks (GANs) , and then introduce some of the existing works on adversarial transfer learning. GANs attempt to learn the distribution of the input data via a minimax framework where two networks are competing: a discriminator D and a generator G. The generator tries to create fake samples from a randomly sampled latent variable that fool the discriminator, while the discriminator tries to catch these fake samples and discriminate them from the real ones. Therefore, the generator wants to minimize its error, while the discriminator wants to maximize its accuracy: A majority of literature on adversarial transfer learning are for transductive transfer learning where the source domain is labeled while the target domain is unlabeled. Transductive transfer learning, often referred to as domain adaptation, is the most common scenario in transfer learning. Various methods have been proposed for adversarial transductive transfer learning in different applications such as image segmentation ), image classification ), speech recognition , domain adaptation under label-shift , partial domain adaptation , and multiple domain adaptation . The idea of these methods is that features extracted from source and target samples should be similar enough to fool a global discriminator and/or class-wise discriminators. The goal of precision oncology is to tailor a treatment for a cancer patient using genomic information of that patient. However, currently, only about 5% of the patients can benefit from precision oncology because response to a drug is a highly complex phenotype and it depends on diverse genetic and/or non-genetic factors . Pre-clinical pharmacogenomics datasets such as cancer cell lines and patientderived xenografts (PDX) are reliable proxies to study the associations between the genomic landscape and the response to a cancer treatment. The advantage of these resources is that they can be screened with hundreds of drugs -chemotherapy agents and targeted therapeutics -which is impossible for patients. For example, in the Genomics of Drug Sensitivity in Cancer (GDSC) dataset over 1000 pan-cancer cell lines screened with 265 chemotherapy and targeted drugs are available. Another advantage of the pre-clinical datasets is that they are often significantly larger than patient datasets with known drug response (labels). These advantages of pre-clinical datasets make them a suitable resource to develop computational methods for drug response prediction . Various methods have been developed to predict drug response from single or multiple types of genomic data. For example, proposed a ridge-regression method to predict drug response based on gene expression data. showed that integrating multiple data types with deep neural networks and transfer learning via sample transfer improves the accuracy of drug response prediction. Given a labeled source domain DM S with a learning task T S and a labeled target domain DM T with a learning task T T, where T T = T S, and p(X T) = p(X S), where X S, X T ∈ X, we assume that the source and the target domains are not the same due to different probability distributions. The goal of Adversarial Inductive Transfer Learning (AITL) is to utilize the source and target domains and their tasks in order to improve the learning of In the area of pharmacogenomics, the source domain is the gene expression data obtained from the cell lines, and the source task is to predict the drug response in the form of IC50 values. The target domain consists of gene expression data obtained from patients, and the target task is to predict drug response in a different form -often change in the size of tumor after receiving the drug. In this setting, p(X T) = p(X S) because cell lines are different from patients even with the same set of genes. Additionally, Y T = Y S because for the target task Y T ∈ {0, 1}, drug response in patients is a binary outcome, but for the source task Y S ∈ R +, drug response in cell lines is a continuous outcome. As a , AITL needs to address these discrepancies in the input and output spaces. Our proposed AITL method takes input data from the source and target domains, and achieves the following three objectives: first, it makes predictions for the target domain using both of the input domains and their corresponding tasks, second, it addresses the discrepancy in the output space between the source and target tasks, and third, it addresses the discrepancy in the input space. AITL is a neural network consisting of four components: • The feature extractor receives the input data from the source and target domains and extracts salient features, which are then sent to the multi-task subnetwork component. • The multi-task subnetwork takes the extracted features of source and target samples and maps them to their corresponding labels and makes predictions for them. This component has a shared layer and two task-specific towers for regression (source task) and classification (target task). Therefore, by training the multi-task subnetwork on the source and target samples, it addresses the small sample size challenge in the target domain. In addition, it also addresses the discrepancy in the output space by assigning cross-domain labels (binary labels in this case) to the source samples (for which only continuous labels are available) using its classification tower. • The global discriminator receives extracted features of source and target samples and predicts if an input sample is from the source or the target domain. To address the discrepancy in the input space, these features should be domain-invariant so that the global discriminator cannot predict their domain labels accurately. This goal is achieved by adversarial learning. • The class-wise discriminators further reduce the discrepancy in the input space by adversarial learning at the level of the different classes, i.e., extracted features of source and target samples from the same class go to the discriminator for that class and this discriminator should not be able to predict if an input sample from a given class is from the source or the target domain. The AITL cost function consists of a classification loss, a regression loss, and global and class-wise discriminator adversarial losses and is optimized end-to-end. An overview of the proposed method is presented in figure 1. Figure 1: Overview of AITL: First, the feature extractor receives source and target samples and learns feature for them. Then, the multi-task subnetwork uses these features to make predictions for the source and target samples and also assigns cross-domain labels to the source samples. The multi-task subnetwork addresses the discrepancy in the output space. Finally, to address the input space discrepancy, global and class-wise discriminators receive the extracted features and regularize the feature extractor to learn domain-invariant features. To learn salient features in lower dimensions for the input data, we design a feature extractor component. The feature extractor is a one-layer fully-connected subnetwork with batch normalization and the ReLU activation function that receives both the source and target samples as input. We denote the feature extractor as f : where Z denotes the extracted features for input X which is from either the source (S) or the target (T) domain. In our driving application, the feature extractor learns features for the cell line and patient data. After extracting features of the input samples, we want to use these learned features to 1) make predictions for target samples, and 2) address the discrepancy between the source and the target domains in the output space. To achieve these goals, a multi-task subnetwork with a shared layer g and two task-specific towers M S and M T is designed, where M S is for regression (the source task) and M T is for classification (the target task): The performance of the multi-task subnetwork component is evaluated based on a binary-cross entropy loss for the classification task on the target samples and a mean squared loss for the regression task on the source samples: Where Y S and Y T are the true labels of the source and the target samples, respectively, and L BCE and L M SE are the corresponding losses for the target and the source domains, respectively. The multi-task subnetwork component outputs 1) the predicted labels for the target samples, and 2) the assigned cross-domain labels for the source samples. The classification tower in the multitask subnetwork makes predictions for the source samples and assigns binary labels (responder or non-responder) because such labels do not exist for the source samples. Therefore, the multi-task subnetwork adapts the output space of the source and the target domains by assigning cross-domain labels to the source domain. The multi-task subnetwork has a shared fully-connected layer with batch normalization and the ReLU activation function. The regression tower has two layers with batch normalization and the ReLU activation function. The classification tower also has two fully connected layer with batch normalization and the ReLU activation function in the first layer and the Sigmoid activation function in the second layer. In our driving application the multi-task subnetwork predicts IC50 values for the cell lines and the binary response outcome for the patients. Moreover, it also assigns binary labels to the cell lines which is similar to those of the patients. The goal of this component is to address the discrepancy in the input space by adversarial learning of domain-invariant features. To achieve this goal, a discriminator receives source and target extracted features from the feature extractor and classifies them into their corresponding domain. The feature extractor should learn domain-invariant features to fool the global discriminator. In our driving application the global discriminator should not be able to recognize if the extracted features of a sample are from a cell line or a patient. This discriminator is a one-layer subnetwork with the Sigmoid activation function denoted by D G . The adversarial loss for D G is as follows: With cross-domain binary labels available for the source domain, AITL further reduces the discrepancy between the input domains via class-wise discriminators. The goal is to learn domain-invariant features with respect to specific class labels such that they fool corresponding class-wise discriminators. Therefore, extracted features of the target samples in class i, and those of the source domain which the multi-task subnetwork assigned to class i, will go to the discriminator for class i. We denote such a class-wise discriminator as DC i. The adversarial loss for DC i is as follows: In our driving application the class-wise discriminator for the responder samples should not be able to recognize if the extracted features of a responder sample are from a cell line or a patient (similarly for a non-responder sample). Similar to the global discriminator, class-wise discriminators are also one-layer fully-connected subnetworks with the Sigmoid activation function. To optimize the entire network in an end-to-end fashion, we design the cost function as follows: Where, λ G and λ DC are adversarial regularization coefficients for the global and class-wise discriminators, respectively. In our experiments, we used the following datasets (See Table A2 in the Appendix for more detail): The GDSC dataset was used as the source domain, and all the other datasets were used as the target domain. For the GDSC dataset, raw gene expression data were downloaded from ArrayExpress (E-MTAB-3610) and release 7.0 of the dataset was used to obtain the response outcome. Gene expression data of TCGA patients were downloaded from the Firehose Broad GDAC and the response outcome was obtained from . Patient datasets from clinical trials were obtained from the Gene Expression Omnibus (GEO), and the PDX dataset was obtained from the supplementary material of . For each drug, we selected those patient datasets that applied a comparable measure of the drug response. For preprocessing, the same procedure was adopted as described in the supplementary material of for the raw gene expression data (normalized and z-score transformed) and the drug response data. After the preprocessing, source and target domains had the same number of genes. We designed our experiments to answer the following three questions: 1. Does AITL outperform baselines that are trained only on cell lines and then evaluated on patients (without transfer learning)? To answer this question, we compared AITL against and ) (MOLI) which are state-of-the-art methods of drug response prediction that do not perform domain adaptation. 2. Does AITL outperform baselines that adopt adversarial transductive transfer learning (without adaptation of the output space)? To answer this question, we compared AITL against ) (ADDA) and, state-of-the-art methods of adversarial transductive transfer learning with global and class-wise discriminators, respectively. 3. Does AITL outperform a baseline for inductive transfer learning? To answer this last question, we compared AITL against (ProtoNet) which is the state-of-the-art inductive transfer learning method for small numbers of examples per class. Based on the availability of patient/PDX datasets for a drug, we experimented with four different drugs: Bortezomib, Cisplatin, Docetaxel, and Paclitaxel. It is important to note that these drugs have different mechanisms and are being prescribed for different cancers. For example, Docetaxel is a chemotherapy drug mostly known for treating breast cancer patients , while Bortezomib is a targeted drug mostly used for multiple myeloma patients . Therefore, the datasets we have selected cover different types of anti-cancer drugs. In addition to the experimental comparison against published methods, we also performed an ablation study to investigate the impact of the different AITL components separately. AITL−AD denotes a version of AITL without the adversarial adaptation components, which means the network only contains the multi-task subnetwork. AITL−D G denotes a version of AITL without the 0.54±0.07 0.60±0.14 0.52±0.02 0.58±0.04 ADDA 0.51±0.06 0.56±0.06 0.48±0.06 did not converge ProtoNet 0 global discriminator, which means the network only employs the multi-task subnetwork and classwise discriminators. AITL−DC denotes a version of AITL without the class-wise discriminators, which means the network only contains the multi-task subnetwork and the global discriminator. All of the baselines were trained on the same data, tested on patients/PDX for these drugs, and eventually compared to AITL in terms of prediction AUROC and AUPR. Since the majority of the studied baselines cannot use the continuous IC50 values in the source domain, binarized IC50 labels provided by using the Waterfall approach were used to train them. Finally, for the minimax optimization, a gradient reversal layer was employed by AITL and the adversarial baselines . We performed 3-fold cross validation in the experiments to tune the hyper-parameters of AITL and the baselines based on the AUROC. Two folds of the source samples were used for training and the third fold for validation, similarly, two folds of the target samples were used for training and validation, and the third one for the test. The hyper-parameters tuned for AITL were the number of nodes in the hidden layers, learning rates, mini-batch size, weight decay coefficient, the dropout rate, number of epochs, and the regularization coefficients. We considered different ranges for each hyper-parameter and the final selected hyper-parameter settings for each drug and each method are provided in Section A.2 in the Appendix. Finally, each network was re-trained on the selected settings using the train and validation data together for each drug. We used Adagrad for optimizing the parameters of AITL and the baselines implemented in the PyTorch framework, except for the method of which was implemented in R. We used the author's implementations for the method of , MOLI, and ProtoNet. For ADDA, we used an existing implementation from https://github.com/jvanvugt/pytorch-domain-adaptation, and we implemented the method of from scratch. Tables 1 and A3 (Appendix) and Figure 2 report the performance of AITL and the baselines in terms of AUROC and AUPR, respectively. To answer the first experimental question, AITL was compared to the baselines which do not use any adaptation (neither the input nor the output space), i.e. the method of and MOLI, and AITL demonstrated a better performance in both AUROC and AUPR for all of the studied drugs. This indicates that addressing the discrepancies in the input and output spaces leads to better performance compared to training a model on the source domain and testing it on the target domain. To answer the second experimental question, AITL was compared to state-of-the-art methods of adversarial transductive transfer learning, i.e. ADDA and the method of, which address the discrepancy only in the input space. AITL achieved significantly better performance in AUROC for all of the drugs and for three out of four drugs in AUPR (the of for Cisplatin were very competitive with AITL). This indicates that addressing the discrepancies in the both spaces outperforms addressing only the input space discrepancy. Finally, to answer the last experimental question, AITL was compared to ProtoNet ) -a representative of inductive transfer learning with input space adaptation via few-shot learning. AITL outperformed this method in all of the metrics for all of the drugs. We note that the methods of drug response prediction without adaptation, namely the method of and MOLI, outperformed the method of inductive transfer learning based on few-shot learning (ProtoNet). Moreover, these two methods also showed a very competitive performance compared to the methods of adversarial transductive transfer learning (ADDA and the method of). For Paclitaxel, ADDA did not converge in the first step (training a classifier on the source domain), which was also observed in another study. ProtoNet also did not converge for this drug. We observed that AITL, using all of its components together, outperforms all the additional baselines omitting some of the components. This indicates the importance of both input and output space adaptation. The only exception was for the drug Paclitaxel, where AITL−D G outperforms AITL. We believe the reason for this is that this drug has the most heterogeneous target domain (see Table A1 in the appendix), and therefore the global discriminator component of AITL causes a minor decrease in the performance. All these indicate that addressing the discrepancies in the input and output spaces between the source and target domains, via the AITL method, leads to a better prediction performance. To our surprise, ProtoNet and ADDA could not outperform the method of and MOLI baselines. For ProtoNet, this may be due to the depth of the backbone network. A recent study has shown that a deeper backbone improves ProtoNet performance drastically in image classification. However, in pharmacogenomics, employing a deep backbone is not realistic because of the much smaller sample size compared to an image classification application. Another limitation for ProtoNet is the imbalanced number of training examples in different classes in pharmacogenomics datasets. Specifically, the number of examples per class in the training episodes is limited to the number of samples of the minority class as ProtoNet requires the same number of examples from each class. For ADDA, this lower performance may be due to the lack of end-to-end training of the classifier along with the global discriminator of this method. The reason is that end-to-end training of the classifier along with the discriminators improved the performance of the second adversarial baseline in AUROC and AUPR compared to ADDA. Moreover, the method of ) also showed a relatively better performance in AUPR compared to the method of and MOLI. In pharmacogenomics, patient datasets are small or not publicly available due to privacy and/or data sharing issues. We believe including more patient samples and more drugs will increase generalization capability. In addition, recent studies in pharmacogenomics have shown that using multiple genomic data types (known as multi-omics in genomics) works better than using only gene expression. In this work, we did not consider such data due to the lack of patient samples with multi-omics and drug response data publicly available; however, in principle, AITL also works with such data. Last but not least, we used pharmacogenomics as our motivating application for this new problem of transfer learning, but we believe that AITL can also be employed in other applications. For example, in slow progressing cancers such as prostate cancer, large patient datasets with gene expression and short-term clinical data (source domain) are available, however, patient datasets with long-term clinical data (target domain) are small. AITL may be beneficial to learn a model to predict these long-term clinical labels using the source domain and its short-term clinical labels (a). Moreover, AITL can also be applied to the diagnosis of rare cancers with a small sample size. Gene expression data of prevalent cancers with a large sample size, such as breast cancer, may be beneficial to learn a model to diagnose these rare cancers. In this paper, we introduced a new problem in transfer learning motivated by applications in pharmacogenomics. Unlike domain adaptation that only requires adaptation in the input space, this new problem requires adaptation in both the input and output spaces. To address this problem, we proposed AITL, an Adversarial Inductive Transfer Learning method which, to the best of our knowledge, is the first method that addresses the discrepancies in both the input and output spaces. AITL uses a feature extractor to learn features for target and source samples. Then, to address the discrepancy in the output space, AITL utilizes these features as input of a multi-task subnetwork that makes predictions for the target samples and assign cross-domain labels to the source samples. Finally, to address the input space discrepancy, AITL employs global and class-wise discriminators for learning domain-invariant features. In our motivating application, pharmacogenomics, AITL adapts the gene expression data obtained from cell lines and patients in the input space, and also adapts different measures of the drug response between cell lines and patients in the output space. In addition, AITL can also be applied to other applications such as rare cancer diagnosis or predicting long-term clinical labels for slow progressing cancers. We evaluated AITL on four different drugs and compared it against state-of-the-art baselines from three categories in terms of AUROC and AUPR. The empirical indicated that AITL achieved a significantly better performance compared to the baselines showing the benefits of addressing the discrepancies in both the input and output spaces. We conclude that AITL may be beneficial in pharmacogenomics, a crucial task in precision oncology. For future research directions, we believe that the TCGA dataset consisting of gene expression data of more than 12,000 patients (without drug response outcome) can be incorporated in an unsupervised transfer learning setting to learn better domain-invariant features between cell lines and cancer patients. In addition, we did not explore the impact of the chemical structures of the studied drugs in the prediction performance. We believe incorporating this input with transfer learning in the genomic level can lead to a better performance. Currently, AITL borrows information between the input domains indirectly via its multi-task subnetwork and assignment of cross-domain labels. An interesting future direction can be to exchange this information between domains in a more explicit way. Moreover, we also did not perform theoretical analysis on this new problem of transfer learning and we leave it for future work. Finally, we did not distinguish between different losses in the multi-task subnetwork, however, in reality patients are more important than cell lines, and considering a higher weight for the corresponding loss in the cost function can improve the prediction performance. A.1 SUPPLEMENTARY TABLES Patient-Derived Xenografts (PDX) Tumor tissue taken from a patient and implanted into mice to mimic the microenvironment around the tumor. Chemotherapy drugs A type of treatment that stops cancer cells' growth by killing them or stopping them from dividing. Targeted drugs A type of treatment that is designed for a specific type(s) of cancer cells with minor effect on the other cell types. clinical trial Bortezomib target 169 GDSC cell line Bortezomib source 391 GSE18864 clinical trial Cisplatin target 24 GSE23554 clinical trial Cisplatin target 28 TCGA patient Cisplatin target 66 GDSC cell line Cisplatin source 829 GSE25065 clinical trial Docetaxel target 49 GSE28796 clinical trial Docetaxel target 12 GSE6434 clinical trial Docetaxel target 24 TCGA patient Docetaxel target 16 GDSC cell line Docetaxel source 829 GSE15622 clinical trial Paclitaxel target 20 GSE22513 clinical trial Paclitaxel target 14 GSE25065 clinical trial Paclitaxel target 84 PDX animal (mouse) Paclitaxel target 43 TCGA patient Paclitaxel target 35 GDSC cell line Paclitaxel source 389

Represent the short summary for finding the corresponding scientific article (long)

A novel method of inductive transfer learning that employs adversarial learning and multi-task learning to address the discrepancy in input and output space

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR.

Question: Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR). Recent work has demonstrated that it is beneficial to learn these tasks jointly, which avoids the propagation of error inherent in pipeline-based systems and improves performance. However, state-of-the-art joint models typically rely on external natural language processing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well. The few neural, end-to-end models that have been proposed are trained almost completely from scratch. In this paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model. Because the bulk of our model's parameters are pre-trained and we eschew recurrence for self-attention, our model is fast to train. On 5 datasets across 3 domains, our model matches or exceeds state-of-the-art performance, sometimes by a large margin. The extraction of named entities (named entity recognition, NER) and their semantic relations (relation extraction, RE) are key tasks in information extraction and retrieval (IE & IR). Given a sequence of text (usually a sentence), the objective is to identify both the named entities and the relations between them. This information is useful in a variety of NLP tasks such as question answering, knowledge base population, and semantic search . In the biomedical domain, NER and RE facilitate large-scale biomedical data analysis, such as network biology , gene prioritization , drug repositioning and the creation of curated databases . In the clinical domain, NER and RE can aid in disease and treatment prediction, readmission prediction, de-identification, and patient cohort identification . Most commonly, the tasks of NER and RE are approached as a pipeline, with NER preceding RE. There are two main drawbacks to this approach: Pipeline systems are prone to error propagation between the NER and RE systems. One task is not able to exploit useful information from the other (e.g. the type of relation identified by the RE system may be useful to the NER system for determining the type of entities involved in the relation, and vice versa). More recently, joint models that simultaneously learn to extract entities and relations have been proposed, alleviating the aforementioned issues and achieving state-of-the-art performance (; ; ; ; Adel & Schütze, 2017; a; b; ;). Many of the proposed joint models for entity and relation extraction rely heavily on external natural language processing (NLP) tools such as dependency parsers. For instance, propose a recurrent neural network (RNN)-based joint model that uses a bidirectional long-short term memory network (BiLSTM) to model the entities and a tree-LSTM to model the relations between entities; propose a similar model for biomedical text. The tree-LSTM uses dependency tree information extracted using an external dependency parser to model relations between entities. The use of these external NLP tools limits the effectiveness of a model to domains (e.g. news) where those NLP tools perform well. As a remedy to this problem, Bekoulis et al. (2018a) proposes a neural, end-to-end system that jointly learns to extract entities and relations without relying on external NLP tools. In Bekoulis et al. (2018b), they augment this model with adversarial training. propose a different, albeit similar end-to-end neural model which makes use of deep biaffine attention . approach the problem with multi-turn question answering, posing templated queries to a BERT-based QA model whose answers constitute extracted entities and their relations and achieve state-of-the-art on three popular benchmark datasets. While demonstrating strong performance, end-to-end systems like Bekoulis et al. (2018a; b) and suffer from two main drawbacks. The first is that most of the models parameters are trained from scratch. For large datasets, this can lead to long training times. For small datasets, which are common in the biomedical and clinical domains where it is particularly challenging to acquire labelled data, this can lead to poor performance and/or overfitting. The second is that these systems typically contain RNNs, which are sequential in nature and cannot be parallelized within training examples. The multi-pass QA model proposed in alleviates these issues by incorporating a pre-trained language model, BERT , which eschews recurrence for self-attention. The main limitation of their approach is that it relies on handcrafted question templates to achieve maximum performance. This may become a limiting factor where domain expertise is required to craft such questions (e.g., for biomedical or clinical corpora). Additionally, one has to create a question template for each entity and relation type of interest. In this study, we propose an end-to-end model for joint NER and RE which addresses all of these issues. Similar to past work, our model can be viewed as a mixture of a NER module and a RE module (Figure 1). Unlike most previous works, we include a pre-trained, transformer-based language model, specifically BERT , which achieved state-of-the-art performance across many NLP tasks. The weights of the BERT model are fine-tuned during training, and the entire model is trained in an end-to-end fashion. Our main contributions are as follows: Our solution is truly end-to-end, relying on no handcrafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers). Our model is fast to train (e.g. under 10 minutes on a single GPU for the CoNLL04 corpus), as most of its parameters are pre-trained and we avoid recurrence. We match or exceed state-of-the-art performance for joint NER and RE on 5 datasets across 3 domains. Figure 1 illustrates the architecture of our approach. Our model is composed of an NER module and an RE module. The NER module is identical to the one proposed by. For a given input sequence s of N word tokens w 1, w 2,..., w N, the pre-trained BERT BASE model first produces a sequence of vectors, x which are then fed to a feed-forward neural network (FFNN) for classification. The output size of this layer is the number of BIOES-based NER labels in the training data, |C (NER) |. In the BIOES tag scheme, each token is assigned a label, where the B-tag indicates the beginning of an entity span, I-the inside, E-the end and S-is used for any single-token entity. All other tokens are assigned the label O. During training, a cross-entropy loss is computed for the NER objective, where s (NER) n is the predicted score that token n ∈ N belongs to the ground-truth entity class and s is the predicted score for token n belonging to the entity class c ∈ C (NER). In the RE module, the predicted entity labels are obtained by taking the argmax of each score vector s. The predicted entity labels are then embedded to produce a sequence of fixed-length, continuous vectors, e,..., e (NER) N which are concatenated with the hidden states from the final layer in the BERT model and learned jointly with the rest of the models parameters. and , we incrementally construct the set of relation candidates, R, using all possible combinations of the last word tokens of predicted entities, i.e. words with E-or S-labels. An entity pair is assigned to a negative relation class (NEG) when the pair has no relation or when the predicted entities are not correct. Once relation candidates are constructed, classification is performed with a deep bilinear attention mechanism , as proposed by. To encode directionality, the mechanism uses FFNNs to project each x (RE) i into head and tail vector representations, corresponding to whether the i th word serves as head or tail argument of the relation. These projections are then fed to a biaffine classifier, where U is an m × |C (RE) | × m tensor, W is a |C (RE) | × (2 * m) matrix, and b is a bias vector. Here, m is the size of the output layers of FFNN head and FFNN tail and C (RE) is the set of all relation classes (including NEG). During training, a second cross-entropy loss is computed for the RE objective where is the predicted score that relation candidate r ∈ R belongs to the ground-truth relation class and s (RE) r,c is the predicted score for relation r belonging to the relation class c ∈ C (RE). The model is trained in an end-to-end fashion to minimize the sum of the NER and RE losses. 2.1 , entity pre-training is proposed as a solution to the problem of lowperformance entity detection in the early stages of training. It is implemented by delaying the training of the RE module by some number of epochs, before training the entire model jointly. Our implementation of entity pretraining is slightly different. Instead of delaying training of the RE module by some number of epochs, we weight the contribution of L RE to the total loss during the first epoch of training where λ is increased linearly from 0 to 1 during the first epoch and set to 1 for the remaining epochs. We chose this scheme because the NER module quickly achieves good performance for all datasets (i.e. within one epoch). In early experiments, we found this scheme to outperform a delay of a full epoch. We implemented our model in PyTorch using the BERT BASE model from the PyTorch Transformers library 1. Our model is available at our GitHub repository 2. Furthermore, we use NVIDIAs automatic mixed precision (AMP) library Apex 3 to speed up training and reduce memory usage without affecting task-specific performance. To demonstrate the generalizability of our model, we evaluate it on 5 commonly used benchmark corpora across 3 domains. All corpora are in English. Detailed corpus statistics are presented in Table A.1 of the appendix. The Automatic Content Extraction (ACE04) corpus was introduced by , and is commonly used to benchmark NER and RE methods. There are 7 entity types and 7 relation types. ACE05 builds on ACE04, splitting the Physical relation into two classes (Physical and PartWhole), removing the Discourse relation class and merging Employment-Membership-Subsidiary and Person-Organization-Affiliation into one class (Employment-Membership-Subsidiary). For ACE04, we follow by removing the Discourse relation and evaluating our model using 5-fold cross-validation on the bnews and nwire subsets, where 10% of the data was held out within each fold as a validation set. For ACE05, we use the same test split as. We use 5-fold cross-validation on the remaining data to choose the hyperparameters. Once hyperparameters are chosen, we train on the combined data from all the folds and evaluate on the test set. For both corpora, we report the micro-averaged F 1 score. We obtained the pre-processing scripts from 4. The CoNLL04 corpus was introduced in and consists of articles from the Wall Street Journal (WSJ) and Associated Press (AP). There are 4 entity types and 5 relation types. We use the same test set split as 5. We use 5-fold cross-validation on the remaining data to choose hyperparameters. Once hyperparameters are chosen, we train on the combined data from all folds and evaluate on the test set, reporting the micro-averaged F 1 score. The adverse drug event corpus was introduced by to serve as a benchmark for systems that aim to identify adverse drug events from free-text. It consists of the abstracts of medical case reports retrieved from PubMed 6. There are two entity types, Drug and Adverse effect and one relation type, Adverse drug event. Similar to previous work (; b), we remove ∼130 relations with overlapping entities and evaluate our model using 10-fold cross-validation, where 10% of the data within each fold was used as a validation set, 10% as a test set and the remaining data is used as a train set. We report the macro F 1 score averaged across all folds. The 2010 i2b2/VA dataset was introduced by for the 2010 i2b2/Va Workshop on Natural Language Processing Challenges for Clinical Records. The workshop contained an NER task focused on the extraction of 3 medical entity types (Problem, Treatment, Test) and an RE task for 8 relation types. In the official splits, the test set contains roughly twice as many examples as the train set. To increase the number of training examples while maintaining a rigorous evaluation, we elected to perform 5-fold cross-validation on the combined data from both partitions. We used 10% of the data within each fold as a validation set, 20% as a test set and the remaining data was used as a train set. We report the micro F 1 score averaged across all folds. To the best of our knowledge, we are the first to evaluate a joint NER and RE model on the 2010 i2b2/VA dataset. Therefore, we decided to compare to scores obtained by independent NER and RE systems. We note, however, that the scores of independent RE systems are not directly comparable to the scores we report in this paper. This is because RE is traditionally framed as a sentence-level classification problem. During pre-processing, each example is permutated into processed examples containing two "blinded" entities and labelled for one relation class. E.g. the example: "His PCP had recently started ciprofloxacin TREATMENT for a UTI PROBLEM " becomes "His PCP had recently started @TREATMENT$ for a @PROBLEM$", where the model is trained to predict the target relation type, "Treatment is administered for medical problem" (TrAP). This task is inherently easier than the joint setup, for two reasons: relation predictions are made on ground-truth entities, as opposed to predicted entities (which are noisy) and the model is only required to make one classification decision per pre-processed sentence. In the joint setup, a model must identify any number of relations (or the lack thereof) between all unique pairs of predicted entities in a given input sentence. To control for the first of these differences, we report scores from our model in two settings, once when predicted entities are used as input to the RE module, and once when ground-truth entities are used. Besides batch size, learning rate and number of training epochs, we used the same hyperparameters across all experiments (see Table A .2). Similar to , learning rate and batch size were selected for each dataset using a minimal grid search (see See Table A. 3). One hyperparameter selected by hand was the choice of the pre-trained weights used to initialize the BERT BASE model. For general domain corpora, we found the cased BERT BASE weights from to work well. For biomedical corpora, we used the weights from BioBERT , which recently demonstrated state-of-the-art performance for biomedical NER, RE and QA. Similarly, for clinical corpora we use the weights provided by , who pre-trained BERT BASE on PubMed abstracts and clinical notes from MIMIC-III 7. 4.1 JOINTLY LEARNING NER AND RE Table 1 shows our in comparison to previously published , grouped by the domain of the evaluated corpus. We find that on every dataset besides i2b2, our model improves NER performance, for an average improvement of ∼2%. This improvement is particularly large on the ACE04 and ACE05 corpora (3.98% and 2.41% respectively). On i2b2, our joint model performs within 0.29% of the best independent NER solution. For relation extraction, we outperform previous methods on 2 datasets and come within ∼2% on both ACE05 and CoNLL04. In two cases, our performance improvement is substantial, with improvements of 4.59% and 10.25% on the ACE04 and ADE corpora respectively. For i2b2, our score is not directly comparable to previous systems (as discussed in section 3.1.4) but will facilitate future comparisons of joint NER and RE methods on this dataset. By comparing overall performance, we find that our approach achieves new state-of-the-art performance for 3 popular benchmark datasets (ACE04, ACE05, ADE) and comes within 0.2% for CoNLL04. To determine which training strategies and components are responsible for our models performance, we conduct an ablation analysis on the CoNLL04 corpus (Table 2) Removing FFNN head and FFNN tail has, by far, the largest negative impact on performance. Interestingly, however, replacing FFNN head and FFNN tail with a single FFNN has only a small negative impact. This suggests that while these layers are very important for model performance, using distinct FFNNs for the projection of head and tail entities (as opposed to the same FFNN) is relatively much less important. The next most impactful ablation was entity pre-training, suggesting that low-performance entity detection during the early stages of training is detrimental to learning (see section 2.1). Finally, we note that the importance of entity embeddings is surprising, as a previous study has found that entity embeddings did not help performance on the CoNLL04 corpus (a), although their architecture was markedly different. We conclude that each of our ablated components is necessary to achieve maximum performance. One advantage of including a transformer-based language model is that we can easily visualize the attention weights with respect to some input. This visualization is useful, for example, in detecting model bias and locating relevant attention heads . Previous works have used such visualizations to demonstrate that specific attention heads mark syntactic dependency relations and that lower layers tend to learn more about syntax while higher layers tend to encode more semantics . In Figure 2 we visualize the attention weights of select layers and attention heads from an instance of BERT fine-tuned within our model on the CoNLL04 corpus. We display four patterns that are easily 14 -* To the best of our knowledge, there are no published joint NER and RE models that evaluate on the i2b2 2010 corpus. We compare our model to the state-of-the-art for each individual task (see section 3.1.4). ** We compare to the scores achieved by their BERT BASE model. interpreted: paying attention to the next and previous words, paying attention to the word itself, and paying attention to the end of the sentence. These same patterns have been found in pre-trained BERT models that have not been fine-tuned on a specific, supervised task , and therefore, are retained after our fine-tuning procedure. To facilitate further analysis of our learned model, we make available Jupyter and Google Colaboratory notebooks on our GitHub repository 8, where users can use multiple views to explore the learned attention weights of our models. We use the BertViz library to render the interactive, HTML-based views and to access the attention weights used to plot the heat maps. In this paper, we introduced an end-to-end model for entity and relation extraction. Our key contributions are: No reliance on any hand-crafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers). Integration of a pre-trained, transformer-based language model. State-of-the-art performance on 5 datasets across 3 domains. Furthermore, our model is inherently modular. One can easily initialize the language model with pre-trained weights better suited for a domain of interest (e.g. BioBERT for biomedical corpora) or swap BERT for a comparable language model (e.g. XLNet ). Finally, because of, our model is fast to train, converging in approximately 1 hour or less on a single GPU for all datasets used in this study. Our model out-performed previous state-of-the-art performance on ADE by the largest margin (6.53%). While exciting, we believe this corpus was particularly easy to learn. The majority of sentences (∼68%) are annotated for two entities (drug and adverse effect, and one relation (adverse drug event). Ostensibly, a model should be able to exploit this pattern to get near-perfect performance on the majority of sentences in the corpus. As a test, we ran our model again, this time using ground-truth entities in the RE module (as opposed to predicted entities) and found that the model very quickly reached almost perfect performance for RE on the test set (∼98%). As such, high performance on the ADE corpus is not likely to transfer to real-world scenarios involving the large-scale annotation of diverse biomedical articles. In our experiments, we consider only intra-sentence relations. However, the multiple entities within a document generally exhibit complex, inter-sentence relations. Our model is not currently capable of extracting such inter-sentence relations and therefore our restriction to intra-sentence relations will limit its usefulness for certain downstream tasks, such as knowledge base creation. We also ignore the problem of nested entities, which are common in biomedical corpora. In the future, we would like to extend our model to handle both nested entities and inter-sentence relations. Finally, given that multilingual, pre-trained weights for BERT exist, we would also expect our model's performance to hold across multiple languages. We leave this question to future work. A.1 CORPUS STATISTICS . Gradient normalization Γ = 1 Rescales the gradient whenever the norm goes over some threshold Γ . Weight decay 0.1 L2 weight decay.

Represent the short summary for finding the corresponding scientific article (long).

A novel, high-performing architecture for end-to-end named entity recognition and relation extraction that is fast to train.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared. Recurrent Neural Networks (RNNs) achieve state-of-the-art performance on a wide range of sequence prediction tasks BID0 BID22 BID50 BID32. In this work we examine how to add uncertainty and regularisation to RNNs by means of applying Bayesian methods to training. This approach allows the network to express uncertainty via its parameters. At the same time, by using a prior to integrate out the parameters to average across many models during training, it gives a regularisation effect to the network. Recent approaches either justify dropout BID43 and weight decay as a variational inference scheme BID12, or apply Stochastic Gradient Langevin dynamics (, SGLD) to truncated backpropagation in time directly BID13. Interestingly, recent work has not explored further directly applying a variational Bayes inference scheme BID3 for RNNs as was done in BID14. We derive a straightforward approach based upon Bayes by Backprop that we show works well on large scale problems. Our strategy is a simple alteration to truncated backpropagation through time that in an estimate of the posterior distribution on the weights of the RNN. This formulation explicitly leads to a cost function with an information theoretic justification by means of a bits-back argument BID18 where a KL divergence acts as a regulariser. The form of the posterior in variational inference shapes the quality of the uncertainty estimates and hence the overall performance of the model. We shall show how performance of the RNN can be improved by means of adapting ("sharpening") the posterior locally to a batch. This sharpening adapts the variational posterior to a batch of data using gradients based upon the batch. This can be viewed as a hierarchical distribution, where a local batch gradient is used to adapt a global posterior, forming a local approximation for each batch. This gives a more flexible form to the typical assumption of Gaussian posterior when variational inference is applied to neural networks, which reduces variance. This technique can be applied more widely across other Bayesian models. The contributions of our work are as follows:• We show how Bayes by Backprop (BBB) can be efficiently applied to RNNs.• We develop a novel technique which reduces the variance of BBB, and which can be widely adopted in other maximum likelihood frameworks.• We improve performance on two widely studied benchmarks outperforming established regularisation techniques such as dropout by a big margin.• We introduce a new benchmark for studying uncertainty of language models. Bayes by Backprop BID14 is a variational inference BID46 scheme for learning the posterior distribution on the weights θ ∈ R d of a neural network. This posterior distribution is typically taken to be a Gaussian with mean parameter µ ∈ R d and standard deviation parameter σ ∈ R d, denoted N (θ|µ, σ 2). Note that we use a diagonal covariance matrix, and d -the dimensionality of the parameters of the network -is typically in the order of millions. Let log p(y|θ, x) be the log-likelihood of the model, then the network is trained by minimising the variational free energy: DISPLAYFORM0 where p(θ) is a prior on the parameters. Minimising the variational free energy is equivalent to maximising the log-likelihood log p(y|θ, x) subject to a KL complexity term on the parameters of the network that acts as a regulariser: DISPLAYFORM1 In the Gaussian case with a zero mean prior, the KL term can be seen as a form of weight decay on the mean parameters, where the rate of weight decay is automatically tuned by the standard deviation parameters of the prior and posterior. Please refer to the supplementary material for the algorithmic details on Bayes by Backprop. The uncertainty afforded by Bayes by Backprop trained networks has been used successfully for training feedforward models for supervised learning and to aid exploration by reinforcement learning agents BID30 BID21, but as yet, it has not been applied to recurrent neural networks. The core of an RNN, f, is a neural network that maps the RNN state s t at step t, and an input observation x t to a new RNN state s t+1, f: (s t, x t) → s t+1. The exact equations of an LSTM core can be found in the supplemental material Sec A.2.An RNN can be trained on a sequence of length T by backpropagation through by unrolling T times into a feedforward network. Explicitly, we set s i = f (s i−1, x i), for i = 1,..., T. We shall refer to an RNN core unrolled for T steps by s 1:T = F T (x 1:T, s 0). Note that the truncated version of the algorithm can be seen as taking s 0 as the last state of the previous batch, s T.RNN parameters are learnt in much the same way as in a feedforward neural network. A loss (typically after further layers) is applied to the states s 1:T of the RNN, and then backpropagation is used to update the weights of the network. Crucially, the weights at each of the unrolled steps are shared. Thus each weight of the RNN core receives T gradient contributions when the RNN is unrolled for T steps. Applying BBB to RNNs is depicted in FIG0 where the weight matrices of the RNN are drawn from a distribution (learnt by BBB). However, this direct application raises two questions: when to sample the parameters of the RNN, and how to weight the contribution of the KL regulariser of. We shall briefly justify the adaptation of BBB to RNNs, given in FIG0. The variational free energy of for an RNN on a sequence of length T is: DISPLAYFORM0 where p(y 1:T |θ, x 1:T) is the likelihood of a sequence produced when the states of an unrolled RNN F T are fed into an appropriate probability distribution. The parameters of the entire network are Sample ∼ N (0, I), ∈ R d, and set network parameters to θ = µ + σ. Sample a minibatch of truncated sequences (x, y). Do forward and backward propagation as normal, and let g be the gradient w.r.t θ. DISPLAYFORM0 be the gradients of log N (θ|µ, σ 2) − log p(θ) w.r.t. θ, µ and σ respectively. Update µ using the gradient DISPLAYFORM1 Update σ using the gradient θ. Although the RNN is unrolled T times, each weight is penalised just once by the KL term, rather than T times. Also clear from is that when a Monte Carlo approximation is taken to the expectation, the parameters θ should be held fixed throughout the entire sequence. DISPLAYFORM2 Two complications arise to the above naive derivation in practice: firstly, sequences are often long enough and models sufficiently large, that unrolling the RNN for the whole sequence is prohibitive. Secondly, to reduce variance in the gradients, more than one sequence is trained at a time. Thus the typical regime for training RNNs involves training on mini-batches of truncated sequences. Let B be the number of mini-batches and C the number of truncated sequences ("cuts"), then we can write as: DISPLAYFORM3 where the (b, c) superscript denotes elements of cth truncated sequence in the bth minibatch. Thus the free energy of mini-batch b of a truncated sequence c can be written as: DISPLAYFORM4 where w Finally, the question of when to sample weights follows naturally from taking a Monte Carlo approximations to: for each minibatch, sample a fresh set of parameters. The choice of variational posterior q(θ) as described in Section 3 can be enhanced by adding side information that makes the posterior over the parameters more accurate, thus reducing variance of the learning process. Akin to Variational Auto Encoders (VAEs) BID25 BID41, which propose a powerful distribution q(z|x) to improve the gradient estimates of the (intractable) likelihood function p(x), here we propose a similar approach. Namely, for a given minibatch of data (inputs and targets) (x, y) sampled from the training set, we construct such q(θ|(x, y)). Thus, we compute a proposal distribution where the latents (z in VAEs) are the parameters θ (which we wish to integrate out), and the "privileged" information upon which we condition is a minibatch of data. We could have chosen to condition on a single example (x, y) instead of a batch, but this would have yielded different parameter vectors θ per example. Conditioning on the full minibatch has the advantage of producing a single θ per minibatch, so that matrix-matrix operations can still be carried. This "sharpened" posterior yields more stable optimisation, a common pitfall of Bayesian approaches to train neural networks, and the justification of this method follows from strong empirical evidence and extensive work on VAEs. A challenging aspect of modelling the variational posterior q(θ|(x, y)) is the large number of dimensions of θ ∈ R d. When the dimensionality is not in the order of millions, a powerful non-linear function (such as a neural network) can be used which transforms observations (x, y) to the parameters of a Gaussian distribution, as proposed in BID25; BID41. Unfortunately, this neural network would have far too many parameters, making this approach unfeasible. Given that the loss − log p(y|θ, x) is differentiable with respect to θ, we propose to parameterise q as a linear combination of θ and g θ = −∇ θ log p(y|θ, x), both d-dimensional vectors. Thus, we can define a hierarchical posterior of the form DISPLAYFORM0 with µ, σ ∈ R d, and q(ϕ) = N (ϕ|µ, σ) -the same as in the standard BBB method. Finally, let * denote element-wise multiplication, we then have DISPLAYFORM1 where η ∈ R d is a free parameter to be learnt and σ 0 a scalar hyper-parameter of our model. η can be interpreted as a per-parameter learning rate. During training, we get θ ∼ q(θ|(x, y)) via ancestral sampling to optimise the loss DISPLAYFORM2 DISPLAYFORM3 where µ, σ, η are our model parameters, and p are the priors for the distributions defining q (for exact details of these distributions see Section 6). The constant C is the number of truncated sequences as defined in Section3. The bound on the true data likelihood which yields eq. is derived in Sec 4.1. Algorithm 1 presents how learning is performed in practice. Sample a minibatch (x, y) of truncated sequences. DISPLAYFORM0 As long as the improvement of the log likelihood log p(y|θ, x) term along the gradient g ϕ is greater than the KL cost added for posterior sharpening (KL [q(θ|ϕ, (x, y)) || p(θ|ϕ)]), then the lower bound in will improve. This justifies the effectiveness of the posterior over the parameters proposed in eq. 7 which will be effective as long as the curvature of log p(y|θ, x) is large. Since η is learnt, it controls the tradeoff between curvature improvement and KL loss. Studying more powerful parameterisations is part of future research. Unlike regular BBB where the KL terms can be ignored during inference, there are two options for doing inference under posterior sharpening. The first involves using q(ϕ) and ignoring any KL terms, similar to regular BBB. The second involves using q(θ|ϕ, (x, y)) which requires using the term KL [q(θ|ϕ, (x, y)) || p(θ|ϕ)] yielding an upper bound on perplexity (lower bound in log probability; see Section 4.2 for details). This parameterisation involves computing an extra gradient and incurs a penalty in training speed. A comparison of the two inference methods is provided in Section 6. Furthermore, in the case of RNNs, the exact gradient cannot be efficiently computed, so BPTT is used. Here we turn to deriving the training loss function we use for posterior sharpening. The basic idea is to take a variational approximation to the marginal likelihood p(x) that factorises hierarchically. Hierarchical variational schemes for topic models have been studied previously in BID40. Here, we shall assume a hierarchical prior for the parameters such that p(x) = p(x|θ)p(θ|ϕ)p(ϕ)dθdϕ. Then we pick a variational posterior that conditions upon x, and factorises as q(θ, ϕ|x) = q(θ|ϕ, x)q(ϕ). The expected lower bound on p(x) is then as follows: DISPLAYFORM0 DISPLAYFORM1 We note that the procedure of sharpening the posterior as explained above has similarities with other techniques. Perhaps the most obvious one is line search: indeed, η is a trained parameter that does line search along the gradient direction. Probabilistic interpretations have been given to line search in e.g. BID34, but ours is the first that uses a variational posterior with the reparametrization trick/perturbation analysis gradient. Also, the probabilistic treatment to line search can also be interpreted as a trust region method. Another related technique is dynamic evaluation BID37, which trains an RNN during evaluation of the model with a fixed learning rate. The update applied in this case is cumulative, and only uses previously seen data. Thus, they can take a purely deterministic approach and ignore any KL between a posterior with privileged information and a prior. As we will show in Section 6, performance gains can be significant as the data exhibits many short term correlations. Lastly, learning to optimise (or learning to learn) BID28 BID1 is related in that a learning rate is learned so that it produces better updates than those provided by e.g. AdaGrad BID9 or Adam BID24. Whilst they train a parametric model, we treat these as free parameters (so that they can adapt more quickly to the non-stationary distribution w.r.t. parameters). Notably, we use gradient information to inform a variational posterior so as to reduce variance of Bayesian Neural Networks. Thus, although similar in flavour, the underlying motivations are quite different. Applying Bayesian methods to neural networks has a long history, with most common approximations having been tried. BID5 propose various maximum a posteriori schemes for neural networks, including an approximate posterior centered at the mode. also suggest using second order derivatives in the prior to encourage smoothness of the ing network. BID18 proposed using variational methods for compressing the weights of neural networks as a regulariser. BID20 suggest an MDL loss for single layer networks that penalises non-robust weights by means of an approximate penalty based upon perturbations of the weights on the outputs.; investigated using the Laplace approximation for capturing the posterior of neural networks. investigated the use of hybrid Monte Carlo for training neural networks, although it has so far been difficult to apply these to the large sizes of networks considered here. More recently derived a variational inference scheme for neural networks and Blundell et al. FORMULA0 extended this with an update for the variance that is unbiased and simpler to compute. derives a similar algorithm in the case of a mixture posterior. Several authors have claimed that dropout BID43 and Gaussian dropout BID47 can be viewed as approximate variational inference schemes BID11 , respectively FORMULA0 proposed a variational scheme with biased gradients for the variance parameter using the Fisher matrix. Our work extends this by using an unbiased gradient estimator without need for approximating the Fisher and also add a novel posterior approximation. Variational methods typically underestimate the uncertainty in the posterior (as they are mode seeking, akin to the Laplace approximation), whereas expectation propagation methods often average over modes and so tend to overestimate uncertainty (although there are counter examples for each depending upon the particular factorisation and approximations used; see for example BID44). Nonetheless, several papers explore applying expectation propagation to neural networks: BID42 derive a closed form approximate online expectation propagation algorithm, whereas Hernández- proposed using multiple passes of assumed density filtering (in combination with early stopping) attaining good performance on a number of small data sets. BID16 derive a distributed expectation propagation scheme with SGLD BID48 as an inner loop. Others have also considered applying SGLD to neural networks BID27 and BID13 more recently used SGLD for LSTMs (we compare to these in our experiments). We present the of our method for a language modelling and an image caption generation task. We evaluated our model on the Penn Treebank BID35 benchmark, a task consisting on next word prediction. We used the network architecture from BID50, a simple yet strong baseline on this task, and for which there is an open source implementation 1. The baseline consists of an RNN with LSTM cells and a special regularisation technique, where the dropout operator is only applied to the non-recurrent connections. We keep the network configuration unchanged, but instead of using dropout we apply our Bayes by Backprop formulation. Our goal is to demonstrate the effect of applying BBB to a pre-existing, well studied architecture. To train our models, we tuned the parameters on the prior distribution, the learning rate and its decay. The weights were initialised randomly and we used gradient descent with gradient clipping for optimisation, closely following BID50's "medium" LSTM configuration (2 layers with 650 units each). As in, the prior of the network weights θ was taken to be a scalar mixture of two Gaussian densities with zero mean and variances σ 2 1 and σ 2 2, explicitly DISPLAYFORM0 where θ j is the j-th weight of the network. We searched π ∈ {0.25, 0.5, 0.75}, log σ 1 ∈ {0, −1, −2} and log σ 2 ∈ {−6, −7, −8}.For speed purposes, during training we used one sample from the posterior for estimating the gradients and computing the (approximate) KL-divergence. For prediction, we experimented with either computing the expected loss via Monte Carlo sampling, or using the mean of the posterior distribution as the parameters of the network (MAP estimate). We observed that the improved as we increased the number of samples but they were not significantly better than taking the mean (as was also reported by BID14). For convenience, in TAB2 we report our numbers using the mean of the converged distribution, as there is no computation overhead w.r.t. a standard LSTM model. TAB2 compares our to the LSTM dropout baseline BID50 we built from, and to the Variational LSTMs BID12, which is another Bayesian approach to this task. Finally, we added dynamic evaluation BID37 with a learning rate of 0.1, which was found via cross validation. As with other VAE-related RNNs BID10; BID2 BID7 perplexities using posterior sharpening are reported including a KL penalty KL [q(θ|ϕ, (x, y)) || p(θ|ϕ)] in the log likelihood term (the KL is computed exactly, not sampled). For posterior sharpening we use a hierarchical prior for θ: p(θ|ϕ) = N (θ|ϕ, σ 2 0 I) which expresses our belief that a priori, the network parameters θ will be much like the data independent parameters ϕ with some small Gaussian perturbation. In our experiments we swept over σ 0 on the validation set, and found σ 0 = 0.02 to perform well, although were not particularly sensitive to this. Note that with posterior sharpening, the perplexities reported are upper bounds (as the likelihoods are lower bounds).Lastly, we tested the variance reduction capabilities of posterior sharpening by analysing the perplexity attained by the best models reported in TAB2. Standard BBB yields 258 perplexity after only one epoch, whereas the model with posterior sharpening is better at 227. We also implemented it on MNIST following, and obtained small but consistent speed ups. Lower perplexities on the Penn Treebank task can be achieved by varying the model architecture, which should be complementary to our work of treating weights as random variables-we are simply interested in assessing the impact of our method on an existing architecture, rather than absolute state-of-the-art. See BID23; Zilly et al. FORMULA0; BID36, for a report on recent advances on this benchmark, where they achieve perplexities of 70.9 on the test set. Furthermore we note that the speed of our naïve implementation of Bayesian RNNs was 0.7 times the original speed and 0.4 times the original speed for posterior sharpening. Notably, FIG2 shows the effect of weight pruning: weights were ordered by their signal-to-noise ratio (|µ i |/σ i) and removed (set to zero) in reverse order. We evaluated the validation set perplexity for each proportion of weights dropped. As can be seen, around 80% of the weights can be removed from the network with little impact on validation perplexity. Additional analysis on the existing patterns of the dropped weights can be found in the supplementary material A.3. We used the Penn Treebank test set, which is a long sequence of ≈ 80K words, and reversed it. Thus, the "reversed" test set first few words are: "us with here them see not may we..." which correspond to the last words of the standard test set: "... we may not see them here with us".Let V be the vocabulary of this task. For a given input sequence x = x 1:T and a probabilistic model p, we define the entropy of x under p, H p [x], by DISPLAYFORM0 DISPLAYFORM1, i.e., the per word entropy. Let X be the standard Penn Treebank test set, and X rev the reversed one. For a given probabilistic model p, we define the entropy gap ∆H p by DISPLAYFORM2 Since X rev clearly does not come from the training data distribution (reversed English does not look like proper English), we expect ∆H p to be positive and large. Namely, if we take the per word entropy of a model as a proxy for the models' certainty (low entropy means the model is confident about its prediction), then the overall certainty of well calibrated models over X rev should be lower than over X. Thus, DISPLAYFORM3. When comparing two distributions, we expect the better calibrated one to have a larger ∆H p.In FIG3, we plotted ∆H p for the BBB and the baseline dropout LSTM model. The BBB model has a gap of about 0.67 nats/word when taking 10 samples, and slightly below 0.65 when using the posterior mean. In contrast, the model using MC Dropout BID11 is less well calibrated and is below 0.58 nats/word. However, when "turning off" dropout (i.e., using the mean field approximation), ∆H p improves to below 0.62 nats/word. We note that with the empirical likelihood of the words in the test set with size T (where for each word w ∈ V, p(w) = (# occurrences of w) T ), we get an entropy of 6.33 nats/word. The BBB mean model has entropy of 4.48 nats/word on the reversed set which is still far below the entropy we get by using the empirical likelihood distribution. We also applied Bayes by Backprop for RNNs to image captioning. Our experiments were based upon the model described in, where a state-of-the-art pre-trained convolutional neural network (CNN) was used to map an image to a high dimensional space, and this representation was taken to be the initial state of an LSTM. The LSTM model was trained to predict the next word on a sentence conditioned on the image representation and all the previous words in the image caption. We kept the CNN architecture unchanged, and used an LSTM trained using Bayes by Backprop rather than the traditional LSTM with dropout regularisation. As in the case for language modelling, this work conveniently provides an open source implementation 2. We used the same prior distribution on the weights of the network as we did for the language modelling task, and searched over the same hyper-parameters. We used the MSCOCO BID29 ) data set and report perplexity, BLUE-4, and CIDER scores on compared to the Show and Tell model We observe significant improvements in BLUE and CIDER, outperforming the dropout baseline by a large margin. Moreover, a random sample of the captions that were different for both the baseline and BBB is shown in FIG4. Besides the clear quantitative improvement, it is useful to visualise qualitatively the performance of BBB, which indeed generally outperforms the strong baseline, winning in most cases. As in the case of Penn Treebank, we chose a performant, open source model. Captioning models that use spatial attention, combined with losses that optimise CIDER directly (rather than a surrogate loss as we do) achieve over 100 CIDER points BID32 BID31. We have shown how to apply the Bayes by Backprop (BBB) technique to RNNs. We enhanced it further by introducing the idea of posterior sharpening: a hierarchical posterior on the weights of neural networks that allows a network to adapt locally to batches of data by a gradient of the model. We showed improvements over two open source, widely available models in the language modelling and image captioning domains. We demonstrated that not only do BBB RNNs often have superior performance to their corresponding baseline model, but are also better regularised and have superior uncertainty properties in terms of uncertainty on out-of-distribution data. Furthermore, BBB RNNs through their uncertainty estimates show signs of knowing what they know, and when they do not, a critical property for many real world applications such as self-driving cars, healthcare, game playing, and robotics. Everything from our work can be applied on top of other enhancements to RNN/LSTM models (and other non-recurrent architectures), and the empirical evidence combined with improvements such as posterior sharpening makes variational Bayes methods look very promising. We are exploring further research directions and wider adoption of the techniques presented in our work. The core of an RNN, f, is a neural network that maps the RNN state at step t, s t and an input observation x t to a new RNN state s t+1, f: (s t, x t) → s t+1.An LSTM core has a state s t = (c t, h t) where c is an internal core state and h is the exposed state. Intermediate gates modulate the effect of the inputs on the outputs, namely the input gate i t, forget gate f t and output gate o t. The relationship between the inputs, outputs and internal gates of an LSTM cell (without peephole connections) are as follows: As discussed in Section 6.1, for the Penn Treebank task, we have taken the converged model an performed weight pruning on the parameters of the network. Weights were ordered by their signalto-noise ratio (|µ i |/σ i) and removed (set to zero) in reverse order. It was observed that around 80% of the weights can be removed from the network with little impact on validation perplexity. In Figure 5, we show the patterns of the weights dropped for one of the LSTM cells from the model. DISPLAYFORM0 Figure 5: Pruning patterns for one LSTM cell (with 650 untis) from converged model with 80% of total weights dropped. A white dot indicates that particular parameter was dropped. In the middle column, a horizontal white line means that row was set to zero. Finally, the last column indicates the total number of weights removed for each row.

represent the tldr for finding the corresponding scientific article

Variational Bayes scheme for Recurrent Neural Networks

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.