Question: Represent the scientific article to find a short TLDR:

Question: Existing deep learning approaches for learning visual features tend to extract more information than what is required for the task at hand. From a privacy preservation perspective, the input visual information is not protected from the model; enabling the model to become more intelligent than it is trained to be. Existing approaches for suppressing additional task learning assume the presence of ground truth labels for the tasks to be suppressed during training time. In this research, we propose a three-fold novel contribution: (i) a novel metric to measure the trust score of a trained deep learning model, (ii) a model-agnostic solution framework for trust score improvement by suppressing all the unwanted tasks, and (iii) a simulated benchmark dataset, PreserveTask, having five different fundamental image classification tasks to study the generalization nature of models. In the first set of experiments, we measure and improve the trust scores of five popular deep learning models: VGG16, VGG19, Inception-v1, MobileNet, and DenseNet and demonstrate that Inception-v1 is having the lowest trust score. Additionally, we show of our framework on color-MNIST dataset and practical applications of face attribute preservation in Diversity in Faces (DiF) and IMDB-Wiki dataset. The primary objective of artificial intelligence is to imitate human intelligence tabular rasa. Especially, with the advent of deep learning (DL), the models are striving to perform composite tasks by learning complex relationships and patterns available in noisy, unstructured data . With this sudden growth in the consumption of data by models, there has been a lot of study on the privacy and security of the learnt model . Data governance and model governance frameworks, control and protect sharing of data and model meta information between two entities and also their social implications . The premise of model privacy has majorly revolved around preserving the model content from human (man-in-the-middle) adversarial attacks . However, the model itself could learn all the private information from the data and become much more intelligent than the original intent it was trained for. With the strive for model generalization, including techniques for transfer learning and multi-task learning, the model is encouraged to learn more and more generic features from the data that could be used for more than one task (Søgaard &). Consider the example described in Figure 1, where a classifier is trained to detect the shape of an object from images. However, using the features extracted by the above classifier, the size and location of the object in the image can also be predicted. Thus, a shape classifier is more intelligent than its objective of only predicting the shape of the object. While in certain applications, this is a required property of classification models (such as in, transfer learning and domain adaptation), in most of the privacy preserving applications, the data and its other visual attributes have to be kept private from the model itself. As an additional real-world example, we train a DL model to predict the gender from a face image. However, the DL model learns most generic features from the face image, enabling it to predict the age and the identity of the person. The input face image could be saved securely from a human attacker, however, there is not much focus on securing from the model itself. Additionally as shown in Figure 1 (a), the task of debiasing is to remove the the bias (color) in learning a specific task (shape). This happens due to the high correlation between the color and shapes in the input images. However, as shown in Figure 1 (b), our task in model trust is to forcefully The fundamental research motivation in this work is to study if a learning model could be restricted to perform only one or a specific group of tasks. ensure that the model learns to perform only one or few selected tasks (shape) from the input images and unlearn all other tasks (color, size, location). If multi-class classification tasks could be done from the same image, the research question is, "How can we ensure that the model is learnt only for one or a few tasks (called as, preserved tasks), and is strictly not learnt for the other tasks (called as, suppressed tasks)?". To pursue research on this problem, there are few evident challenges: (i) there is a lack of a balanced and properly curated image dataset where multiple classification tasks could be performed on the same image, (ii) the complete knowledge of both the preserved tasks and the suppressed tasks should be known apriori, that is, we cannot suppress those tasks that we don't have information about, and (iii) presence of very few model agnostic studies to preserve and suppress different task groups. In this research, we propose a novel framework to measure the trust score of a trained DL model and a solution approach to improve the trust score during training. The major research contributions are summarized as follows: 1. A simulated, class-balanced, multi-task dataset, PreserveTask with five tasks that could be performed on each image: shape, size, color, location, and color classification. 2. A novel metric to measure the trustworthiness score of a trained DL model. The trust scores of five popular DL models are measured and compared: VGG16, VGG19, Inception-v1, MobileNet, and DenseNet. A generic model-agnostic solution framework to improve the trust scores of DL models during training by preserving a few tasks and suppressing other tasks on the same image. 3. Experimental analysis are performed for the proposed framework in comparison with other existing approaches under different settings. Experimentally, we considered the model with the least trust score, Inception-v1, and showed that the proposed framework aids in improving the overall trust score 1. 4. To demonstrate the practical applications and generalizability of the metric and the solution framework, we show additionally in colored MNIST dataset and face attribute preservation using two datasets: (i) Diversity in Faces (DiF) (Merler et al.) (ii) IMDBWiki . There are broadly two different groups of work related to the research problem at hand: (i) kanonymity preservation and (ii) attribute suppression. k-anonymity Preservation: The objective here is to preserve the anonymity of certain attributes from being predicted by the model. To quote some earlier works, , studied to mask out potentially sensitive information from video feeds. In the last decade, face recognition has become an important commercial applications and also an application that demanded discussion regarding privacy preservation. Studies focused on extracting only the required meta information from face images while not extracting the identity. This was a required step to make face as a usable biometric. Studies such as , , and focused on preserving the identity of the face image from the model by performing face de-identification. Studies such as and focused on anonymizing the face gender information while models could extract the identity. Attribute Suppression: The aim of this group of techniques is to explicitly suppress a few attributes by perturbing the input data to the model. Studies such as and test if the learnt models are robust and protected against adversarial attacks. suggested using a constrained generative adversarial network (GAN) to perturb the input face image and suppress the required attribute. The GANs will generate the attribute free face image of the original face image. The closest related work to our approach, is the study by where the visual attributes are decorrelated using a negative gradient in the model. The demonstrate that the classification task could be performed by preserving specific attributes in the image while suppressing the influence of the remaining. Additionally, there is a good amount of research in bias mitigation while learning models . The primary aim is to debias the model learning from any kind of correlated attributes , which is different from our aim of improving the model's trust. The major gaps in the existing research works are: (i) most of the techniques focus on data perturbation, that is, changing the input data from x to x such that the suppressed task information is not available in the data. There is not much focus on model perturbation without altering the input data, (ii) most of the existing datasets have only binary attributes and hence suppressing and preserving a few tasks does not actually translate to the classification complexity of multi-class tasks, and (iii) there is a lack of a well curated benchmark dataset to evaluate the privacy preserving capacity of DL models. Shared tasks performed on the same image carry some common attributes which are often extracted by complex deep learning models. The objective of this is to untangle the shared tasks and enable deep learning models to perform only one (or few) of those tasks. In order to evaluate the performance of such a framework, the dataset should have the following properties: • Should perform multiple tasks on the same image and each task should have varying number of classes, in order to study the relationship of complexity of classification tasks. Figure 3: (a) A deep learning model learning features suited for multiple tasks, more than the intended shape classification task, (b) Existing approaches suppress other known tasks, such as size classification by backpropagation of negative loss or gradient, (c) Proposed approach of suppressing all possible n-class classification task by using random class labels. • As this research area is nascent, the dataset should be noise-free and class balanced, to avoid other complexities that could influence classification performance. • Tasks should be designed in such a way that certain tasks, share common attributes and features, while certain tasks should be independent of each other. There are some similar publicly available datasets in the literature. LFW , CelebA , IMDB-Wiki , AwA 2 , and CUB datasets have multiple binary classification tasks, while only one nonbinary classification task. It is challenging to study the influence of complexity of classification tasks using these datasets and hence is not extendable to practical applications. CLEVR dataset provides with four different tasks with variable number of classes. However, each image contains multiple objects with different shape, color, and textures, allowing multiple labels for each task. Task suppression in multi-label, multi-task classification setting provides a very challenging experimental setting. Inspired from the CLEVR dataset, we create a new PreserveTask dataset, which is a multi-task dataset exclusively designed for the purpose of bench-marking models against preserving task privacy. The primary objective is to create easy-to-perform multi-task dataset, where the performance of the individual tasks is high. As shown in Figure 2, PreserveTask dataset has five different classification tasks, as follows: (i) Shape Classification: circle, triangle, diamond, pentagon, hexagon, (ii) Color Classification: violent, indigo, blue, green, yellow, orange, red, (iii) Size Classification: small, medium, large, (iv) Location Classification: quadrant 1, quadrant 2, quadrant 3, quadrant 4, (v) Background Color Classification: white, black, or colored. These five tasks are chosen such that few tasks are highly correlated (size, shape), while few tasks are ideally independent of each other (size, color). All the images are generated as 256 × 256 colored images. There are 5 (shapes) * 7 (color) * 3 (size) * 4 (location) * 3 ( color) = 1260 variations, with 50 images for training and 10 images for testing for each variation, generating a total of 63, 000 training and 12, 600 images. This ensures that there is a perfect class balance across all tasks. It is to be noted that the task of suppression of unknown shared task is a fairly open research problem. Hence, in order to set the benchmark of different frameworks, an easy, straightforward PreserveTask dataset is created as a conscious decision without having much noise, such as in DeepFashion dataset. As the problem area matures, further extensions of this dataset could be generated and more real world natural objects could be added. To understand the current scenario of shared task learning, consider any deep learning model as shown in Figure 3 (a). Assume a deep learning model, say VGG19, is trained for predicting the shape of objects in images. Ideally, the features f 1 obtained from the model should be good for object shape prediction. However, it is observed that different size, color, location prediction classifiers could be trained on top of f 1 demonstrating that f 1 contains more information about the object than just its shape. While this is a required property in multi-task learning and in applications of domain adaptation, from a task privacy preservation perspective this should be controlled. In literature, few technique variants exist to suppress the model from learning a few attributes or tasks . As shown in Figure 3 (b), if the model has to be suppressed from learning the size of the objects, a negative loss or negative gradient is applied to enable features f 2 to not carry any information about the size of the object while retaining all the information about the shape of the object. This comes with an assumption that the information about the tasks to be suppressed are available during training time along with its ground truth class labels for the entire training data. In our proposed framework, we overcome this assumption and do not expect the suppression task information to be available during model training time. Additionally, we provide a model agnostic approach of suppressing task learning so that the framework could be directly applied to any deep learning model. Let x ∈ X be the input data and y x ∈ Y (n) be the n different tasks that could be performed on the image. We learn a model,, be the feature representation for the given task. Ideally, while only g: provides high classification accuracy in most cases. To overcome this challenge, we generate random n-class labels in the gradient reversal (GR) branch in order to suppress any other n-class classification, as shown in Figure 3 (c),. Multiple gradient reversal branches could be built for varying values of n to suppress all possible other classification tasks. The DL model is trained by a custom loss function as follows, where L m is the loss of the model branch trained for the task to be preserved. L is the sum of individual losses (L i) which are to be maximized (task suppressed). λ and (1 − λ) are the weights given for the minimization and maximization losses which can be chosen based on the amount of sharing between the tasks. Additionally, each of the individual L i could be distinct loss functions in the model, depending on the task performed. Thus, it can be observed that the proposed framework is both DL model agnostic and loss function agnostic. PreserveTask will be used as the benchmark dataset against which the trust score of any trained DL model could be extracted. The trained DL model is evaluated against different tasks in the PreserveTask and the entire confusion matrix of performance accuracy is obtained (5 × 5 corresponding to the five tasks). The behavior of an ideal DL model, would provide 100% accuracy on the leading diagonal i.e., the tasks it was trained for, while providing, random classification accuracies for other tasks. The confusion matrix for such an ideal DL model is shown in Figure 4. For example in the first row, the DL model was trained to learn and predict the color of the object. Hence, color prediction performance should be 1 (denoting, 100% accuracy), while other tasks should provide random 1/n accuracy, where n is the number of classes. Let the ideal performance matrix be denoted as M and the obtained performance matrix for a given trained DL model be T. By intuition, the matrix T that does not deviate much from the ideal matrix M should have a higher trust score. The trust score is mathematically computed as follows, where, W = 4 × I 5 · 1 5 provides the weight corresponding to each task pair, I is an identity matrix and 1 5 is a ones matrix, each of dimensionality 5 × 5. Since for each preserving task, there are four suppressing tasks, the deviation of the preserving task from the ideal matrix is scaled by a factor of four to normalize the computation. Note that if the diagonal elements perform poorly, the concern is on the performance of the model. On the contrary, if the non-diagonal elements has a higher performance, the concern is on the trust of a model from a privacy preservation perspective. The proposed metric implements this notion to compute the trustworthiness of a trained DL model. The trust score is bounded between. By Trust scores obtained after various suppression techniques for Inception-v1. It can be observed that using random labels for unknown tasks, we could improve the trustworthiness. empirical analysis, we observe that a trust score above 0.9 is highly desirable, a trust score between 0.8 and 0.9 is practically acceptable, and any score below 0.8 is considered poor. The trust score of the ideal matrix is 1, while the trust score of a 1 5 (all task classification performance is 100%) is 0.6259. To understand the sensitivity of the proposed metric, let us assume that in the ideal matrix, any one non-diagonal element is changed to 1 which in a trust score of 0.98125. Thus, any reduction of (1 -0.98125) = 0.0175 in the trust score corresponds to one additional unwanted task being learnt by the classifier. In this section, we show the experimental and perform analysis of the proposed framework. Initially, we measure the trustworthiness of the existing models. We then experimentally demonstrate suppression of different tasks in various experimental settings. All the experiments are performed using the PreserveTask dataset. For additional and detailed comparison with other techniques, please refer to the appendix. Consider a popular deep learning model, Inception-v1 consisting of 22 computational layers. The model was trained from scratch using the PreserveTask for the task of shape classification, providing 99.98%. In order to study, if this deep learning model learnt additional visual attributes, as well, the last flatten layer's output (4096 × 1) were extracted. Four different two-hidden layer neural network classifiers were trained 2 using the extracted features to predict size, color, location, and color of the objects. The prediction accuracies were 97.29%, 51.25%, 99.98%, 92.05%, respectively for the four tasks. It can be observed that the performance of size, location, and prediction are really high proving that the features obtained from Inception v1 model has features corresponding to these tasks as well. Also, it can be observed that the color prediction performance is very low, as shape and color prediction are inherently independent tasks. The similar experiment is repeated for training the Inception v1 model on one task and using the learnt feature to predict the performance of other tasks, and the are shown in Figure 4. Ideally, only the diagonal elements of this confusion matrix should have higher accuracies (red in color) while the rest of the prediction should have lower accuracies (green in color). Accordingly, the trust score of the trained Inception-v1 model (proposed in section 4.1) was found to be 0.7530, which is very poor. In order to further demonstrate that this additional intelligence is not a property of just Inception-v1 model, similar experiments are performed using four other popular deep learning models: VGG16, VGG19, MobileNet, and DenseNet. The trust scores of all the DL models are shown in Figure 5. It can be observed that out of these five models, Inception-v1 and DenseNet has the lowest trust score while MobileNet has the highest trust score. While one could argue that the Inception-v1 model learns highly generic features supporting multi-task and transfer learning, from a privacy preservation perspective, the model is found to have a poor trust score. This leads to the open question, "Do models always needs to be additionally intelligent, and if not, how to suppress them?" In this section, we perform experiments to suppress the tasks that are known apriori during training, that is, the ground truth labels of the suppression task is available. For simplicity, in demonstrating the experimental , we assume that one task is to be preserved and one task is to be suppressed, using the Inception-v1 model. This experimental setting is similar to the approach explained in Figure 3 (b). The gradient reversal (GR) layer unlearns the suppressed task, while learning the preserved task. In order to compare the performance of GR, we also use a customized negative loss function which minimizes the loss obtained for the preserved task while maximizing the loss obtained for the suppressed task, weighted by a constant factor. The features eventually extracted from the flatten layer has to show similar performance on the preserved task while reduced performance on the suppressed task. Colour accuracy across experiments Figure 7: Comparison of color prediction performance with and without using the different task suppression mechanisms. It can be observed that using random labels reduces the performance of color prediction irrespective of whether the preserved task was shape or size prediction. corresponding trust scores are shown in Figure 5. It can be observed that suppressing known tasks using GR layer improves the trust of the baseline model from 0.7530 to 0.8563. The obtained in the previous section made the assumption that the ground truth labels of the suppression task have to be available while training the Inception-v1 model. In an attempt to break that assumption, the experimental setting discussed in Figure 3 (c) is performed. Instead of the actual ground truth labels of a particular task, randomly generated n-class labels are used during every mini-batch. Thus, for the same mini-batch training in the next epoch, a different set of random class labels are generated to be maximized. This ensures that the model does not memorize a single suppression task, but, learns to suppress all possible n-class classification tasks. Figure 6 (c) and (d) demonstrates the obtained by using random class labels. In comparison with Figure 4, it can be observed that using random class performs well in certain settings. For example, while trying to preserve the shape features and suppressing the prediction capacity of color, the original model's prediction performance of 92.05% reduced to 87.06% by using the actual labels of color, while further reduced to 33.37% while using random 3-class labels. It is further highlighted in Figure 7 where color prediction is chosen as the task to be suppressed, while shape and size are independently being preserved. It can be observed that the proposed framework of using random labels, reduces the performance of color prediction from 51.25% to 26.83% when using actual labels and 17.94% when using random labels, when shape prediction was the preserved task. A similar performance reduction from 35.59% to 14.29% is observed when size prediction was the preserved task. We conclude that using random labels for task suppression produces a comparable trust score to using known labels while producing better than the baseline trust score of a DL model. Colored MNIST Dataset: We introduced two additional tasks of foreground and color prediction tasks into the MNIST dataset. As shown in Figure 8, colored MNIST images are created by randomly assigning one of the 10 possible foreground colors and one of the different 10 possible colors. Similar assignment is performed in both training and test dataset, to maintain the standard experimental protocol. MobileNet model was trained from scratch to obtained a baseline trust score of 0.756. After using our framework for task suppression with random labels and gradient reversal based training on the suppression branch, we observed that the MobileNet models trust scores increased to 0.824. In Figure 8 (middle), the TSNE plot shows that when the model is learnt only for shapes, the features for'red' and'cyan' colored images are still separable. However, after suppressing the color prediction task using the proposed framework, the features'red' and'cyan' colored images are scattered and no longer separable, as shown in Figure 8 (right). Diversity in Faces (DiF) Dataset: In DiF dataset (Merler et al.), we considered the tasks of gender (two class) and pose (three class) classification. The aim is learn (preserve) only one of these while suppressing the other. Since, the dataset was highly skewed for different classes, we considered a subset of 39296 images with equal class balance 3. We trained Inception-v1 model on this dataset from scratch and obtained a trust score of 0.7497. Using our framework for task suppression with GR layer and known class labels, the trust score of the model increased to 0.8606. Additionally, with random unknown class labels, we observed that the model's trust scores increased to 0.9069. In IMDB-Wiki dataset , we considered the tasks of gender (two class) and age (ten class) classification. The cropped face images of the Wiki dataset are used to train the DenseNet model (the second least trusted model according to our trust scores). The trained model provided a baseline trust score of 0.7846. After using our framework for task suppression and known class labels, the trust score of DenseNet model increased to 0.7883. Also, with random unknown class labels, we observed that the model's trust scores increased to 0.7860. Thus, our framework for measuring and improving a DL model's trust has lots of practical applications. A face recognition system or a face image based gender recognition system can now be deployed with an additional trust on the model's intelligence level. In this research, we showcased a model-agnostic framework for measuring and improving the trustworthiness of a model from a privacy preservation perspective. The proposed framework did not assume the need for the suppression task labels during train time, while, similar performance could be obtained by training using random classification boundaries. A novel simulated benchmark dataset called PreserveTask was created to methodically evaluate and analyze a DL model's capability in suppressing shared task learning. This dataset opens up further research opportunities in this important and practically necessary research domain. Experimentally, it was shown that popular DL models such as VGG16, VGG19, Inception-v1, DenseNet, and MobileNet show poor trust scores and tend to be more intelligent than they were trained for. Also, we show a practical case study of our proposed approach in face attribute classification using: (i) Diversity in Faces (DiF) and (ii) IMDB-Wiki datasets. We would like to extend this work by studying the effect of multi-label classification tasks during suppression. This supplementary material contains all the detailed hyper-parameters used by different models that we trained, to aid in reproducing the that we showed in the research paper. Additionally, we provide more detailed analysis and visualizations of the , that could not be included in the paper due to space constraints. Five different baseline deep learning models were used in the experiments: Inception-v1, VGG16, VGG19, DenseNet, and MobileNet. The different parameters and the training process used in these experiments are shown below: • The data is z-normalized to have a zero mean and unit standard deviation, before being provided to the models for training. • The standard architectures of Inception-v1, VGG16, VGG19, DenseNet, and MobileNet are borrowed from the default implementations in the Keras library. • The deep learning models were trained with categorical cross-entropy and Adam optimizer with parameters as learning rate = 0.0001 and amsgrad set as F alse. For all the experiments, a two hidden layer neural network is used as a classifier. This is to maintain consistency of the same classifier across all the experiments. • The architecture is Dense → Dropout (0.5) → Dense → Dropout (0.3) → Dense (num of classes) • Each of the Dense layer has a ReLU activation function. • categorical cross-entropy is used as the loss function with Adam as the optimizer, having parameter values as learning rate = 0.0001 and amsgrad set as F alse. • 20% of the data is used as validation data and the model is trained for 100 epochs with early stopping. • Batch size of 32 was used to make the computation faster and the experiments were run using 1 × K80 GPU. In this section, we are including additional analysis, visualizations, and charts of the presented in the main paper. In order to aid better comparison, we include the charts and presented in the main paper also here, so that the supplementary could be read in an independent manner. Figure 9: Trust scores obtained for various DL models. It can be observed that, of the five models, the Inception-v1 and DenseNet has the least trust score while MobileNet has the highest. Figure 15: Trust scores obtained after various suppression techniques. It can be observed that even using random labels for unknown tasks, we could improve the trustworthiness of the Inception-v1 model on the PreserveTask dataset. Figure 16: The performance matrix heat-map, after suppressing a known task using negative loss, detailing the shared task performance of Inception-v1 model on the PreserveTask dataset. Figure 17: The performance matrix heat-map, after suppressing a known task using GR layer, detailing the shared task performance of Inception-v1 model on the PreserveTask dataset. Figure 18: The performance matrix heat-map, after suppressing a unknown task using negative loss, detailing the shared task performance of Inception-v1 model on the PreserveTask dataset. Figure 19: The performance matrix heat-map, after suppressing a unknown task using GR layer, detailing the shared task performance of Inception-v1 model on the PreserveTask dataset. Figure 20: Trust scores obtained in the Diversity in Faces (DiF) dataset after various suppression techniques. It can be observed that even using random labels for unknown tasks, we could improve the trustworthiness of the Inception-v1 model.

represent the tldr for finding the corresponding scientific article:

Can we trust our deep learning models? A framework to measure and improve a deep learning model's trust during training.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Examples:

"This paper proposes an effective generic sequence model which leverages the strengths of both RNNs and Multi-head attention." == "Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. Recurrent Neural Networks (RNNs) especially its variants such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have achieved great success in a wide range of sequence learning tasks including language modeling, speech recognition, recommendation, etc (; ; ; ;). Despite their success, however, the recurrent structure is often troubled by two notorious issues. First, it easily suffers from gradient vanishing and exploding problems, which largely limits their ability to learn very long-term dependencies . Second, the sequential nature of both forward and backward passes makes it extremely difficult, if not impossible, to parallelize the computation, which dramatically increases the time complexity in both training and testing procedure. Therefore, many recently developed sequence learning models have completely jettisoned the recurrent structure and only rely on convolution operation or attention mechanism that are easy to parallelize and allow the information flow at an arbitrary length. Two representative models that have drawn great attention are Temporal Convolution Networks(TCN) and Transformer . In a variety of sequence learning tasks, they have demonstrated comparable or even better performance than that of RNNs (; ;). The remarkable performance achieved by such models largely comes from their ability to capture long-term dependencies in sequences. In particular, the multi-head attention mechanism in Transformer allows every position to be directly connected to any other positions in a sequence. Thus, the information can flow across positions without any intermediate loss. Nevertheless, there are two issues that can harm the effectiveness of multi-head attention mechanism for sequence learning. The first comes from the loss of sequential information of positions as it treats every position identically. To mitigate this problem, Transformer introduces position embeddings, whose effects, The illustration of one layer of R-Transformer. There are three different networks that are arranged hierarchically. In particular, the lower-level is localRNNs that process positions in a local window sequentially (This figure shows an example of local window of size 3); The middle-level is multi-head attention networks which capture the global long-term dependencies; The upper-level is Position-wise feedforward networks that conduct non-linear feature transformation. These three networks are connected by a residual and layer normalization operation. The circles with dash line are the paddings of the input sequence however, have been shown to be limited . In addition, it requires considerable amount of efforts to design more effective position embeddings or different ways to incorporate them in the learning process . Second, while multi-head attention mechanism is able to learn the global dependencies, we argue that it ignores the local structures that are inherently important in sequences such as natural languages. Even with the help of position embeddings, the signals at local positions can still be very weak as the number of other positions is significantly more. To address the aforementioned limitations of the standard Transformer, in this paper, we propose a novel sequence learning model, termed as R-Transformer. It is a multi-layer architecture built on RNNs and the standard Transformer, and enjoys the advantages of both worlds while naturally avoids their respective drawbacks. More specifically, before computing global dependencies of positions with the multi-head attention mechanism, we firstly refine the representation of each position such that the sequential and local information within its neighborhood can be compressed in the representation. To do this, we introduce a local recurrent neural network, referred to as LocalRNN, to process signals within a local window ending at a given position. In addition, the LocalRNN operates on local windows of all the positions identically and independently and produces a latent representation for each of them. In this way, the locality in the sequence is explicitly captured. In addition, as the local window is sliding along the sequence one position by one position, the global sequential information is also incorporated. More importantly, because the localRNN is only applied to local windows, the aforementioned two drawbacks of RNNs can be naturally mitigated. We evaluate the effectiveness of R-Transformer with a various of sequence learning tasks from different domains and the empirical demonstrate that R-Transformer achieves much stronger performance than both TCN and standard Transformer as well as other state-of-the-art sequence models. The rest of the paper is organized as follows: Section 2 discusses the sequence modeling problem we aim to solve; The proposed R-Transformer model is presented in Section 3. In Section 4, we describe the experimental details and discuss the . The related work is briefly reviewed in Section 5. Section 6 concludes this work. Before introducing the proposed R-Transformer model, we formally describe the sequence modeling problem. Given a sequence of length N: x 1, x 2, · · ·, x N, we aim to learn a function that maps the input sequence into a label space Y: where y ∈ Y is the label of the input sequence. Depending on the definition of label y, many tasks can be formatted as the sequence modeling problem defined above. For example, in language modeling task, x t is the character/word in a textual sentence and y is the character/word at next position ; in session-based recommendation, x t is the user-item interaction in a session and y is the future item that users will interact with ; when x t is a nucleotide in a DNA sequence and y is its function, this problem becomes a DNA function prediction task . Note that, in this paper, we do not consider the sequence-to-sequence learning problems. However, the proposed model can be easily extended to solve these problems and we will leave it as one future work. The proposed R-Transformer consists of a stack of identical layers. Each layer has 3 components that are organized hierarchically and the architecture of the layer structure is shown in Figure 1. As shown in the figure, the lower level is the local recurrent neural networks that are designed to model local structures in a sequence; the middle level is a multi-head attention that is able to capture global long-term dependencies; and the upper level is a position-wise feedforward networks which conducts a non-linear feature transformation. Next, we describe each level in detail. Sequential data such as natural language inherently exhibits strong local structures. Thus, it is desirable and necessary to design components to model such locality. In this subsection, we propose to take the advantage of RNNs to achieve this. Unlike previous works where RNNs are often applied to the whole sequence, we instead reorganize the original long sequence into many short sequences which only contain local information and are processed by a shared RNN independently and identically. In particular, we construct a local window of size M for each target position such that the local window includes M consecutive positions and ends at the target position. Thus, positions in each local window form a local short sequence, from which the shared RNN will learn a latent representation. In this way, the local structure information of each local region of the sequence is explicitly incorporated in the learned latent representations. We refer to the shared RNN as LocalRNN. Comparing to original RNN operation, LocalRNN only focuses on local short-term dependencies without considering any long-term dependencies. Figure 2 shows the different between original RNN and LocalRNN operations. Concretely, given the positions x t−M +1, x t−M +2, · · ·, x t of a local short sequence of length M, the LocalRNN processes them sequentially and outputs M hidden states, the last of which is used as the representation of the local short sequences: where RNN denotes any RNN cell such as Vanilla RNN cell, LSTM, GRU, etc. To enable the model to process the sequence in an auto-regressive manner and take care that no future information is available when processing one position, we pad the input sequence by (M − 1) positions before the start of a sequence. Thus, from sequence perspective, the LocalRNN takes an input sequence and outputs a sequence of hidden representations that incorporate information of local regions: The localRNN is analogous to 1-D Convolution Neural Networks where each local window is processed by convolution operations. However, the convolution operation completely ignores the sequential information of positions within the local window. Although the position embeddings have been proposed to mitigate this problem, a major deficiency of this approach is that the effectiveness of the position embedding could be limited; thus it requires considerable amount of extra efforts . On the other hand, the LocalRNN is able to fully capture the sequential information within each window. In addition, the one-by-one sliding operation also naturally incorporates the global sequential information. Discussion: RNNs have long been a dominating choice for sequence modeling but it severely suffers from two problems -The first one is its limited ability to capture the long-term dependencies and the second one is the time complexity, which is linear to the sequence length. However, in LocalRNN, these problems are naturally mitigated. Because the LocalRNN is applied to a short sequence within a local window of fixed size, where no long-term dependency is needed to capture. In addition, the computation procedures for processing the short sequences are independent of each other. Therefore, it is very straightforward for the parallel implementation (e.g., using GPUs), which can greatly improve the computation efficiency. The RNNs at the lower level introduced in the previous subsection will refine representation of each positions such that it incorporates its local information. In this subsection, we build a sub-layer on top of the LocalRNN to capture the global long-term dependencies. We term it as pooling sublayer because it functions similarly to the pooling operation in CNNs. Recent works have shown that the multi-head attention mechanism is extremely effective to learn the long-term dependencies, as it allows a direct connection between every pair of positions. More specifically, in the multihead attention mechanism, each position will attend to all the positions in the past and obtains a set of attention scores that are used to refine its representation. Mathematically, given current representations h 1, h 2, · · ·, h t, the refined new representations u t are calculated as: where head k (h t) is the of k th attention pooling and W o is a linear projection matrix. Considering both efficiency and effectiveness, the scaled dot product is used as the attention function . Specifically, head i (h t) is the weighted sum of all value vectors and the weights are calculated by applying attention function to all the query, key pairs: where q, k i, and v i are the query, key, and value vectors and d k is the dimension of k i. Moreover, q, k i, and v i are obtained by projecting the input vectors into query, key and value spaces, respectively . They are formally defined as: where W q, W k and W v are the projection matrices and each attention pooling head i has its own projection matrices. As shown in Eq., each head i is obtained by letting h t attending to all the "past" positions, thus any long-term dependencies between h t and h i can be captured. In addition, different heads will focus on dependencies in different aspects. After obtaining the refined representation of each position by the multi-head attention mechanism, we add a position-wise fully connected feed-forward network sub-layer, which is applied to each position independently and identically. This feedforward network transforms the features non-linearly and is defined as follows: Following , We add a residual and layernorm connection between all the sub-layers. With all the aforementioned model components, we can now give a formal description of the overall architecture of an N -layer R-Transformer. For the i th layer (i ∈ {1, 2, · · · N}): where T is the length of the input sequence and x i t is the input position of the layer i at time step t. Comparing with TCN: R-Transformer is partly motivated by the hierarchical structure in , thus, we make a detailed comparison here. In TCN, the locality in sequences in captured by convolution filters. However, the sequential information within each receptive field is ignored by convolution operations. In contrast, the LocalRNN structure in R-Transformer can fully incorporate it by the sequential nature of RNNs. For modeling global long-term dependencies, TCN achieves it with dilated convolutions that operate on nonconsecutive positions. Although such operation leads to larger receptive fields in lower-level layers, it misses considerable amount of information from a large portion of positions in each layer. On the other hand, the multi-head attention pooling in R-Transformer considers every past positions and takes much more information into consideration than TCN. The proposed R-Transformer and standard Transformer enjoys similar long-term memorization capacities thanks to the multi-head attention mechanism . Nevertheless, two important features distinguish R-Transformer from the standard Transformer. First, R-Transformer explicitly and effectively captures the locality in sequences with the novel LocalRNN structure while standard Transformer models it very vaguely with multi-head attention that operates on all of the positions. Second, R-Transformer does not rely on any position embeddings as Transformer does. In fact, the benefits of simple position embeddings are very . In the next section, we will empirically demonstrate the advantages of R-Transformer over both TCN and the standard Transformer. Since the R-Transformer is a general sequential learning framework, we evaluate it with sequential data from various domains including images, audios and natural languages. We mainly compare it with canonical recurrent architectures (Vanilla RNN, GRU, LSTM) and two of the most popular generic sequence models that do not have any recurrent structures, namely, TCN and Transformer. However, since the majority of existing efforts to enhance Transformer are for natural languages, in the natural language evaluation, we also include one recent advanced Transformer, i.e., Transformer-XL. For all the tasks, Transformer and R-Transformer were implemented with Pytorch and the for canonical recurrent architectures and TCN were directly copied from as we follow the same experimental settings. In addition, to make the comparison fair, we use the same set of hyperparameters (i.e, hidden size, number of layers, number of heads) for R-Transformer and Transformer. Moreover, unless specified otherwise, for training, all models are trained with same optimizer and learning rate is chosen from the same set of values according to validation performance. In addition, the learning rate annealed such that it is reduced when validation performance reaches plateau. This task is designed to test model ability to memorize long-term dependencies. It was firstly proposed by and has been used by many previous works (; ;). Following previous settings, we rescale each 28 × 28 image in MNIST dataset into a 784 × 1 sequence, which will be classified into ten categories (each image corresponds to one of the digits from 0 to 9) by the sequence models. Since the rescaling could make pixels that are connected in the origin images far apart from each other, it requires the sequence models to learn very long-term dependencies to understand the content of each sequence. The dataset is split into training and testing sets as same as the default ones in Pytorch(version 1.0.0) 2. The model hyperparameters and classification accuracy are reported in Table 1. From the table, it can be observed that firstly, RNNs based methods generally perform worse than others. This is because the input sequences exhibit very long-term dependencies and it is extremely difficult for RNNs to memorize them. On the other hand, methods that build direct connections among positions, i.e., Transformer, TCN, achieve much better . It is also interesting to see that TCN is slightly better than Transformer, we argue that this is because the standard Transformer cannot model the locality very well. However, our proposed R-Transformer that leverages LocalRNN to incorporate local information, has achieved better performance than TCN. -3.46 LSTM -3.29 TCN 4 /150 3.07 Transformer 3/160 3.34 R-Transformer 3/160 2.37 Next, we evaluate R-Transformer on the task of polyphonic music modeling with Nottingham dataset . This dataset collects British and American folk tunes and has been commonly used in previous works to investigate the model's ability for polyphonic music modeling (; ;). Following the same setting in , we split the data into training, validation, and testing sets which contains 694, 173 and 170 tunes, respectively. The learning rate is chosen from {5e −4, 5e −5, 5e −6} and dropout with probability of 0.1 is used to avoid overfitting. Moreover, gradient clipping is used during the training process. We choose negative log-likelihood (NLL) as the evaluation metrics and lower value indicates better performance. The experimental are shown in Table 2. Both LTSM and TCN outperform Transformer in this task. We suspect this is because these music tunes exhibit strong local structures. While Transformer is equipped with multi-head attention mechanism that is effective to capture long-term dependencies, it fails to capture local structures in sequences that could provide strong signals. On the other hand, R-Transformer enhanced by LocalRNN has achieved much better than Transformer. In addition, it also outperforms TCN by a large margin. This is expected because TCN tends to ignore the sequential information in the local structure, which can play an important role as suggested by . In this subsection, we further evaluate R-Transformer's ability on both character-level and wordlevel language modeling tasks. The dataset we use is PennTreebank(PTB) that contains 1 million words and has been extensively used by previous works to investigate sequence models (; ; ;). For character-level language modeling task, the model is required to predict the next character given a context. Following the experimental settings in , we split the dataset into train-ing, validation and testing sets that contains 5059K, 396K and 446K characters, respectively. For Transformer and R-Transformer, the learning rate is chosen from {1, 2, 3} and dropout rate is 0.15. Gradient clipping is also used during the training process. The bpc is used to measure the predicting performance. For word-level language modeling, the models are required to predict the next word given the contextual words. Similarly, we follow previous works and split PTB into training, validation, and testing sets with 888K, 70K and 79K words, respectively. The vocabulary size of PTB is 10K. As with character-level language modeling,the learning rate is chosen from {1, 2, 3} for Transformer and R-Transformer and dropout rate is 0.35. In this task, we also add Transformer-XL as one baseline, which has been particularly designed for language modeling tasks and has achieved state-of-the-art performance. Note that to make the comparison fair, we apply the same model configuration, i.e., number of layers, to Transformer-XL. All other settings such as optimizer are the same as its original ones. The learning rate is chosen from {0.01, 0.001, 0.0001} and its best validation performance is achieved with 0.001. Note that, except dropout, no other regularization tricks such as variational dropout and weight dropout are applied. The prediction performance is evaluated with perplexity, the lower value of which denotes better performance. The experimental of character-level and word-level language modeling tasks are shown in Table 3 and Table 4, respectively. Several observations can be made from the Table 3. First, Transformer performs only slightly better than RNNs while much worse than other models. The reason for this observation is similar to the case of polyphonic music modeling task that language exhibits strong local structures and standard Transformer can not fully capture them. Second, TCN achieves better than all of the RNNs, which is attributed to its ability to capture both local structures and long-term dependencies in languages. Notably, for both local structures and longterm dependencies, R-Transformer has more powerful components than TCN, i.e., LocalRNN and Multi-head attention. Therefore, it is not surprising to see that R-Transformer achieves significantly better . Table 4 presents the for word-level language modeling. Similar trends are observed, with the only exception that LSTM achieves the best among all the methods. In addition, the of Transformer-XL is only slightly better than R-transformer. Considering the fact that Transformer-XL is specifically designed for language modeling and employs the recurrent connection of segments , this suggests the limited contribution of engineered positional embeddings. In summary, experimental have shown that the standard Transformer can achieve better than RNNs when sequences exhibit very long-term dependencies, i.e., sequential MNIST while its performance can drop dramatically when strong locality exists in sequences, i.e., polyphonic music and language. Meanwhile, TCN is a very strong sequence model that can effectively learn both local structures and long-term dependencies and has very stable performance in different tasks. More importantly, the proposed R-Transformer that combines a lower level LocalRNN and a higher level multi-head attention, outperforms both TCN and Transformer by a large margin consistently in most of the tasks. The experiments are conducted on various sequential learning tasks with datasets from different domains. Moreover, all experimental settings are fair to all baselines. Thus, the observations from the experiments are reliable with the current experimental settings. However, due to the computational limitations, we are currently restricted our evaluation settings to moderate model and dataset sizes. Thus, more evaluations on big models and large datasets can make the more convincing. We would like to leave this as one future work. Recurrent Neural Networks including its variants such LSTM and GRU have long been the default choices for generic sequence modeling. A RNN sequentially processes each position in a sequence and maintains an internal hidden state to compresses information of positions that have been seen. While its design is appealing and it has been successfully applied in various tasks, several problems caused by its recursive structures including low computation efficiency and gradient exploding or vanishing make it ineffective when learning long sequences. Therefore, in recent years, a lot of efforts has been made to develop models 3 /700 78.93 TCN 4 without recursive structures and they can be roughly divided into two categories depending whether they rely on convolutions operations or not. The first category includes models that mainly built on convolution operations. For example, van den Oord et al. have designed an autoregressive WaveNet that is based on causal filters and dilated convolution to capture both global and local information in raw audios . Ghring et al. has successfully replace traditional RNN based encoder and decoder with convolutional ones and outperforms LSTM setup in neural machine translation tasks (; . Moreover, researchers introduced gate mechanism into convolutions structures to model sequential dependencies in languages . Most recently, a generic architecture for sequence modeling, termed as Temporal Convolutional Networks (TCN), that combines components from previous works has been proposed in . Authors in have systematically compared TCN with canonical recurrent networks in a wide range of tasks and TCN is able achieve better performance in most cases. Our R-transformer is motivated by works in this group in a sense that we firstly models local information and then focus on global ones. The most popular works in second category are those based on multi-head attention mechanism. The multi-head attention mechanism was firstly proposed in , where impressive performance in machine translation task has been achieved with Transformer. It was then frequently used in other sequence learning models (; ;). The success of multi-head attention largely comes from its ability to learn long-term dependencies through direct connections between any pair of positions. However, it heavily relies on position embeddings that have limited effects and require a fair amount of effort to design effective ones. In addition, our empirical shown that the local information could easily to be ignored by multi-head attention even with the existence of position embeddings. Unlike previously proposed Transformer-like models, R-Transformer in this work leverages the strength of RNN and is able model the local structures effectively without the need of any position embeddings. In this paper, we propose a novel generic sequence model that enjoys the advantages of both RNN and the multi-head attention while mitigating their disadvantages. Specifically, it consists of a LocalRNN that learns the local structures without suffering from any of the weaknesses of RNN and a multi-head attention pooling that effectively captures long-term dependencies without any help of position embeddings. In addition, the model can be easily implemented with full parallelization over the positions in a sequence. The empirical on sequence modeling tasks from a wide range of domains have demonstrated the remarkable advantages of R-Transformer over state-of-the-art nonrecurrent sequence models such as TCN and standard Transformer as well as canonical recurrent architectures."

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.