Question: Represent the scientific article to find a short TLDR

Question: The carbon footprint of natural language processing (NLP) research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation to improve the efficiency of the Biaffine dependency parser which obtains state-of-the-art performance with respect to accuracy and parsing speed . When distilling to 20% of the original model’s trainable parameters, we only observe an average decrease of ∼1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.26x (1.21x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80% for some treebanks. Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank. Ethical NLP research has recently gained attention . For example, the environmental cost of AI research has become a focus of the community, especially with regards to the development of deep neural networks . Beyond developing systems to be greener, increasing the efficiency of models makes them more cost-effective, which is a compelling argument even for people who might downplay the extent of anthropogenic climate change. In conjunction with this push for greener AI, NLP practitioners have turned to the problem of developing models that are not only accurate but also efficient, so as to make them more readily deployable across different machines with varying computational capabilities (; ;). This is in contrast with the recently popular principle of make it bigger, make it better . Here we explore teacher-student distillation as a means of increasing the efficiency of neural network systems used to undertake a core task in NLP, dependency parsing. To do so, we take a state-of-theart (SoTA) Biaffine parser from. The Biaffine parser is not only one of the most accurate parsers, it is the fastest implementation by almost an order of magnitude among state-of-the-art performing parsers. Contribution We utilise teacher-student distillation to compress Biaffine parsers trained on a diverse subset of Universal Dependency (UD) treebanks. We find that distillation maintains accuracy performance close to that of the full model and obtains far better accuracy than simply implementing equivalent model size reductions by changing the parser's network size and training regularly. Furthermore, we can compress a parser to 20% of its trainable parameters with minimal loss in accuracy and with a speed 2.26x (1.21x) faster than that of the original model on CPU (GPU). Dependency parsing is a core NLP task where the syntactic relations of words in a sentence are encoded as a well-formed tree with each word attached to a head via a labelled arc. Figure 1 shows an example of such a tree. The syntactic information attained from parsers has been shown to benefit a number of other NLP tasks such as relation extraction, machine translation , and sentiment analysis . The son of the cat hunts the rat. Table 1 shows performance details of current SoTA dependency parsers on the English Penn Treebank (PTB) with predicted POS tags from the Stanford POS tagger . The Biaffine parser of offers the best trade-off between accuracy and parsing speed with the HPSG parser of achieving the absolute best reported accuracy but with a reported parsing speed of roughly one third of the Biaffine's parsing speed. It is important to note that direct comparisons between systems with respect to parsing speed are wrought with compounding variables, e.g. different GPUs or CPUs used, different number of CPU cores, different batch sizes, and often hardware is not even reported. Biaffine Table 1: Speed and accuracy performance for SoTA parsers and parsers from our distillation method, Biaffine-Dπ compressing to π% of the original model, for the English PTB with POS tags predicted from the Stanford POS tagger. In the first table block, † denotes values taken from the original paper, ‡ from. Values with no superscript (second and third blocks) are from running the models on our system locally with a single CPU core for both CPU and GPU speeds (averaged over 5 runs) and with a batch size of 4096 with GloVe 100 dimension embeddings. We therefore run a subset of parsers locally to achieve speed measurements in a controlled environment, also shown in Table 1: we compare a PyTorch implentation of the Biaffine parser (which runs more than twice as fast as the reported speed of the original implementation); the UUParser from which is one of the leading parsers for Universal Dependency (UD) parsing; a sequence-labelling dependency parser from which has the fastest reported parsing speed amongst modern parsers; and also distilled Biaffine parsers from our implementation described below. All speeds measured here are with the system run with a single CPU core for both GPU and CPU runs. Biaffine parser is a graph-based parser extended from the graph-based BIST parser to use a deep self-attention mechanism. This in a fast and accurate parser, as described above, and is used as the parser architecture for our experiments. More details of the system can be found in. Model compression has been under consideration for almost as long as neural networks have been utilised, e.g. introduced a pruning technique which removed weights based on a locally predicted contribution from each weight so as to minimise the perturbation to the error function. More recently, introduced a means of pruning a network up to 40 times smaller with minimal affect on performance. and utilised magnitude-based pruning to increase network generalisation. More specific to used absolute-magnitude pruning to compress neural machine translation systems by 40% with minimal loss in performance. However, pruning networks leaves them in an irregularly sparse state which cannot be trivially re-cast into less sparse architectures. Sparse tensors could be used for network layers to obtain real-life decreases in computational complexity, however, current deep learning libraries lack this feature. introduced structured pruning to account for this, but this kernel-based technique is restricted to convolutional networks. More recently pruned the heads of the attention mechanism in their neural machine translation system and found that the remaining heads were linguistically salient with respect to syntax, suggesting that pruning could also be used to undertake more interesting analyses beyond merely compressing models and helping generalisation. and developed distillation as a means of network compression from the work of , who compressed a large ensemble of networks into one smaller network. Teacher-student distillation is the process of taking a large network, the teacher, and transferring its knowledge to a smaller network, the student. Teacher-student distillation has successfully been exploited in NLP for machine translation, language modelling, and speech recognition (; ;). Latterly, it has also been used to distill task-specific knowledge from BERT. Other compression techniques have been used such as low-rank approximation decomposition , vector quantisation , and Huffman coding . For a more thorough survey of current neural network compression methods see. The essence of model distillation is to train a model and subsequently use the patterns it learnt to influence the training of a smaller model. For teacher-student distillation, the smaller model, the student, explicitly uses the information learnt by the larger original model, the teacher, by comparing the distribution of each model's output layer. We use the Kullback-Leibler divergence to calculate the loss between the teacher and the student: where P is the probability distribution from the teacher's softmax layer, Q is the probability distribution from the student's, and x is the input to the target layer for token w x in a given tree, t. For our implementation, there are two probability distributions for each model, one for the arc prediction and one for the label prediction. By using the distributions of the teacher rather than just using the predicted arc and label, the student can learn more comprehensively about which arcs and labels are very unlikely in a given context, i.e. if the teacher makes a mistake in its prediction, the distribution might still carry useful information such as having a similar probability for y g and y p which can help guide the student better rather than just learning to copy the teacher's predictions. In addition to the loss with respect to the teacher's distributions, the student model is also trained using the loss on the gold labels in the training data. We use cross entropy to calculate the loss on the student's predicted head classifications: where t is a tree in the treebank T, h is a head position for the set of heads H for a given tree, and h is the head position predicted by the student model. Similarly, cross entropy is used to calculate the loss on the predicted arc labels for the student model. The total loss for the student model is therefore: where L CE (h) is the loss for the student's predicted head positions, L CE (lab) is the loss for the student's predicted arc label, L KL (T h, S h) is the loss between the teacher's probability distribution for arc predictions and that of the student, and L KL (T lab, S lab) is the loss between label distributions. We train a Biaffine parser for a number of Universal Treebanks v2.4 (UD) and apply the teacher-student distillation method to compress these models into a number of different sizes. We use the hyperparameters from , but use a PyTorch implementation for our experiments which obtains the same parsing and runs faster than the reported speed of the original (see Table 1). 2 The hyperparameter values can be seen in Table 4. During distillation dropout is not used. Beyond lexical features, the model only utilises universal part-ofspeech (UPOS) tags. Gold UPOS tags were used for training and at runtime. Also, we used gold sentence segmentation and tokenisation. We opted to use these settings to compare models under homogeneous settings, so as to make reproducibility of and comparability with our easier. Data We use the subset of UD treebanks suggested by de from v2.4, so as to cover a wide range of linguistic features, linguistic typologies, and different dataset sizes. We make some changes as this set of treebanks was chosen from a previous UD version. We exchange Kazakh with Uyghur because the Kazakh data does not include a development set and Uyghur is a closely related language. We also exchange Ancient-Greek-Proiel for Ancient-Greek-Perseus because it contains more non-projective arcs (the number of arcs which cross another arc in a given tree) as this was the original justification for including Ancient Greek. We also included Wolof as African languages were wholly unrepresented in the original collection of suggested treebanks. Details of the treebanks pertinent to parsing can be seen in Table 2. We use pretrained word embeddings from FastText for all but Ancient Greek, for which we used embeddings from , and Wolof, for which we used embeddings from. When necessary, we used the algorithm of to reduce the embeddings to 100 dimensions. For each treebank we then acquired the following models: i Baseline 1: Full-sized model is trained as normal and undergoes no compression technique. ii Baseline 2: Model is trained as normal but with equivalent sizes of the distilled models (20%, 40%, 60%, and 80% of the original size) and undergoes no compression technique. These models have the same overall structure of baseline 1, with just the number of dimensions of each layer changed to in a specific percentage of trainable parameters of the full model. iii Distilled: Model is distilled using the teacher-student method. We have four models were the first is distilled into a smaller network with 20% of the parameters of the original, the second 40%, the third 60%, and the last 80%. The network structure and parameters of the distilled models are the exact same as those of the baseline 2 models. Table 2: Statistics for salient features with respect to parsing difficulty for each UD treebank used: number of trees, the number of data instances; average tree length, the length of each data instance on average; average arc length, the mean distance between heads and dependents; non.proj. arc pct, the percentage of non-projective arcs in a treebank. Base E, the baseline models of equivalent size to the distilled models; Distill, the distilled models; Base, the performance of the original full-sized model. Hardware For evaluating the speed of each model when parsing the test sets of each treebank we set the number of CPU cores to be one and either ran the parser using that solitary core or using a GPU (using a single CPU core too). The CPU used was an Intel Core i7-7700 and the GPU was an Nvidia GeForce GTX 1080. Experiment We compare the performance of each model on the aforementioned UD treebanks with respect to the unlabelled attachment score (UAS) which evaluates the accuracy of the arcs, and the labelled attachment score (LAS) which also includes the accuracy of the arc labels. We also evaluate the differences in inference time for each model on CPU and GPU with respect to sentences per second and tokens per second. We report sentences per second as this has been the measurement traditionally used in most of the literature, but we also use tokens per second as this more readily captures the difference in speed across parsers for different treebanks where the sentence length varies considerably. We also report the number of trainable parameters of each distilled model and how they compare to the baseline, as this is considered a good measure of how green a model is in lieu of the number of floating point operations (FPO) . 6 AND DISCUSSION Figure 2a shows the average attachment scores across all treebanks for the distilled models and the equivalent-sized base models against the size of the model relative to the original full model. There is a clear gap in performance between these two sets of models with roughly 2 points of UAS and LAS more for the distilled models. This shows that the distilled models do actually manage to leverage the information from the original full model. The full model's scores are also shown and it is clear that on average the model can be distilled to 60% with no loss in performance. When compressing to 20% of the full model, the performance only decreases by about 1 point for both UAS and LAS. Figures 3a and 3b show the differences in UAS and LAS for the models distilled to 20% and 80% respectively for each treebank when compared to the equivalent sized baseline model and the full baseline model. The distilled models far outperform the equivalent-sized baselines for all treebanks. It is clear that for the smaller model that some treebanks suffer more when compressed to 20% than others when compared to the full baseline model, e.g. Finnish-TDT and Ancient-Greek-Perseus. These two treebanks have the largest percentage of non-projective arcs (as can be seen in Table 2) which could account for the decrease in performance, with a more powerful model required to account for this added syntactic complexity. However, the two smallest treebanks, Tamil-TTB and Wolof-WTB, actually increase in accuracy when using distillation, especially Tamil-TTB, which is by far the smallest treebank, with an increase in UAS and LAS of about 4 points over the full base model. This is likely the of over-fitting when using the larger, more powerful model, so that reducing the model size actually helps with generalisation. These observations are echoed in the for the model distilled to 80%, where most treebanks lose less than a point for UAS and LAS against the full baseline, but have a smaller increase in performance over the equivalent-sized baseline. This makes sense as the model is still close in size to the full baseline and still similarly powerful. The increase in performance for Tamil-TTB and Wolof-WTB are greater for this distilled model, which suggests the full model doesn't need to be compressed to such a small model to help with generalisation. The full set of attachment scores from our experiments can be seen in Table 5 in the Appendix. With respect to how green our distilled models are, Table 3 shows the number of trainable parameters for each distilled model for each treebank alongside its corresponding full-scale baseline. We report these in lieu of FPO as, to our knowledge, no packages exist to calculate the FPO for neural network layers like LSTMs which are used in our network. These numbers do not depend on the hardware used and strongly correlate with the amount of memory a model consumes. Different algorithms do utilise parameters differently, however, the models compared here are of the same structure and use the same algorithm, so comparisons of the number of trainable model parameters do relate to how much work each respective model does compared to another. Figures 4 and 5 show the parsing speeds on CPU and GPU for the distilled models and for the full baseline model for sentence per second and token per second, respectively. The speeds are reported for different batch sizes as this obviously affects the speed at which a neural network can make predictions, but the maximum batch size that can be used on different systems varies significantly. As can be seen in Figures 4a and 5a, the limiting factor in parsing speed is the bottleneck of loading the data onto the GPU when using a batch size less than ∼1000 sentences. However, with a batch size of 4096 sentences, we achieve an increase in parsing speed of 21% over the full baseline model when considering tokens per second. As expected, a much smaller batch size is required to achieve increases in parsing speed when using a CPU. Even with a batch size of 32 sentences, the smallest model more than doubles the speed of the baseline. For a batch size of 4096, the distilled model compressed to 20% increases the speed of the baseline by 126% when considering tokens per second. A full breakdown of the parsing speeds for each treebank and each model when using a batch size of 4096 sentences is given in Table 6 in the Appendix. Figure 6 shows the attachment scores and the corresponding parsing speed against model size for the distilled model and the full baseline model. These plots clearly show that the cost in accuracy is neglible when compared to the large increase in parsing speed. So not only does this teacher-student distillation technique maintain the accuracy of the baseline model, but it achieves real compression and with it practical increases in parsing speed and with a greener implementation. In absolute terms, our distilled models are faster than the previously fastest parser using sequence labelling, as can be seen explicitly in Table 1 for PTB, and outperforms it by over 1 point with respect to UAS and LAS when compressing to 40%. Distilling to 20% in a speed 4x that of the sequence labelling model on CPU but comes at a cost of 0.62 points for UAS and 0.76 for LAS compared to the sequence labelling accuracies. Furthermore, the increase in parsing accuracy for the smaller treebanks suggests that distillation could be used as a more efficient way of finding optimal hyperparameters depending on the available data, rather than training numerous models with varying hyperparameter settings. There are numerous ways in which this distillation technique could be augmented to potentially retain more performance and even outperform the large baseline models, such as using teacher annealing introduced by where the distillation process gradually secedes to standard training. Beyond this, the structure of the distilled models can be altered, e.g. student models which are more shallow than the teacher models . This technique could further improve the efficiency of models and make them more environmentally friendly by reducing the depth of the models and therefore the total number of trainable parameters. Distillation techniques can also be easily expanded to other NLP tasks. Already attempts have been made to make BERT more wieldy by compressing the information it contains into task-specific models. But this can be extended to other tasks more specifically and potentially reduce the environmental impact of NLP research and deployable NLP systems. We have shown the efficacy of using the teacher-student distillation technique for dependency parsing by distilling a state-of-the-art parser implementation. The parser used for our experiments was not only accurate but already fast, meaning it was a strong baseline from which to see improvements. We obtained parsing speeds up to 2.26x (1.21x) faster on CPU (GPU) while only losing ∼1 point for both UAS and LAS when compared to the original sized model. Furthermore, the smallest model which obtains these only has 20% of the original model's trainable parameters, vastly reducing its environmental impact. A APPENDIX

Represent the TLDR for finding the corresponding scientific article

We increase the efficiency of neural network dependency parsers with teacher-student distillation.

represent this text to find a much much longer article about it with say hundreds-thousands of words
Examples:

"An unsupervised structure learning method for Parsimonious Deep Feed-forward Networks." == "Convolutional neural networks and recurrent neural networks are designed with network structures well suited to the nature of spacial and sequential data respectively. However, the structure of standard feed-forward neural networks (FNNs) is simply a stack of fully connected layers, regardless of the feature correlations in data. In addition, the number of layers and the number of neurons are manually tuned on validation data, which is time-consuming and may lead to suboptimal networks. In this paper, we propose an unsupervised structure learning method for learning parsimonious deep FNNs. Our method determines the number of layers, the number of neurons at each layer, and the sparse connectivity between adjacent layers automatically from data. The ing models are called Backbone-Skippath Neural Networks (BSNNs). Experiments on 17 tasks show that, in comparison with FNNs, BSNNs can achieve better or comparable classification performance with much fewer parameters. The interpretability of BSNNs is also shown to be better than that of FNNs. Deep neural networks have made breakthroughs in all kinds of machine learning tasks BID13 BID22, specifically with convolutional neural networks (CNNs) for tasks with spacial data BID17 and recurrent neural networks (RNNs) for tasks with sequential data. One of the key reasons for the effectiveness of CNNs and RNNs is the well-designed network structures together with the parameter sharing schemes. For example, in the convolution layers of CNNs, each neuron is connected to a local region in the input volume instead of all the input neurons. Besides, the neurons in the same channel share the same set of weights. This design utilizes the local and "stationary" properties of spacial data and consequently forms effective feature extractors. In addition, it also prevents CNNs from having an exploding number of parameters when the networks become deeper and deeper. However, in practice, there are also many data which are neither spacial nor sequential, and hence the only applicable neural networks are the standard feed-forward neural networks (FNNs). In contrast to CNN and RNN, FNN's network structure is simple. It consists of multiple layers of neurons and each layer is fully connected to the next layer up, without considering any correlations in data or among neurons. The network structure has two main shortcomings. The first is that, there can be high connection redundancies. As the number of layers and the number of neuron at each layer increase, the number of parameters increases quickly, which can cause severe overfitting. The other shortcoming is that, ignoring all the correlations existing in data weakens the model's strength (as a feature extractor) and hurts the model's interpretability. We are interested in learning parsimonious deep feed-forward neural networks. The goal is to learn FNNs which contain as few parameters as possible. Parsimonious FNNs are desirable for several reasons. Firstly, fewer parameters can ease overfitting. Secondly, parsimonious FNNs require less storage and computation than FNNs, which makes it possible to be run on devices like mobile phones. Lastly, parsimonious FNNs can have very flexible and different structures from each other depending on the specific tasks and data. This would help the models fit the data well and also have good interpretability. In general, it is desirable to solve a problem using the simplest model possible because it implies a good understanding of the problem. connections (x − h 1, h 1 − h 2) form the Backbone path. The narrow fully-connected layers (x − h 3, h 1 − h 3, h 2 − h 3) are the Skip-paths. The number of units at h 3 is relatively smaller than that at x, h 1 and h 2.Learning parsimonious FNNs is challenging mainly because we need to determine the sparse connectivity between layers. Network pruning is a potential way to achieve this. However, it requires to start from a network which is much larger than necessary for the task at hand. This can cause a lot of computations wasted on those useless connections. In addition, network pruning is not able to learn the number of units and number of layers. In this paper, we assume that data are generated by a sparse probabilistic model with multiple layers of latent variables, and view the feed-forward network to be built as a way to approximate the relationships between the observed variables and the top-level latent variables in the probabilistic model. The level 1 latent variables induce correlations among the observed variables. Therefore, it is possible to determine them by analysing how the observed variables are correlated. Similarly, by analysing how the level 1 latent variables are correlated, we can determine the level 2 latent variables, and so on. We empirically show that our method can significantly reduce the number of parameters in FNNs, and the ing model still achieves better or comparable than FNNs in 17 classification tasks. Network Structure Learning One early attempt to learn network structure for FNNs is the approach based on constructive algorithms BID1 BID3 BID18. These algorithms start from a small network and gradually add new neurons to the network until some stopping criterion are met (e.g. no more performance gain is observed). They require manuallydesigned strategies to decide how to connect new neurons to the existing network. Besides, each time when new neurons are introduced, the network needs to be retrained completely or partially. Lately, BID0 proposes to learn the structure of deep belief networks by using cascading Indian buffet process, which is very time-consuming. In BID6, the authors propose a structure learning method, based on hierarchical latent tree analysis BID21 BID5, for RBM-like models. The method automatically determines the number of hidden units and the sparse connections between layers. However, it is not tested on deep models and in supervised learning tasks. Recently, reinforcement learning BID2 BID34 and genetic algorithms BID27 BID31 are also applied to learning complex structures for CNNs. Generally, these methods require tens of thousands of full training runs before giving a feasible network structure, which is prohibitive for many applications. Network Pruning In contrast to constructive algorithms, network pruning starts from a large network and prune connections or neurons to achieve structure learning. Optimal Brain Damage BID8 and Optimal Brain Surgeon BID12 prune connections based on the Hessian matrix of the loss function. Recently, BID11 proposes to conduct pruning by iteratively pruning connections with absolute weight value smaller than a threshold and retraining the network. One drawback of the method is that the retraining process is time-consuming. BID10 proposes Dynamic Network Surgery which conducts parameter learning and connection pruning simultaneously and avoids the retraining process. Moreover, it also allows mistakenly pruned connections to be rebuilt in subsequent training. Similar to connection pruning, neurons pruning methods are proposed and tested in BID28; BID20. The main drawback of all these pruning methods is that, they require to start from a network which is larger than necessary for the task at hand. This causes some wasted computations on the useless connections or neurons. In addition, the number of layers is still set manually instead of learned from data. In this section, we present a method for learning parsimonious deep FNNs. The method is called Parsimonious Structure Analysis (PSA). PSA learns a model which contains two parts as shown in FIG0. The first is the main part of the model, called the Backbone. It is a wide, deep but sparse feed-forward path in the network. The second part is the Skip-paths. It consists of multiple narrow paths, each of which is a fully-connected layer. We call the ing model Backbone-Skippath Neural Network (BSNN). We will introduce how PSA learns the Backbone and the Skip-paths in Section 3.1 and Section 3.2 respectively. Structure learning for neural networks is challenging since generally the features in data do not always have apparent relationships as the units in convolutional networks. In a convolutional layer, units in a feature map are only connected to a group of units strongly correlated in the spacial dimension at the layer below. This significantly reduces the number of parameters in CNNs and is essential if we want to learn a very sparse structure. The same intuition can be applied to general data other than images in feed-forward neural networks. A hidden unit, detecting one particular feature such as co-occurrence pattern, should only be connected to a group of units that are strongly correlated in the layer below. However, unlike CNNs where the spatial correlation is apparent, the correlations of units in feed-forward neural networks are not easy to discover. In PSA, we propose to apply Hierarchical Latent Tree Analysis (HLTA) BID21 BID5 to identify the co-occurrence patterns among units and construct hidden units to explain the co-occurrence patterns. PSA treats the input features as a set of isolated random variables as in Figure 3 (a). Although no apparent spacial or sequential relationships exist among the variables, PSA seeks to discover the correlations among the variables and groups the highly correlated ones together. It starts from finding two most correlated variables to form one group and keeps expanding the group if necessary. Let S denotes the set of observed variables which haven't been included into any variable groups. PSA firstly computes the mutual information between each pair of observed variables. Then it picks the pair in S with the highest mutual information and uses them as the seeds of a new variable group G. New variables from S are then added to G one by one in descending order of their mutual information with variables already in G. Each time when a new variable is added into G, PSA builds two models (M 1 and M 2) with G as the observed variables. The two models are the best models with one single latent variable and two latent variables respectively, as shown in Figure 2. PSA computes the BIC scores of the two models and tests whether the following condition is met: DISPLAYFORM0 where D is the dataset and δ is a threshold which is usually set at 3 BID5. When the condition is met, the two latent variable model M 2 is not significantly better than the one latent variable model M 1. Correlations among variables in G are still well modeled using a single latent variable. Then PSA keeps on adding new variables to G. If the test fails, PSA takes the subtree in M 2 which doesn't contain the newly added variable and identifies the observed variables in it as a finalized variable group. The group is then removed from S. And the above process is repeated on S until all the variables in S are partitioned into disjoint groups. An efficient algorithm progressive EM ) is used to estimate the parameters in M 1 and M 2.As shown in Figure 2 (b), after the above process, all the observed variables are partitioned into disjoint groups such that the variables in each group are strongly correlated and their correlations can be explained using a single latent variables. Then PSA introduces a latent variable for each group and computes the mutual information among the latent variables. After that, it links up the Figure 4: Expanding the tree structure for the Backbone path: A three-layer structure is first learned (left). New connections are added to all the layers according to empirical conditional mutual information (middle). The connections between variables at the top layer are removed and the structure is finalized (right). latent variables to form a Chow-Liu tree BID7. The is a latent tree model BID26 BID32, as shown in Figure 2 (c). Parameter estimation for the model is done using the EM algorithm. Since the model is tree-structured, EM is efficient in this process. While the above procedure gives us a one-layer network, we seek to build deep model to capture the long-range correlations among variables. We perform the construction of deep structure in a layerwise manner. Using the obtained one-layer model, PSA converts the latent variables into observed ones through data completion. With this, another layer of latent variables can be learned in the same manner as the first layer by grouping the first-layer latent variables and linking up the groups, as in Figure 2 (d). Then the two models can be stacked up to form a three-layer network, with the latent variables in the higher layer capturing longer-range correlations of the observed variables. This procedure can be recursively conducted to build deep hierarchy until the number of variables at the top layer falls below a threshold K. And it in a hierarchical latent tree model BID21 BID5. While the above deep structure captures the most important correlations among the observed variables, the tree structure might cause underfitting for discovering non-trivial correlations. Thus we introduce additional links to model the salient interactions that are not captured by the tree model. For each latent variable V l at level l, PSA considers adding connections to link it to more nodes at level l − 1. To do so, PSA considers how closely V l is related to each node V l−1 at level l − 1 given the parent variable Z of V l−1. The strength of correlation is measured using the conditional mutual information: DISPLAYFORM0 The top N nodes with the highest I(V l, V l−1 |Z) are then connected to V l. After expanding the connections for all the layers, PSA removes the links among the variables at the top layer and uses the ing structure for the Backbone. The process of expanding tree structure is illustrated in Figure 4. Although the Backbone path is deep and wide, its sparsity can easily lead to model which cannot capture global features. For example, suppose there is an essential feature which is correlated to all the input features. When the Backbone path is very sparse, even after multiple layers of projections, it is still unlikely that there will be a feature in the model which is projected from all the input features. To tackle the above problem, we introduce Skip-paths to our BSNN. FIG0 shows the whole model structure of BSNN. The path from x to h 2 illustrates the Backbone path whose sparse structure is learned using the method we propose. To complement the the model's power of extracting features, narrow Skip-paths (x − h 3, h 1 − h 3, h 2 − h 3) are added to the model. The Skip-paths take all the feature layers in the Backbone as input and compress them to layers with a small number of units through fully-connected projections. After the structure for the Backbone path and the Skip-paths are determined, a classification layer or regression layer can then be added to the top of all the paths, utilizing all the features extracted. The network can then be trained using back-propagation algorithms as in normal neural networks. In experiment, we evaluate our method in 17 classification tasks. We consider applications where the data is neither spacial nor sequential. Unlike CNNs or RNNs where the structure is designed to exploit spatial or sequential correlation, few effort has been put to learn the structure of feedfoward neural networks, which have highly redudant parameters and is prone to overfit. Our proposed method learns the structure of feedforward neural network from data. It significantly reduces the model complexity and parameters while achieving better or comparable classification performance, and leads to models which are more interpretable. TAB0 gives a summary of all the datasets used in the experiment. We choose 12 tasks for chemical compounds classification and 5 tasks for text classification. All the datasets are published by previous researchers and are available to the public. 1 There are about 12,000 environmental chemical compounds in the dataset, each represented as its chemical structure. The tasks are to predict 12 different toxic effects for the chemical compounds. We treat them as 12 binary classification tasks. We filter out sparse features which are present in fewer than 5% compounds, and rescale the remaining 1,644 features to zero mean and unit variance. The dataset contains a training set and a test set, and we randomly sample 500 compounds from training data to build the validation set. All the experiments are run for three times and we report the average AUC together with the standard deviations. 2 We use 5 text classification datasets from BID33. After removing stop words, the top 10,000 frequent words in each dataset are selected as the vocabulary respectively and each document is represented as bag-of-words over the vocabulary. The validation set is randomly sampled from the original training samples. We run all the experiments for three times and report the average classification accuracies with the standard deviations. We compare our model with standard feed-forward neural networks (FNNs) and sparse neural networks whose weak connections are pruned (Pruned FNNs) in the 17 classification tasks. The models involved in the experiment are as follows:• BSNN: Backbone-Skippath Neural Network is the ing model of our method PSA. For all the tasks, we keep only 5% of the connections in the Backbone path and limit the number of units in the narrow Skip-paths to 100.• FNN: Feed-forward Neural Network is a standard fully-connected neural network. It is mainly composed of linear layers and activation functions. Each hidden unit is connected to all neurons in the previous layer. Information flows from low layers to high layers in a feed-forward manner.• Pruned FNN: Pruned Feed-forward Neural Network is trained by using the method proposed in BID11. We Firstly train a fully-connected FNN from scratch, and then prune out the weak connections with small absolute weight values. The pruned network is then retrained from the initial training phase by keeping the surviving weight parameters. We learn the structure of BSNNs using PSA. The number of layers, the number of hidden units in each layer and the sparse connections between adjacent layers are automatically determined. After structure learning, we train the sparse model from scratch by random initialization of weights. As for FNNs, we treat the number of hidden units and number of layers as hyper-parameters of network and determine the best structure by grid-search over all the combinations using validation data. TAB1 shows the space of network structures considered. Following the method in BID16, both "rectangle " and "conic" network shapes are tested. In FNNs with rectangle shape, all the hidden layers have constant number of units. FNNs with conic shape start with the given number of hidden units and decrease it layer by layer in a geometric progression manner towards the output layer. For Pruned FNNs, we take the best FNNs as the initial model and perform pruning as in BID11. The pruned model is then retrained for final model. We implement all the experiments using PyTroch 3 which is a flexible deep learning framework. We use ReLUs BID25 BID9 as the non-linear activation functions in all the networks. Dropout BID14 BID29 ) with rate 0.5 is applied after each non-linear projection. We use Adam BID15 as the optimizer to optimize the training objective function. During training, we select models by monitoring validation loss. Codes will be released after the paper is accepted to the conference. TAB2 shows the classification of BSNNs and FNNs on Tox21 dataset. The structures of FNNs are tuned individually for each task. It is clear that BSNNs achieve better AUC scores on 10 out of the 12 classification tasks. Even when it is not better, the average AUC value of BSNNs, e.g. on task SR.MMP, is also very close to that of FNNs. More importantly, BSNNs always contain much fewer parameters than FNNs, with the ratios of parameter number ranging from 7% to 40.11%. TAB3 shows the of BSNNs and FNNs over the 5 text classification tasks. Although BSNNs contain much fewer parameters than FNNs, BSNNs still achieve higher classification accuracy in the first two tasks, and comparable accuracy in the remaining tasks. Note that the ratios of parameter number ranges from 6.25% to 32.07%. This again confirms that our method learns good parsimonious deep models which can achieve high classification performance with much fewer parameters than standard FNNs. To validate our assumption that the backbone path in BSNNs captures most of the information in data and acts as a main part of the model, we remove the narrow skip-paths in BSNNs and train the model to test its performance in classification tasks. TAB4 shows the . As we can see from the , the backbone path alone already achieves AUC scores or accuracies which are only slightly worse than BSNNs. Note that the number of parameters in the sparse path is even much smaller than BSNNs. Compared with FNNs, the number of parameters is only 2% 11%, significantly smaller than that of FNNs. However, without the backbone, the performance of the model will be significantly worse due to the insufficient capability of the other narrow path. The not only show the importance of the backbone path in BSNNs, but also shows that our structure learning method in the backbone path is effective enough. To further show the effectiveness of our structure learning method, we introduce a new model called BSNN-FC. For each specific task, the structure of BSNN-FC is completely the same as that of BSNN, except that the layers in the sparse Backbone path are changed to fully-connected layers. We train BSNN-FC for all the tasks in Tox21 dataset and the are shown in TAB5. From the table we can see that, although BSNN keeps only 5% of the connections in the sparse path, it gives classification which are very similar to that of BSNN-FC. It shows that our structure learning method successfully removes the useless connections in BSNN-FC.We also compare BSNNs with Pruned FNNs whose weak connections are pruned using the method in BID11. We start from the fully pretrained FNNs reported in TAB2, and prune the connections with the smallest absolute weight values. After pruning, the number of remaining parameters in each FNN is the same as that in the corresponding BSNN for the same task. The comparison between BSNNs and pruned FNNs is shown in TAB5. Again BSNNs give higher AUC scores than pruned FNNs in 10 of the 12 classification tasks. Next we compare the interpretability of BSNNs with FNNs and Pruned FNNs on the text datasets. Here is how we interpret hidden units. We feed the data to the networks and do forward propagation to get the values of the hidden units corresponding to each data sample. Then for each hidden unit, we sort the words in descending order of the correlations between the words and the hidden unit. The top 10 words with the highest correlations are chosen to characterize the hidden unit. Following BID6, we measure the interpretability of a hidden unit by considering how similar pairs of words in the top-10 list are. The similarity between two words is determined using a word2vec model BID23 b) trained on part of the Google News datasets, where each word is mapped to a high dimensional vector. The similarity between two words is defined as the cosine similarity of the two corresponding vectors. High similarity suggests that the two words appear in similar contexts. The interpretability score of a hidden unit is defined as the compactness of its characterizing words and is computed as the average similarity of all pairs of words. The interpretability score of a model is defined as the average of interpretability scores of all hidden units. Table 7 reports the interpretability scores of BSNNs, FNNs and Pruned FNNs for different datasets. Sogounews dataset is not included in the experiment since its vocabulary are Chinese pingyin characters and most of them do not appear in the Google News word2vec model. We measure the interpretability scores by considering the top-layer hidden units. For the fair of comparison, all models have approximately the same number of top-layer hidden units. As it can be seen that BSNNs significantly outperform the FNNs and Pruned FNNs in most cases and is comparable if not better, showing superior coherency and compactness in the characterizations of the hidden units and thus better model interpretability. Pruned FNNs, on the other hand, reduce the interpretability of FNNs with the pruning strategy. Table 8 shows the qualitative interpretability by presenting the characterization words of hidden units with high interpretability scores in BSNNs. The hidden units are very meaningful for different datasets. For example, in Yelp Review dataset, the first hidden unit represents negative opinions on food with words "tasteless" and"flavorless"; the second hidden unit is more related to food like "paprika", "crust" and "unagi". In DBPedia, the first hidden unit is found out to have closer relationship with music, while the second one is more closely related to sport. Similar phenomena can be found in the rest of the table. This shows that the proposed BSNNs, with the statistical property, have better model interpretability and make a step further towards understandable deep learning models. Structure learning for deep neural network is a challenging and interesting research problem. We have proposed an unsupervised structure learning method which utilizes the correlation information in data for learning parsimonious deep feed-forward networks. In comparison with standard FNN, although the ing model of our method contains much fewer parameters, it achieves better or comparable classification performance in all kinds of tasks. Our method is also shown to learn models with better interpretability, which is also an important problem in deep learning. In the future, we will generalize our method to other networks like RNNs and CNNs."

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: While autoencoders are a key technique in representation learning for continuous structures, such as images or wave forms, developing general-purpose autoencoders for discrete structures, such as text sequence or discretized images, has proven to be more challenging. In particular, discrete inputs make it more difficult to learn a smooth encoder that preserves the complex local relationships in the input space. In this work, we propose an adversarially regularized autoencoder (ARAE) with the goal of learning more robust discrete-space representations. ARAE jointly trains both a rich discrete-space encoder, such as an RNN, and a simpler continuous space generator function, while using generative adversarial network (GAN) training to constrain the distributions to be similar. This method yields a smoother contracted code space that maps similar inputs to nearby codes, and also an implicit latent variable GAN model for generation. Experiments on text and discretized images demonstrate that the GAN model produces clean interpolations and captures the multimodality of the original space, and that the autoencoder produces improvements in semi-supervised learning as well as state-of-the-art in unaligned text style transfer task using only a shared continuous-space representation. Recent work on regularized autoencoders, such as variational BID15 BID29 and denoising BID37 variants, has shown significant progress in learning smooth representations of complex, high-dimensional continuous data such as images. These codespace representations facilitate the ability to apply smoother transformations in latent space in order to produce complex modifications of generated outputs, while still remaining on the data manifold. Unfortunately, learning similar latent representations of discrete structures, such as text sequences or discretized images, remains a challenging problem. Initial work on VAEs for text has shown that optimization is difficult, as the decoder can easily degenerate into a unconditional language model BID2. Recent work on generative adversarial networks (GANs) for text has mostly focused on getting around the use of discrete structures either through policy gradient methods BID40 or with the Gumbel-Softmax distribution BID17. However, neither approach can yet produce robust representations directly. A major difficulty of discrete autoencoders is mapping a discrete structure to a continuous code vector while also smoothly capturing the complex local relationships of the input space. Inspired by recent work combining pretrained autoencoders with deep latent variable models, we propose to target this issue with an adversarially regularized autoencoder (ARAE). Specifically we jointly train a discrete structure encoder and continuous space generator, while constraining the two models with a discriminator to agree in distribution. This approach allows us to utilize a complex encoder model, such as an RNN, and still constrain it with a very flexible, but more limited generator distribution. The full model can be then used as a smoother discrete structure autoencoder or as a latent variable GAN model where a sample can be decoded, with the same decoder, to a discrete output. Since the system produces a single continuous coded representation-in contrast to methods that act on each RNN state-it can easily be further regularized with problem-specific invariants, for instance to learn to ignore style, sentiment or other attributes for transfer tasks. Experiments apply ARAE to discretized images and sentences, and demonstrate that the key properties of the model. Using the latent variable model (ARAE-GAN), the model is able to generate varied samples that can be quantitatively shown to cover the input spaces and to generate consistent image and sentence manipulations by moving around in the latent space via interpolation and offset vector arithmetic. Using the discrete encoder, the model can be used in a semi-supervised setting to give improvement in a sentence inference task. When the ARAE model is trained with task-specific adversarial regularization, the model improves the current best on sentiment transfer reported in BID33 and produces compelling outputs on a topic transfer task using only a single shared code space. All outputs are listed in the Appendix 9 and code is available at (removed for review). In practice unregularized autoencoders often learn a degenerate identity mapping where the latent code space is free of any structure, so it is necessary to apply some method of regularization. A popular approach is to regularize through an explicit prior on the code space and use a variational approximation to the posterior, leading to a family of models called variational autoencoders (VAE) BID15 BID29. Unfortunately VAEs for discrete text sequences can be challenging to train-for example, if the training procedure is not carefully tuned with techniques like word dropout and KL annealing BID2, the decoder simply becomes a language model and ignores the latent code (although there has been some recent successes with convolutional models BID32 BID39). One possible reason for the difficulty in training VAEs is due to the strictness of the prior (usually a spherical Gaussian) and/or the parameterization of the posterior. There has been some work on making the prior/posterior more flexible through explicit parameterization BID28 BID16 BID4. A notable technique is adversarial autoencoders (AAE) BID23 which attempt to imbue the model with a more flexible prior implicitly through adversarial training. In AAE framework, the discriminator is trained to distinguish between samples from a fixed prior distribution and the input encoding, thereby pushing the code distribution to match the prior. While this adds more flexibility, it has similar issues for modeling text sequences and suffers from mode-collapse in our experiments. Our approach has similar motivation, but notably we do not sample from a fixed prior distribution-our'prior' is instead parameterized through a flexible generator. Nonetheless, this view (which has been observed by various researchers BID35 BID24 BID22) provides an interesting connection between VAEs and GANs. The success of GANs on images have led many researchers to consider applying GANs to discrete data such as text. Policy gradient methods are a natural way to deal with the ing non-differentiable generator objective when training directly in discrete space BID7 BID38. When trained on text data however, such methods often require pre-training/co-training with a maximum likelihood (i.e. language modeling) objective BID40. This precludes there being a latent encoding of the sentence, and is also a potential disadvantage of existing language models (which can otherwise generate locally-coherent samples). Another direction of work has been through reparameterizing the categorical distribution with the Gumbel-Softmax trick BID13 BID21 )-while initial experiments were encouraging on a synthetic task BID17, scaling them to work on natural language is a challenging open problem. There has also been a flurry of recent, related approaches that work directly with the soft outputs from a generator BID9; BID33 BID26. For example, Shen et al. BID33 exploits adversarial loss for unaligned style transfer between text by having the discriminator act on the RNN hidden states and using the soft outputs at each step as input to an RNN generator, utilizing the Professor-forcing framework BID18. Our approach instead works entirely in code space and does not require utilizing RNN hidden states directly. Discrete Structure Autoencoders Define X = V n to be a set of discrete structures where V is a vocabulary of symbols and P x to be a distribution over this space. For instance, for binarized images V = {0, 1} and n is the number of pixels, while for sentences V is the vocabulary and n is the sentence length. A discrete autoencoder consists of two parameterized functions: a deterministic encoder function enc φ: X → C with parameters φ that maps from input to code space and a conditional decoder distribution p ψ (x | c) over structures X with parameters ψ. The parameters are trained on a cross-entropy reconstruction loss: DISPLAYFORM0 The choice of the encoder and decoder parameterization is specific to the structure of interest, for example we use RNNs for sequences. We use the notation,x = arg max x p ψ (x | enc φ (x)) for the (approximate) decoder mode. When x =x the autoencoder is said to perfectly reconstruct x. Generative Adversarial Networks GANs are a class of parameterized implicit generative models BID8. The method approximates drawing samples from a true distribution c ∼ P r by instead employing a latent variable z and a parameterized deterministic generator functioñ c = g θ (z) to produce samplesc ∼ P g. Initial work on GANs minimizes the Jensen-Shannon divergence between the distributions. Recent work on Wasserstein GAN (WGAN), replaces this with the Earth-Mover (Wasserstein-1) distance. GAN training utilizes two separate models: a generator g θ (z) maps a latent vector from some easy-to-sample source distribution to a sample and a critic/discriminator f w (c) aims to distinguish real data and generated samples from g θ. Informally, the generator is trained to fool the critic, and the critic to tell real from generated. WGAN training uses the following min-max optimization over generator parameters θ and critic parameters w, DISPLAYFORM1 where f w: C → R denotes the critic function,c is obtained from the generator,c = g θ (z), and P r and P g are real and generated distributions. If the critic parameters w are restricted to an 1-Lipschitz function set W, this term correspond to minimizing Wasserstein-1 distance W (P r, P g). We use a naive approximation to enforce this property by weight-clipping, i.e.. DISPLAYFORM2 Ideally, a discrete autoencoder should be able to reconstruct x from c, but also smoothly assign similar codes c and c to similar x and x. For continuous autoencoders, this property can be enforced directly through explicit regularization. For instance, contractive autoencoders BID30 regularize their loss by the functional smoothness of enc φ. However, this criteria does not apply when inputs are discrete and we lack even a metric on the input space. How can we enforce that similar discrete structures map to nearby codes?Adversarially regularized autoencoders target this issue by learning a parallel continuous-space generator with a restricted functional form to act as a smoother reference encoding. The joint objective regularizes the autoencoder to constrain the discrete encoder to agree in distribution with its continuous counterpart: DISPLAYFORM0 Above W is the Wasserstein-1 distance between P r the distribution of codes from the discrete encoder model (enc φ (x) where x ∼ P(x)) and P g is the distribution of codes from the continuous generator model (g θ (z) for some z, e.g. z ∼ N (0, I)). To approximate Wasserstein-1 term, the W function includes an embedded critic function which is optimized adversarially to the encoder and generator as described in the . The full model is shown in Figure 1.To train the model, we use a block coordinate descent to alternate between optimizing different parts of the model: the encoder and decoder to minimize reconstruction loss, the WGAN critic function to approximate the W term, the encoder and generator to adversarially fool the critic to minimize W: DISPLAYFORM1 The full training algorithm is shown in Algorithm 1. discrete struct. encoder code (P r) decoder reconstruction loss DISPLAYFORM2 Figure 1: ARAE architecture. The model can be used as an autoencoder, where a structure x is encoded and decoded to producex, and as a GAN (ARAE-GAN), where a sample z is passed though a generator g θ to produce a code vector, which is similarly decoded tox. The critic function fw is only used at training to help approximate W. for number of training iterations do Train the autoencoder for reconstruction DISPLAYFORM0 Backpropagate reconstruction loss, DISPLAYFORM1, and update. Sample DISPLAYFORM2 DISPLAYFORM3 Backpropagate adversarial loss DISPLAYFORM4 ) and update. Extension: Code Space Transfer One benefit of the ARAE framework is that it compresses the input to a single code vector. This framework makes it ideal for manipulating discrete objects while in continuous code space. For example, consider the problem of unaligned transfer, where we want to change an attribute of a discrete input without supervised examples, e.g. to change the topic or sentiment of a sentence. First, we extend the decoder to condition on a transfer variable denoting this attribute y which is known during training, to learn p ψ (x | c, y). Next, we train the code space to be invariant to this attribute, to force it to be learned fully by the decoder. Specifically, we further regularize the code space to map similar x with different attribute labels y near enough to fool a code space attribute classifier, i.e.: DISPLAYFORM5 where L class (φ, u) is the loss of a classifier p u (y | c) from code space to labels (in our experiments we always set λ = 1). To incorporate this additional regularization, we simply add two more gradient update steps: (2b) training a classifier to discriminate codes, and (3b) adversarially training the encoder to fool this classifier. The algorithm is shown in Algorithm 2. Note that similar technique has been introduced in other domains, notably in images BID19 and video modeling BID6. We experiment with three different ARAE models: an autoencoder for discretized images trained on the binarized version of MNIST, an autoencoder for text sequences trained using the Stanford Natural Language Inference (SNLI) corpus BID1, and an autoencoder trained DISPLAYFORM0, and compute code-vectors c DISPLAYFORM1 Backpropagate adversarial classifier loss DISPLAYFORM2 for text transfer (Section 6.2) based on the Yelp and Yahoo datasets for unaligned sentiment and topic transfer. All three models utilize the same generator architecture, g θ. The generator architecture uses a low dimensional z with a Gaussian prior p(z) = N (0, I), and maps it to c. Both the critic f w and the generator g θ are parameterized as feed-forward MLPs. The image model uses fully-connected NN to autoencode binarized images. Here X = {0, 1} n where n is the image size. The encoder used is a feed-forward MLP network mapping from {0, DISPLAYFORM3 The text model uses a recurrent neural network (RNN) for both the encoder and decoder. Here X = V n where n is the sentence length and V is the vocabulary of the underlying language. Define an RNN as a parameterized recurrent function h j = RNN(x j, h j−1 ; φ) for j = 1... n (with h 0 = 0) that maps a discrete input structure x to hidden vectors h 1... h n. For the encoder, we define enc φ (x) = h n = c. For decoding we feed c as an additional input to the decoder RNN at each time step, i.e.h j = RNN(x j,h j−1, c; ψ), and further calculate the distribution over V at each time step via softmax, p ψ (x | c) = n j=1 softmax(Wh j + b) xj where W and b are parameters (part of ψ). Finding the most likely sequencex under this distribution is intractable, but it is possible to approximate it using greedy search or beam search. In our experiments we use an LSTM architecture BID12 for both the encoder/decoder and decode using greedy search. The text transfer model uses the same architecture as the text model but extends it with a code space classifier p(y|c) which is modeled using an MLP and trained to minimize cross-entropy. Our baselines utilize a standard autoencoder (AE) and the cross-aligned autoencoder BID33 for transfer. Note that in both our ARAE and standard AE experiments, the encoded code from the encoder is normalized to lie on the unit sphere, and the generated code is bounded to lie in (−1, 1) n by the tanh function at output layer. We additionally experimented with the sequence VAE introduced by BID2 and the adversarial autoencoder (AAE) model BID23 on the SNLI dataset. However despite extensive parameter tuning we found that neither model was able to learn meaningful latent representations-the VAE simply ignored the latent code and the AAE experienced mode-collapse and repeatedly generated the same samples. The Appendix 12 includes detailed descriptions of the hyperparameters, model architecture, and training regimes. Our experiments consider three aspects of the model. First we measure the empirical impact of regularization on the autoencoder. Next we apply the discrete autoencoder to two applications, unaligned style transfer and semi-supervised learning. Finally we employ the learned generator network as an implicit latent variable model (ARAE-GAN) over discrete sequences. Our main goal for ARAE is to regularize the model produce a smoother encoder by requiring the distribution from the encoder to match the distribution from the continuous generator over a simple latent variable. To examine this claim we consider two basic statistical properties of the code space during training of the text model on SNLI, shown in FIG1. On the left, we see that the 2 norm of c and codec converge quickly in ARAE training. The encoder code is always restricted to be on the unit sphere, and the generated codec quickly learns to match it. The middle plot shows the convergence of the trace of the covariance matrix between the generator and the encoder as training progresses. We find that variance of the encoder and the generator match after several epochs. To check the smoothness of the model, for both ARAE/AE, we take a sentence and calculate the average cosine similarity of 100 randomly-selected sentences that had an edit-distance of at most 5 to the original sentence. We do this for 250 sentences and calculate the mean of the average cosine similarity. FIG1 (right) shows that the cosine similarity of nearby sentences is quite high for the ARAE than in the case for the AE. Edit-distance is not an ideal proxy for similarity in sentences, but it is often a sufficient condition. Finally an ideal representation should be robust to small changes of the input around the training examples in code space BID30. We can test this property by feeding a noised input to the encoder and (i) calculating the score given to the original input, and (ii) checking the reconstructions. TAB2 (right) shows an experiment for text where we add noise by permuting k words in each sentence. We observe that the ARAE is able to map a noised sentence to a natural sentence, (though not necessarily the denoised sentence). TAB2 (left) shows empirical for these experiments. We obtain the reconstruction error (i.e. negative log likelihood) of the original (non-noised) sentence under the decoder, utilizing the noised code. We find that when k = 0 (i.e. no swaps), the regular AE better reconstructs the input as expected. However, as we increase the number of swaps and push the input further away from the data manifold, the ARAE is more likely to produce the original sentence. We note that unlike denoising autoencoders which require a domain-specific noising function BID10 BID37, the ARAE is not explicitly trained to denoise an input, but learns to do so as a byproduct of adversarial regularization. Unaligned Text Transfer A smooth autoencoder combined with low reconstruction error should make it possible to more robustly manipulate discrete objects through code space without dropping off the data manifold. To test this hypothesis, we experimented with two unaligned text transfer tasks. For these tasks, we attempt to change one attribute of a sentence without aligned examples of this change. To perform this transfer, we learn a code space that can represent an input that is agnostic to this attribute, and a decoder that can incorporate the attribute (as described in Section 4). We experiment with unaligned transfer of sentiment on the Yelp corpus and topic on the Yahoo corpus BID41. we came on the recommendation of a bell boy and the food was amazing. the people who ordered off the menu did n't seem to do much better. ARAE we came on the recommendation and the food was a joke. ARAE the people who work there are super friendly and the menu is good. Cross-AE we went on the car of the time and the chicken was awful.Cross-AE the place, one of the office is always worth you do a business. For sentiment we follow the same setup as BID33 and split the Yelp corpus into two sets of unaligned positive and negative reviews. We train an ARAE as an autoencoder with two separate decoders, one for positive and one for negative sentiment, and incorporate adversarial training of the encoder to remove sentiment information from the code space. We test by encoding in sentences of one class and decoding, greedily, with the opposite decoder. Our evaluation is based on four automatic metrics, shown in Table 2: (i) Transfer: measuring how successful the model is at transferring sentiment based on an automatic classifier (we use the fastText library BID14).(ii) BLEU: measuring the consistency between the transferred text and the original. We expect the model to maintain as much information as possible and transfer only the style; (iii) Perplexity: measuring the fluency of the generated text; (iv) Reverse Perplexity: measuring the extent to which the generations are representative of the underlying data distribution. 1 Both perplexity numbers are obtained by training an RNN language model. We additionally perform human evaluations on the cross-aligned AE and our best ARAE model. We randomly select 1000 sentences (500/500 positive/negative), obtain the corresponding transfers from both models, and ask Amazon Mechanical Turkers to evaluate the sentiment (Positive/Neutral/Negative) and naturalness (1-5, 5 being most natural) of the transferred sentences. We create a separate task in which we show the Turkers the original and the transferred sentences, and ask them to evaluate the similarity based on sentence structure (1-5, 5 being most similar). We explicitly ask the Turkers to disregard sentiment in their similarity assessment. In addition to comparing against the cross-aligned AE of BID33, we also compare against a vanilla AE trained without adversarial regularization. For ARAE, we experimented with different λ weighting on the adversarial loss (see section 4) with λ a = 1, λ b = 10. We generally set λ = 1. Experimentally the adversarial regularization enhances transfer and perplexity, but tends to make the transferred text less similar to the original, compared to the AE. Some randomly selected sentences are shown in figure 6 and more samples are shown available in Appendix 9.The same method can be applied to other style transfer tasks, for instance the more challenging Yahoo QA data BID41 Semi-Supervised Training We further utilize ARAE in a standard AE setup for semi-supervised training. We experiment on a natural language inference task, shown in Table 5 (right). We use 22.2%, 10.8% and 5.25% of the original labeled training data, and use the rest of the training set for unlabeled training. The labeled set is randomly picked. The full SNLI training set contains 543k sentence pairs, and we use supervised sets of 120k, 59k and 28k sentence pairs respectively for the three settings. As a baseline we use an AE trained on the additional data, similar to the setting explored in BID5. For ARAE we use the subset of unsupervised data of length < 15, which roughly includes 655k single sentences (due to the length restriction, this is a subset of 715k sentences that were used for AE training). As observed by BID5, training on unlabeled data with an AE objective improves upon a model just trained on labeled data. Training with adversarial regularization provides further gains. After training, an ARAE can also be used as an implicit latent variable model controlled by z and the generator g θ, which we refer to as ARAE-GAN. While models of this form have been widely used for generation in other modalities, they have been less effective for discrete structures. In this section, we attempt to measure the effectiveness of this induced discrete GAN.A common test for a GANs ability mimic the true distribution P r is to train a simple model on generated samples from P g. While there are pitfalls of this evaluation BID34, it provides a starting point for text modeling. Here we generate 100k samples from (i) ARAE-GAN, (ii) an AE 2, (iii) a RNN LM trained on the same data, and (iv) the real training set (samples from the models are 2 To "sample" from an AE we fit a multivariate Gaussian to the code space after training and generate code vectors from this Gaussian to decode back into sentence space. Medium Table 5 : Left. Semi-Supervised accuracy on the natural language inference (SNLI) test set, respectively using 22.2% (medium), 10.8% (small), 5.25% (tiny) of the supervised labels of the full SNLI training set (rest used for unlabeled AE training). Right. Perplexity (lower is better) of language models trained on the synthetic samples from a GAN/AE/LM, and evaluated on real data (Reverse PPL).A man is on the corner in a sport area. A man is on corner in a road all. A lady is on outside a racetrack. A lady is outside on a racetrack. A lot of people is outdoors in an urban setting.A lot of people is outdoors in an urban setting.A lot of people is outdoors in an urban setting.A man is on a ship path with the woman. A man is on a ship path with the woman. A man is passing on a bridge with the girl. A man is passing on a bridge with the girl. A man is passing on a bridge with the girl. A man is passing on a bridge with the dogs. A man is passing on a bridge with the dogs.A man in a cave is used an escalator.A man in a cave is used an escalator A man in a cave is used chairs. A man in a number is used many equipment A man in a number is posing so on a big rock.People are posing in a rural area. People are posing in a rural area. Figure 3: Sample interpolations from the ARAE-GAN. Constructed by linearly interpolating in the latent space and decoding to the output space. Word changes are highlighted in black. Results of the ARAE. The top block shows output generation of the decoder taking fake hidden codes generated by the GAN; the bottom block shows sample interpolation . shown in Appendix 10). All models are of the same size to allow for fair comparison. We train an RNN language model on generated samples and evaluate on held-out data to calculate the reverse perplexity. As can be seen from Table 5, training on real data (understandably) outperforms training on generated data by a large margin. Surprisingly however, we find that a language model trained on ARAE-GAN data performs slightly better than one trained on LM-generated/AE-generated data. We further found that the reverse PPL of an AAE BID23 was quite high due to mode-collapse. Another property of GANs (and VAEs) is that the Gaussian form of z induces the ability to smoothly interpolate between outputs by exploiting the structure of the latent space. While language models may provide a better estimate of the underlying probability space, constructing this style of interpolation would require combinatorial search, which makes this a useful feature of text GANs. We experiment with this property by sampling two points z 0 and z 1 from p(z) and constructing intermediary points z λ = λz 1 + (1 − λ)z 0. For each we generate the argmax outputx λ. The samples are shown in FIG0 (left) for text and in FIG0 (right) for a discretized MNIST ARAE-GAN.A final intriguing property of image GANs is the ability to move in the latent space via offset vectors (similar to the case with word vectors BID25). For example, Radford et al. BID27 observe that when the mean latent vector for "men with glasses" is subtracted from the mean latent vector for "men without glasses" and applied to an image of a "woman without glasses", the ing image is that of a "woman with glasses". To experiment with this property we generate 1 million sentences from the ARAE-GAN and compute vector transforms in this space to attempt to change main verbs, subjects and modifier (details in Appendix 11). Some examples of successful transformations are shown in FIG2 (right). Quantitative evaluation of the success of the vector transformations is given in FIG2 (left). We present adversarially regularized autoencoders, as a simple approach for training a discrete structure autoencoder jointly with a code-space generative adversarial network. The model learns a improved autoencoder as demonstrated by semi-supervised experiments and improvements on text transfer experiments. It also learns a useful generative model for text that exhibits a robust latent space, as demonstrated by natural interpolations and vector arithmetic. We do note that (as has been frequently observed when training GANs) our model seemed to be quite sensitive to hyperparameters. Finally, while many useful models for text generation already exist, text GANs provide a qualitatively different approach influenced by the underlying latent variable structure. We envision that such a framework could be extended to a conditional setting, combined with other existing decoding schemes, or used to provide a more interpretable model of language. One can interpret the ARAE framework as a dual pathway network mapping two distinct distributions into a similar one; enc φ and g θ both output code vectors that are kept similar in terms of Wasserstein distance as measured by the critic. We provide the following proposition showing that under our parameterization of the encoder and the generator, as the Wasserstein distance converges, the encoder distribution (c ∼ P r) converges to the generator distribution (c ∼ P g), and further, their moments converge. This is ideal since under our setting the generated distribution is simpler than the encoded distribution, because the input to the generator is from a simple distribution (e.g. spherical Gaussian) and the generator possesses less capacity than the encoder. However, it is not so simple that it is overly restrictive (e.g. as in VAEs). Empirically we observe that the first and second moments do indeed converge as training progresses (Section 6.1). Proposition 1. Let P be a distribution on a compact set χ, and (P n) n∈N be a sequence of distributions on χ. Further suppose that W (P n, P) → 0. Then the following statements hold:(i) P n P (i.e. convergence in distribution).(ii) All moments converge, i.e. for all k > 1, k ∈ N, DISPLAYFORM0 Proof. (i) has been proved in BID36 Theorem 6.9.For (ii), using The Portmanteau Theorem, (i) is equivalent to: DISPLAYFORM1 for all bounded and continuous function f: R d → R, where d is the dimension of the random variable. The k-th moment of a distribution is given by DISPLAYFORM2 Our encoded code is bounded as we normalize the encoder output to lie on the unit sphere, and our generated code is also bounded to lie in (−1, 1) n by the tanh function. Hence Original definitely a great choice for sushi in las vegas! Original i was so very disappointed today at lunch. ARAE definitely a _num_ star rating for _num_ sushi in las vegas. ARAE i highly recommend this place today. Cross-AE not a great choice for breakfast in las vegas vegas! Cross-AE i was so very pleased to this. DISPLAYFORM3 Original the best piece of meat i have ever had! Original i have n't received any response to anything. ARAE the worst piece of meat i have ever been to! ARAE i have n't received any problems to please. Cross-AE the worst part of that i have ever had had! Cross-AE i have always the desert vet.Original really good food, super casual and really friendly. Original all the fixes were minor and the bill? ARAE really bad food, really generally really low and decent food. ARAE all the barbers were entertaining and the bill did n't disappoint. Cross-AE really good food, super horrible and not the price. Cross-AE all the flavors were especially and one! Original it has a great atmosphere, with wonderful service. Original small, smokey, dark and rude management. ARAE it has no taste, with a complete jerk. ARAE small, intimate, and cozy friendly staff. Cross-AE it has a great horrible food and run out service. Cross-AE great,,, chips and wine.Original their menu is extensive, even have italian food. Original the restaurant did n't meet our standard though. ARAE their menu is limited, even if i have an option. ARAE the restaurant did n't disappoint our expectations though. Cross-AE their menu is decent, i have gotten italian food. Cross-AE the restaurant is always happy and knowledge.Original everyone who works there is incredibly friendly as well. Original you could not see the stage at all! ARAE everyone who works there is incredibly rude as well. ARAE you could see the difference at the counter! Cross-AE everyone who works there is extremely clean and as well. Cross-AE you could definitely get the fuss! Original there are a couple decent places to drink and eat in here as well. Original room is void of all personality, no pictures or any sort of decorations. ARAE there are a couple slices of options and _num_ wings in the place. ARAE room is eclectic, lots of flavor and all of the best. Cross-AE there are a few night places to eat the car here are a crowd. Cross-AE it's a nice that amazing, that one's some of flavor.Original if you're in the mood to be adventurous, this is your place! Original waited in line to see how long a wait would be for three people. ARAE if you're in the mood to be disappointed, this is not the place. ARAE waited in line for a long wait and totally worth it. Cross-AE if you're in the drive to the work, this is my place! Cross-AE another great job to see and a lot going to be from dinner.Original we came on the recommendation of a bell boy and the food was amazing. Original the people who ordered off the menu did n't seem to do much better. Cross-AE we came on the recommendation and the food was a joke. ARAE the people who work there are super friendly and the menu is good. Cross-AE we went on the car of the time and the chicken was awful. Cross-AE the place, one of the office is always worth you do a business.Original service is good but not quick, just enjoy the wine and your company. Original they told us in the beginning to make sure they do n't eat anything. ARAE service is good but not quick, but the service is horrible. ARAE they told us in the mood to make sure they do great food. Cross-AE service is good, and horrible, is the same and worst time ever. Cross-AE they're us in the next for us as you do n't eat.Original the steak was really juicy with my side of salsa to balance the flavor. Original the person who was teaching me how to control my horse was pretty rude. ARAE the steak was really bland with the sauce and mashed potatoes. ARAE the person who was able to give me a pretty good price. Cross-AE the fish was so much, the most of sauce had got the flavor. Cross-AE the owner's was gorgeous when i had a table and was friendly.Original other than that one hell hole of a star bucks they're all great! Original he was cleaning the table next to us with gloves on and a rag. ARAE other than that one star rating the toilet they're not allowed. ARAE he was prompt and patient with us and the staff is awesome. Cross-AE a wonder our one came in a _num_ months, you're so better! Cross-AE he was like the only thing to get some with with my hair. A woman is seeing a man in the river. There passes a woman near birds in the air. Some ten people is sitting through their office. The man got stolen with young dinner bag. Monks are running in court. The Two boys in glasses are all girl. The man is small sitting in two men that tell a children. The two children are eating the balloon animal. A woman is trying on a microscope. The dogs are sleeping in bed. Two Three woman in a cart tearing over of a tree. A man is hugging and art. The fancy skier is starting under the drag cup in. A dog are <unk> a A man is not standing. The Boys in their swimming. A surfer and a couple waiting for a show. A couple is a kids at a barbecue. The motorcycles is in the ocean loading I's bike is on empty The actor was walking in a a small dog area. no dog is young their mother LM Samples a man walking outside on a dirt road, sitting on the dock. A large group of people is taking a photo for Christmas and at night. Someone is avoiding a soccer game. The man and woman are dressed for a movie. Person in an empty stadium pointing at a mountain. Two children and a little boy are <unk> a man in a blue shirt. A boy rides a bicycle. A girl is running another in the forest. the man is an indian women.Figure 5: Text samples generated from ARAE-GAN, a simple AE, and from a baseline LM trained on the same data. To generate from an AE we fit a multivariate Gaussian to the learned code space and generate code vectors from this Gaussian. We generate 1 million sentences from the ARAE-GAN and parse the sentences to obtain the main verb, subject, and modifier. Then for a given sentence, to change the main verb we subtract the mean latent vector (t) for all other sentences with the same main verb (in the first example in FIG2 this would correspond to all sentences that had "sleeping" as the main verb) and add the mean latent vector for all sentences that have the desired transformation (with the running example this would be all sentences whose main verb was "walking"). We do the same to transform the subject and the modifier. We decode back into sentence space with the transformed latent vector via sampling from p ψ (g(z + t)). Some examples of successful transformations are shown in FIG2 (right). Quantitative evaluation of the success of the vector transformations is given in FIG2 (left). For each original vector z we sample 100 sentences from p ψ (g(z + t)) over the transformed new latent vector and consider it a match if any of the sentences demonstrate the desired transformation. Match % is proportion of original vectors that yield a match post transformation. As we ideally want the generated samples to only differ in the specified transformation, we also calculate the average word precision against the original sentence (Prec) for any match.

Adversarially Regularized Autoencoders learn smooth representations of discrete structures allowing for interesting results in text generation, such as unaligned style transfer, semi-supervised learning, and latent space interpolation and arithmetic.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: When data arise from multiple latent subpopulations, machine learning frameworks typically estimate parameter values independently for each sub-population. In this paper, we propose to overcome these limits by considering samples as tasks in a multitask learning framework.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a method to estimate collections of regression models in which each model is personalized to a single sample.

represent the short summary for finding the corresponding scientific article (long)

The goal of survival clustering is to map subjects into clusters. Without end-of-life signals, this is a challenging task. To address this task we propose a new loss function by modifying the Kuiper statistics.