Question: Represent the text to find a very short summary (1-2 sentences max)

Question: There has been an increasing use of neural networks for music information retrieval tasks. In this paper, we empirically investigate different ways of improving the performance of convolutional neural networks (CNNs) on spectral audio features. More specifically, we explore three aspects of CNN design: depth of the network, the use of residual blocks along with the use of grouped convolution, and global aggregation over time. The application context is singer classification and singing performance embedding and we believe the extend to other types of music analysis using convolutional neural networks. The show that global time aggregation helps to improve the performance of CNNs the most. Another contribution of this paper is the release of a singing recording dataset that can be used for training and evaluation. Deploying deep neural networks to solve music information retrieval problems has benefited from advancements in other areas such as computer vision and natural language processing. In this paper, experiments are designed to investigate whether a few of the recent signature advancements in the deep learning community can improve the learning capability of deep neural networks when applied on time-frequency representations. Because time-frequency representations are frequently treated as 2-D images similarly to image input for computer vision models, convolution layers are popular choices as the first processing layers for time-frequency representations in audio and music analysis applications. One of the recent convolutional layer variants is the residual neural network with a bottleneck design, ResNet. Another variant built upon ResNet is to use grouped convolution inside the bottleneck as a generalization of the Inception Net BID12 BID22, ResNeXt. These two variants have enabled more deepening of the convolutional layers of deep neural networks. Most existing music information retrieval research using convolutional neural networks (CNNs), utilizes vanilla convolutional layers with no more than 5 layers. In this paper, the two convolution layer variants mentioned and a deeper architecture with more than 5 convolution layers is proposed and shown to be effective on audio time-frequency representations. Conceptually, convolution layers take care of learning local patterns (neighboring pixels in images or time frame/frequency bins in time-frequency representations) presented in the input matrices. After learning feature maps from the convolution layers, one of the reoccurring issues, when the input is a time-frequency representation, is how to model or capture temporal relations. Recurrent neural networks has been used to solve this problem BID6 BID1 BID3 BID4. Recent developments from natural language processing in attention mechanisms BID0 BID3 BID15 provide a different approach to model temporal dependencies and relations. In this paper, the attention mechanism is viewed as a special case of a global aggregation operation along the timeaxis that has learnable parameters. Typical aggregation operations such as average or max have no learnable parameters. The effects of global aggregation along the time axis using either average, max or the attention mechanism is investigated experimentally. Two specific applications are investigated in this paper: 1) singer classification of monophonic recordings, and 2) singing performance embedding. The goal of singer classification is to predict the singer's identity given an audio recording as input. A finite set of possible singers is considered so this is a classification task. In singer performance embedding the goal is to create an embedding space in which singers with similar styles can be projected to be closer to each other compared to singers with different styles. Ideally, it should be possible to identify "singing style" or "singing characteristics" by examining (and listening to) the clusters formed from the projections of audio recordings onto the embedding space. Many tasks in music and audio analysis can be formulated in a similar way, in which similarity plays an essential role, therefore we believe that the of this paper generalize to other audio and music tasks. The main challenge and interesting point about this application context is how to isolate the "singer effect" from the "song effect". Classic hand-crafted audio features capture general aspects of similarity. When the same song is performed by different singers audio-based similarity tends to be higher than when different songs are performed -i.e the "song effect" BID14. In order to effectively model singing we need to learn a representation that emphasizes singer similarity while at the same time reduces the effect of song similarity. As an analogy, consider the computer vision problem of face identification. When learning representations for this task we want the information about the identity of the face to be minimally affected by the effect of the environment and pose. The interfering "song effect " is even more dominant in the singing voice case than that of the environment/pose effect in face recognition. Extending this analogy with computer vision, singing performance embedding is analogous to the use of learning an embedded space for face verification BID5 BID17. In this approach, an embedded space of faces is learned with the goal of having pictures of the same person close to each other, and having pictures of different persons away from each other in the learned embedding space. This is accomplished by utilizing a siamese neural network instead of a classifier BID5 BID8 BID16. The large amount of identities make the use of a classifier impractical. By learning an embedding space for singing voice audio recordings that places recordings of the same identity closer to each other, and pushes the ones with different identities away from each other, ideally "singing style" or "singing characteristics" can be identified by examining (and listening to) the clusters formed from the embeddings of audio recordings in the learned embedding space. For both the singer identity classification and the singing performance embedding, we employ an architecture that uses CNNs to extract features followed by a global aggregation layer after which fully connected dense layers are used. The difference between the architectures used for these two tasks is that, for singer identity classification, the output layer is the standard softmax layer that outputs classification probabilities for each singer included in the dataset, but for the singing performance embedding, the output layer is a fully connected linear layer that will embed each input sample into a fixed length vector after which a copy of the network is used to construct a siamese architecture to learn the embedding space. Practically, having a model that embeds singing recordings into short fixed length vectors enables the possibility of fastening the similarity comparison of two long spectrogram sequences (differ in lengths) by calculating the Euclidean distance between their fixed length embedding vectors BID16. This allows a large database of singing recordings to be queried by input query singing recordings more efficiently. In order to evaluate the singing performance embedding model in an unbiased way (not biasing towards the collection of songs sang by a singer), a new set of "balanced" singing recordings are gathered and released. The newly released dataset is an addition to the existing DAMP data set of monophonic vocal music performances. The paper is structured as follows. In section 2, the details of the neural network constructing blocks used in the experiments are described. The dataset used and the experiment details are disclosed in section 3. Discussions and are in Sec.4. The neural network architectures used in the experiments follow a general design pattern depicted in Figure 1. The general design pattern is to feed the input time-frequency features as 2-D images into convolutional layers/blocks, then feed the extracted feature maps to a global time-wise aggregation layer. The output from the global time-wise aggregation layer is fed into dense layers, followed by the output layer. The details of each construction block are described below. The basic convolution layer being used is the vanilla convolution layer that has shared weights and tied biases across channels without any modification. The other variant being used in our experiments is the residual network design with the bottleneck block introduced in ResNet. This variant is extended by using the grouped convolutional block, introduced in ResNeXt BID22, on top of the ResNet. Depictions of the vanilla convolution building block, the ResNet, and ResNeXt are shown in FIG0. Let the outlets in FIG0 be y, inlets be x, and f, g, h be convolution operations. The vanilla convolutional block (a) in FIG0 would be y = g(f (x)), while the ResNet bottleneck block (b) is y = x + f (g(h(x))) and the ResNeXt bottleneck block is y = x + Γ(x) with Γ(·) being the grouped convolution consisting of series of sliced f (g(h(·))) operations over the channel axis of the input. Under the ResNeXt configuration, the ResNet configuration is a special case where the cardinality parameter equals 1 BID22. A max pooling layer with pool size, and stride of is placed between convolutional blocks in the following way: The first convolutional layer is followed immediately by a max pooling layer, while for all the remaining layers the max pooling layers are inserted between every two consecutive convolutional layers/blocks. A distinction between the terms convolution layer and block needs to be made here. A convolutional layer refers to a single vanilla convolution layer, while a convolutional block refers to any of the three architecture patterns show in FIG0. In TAB0, in the column for the number of CNN filters, each number represents the number of output channels for each convolutional layer or block, with normal text for layer and bold text for block. Batch normalizations are applied after each non-linearity activation throughout the convolutional layer/blocks. Global Time-Wise Aggregation Layer Dense Layers Output Layer Input Figure 1: An overview of the neural network architecture used in this paper. Before feeding the output of the convolutional layers to the global time-wise aggregation, the 3 − D feature map having the shape (# of channels, # of time frames, # of frequency bins) as their dimensions will be reshaped as 2-D matrices having the shape (# of time frames, # of channels× # of frequency bins) Originally, the attention mechanism was introduced for sequence-to-sequence learning BID0 in an RNN architecture, that allows the prediction at each time-step to access information from every step in the input hidden sequence in a weighted way. Since the experiments done in this paper do not need sequence-to-sequence prediction, the feed-forward version of attention proposed in BID15 is used instead of the original one. The feed-forward attention is formulated as follows: Given the input matrix X ∈ R N ×D representing N frames of D dimensional feature vectors, a weight vector σ ∈ R N over the time-steps is calculated by DISPLAYFORM0 where DISPLAYFORM1 and f is a non-linear function (tanh for the experiments done in this paper), and w ∈ R D and b ∈ R are the learnable parameters, which can be learned by back-propagation. The outputX of the feed-forward attention layer is then calculated viâ and whereX can be considered a weighted average of X with weights σ, determined by the learnable parameters w and b. This attention operation can also be viewed as an aggregation operation over the time-axis similar to max or average. The idea of aggregation over a specific axis could then be generalized by having the feed-forward attention, max and average all in the same family, except that the later two have no learnable parameters. This family of operations is different from the standard max/average pooling in convolution layers, in that the aggregation is global to the scope of the the input sample i.e the aggregation will reduce the dimension of the aggregation axis to 1. A specific realization of the network architecture including both convolutional and the global aggregation parts can be found in Appendix B. DISPLAYFORM2 The two tasks explored in this paper are singer identity classification and singing performance embedding. In terms of experimentation with different hyper parameters and network architectures, the singer classification problem provides clear evaluation criteria in terms of model performance. That way different hyper parameters and architectural choices can be compared to each other. On the other hand, the embedding task allows a more exploratory way to understand the input in the sense that it is the spatial relationships between the embedded samples that are interesting to us. For both tasks, numerical evaluation metrics, as well as plots of the embedded samples from the singing performance embedding are provided in order for readers to examine the both quantitatively and qualitatively. The dataset being used for the singer identity classification is the DAMP dataset 1. The DAMP dataset has a total of 34620 solo singing recordings by 3462 singers with each singer having 10 recordings. The collections of songs sang by each singer are different, and some singers sing the same song multiples times. Therefore the DAMP dataset is "unbalanced", and making it difficult for the learning algorithm not to be biased to the singer-specific collection of songs when learning to predict the singer identity. Therefore an additional dataset with each singer singing the same collection of songs available for training and evaluation is collected and released. The set of added collections of solo singing recordings is named the DAMP-balanced dataset. DAMP-balanced has a total of 24874 singing recordings sang by 5429 singers. The song collection of the DAMP-balanced has 14 songs. The structure of the DAMP-balanced is that the last 4 songs are designed to be the test set, and the first 10 songs could be partitioned into any 6/4 train/validation split (permutation) that the singers in train and validation set sang the same 6/4 songs collections according to the 6/4 split (the number of total recordings for train and validation set are different from split to split, since there are different number of singers that all sang the same 6/4 split for different split). The DAMPbalanced dataset is suitable for the singing performance embedding task while the original DAMP dataset can be used to train singer identity classification algorithms. The song list and detailed descriptions of the DAMP-balanced is provided in Appendix A. The input to the neural networks are time-frequency representations extracted from raw audio signals. These time-frequency representations are obtained by applying the short-time Fourier transform that extracts frequency information for each short time window analyzed. As a , most time-frequency representations take the form of 2-D matrices with one axis representing time while the other axis represents frequencies. The entries [j, i] of the matrix represent the intensity of a particular frequency i corresponding to a particular time frame j. In this paper, the Mel-scaled magnitude spectrogram (Mel-spectrogram) ) is used as the input feature to the neural network. Mel-spectrogram are used as input to neural network tasks in BID3 BID7 BID4. The other common choice of audio time-frequency input, the constant-Q transformed spectrogram (CQT), which is used extensively in music information retrieval tasks BID20 due to its capability of preserving the constant octave relationships between frequency bins (log-scaled frequency bins). Since all neural network configurations using CQT perform worse than their Mel-spectrogram versions, only a few representative of using CQT are shown in TAB0. The reason why CQT works worse is that although the CQT preserves a linear relationships of the intervals of different pitches, the linear relationships do not apply to the distances between different harmonics of one pitch. Since the audio recording being analyzed here only has one single singing voice at each time frame, the constant octave relationship does not help the neural networks learning the time-frequency patterns for singing voices. The audio recordings are all re-sampled to have 22050Hz sampling rate, then the Mel-scaled magnitude spectrograms are obtained using a Fast Fourier Transform (FFT) with a length of 2048 samples, hop size of 512 samples, a Hanning window and 96 Mel-scaled frequency bins. The extracted Melspectrogram is squared to obtain the power spectrogram which is then transformed into decibels (dB). The values below −60dB are clipped to be zero and an offset is added to the whole power spectrogram in order to have values between 0 and 60.For both tasks, each singing performance audio recording is transformed to a Mel-spectrogram as described above. The Mel-spectrogram of each recording is then chopped into overlapping matrices each of which has a duration of 6 seconds (256 time steps) and 20% hop size. For both tasks the gradient decent is optimized by ADAM BID11 ) with a learning rate 0.0001 and a batch size of 32. A drop out of 10% is applied at the last fully connected dense layers. L 2 weight regularizations with a weight 1e − 6 are applied on all the learnable weights in the neural network. The above hyper parameters are chosen by the Bayesian optimization package SPEARMINT BID19. For both tasks, an early stopping test on the validation set is applied every 50 epochs. For the singer identity classification, the patience is 300 epochs with at least 99.5% improvement, and the patience for the singing performance embedding task is 1000. The non-linear activation function used in all convolution layers and fully connected layers is the rectified linear unit activation function. The convolutional filter sizes are for the first convolutional layer and for all subsequent convolutional layers. For the fully connected dense layer, 3 layers with each having 1024 hidden units are used before the last output layer. A subset of 46 singers (23 males and 23 females) corresponding to 460 solo singing recordings from the DAMP dataset is selected for the singer classification problem. A 10-fold cross validation is used to obtain the test accuracies for different models with each fold using 1 recording from each singer as the test set, while the training is performed on the rest 9 recordings with 1 of them selected randomly as the validation set for early stopping. For the classification task we explore different combinations of neural network configurations in terms of using either the vanilla CNN or ResNeXt building blocks. Also different number of layers and different types of aggregation such as max, average, feed-forward attention or no global aggregation are also investigated. A baseline SVM classifier is also included by having the mean and standard deviation of chroma, MFCC, spectral centroid, spectral roll-off, and spectral flux BID2 extracted from each ∼ 6 second clip as the input. The experimental and associated measures of different models are displayed in TAB0. The number of convolution filters is chosen so that the total number of parameters are on the same scale between different configurations. From TAB0, it can be seen that the baseline method achieved 27% accuracy which is above the random prediction of 2.2% (FORMULA0), while all the neural network models far exceeded the baseline by at least 35%. For all the neural network models, the use of any global aggregation method improved the performance by 5% ∼ 10%. Among the neural network models, global aggregation with average or feed-forward attention has slightly better performance than max except for the shallower CNN. For the singing performance embedding experiment, a subset of 6/4/4 train/validation/test split from the DAMP-balanced is used 2. The total number of recordings and singers for this specific split are 276/88/224 and 46/22/56 respectively. We would like an embedding space that places recordings by the same singer closer to each other and pushes recordings by different singers away from each other, and a siamese neural network architecture BID5 BID8 BID16 ) is used. The inner twin neural network is constructed following the same principle described earlier in section 2. The embedding dimension for the linear fully connected output layer is chosen to be 16 by SPEARMINT. Since a siamese network learns the embedding by shortening or lengthening the distance between pairs of embedded vectors based on their label, pairs of samples from the dataset are arranged and labeled. Denote a pair of samples by x 1, x 2 ∈ R D, and y a binary label that equals 1 when x 1, x 2 have the same identity and equals 0 when their identities are different. The distance metric optimized over the siamese network in this experiment is the squared euclidean distance DISPLAYFORM0 then the contrastive loss BID5 BID8 BID16 ) is used as the optimization goal and is defined as DISPLAYFORM1 where G is the non-linear function that represents the neural network and m is the target margin between embedded vectors having different identities, and m = 1 throughout the experiments. To train the siamese networks, pairs of chopped samples from the same singer or different ones are randomly sampled in a 1: 1 ratio and fed into the siamese networks. The contrastive losses on the test set for different network configurations are shown in TAB1. The cardinalities for the ResNeXt configurations in TAB1 are 4.The training and validation error over epochs are plotted in FIG1. The observation from the training/validation plots are that, 1) feed-forward attention and average aggregation tend to overfit the data more than max and no aggregation by looking at training errors, 2) feed-forward attention and average aggregations reach early stopping earlier than max and no aggregation by looking at the best validation epoch, 3) Shallow architectures work slightly better than deeper ones if their number of parameters are on the same scale. Results showing qualitative characteristic of the embedding are shown in Figure 4. In Figure 4, the embeddings of 40 performances sang by 10 singers with each singer sang the same 4 songs from the test split are plotted. The embedding of a performance is obtained by taking the mean of the embeddings from all the chopped input samples of that performance. A comparison is made between the embeddings from the shallow ResNeXt architecture with/without feed-forward attention and the handcrafted features used in the baseline case for singer classification. Both the embeddings and the extracted handcrafted audio features are projected down to a 2-D space by t-SNE BID13. It is obvious that the baseline handcrafted audio feature captured the "song" effect while the learned embeddings from our singing performance embedding experiment were able to group together the performances by the same singers while invariant to the "song" effect. The t-SNE projections of performed 6-second clips before summarized into songs are shown in FIG4 in Appendix B. To have another quantitative assessment of the embeddings, leave-one-out k-nearest neighbor classifications using the embedded 16-dimensional performance vectors are used as training points. For each k and each network configuration, every sample is used as test sample once and the classification accuracies are obtained by averaging over the outcomes of every test sample for all k and network configurations. For the k-nearest neighbor singer classification, all the 224 performances from 56 singers are used. The classification with multiple ks among the shallow ResNeXt configurations with/without feed-forward attention and the handcrafted features are shown in Figure 5. In addition, k-nearest neighbor classifications on performed songs are also conducted to demonstrate the "song effect". From the k-nearest neighbor classification on singers and songs, it is evidence that the "song" effect exists and singing performance embedding learning is able to dilute the "song" effect while extracting features that are more relevant in terms of characterizing singers. Also the feed-forward global aggregation helped the enhancement of "singer style" while reducing "song effect" slightly by looking at the k-nearest neighbor classification accuracies. It is worth mentioning that the k-nearest neighbor classification on performed songs is only possible due to the "balanced" nature of the dataset. In this paper, empirical investigations into how recent developments in the deep learning community could help solving singer identification and embedding problems were conducted. From the experiment , the obvious take away is that global aggregation over time improves performance by a considerable margin in general. The performances among the three aggregation strategies; max, average and feed-forward attention, are very close. The advantage of using feedforward attention from observing the experiment is that it accelerates the learning process compared to other non-learnable global aggregations. One way to explain such observation is that the feed-forward attention layer learns a "frequency template" for each convolutional channel fed into it. These "frequency templates" are encoded in w and enable each convolutional channel fed into it to focus on different parts along the frequency axis (Since w ∈ R D with D = num of channels × num of frequency bins). In this paper we also have shown that training a deep neural networks having more than 15 convolutional layers on time-frequency input is definitely feasible with the help of global time aggregation. To the authors' knowledge, there is no previous music information retrieval research utilizing neural networks having more than 10 convolutional layers. A dataset consisting of over 20000 single singing voice recordings is also released and described in this paper. The released dataset, DAMP-balanced, could be partitioned in a way that for singer classification, the performed songs for each singer are the same. For future works, replacing max-pooling with striding in convolutional layers which recent works in CNN suggest will be experimented. To improve global-aggregation, taking temporal order into consideration during the global-aggregation operation as suggested in BID21 will also be experimented. Also the proposed neural network configurations will be experimented in other music information retrieval tasks such as music structure segmentation. The DAMP-balanced dataset is a separate dataset from the original DAMP, but comes with the same format of metadata as the original DAMP dataset does. The audio recordings and metadata of the DAMP-balanced dataset are collected by querying the internal database of the Sing! Karaoke app hosted by Smule, Inc., which is the same as the original DAMP dataset. The difference between the DAMP and the DAMP-balanced datasets lies at how the querying is done to collect the audio recordings and the metadata. For the original DAMP, 10 singing performances from 3462 Sing! Karaoke app users are randomly selected. There are no specific constraints on the collections of songs performed by each user. As a , each user sang different collections of songs from each other and one song could be sang multiple times by one user. On the contrary, the queries to retrieve audio recordings and metadata for the DAMP-balanced dataset specifically ask for a group of users that all sang one specific collections of songs at least once, with only one performance returned for each song and each user, per query. 14 popular songs (defined as the more times being sang the more popular) over the past year and are listed in TAB2. For the first 10 songs, 210 × 2 queries were created to retrieve audio recordings and metadata that cover all different combinations of splitting the 10 songs into 6/4 song collections. Each query returns a set of users, along with their singing performances and metadata, such that all users in that returned set has only one performance of each of the songs in the specific 6 or 4 song collection. For example, the train/validation sets used in this paper was the first 6 songs in TAB2 as training set and the following 4 songs as validation set. This specific split has 276 performances for training and 88 performances for validation, and that lead to 46 and 22 singers respectively. Different 6/4 split in different number of singers in each set thus making the total number performances of different songs differ from each other. For example, if instead the first 4 songs and the following 6 songs are taken as the 6/4 split of the first 10 songs, the first 4-song collection will have 459 users and 1836 performances while the following 6-song collection having 3 users and 18 performances. The "balanced" structure of the DAMP-balanced allows possible train/validation rotation within the first 10 songs while leaving the last 4 songs as test set, or provides more possible "balanced" test sets for models training on other datasets.

Using deep learning techniques on singing voice related tasks.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for replacing the softmax layer with a continuous embedding layer. Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax. We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation. We show that our models obtain upto 2.5x speed-up in training time while performing on par with the state-of-the-art models in terms of translation quality. These models are capable of handling very large vocabularies without compromising on translation quality. They also produce more meaningful errors than in the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations. Due to the power law distribution of word frequencies, rare words are extremely common in any language BID45 ). Yet, the majority of language generation tasks-including machine translation BID39 BID1 BID24, summarization BID36 BID37 BID30, dialogue generation BID40, question answering BID44, speech recognition BID13 ), and others-generate words by sampling from a multinomial distribution over a closed output vocabulary. This is done by computing scores for each candidate word and normalizing them to probabilities using a softmax layer. Since softmax is computationally expensive, current systems limit their output vocabulary to a few tens of thousands of most frequent words, sacrificing linguistic diversity by replacing the long tail of rare words by the unknown word token, unk. Unsurprisingly, at test time this leads to an inferior performance when generating rare or out-of-vocabulary words. Despite the fixed output vocabulary, softmax is computationally the slowest layer. Moreover, its computation follows a large matrix multiplication to compute scores over the candidate words; this makes softmax expensive in terms of memory requirements and the number of parameters to learn BID26 BID27 BID8. Several alternatives have been proposed for alleviating these problems, including sampling-based approximations of the softmax function BID2 BID26, approaches proposing a hierarchical structure of the softmax layer BID27 BID7, and changing the vocabulary to frequent subword units, thereby reducing the vocabulary size BID38.We propose a novel technique to generate low-dimensional continuous word representations, or word embeddings BID25 BID31 BID4 instead of a probability distribution over the vocabulary at each output step. We train sequence-to-sequence models with continuous outputs by minimizing the distance between the output vector and the pretrained word embedding of the reference word. At test time, the model generates a vector and then searches for its nearest neighbor in the target embedding space to generate the corresponding word. This general architecture can in principle be used for any language generation (or any recurrent regression) task. In this work, we experiment with neural machine translation, implemented using recurrent sequence-to-sequence models BID39 with attention BID1 BID24.To the best of our knowledge, this is the first work that uses word embeddings-rather than the softmax layer-as outputs in language generation tasks. While this idea is simple and intuitive, in practice, it does not yield competitive performance with standard regression losses like 2. This is because 2 loss implicitly assumes a Gaussian distribution of the output space which is likely false for embeddings. In order to correctly predict the outputs corresponding to new inputs, we must model the correct probability distribution of the target vector conditioned on the input BID3. A major contribution of this work is a new loss function based on defining such a probability distribution over the word embedding space and minimizing its negative log likelihood (§3).We evaluate our proposed model with the new loss function on the task of machine translation, including on datasets with huge vocabulary sizes, in two language pairs, and in two data domains (§4). In §5 we show that our models can be trained up to 2.5x faster than softmax-based models while performing on par with state-of-the-art systems in terms of generation quality. Error analysis (§6) reveals that the models with continuous outputs are better at correctly generating rare words and make errors that are close to the reference texts in the embedding space and are often semantically-related to the reference translation. Traditionally, all sequence to sequence language generation models use one-hot representations for each word in the output vocabulary V. More formally, each word w is represented as a unique vector o(w) ∈ {0, 1} V, where V is the size of the output vocabulary and only one entry id(w) (corresponding the word ID of w in the vocabulary) in o(w) is 1 and the rest are set to 0. The models produce a distribution p t over the output vocabulary at every step t using the softmax function: DISPLAYFORM0 where, s w = W hw h t + b w is the score of the word w given the hidden state h produced by the LSTM cell BID15 at time step t. W ∈ R V xH and b ∈ R v are trainable parameters. H is the size of the hidden layer h. These parameters are trained by minimizing the negative log-likelihood (aka cross-entropy) of this distribution by treating o(w) as the target distribution. The loss function is defined as follows: DISPLAYFORM1 This loss computation involves a normalization proportional to the size of the output vocabulary V. This becomes a bottleneck in natural language generation tasks where the vocabulary size is typically tens of thousands of words. We propose to address this bottleneck by representing words as continuous word vectors instead of one-hot representations and introducing a novel probabilistic loss to train these models as described in §3.2 Here, we briefly summarize prior work that aimed at alleviating the sofmax bottleneck problem. We briefly summarize existing modifications to the sofmax layer, capitalizing on conceptually different approaches. Sampling-Based Approximations Sampling-based approaches completely do away with computing the normalization term of softmax by considering only a small subset of possible outputs. These include approximations like Importance Sampling BID2, Noise Constrastive Estimation BID26, Negative Sampling BID25, and Blackout BID16. These alternatives significantly speed-up training time but degrade generation quality. BID27 replace the flat softmax layer with a hierarchical layer in the form of a binary tree where words are at the leaves. This alleviates the problem of expensive normalization, but these gains are only obtained at training time. At test time, the hierarchical approximations lead to a drop in performance compared to softmax both in time efficiency and in accuracy. BID7 propose to divide the vocabulary into clusters based on their frequencies. Each word is produced by a different part of the hidden layer making the output embedding matrix much sparser. This leads to performance improvement both in training and decoding. However, it assigns fewer parameters to rare words which leads to inferior performance in predicting them BID34. BID0; BID11 add additional terms to the training loss which makes the normalization factor close to 1, obviating the need to explicitly normalize. The evaluation of certain words can be done much faster than in softmax based models which is extremely useful for tasks like language modeling. However, for generation tasks, it is necessary to ensure that the normalization factor is exactly 1 which might not always be the case, and thus it might require explicit normalization. BID18 introduce character-based methods to reduce vocabulary size. While character-based models lead to significant decrease in vocabulary size, they often differentiate poorly between similarly spelled words with different meanings. BID38 find a middle ground between characters and words based on sub-word units obtained using Byte Pair Encoding (BPE). Despite its limitations BID28, BPE achieves good performance while also making the model truly open vocabulary. BPE is the state-of-the art approach currently used in machine translation. We thus use this as a baseline in our experiments. In our proposed model, each word type in the output vocabulary is represented by a continuous vector e(w) ∈ R m where m V. This representation can be obtained by training a word embedding model on a large monolingual corpus BID25 BID31 BID4.At each generation step, the decoder of our model produces a continuous vectorê ∈ R m. The output word is then predicted by searching for the nearest neighbor ofê in the embedding space: DISPLAYFORM0 where V is the output vocabulary, d is a distance function. In other words, the embedding space could be considered to be quantized into V components and the generated continuous vector is mapped to a word based on the quanta in which it lies. The mapped word is then passed to the next step of the decoder BID14. While training this model, we know the target vector e(w), and minimize its distance from the output vectorê. With this formulation, our model is directly trained to optimize towards the information encoded by the embeddings. For example, if the embeddings are primarily semantic, as in BID25 or BID4, the model would tend to output words in a semantic space, that is produced words would either be correct or close synonyms (which we see in our analysis in §6), or if we use synactico-semantic embeddings BID22 BID23, we might be able to also control for syntatic forms. We propose a novel probabilistic loss function-a probabilistic variant of cosine loss-which gives a theoretically grounded regression loss for sequence generation and addresses the limitations of existing empirical losses (described in §4.2). Cosine loss measures the closeness between vector directions. A natural choice for estimating directional distributions is von Mises-Fisher (vMF) defined over a hypersphere of unit norm. That is, a vector close to the mean direction will have high probability. VMF is considered the directional equivalent of Gaussian distribution 3. Given a target word w, its density function is given as follows: DISPLAYFORM1 where µ and e(w) are vectors of dimension m with unit norm, κ is a positive scalar, also called the concentration parameter. κ = 0 defines a uniform distribution over the hypersphere and κ = ∞ defines a point distribution at µ. C m (κ) is the normalization term: DISPLAYFORM2 where I v is called modified Bessel function of the first kind of order v. The output of the model at each step is a vectorê of dimension m. We use κ = ê. Thus the density function becomes: DISPLAYFORM3 It is noteworthy that equation 2 is very similar to softmax computation (except that e(w) is a unit vector), the main difference being that normalization is not done by summing over the vocabulary, which makes it much faster than the softmax computation. More details about it's computation are given in the appendix. The negative log-likelihood of the vMF distribution, which at each output step is given by: DISPLAYFORM4 Regularization of NLLvMF In practice, we observe that the NLLvMF loss puts too much weight on increasing ê, making the second term in the loss function decrease rapidly without significant decrease in the cosine distance. To account for this, we add a regularization term. We experiment with two variants of regularization. NLLvMF reg1: We add λ 1 ê to the loss function, where λ 1 is a scalar hyperparameter. 4 This makes intuitive sense in that the length of the output vector should not increase too much. The regularized loss function is as follows: DISPLAYFORM5 We modify the previous loss function as follows: DISPLAYFORM6 − log C m (ê) decreases slowly as ê increases as compared the second term. Adding a λ 2 < 1 the second term controls for how fast it can decrease. We modify the standard seq2seq models in OpenNMT 6 in PyTorch 7 BID19 ) to implement the architecture described in §3. This model has a bidirectional LSTM encoder with an attentionbased decoder BID24. The encoder has one layer whereas the decoder has 2 layers of 3 A natural choice for many regression tasks would be to use a loss function based on Gaussian distribution itself which is a probabilistic version of 2 loss. But as we describe in §4.2, 2 is not considered a suitable loss for regression on embedding spaces 4 We empirically set λ1 = 0.02 in all our experiments 5 We use λ2 = 0.1 in all our experiments 6 http://opennmt.net/ 7 https://pytorch.org/ size 1024 with the input word embedding size of 512. For the baseline systems, the output at each decoder step multiplies a weight matrix (H × V) followed by softmax. This model is trained until convergence on the validation perplexity. For our proposed models, we replace the softmax layer with the continuous output layer (H × m) where the outputs are m dimensional. We empirically choose m = 300 for all our experiments. Additional hyperparameter settings can be found in the appendix. These models are trained until convergence on the validation loss. Out of vocabulary words are mapped to an unk token 8. We assign unk an embedding equal to the average of embeddings of all the words which are not present in the target vocabulary of the training set but are present in vocabulary on which the word embeddings are trained. Following BID10, after decoding a post-processing step replaces the unk token using a dictionary look-up of the word with highest attention score. If the word does not exist in the dictionary, we back off to copying the source word itself. Bilingual dictionaries are automatically extracted from our parallel training corpus using word alignment BID12 9. We evaluate all the models on the test data using the BLEU score BID29.We evaluate our systems on standard machine translation datasets from IWSLT'16 BID6, on two target languages, English: German→English, French→English and a morphologically richer language French: English→French. The training sets for each of the language pairs contain around 220,000 parallel sentences. We use TED Test 2013+2014 (2,300 sentence pairs) as developments sets and TED Test 2015+2016 (2,200 sentence pairs) as test sets respectively for all the language pairs. All mentioned setups have a total vocabulary size of around 55,000 in the target language of which we choose top 50,000 words by frequency as the target vocabulary 10.We also experiment with a much larger WMT'16 German→English BID5 task whose training set contains around 4.5M sentence pairs with the target vocabulary size of around 800,000. We use newstest2015 and newstest2016 as development and test data respectively. Since with continuous outputs we do not need to perform a time consuming softmax computation, we can train the proposed model with very large target vocabulary without any change in training time per batch. We perform this experiment with WMT'16 de-en dataset with a target vocabulary size of 300,000 (basically all the words in the target vocabulary for which we had trained embeddings). But to able to produce these words, the source vocabulary also needs to be increased to have their translations in the inputs, which would lead to a huge increase in the number of trainable parameters. Instead, we use sub-words computed using BPE as source vocabulary. We use 100,000 merge operations to compute the source vocabulary as we observe using a smaller number leads to too small (and less meaningful) sub-word units which are difficult to align with target words. Both of these datasets contain examples from vastly different domains, while IWSLT'16 contains less formal spoken language, WMT'16 contains data primarily from news. We train target word embeddings for English and French on corpora constructed using WMT'16 BID5 monolingual datasets containing data from Europarl, News Commentary, News Crawl from 2007 to 2015 and News Discussion (everything except Common Crawl due to its large memory requirements). These corpora consist of 4B+ tokens for English and 2B+ tokens for French. We experiment with two embedding models: word2vec BID25 and fasttext Bojanowski et al. FORMULA0 which were trained using the hyper-parameters recommended by the authors. We compare our proposed loss function with standard loss functions used in multivariate regression. Squared Error is the most common distance function used when the model outputs are continuous BID21. For each target word w, it is given as L 2 = ê − e(w) 2 2 penalizes large errors more strongly and therefore is sensitive to outliers. To avoid this we use a square rooted version of 2 loss. But it has been argued that there is a mismatch between the objective function used to learn word representations (maximum likelihood based on inner product), the distance measure for word vectors (cosine similarity), and 2 distance as the objective function 8 Although the proposed model can make decoding open vocabulary, there could still be unknown words, e.g., words for which we do not have pre-trained embeddings; we need unk token to represent these words 9 https://github.com/clab/fast_align 10 Removing the bottom 5,000 words did not make a significant difference in terms of translation quality to learn transformations of word vectors BID41. This argument prompts us to look at cosine loss. ê. e(w). This loss minimizes the distance between the directions of output and target vectors while disregarding their magnitudes. The target embedding space in this case becomes a set of points on a hypersphere of dimension m with unit radius. BID20 argue that using pairwise losses like 2 or cosine distance for learning vectors in high dimensional spaces leads to hubness: word vectors of a subset of words appear as nearest neighbors of many points in the output vector space. To alleviate this, we experiment with a margin-based ranking loss (which has been shown to reduce hubness) to train the model to rank the word vector predictionê for target vector e(w) higher than any other word vector e(w) in the embedding space. L mm = w ∈V,w =w max{0, γ + cos(ê, e(w)) − cos(ê, e(w))} where, γ is a hyperparameter 11 representing the margin and w denotes negative examples. We use only one informative negative example as described in BID20 which is closest tô e and farthest from the target word vector e(w). But, searching for this negative example requires iterating over the vocabulary which brings back the problem of slow loss computation. In the case of empirical losses, we output the word whose target embedding is the nearest neighbor to the vector in terms of the distance (loss) defined. In the case of NLLvMF, we predict the word whose target embedding has the highest value of vMF probability density wrt to the output vector. This predicted word is fed as the input for the next time step. Our nearest-neighbor decoding scheme is equivalent to a greedy decoding; we thus compare to baseline models with beam size of 1. Until now we discussed the embeddings in the output layer. Additionally, decoder in a sequenceto-sequence model has an input embedding matrix as the previous output word is fed as an input to the decoder. Much of the size of the trainable parameters in all the models is occupied by these input embedding weights. We experiment with keeping this embedding layer fixed and tied with pre-trained target output embeddings BID33. This leads to significant reduction in the number of parameters in our model. TAB8 shows the BLEU scores on the test sets for several baseline systems, and various configurations including the types of losses, types of inputs/outputs used (word, BPE, or embedding) 12 and whether the model used tied embeddings in the decoder or not. Since we represent each target word by its embedding, the quality of embeddings should have an impact on the translation quality. We measure this by training our best model with fasttext embeddings BID4, which leads to > 1 BLEU improvement. Tied embeddings are the most effective setups: they not only achieve highest translation quality, but also dramatically reduce parameters requirements and the speed of convergence. 11 We use γ = 0.5 in our experiments. 12 Note that we do not experiment with subword embeddings since the number of merge operations for BPE usually depend on the choice of a language pair which would require the embeddings to be retrained for every language pair. TAB3 shows the average training time per batch. In FIG1 (left), we show how many samples per second our proposed model can process at training time compared to the baseline. As we increase the batch size, the gap between the baseline and the proposed models increases. Our proposed models can process large mini-batches while still training much faster than the baseline models. The largest mini-batch size with which we can train our model is 512, compared to 184 in the baseline model. Using max-margin loss leads to a slight increase in the training time compared to NLLvMF. This is because its computation needs a negative example which requires iterating over the entire vocabulary. Since our model requires look-up of nearest neighbors in the target embedding table while testing, it currently takes similar time as that of softmax-based models. In future work, approximate nearest neighbors algorithms BID17 can be used to improve translation time. We also compare the speed of convergence, using BLEU scores on dev data. In FIG1, we plot the BLEU scores against the number of epochs. Our model convergences much faster than the baseline models leading to an even larger improvement in overall training time (Similar figures for more datasets can be found in the appendix). As a , as shown in table 3, the total training time of our proposed model (until convergence) is less than up-to 2.5x of the total training time of the baseline models. Memory Requirements As shown in TAB3 our best performing model requires less than 1% of the number of parameters in input and output layers, compared to BPE-based baselines. Softmax BPE Emb w/ NLL-vMF fr-en 4h 4.5h 1.9h de-en 3h 3.5h 1.5h en-fr 1.8h 2.8h 1.3 WMT de-en 4.3d 4.5d 1.6d Table 3: Total convergence times in hours(h)/days(d). Translation of Rare Words We evaluate the translation accuracy of words in the test set based on their frequency in the training corpus. Table 5 shows how the F 1 score varies with the word frequency. F 1 score gives a balance between recall (the fraction of words in the reference that the predicted sentence produces right) and precision (the fraction of produced words that are in reference). We show substantial improvements over softmax and BPE baselines in translating less frequent and rare words, which we hypothesize is due to having learned good embeddings of such words from the monolingual target corpus where these words are not as rare. Moreover, in BPE based models, rare words on the source side are split in smaller units which are in some cases not properly translated in subword units on the target side if transparent alignments don't exist. For example, the word saboter in French is translated to sab+ot+tate by the BPE model whereas correctly translated as sabotage by our model. Also, a rare word retraite in French in translated to pension by both Softmax and BPE models (pension is a related word but less rare in the corpus) instead of the expected translation retirement which our model gets right. We conducted a thorough analysis of outputs across our experimental setups. Few examples are shown in the appendix. Interestingly, there are many examples where our models do not exactly match the reference translations (so they do not benefit from in terms of BLEU scores) but produce meaningful translations. This is likely because the model produces nearby words of the target words or paraphrases instead of the target word (which are many times synonyms).Since we are predicting embeddings instead of actual words, the model tends to be weaker sometimes and does not follow a good language model and leads to ungrammatical outputs in cases where the baseline model would perform well. Integrating a pre-trained language model within the decoding framework is one potential avenue for our future work. Another reason for this type of errors could be our choice of target embeddings which are not modeled to (explicitly) capture syntactic relationships. Using syntactically inspired embeddings BID22 BID23 might help reduce these errors. However, such fluency errors are not uncommon also in softmax and BPE-based models either. Table 5: Test set unigram F 1 scores of occurrence in the predicted sentences based on their frequencies in the training corpus for different models for fr-en. This work makes several contributions. We introduce a novel framework of sequence to sequence learning for language generation using word embeddings as outputs. We propose new probabilistic loss functions based on vMF distribution for learning in this framework. We then show that the proposed model trained on the task of machine translation leads to reduction in trainable parameters, to faster convergence, and a dramatic speed-up, up to 2.5x in training time over standard benchmarks. TAB5 visualizes a comparison between different types of softmax approximations and our proposed method. State-of-the-art in softmax-based models are highly optimized after a few years on research in neural machine translation. The that we report are comparable or slightly lower than the strongest baselines, but these setups are only an initial investigation of translation with the continuous output layer. There are numerous possible directions to explore and improve the proposed setups. What are additional loss functions? How to setup beam search? Should we use scheduled sampling? What types of embeddings to use? How to translate with the embedding output into morphologically-rich languages? Can low-resource neural machine translation benefit from translation with continuous outputs if large monolingual corpora are available to pre-train strong target-side embeddings? We will explore these questions in future work. Furthermore, the proposed architecture and the probabilistic loss (NLLvMF) have the potential to benefit other applications which have sequences as outputs, e.g. speech recognition. NLLvMF could be used as an objective function for problems which currently use cosine or 2 distance, such as learning multilingual word embeddings. Since the outputs of our models are continuous (rather than class-based discrete symbols), these models can potentially simplify training of generative adversarial networks for language generation. DISPLAYFORM0 where C m (κ) is given as: DISPLAYFORM1 The normalization constant is not directly differentiable because Bessel function cannot be written in a closed form. The gradient of the first component (log (C m ê)) of the loss is given as DISPLAYFORM2. In Table 9: Translation quality experiments using beam search with BPE based baseline models with a beam size of 5With our proposed models, in principle, it is possible to generate candidates for beam search by using K-Nearest Neighbors. But how to rank the partially generated sequences is not trivial (one could use the loss values themselves to rank, but initial experiments with this setting did not in significant gains). In this work, we focus on enabling training with continuous outputs efficiently and accurately giving us huge gains in training time. The question of decoding with beam search requires substantial investigation and we leave it for future work. Une ducation est critique, mais rgler ce problme va ncessiter que chacun d'entre nous s'engage et soit un meilleur exemple pour les femmes et filles dans nos vies. An education is critical, but tackling this problem is going to require each and everyone of us to step up and be better role models for the women and girls in our own lives. Education is critical, but it's going to require that each of us will come in and if you do a better example for women and girls in our lives. Education is critical, but to to do this is going to require that each of us of to engage and or a better example of the women and girls in our lives. That's critical, but that's that it's going to require that each of us is going to take that the problem and they're going to if you're a better example for women and girls in our lives. Predicted (MaxMargin) Education is critical, but that problem is going to require that every one of us is engaging and is a better example for women and girls in our lives. Predicted (NLLvMF reg) Education is critical, but fixed this problem is going to require that all of us engage and be a better example for women and girls in our lives. TAB8: Translation examples. Red and blue colors highlight translation errors; red are bad and blue are outputs that are good translations, but are considered as errors by the BLEU metric. Our systems tend to generate a lot of such "meaningful" errors. Pourquoi ne sommes nous pas de simples robots qui traitent toutes ces donnes, produisent ces rsultats, sans faire l'exprience de ce film intrieur? Reference Why aren't we just robots who process all this input, produce all that output, without experiencing the inner movie at all? Predicted (BPE) Why don't we have simple robots that are processing all of this data, produce these , without doing the experience of that inner movie? Why are we not that we do that that are technologized and that that that's all these , that they're actually doing these , without do the experience of this film inside? Predicted (Cosine) Why are we not simple robots that all that data and produce these data without the experience of this film inside? Predicted (MaxMargin) Why aren't we just simple robots that have all this data, make these , without making the experience of this inside movie? Predicted (NLLvMF reg) Why are we not simple robots that treat all this data, produce these , without having the experience of this inside film? TAB8: Example of fluency errors in the baseline model. Red and blue colors highlight translation errors; red are bad and blue are outputs that are good translations, but are considered as errors by the BLEU metric.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality

Represent the TLDR for finding the corresponding scientific article

A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory. Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights. Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking. This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers. In the layer decoupling limit applicable to residual networks , we show that the remnant symmetries that survive the non-linear layers are spontaneously broken based on empirical . The Lagrangian for the non-linear and weight layers together has striking similarities with the one in quantum field theory of a scalar. Using from quantum field theory we show that our framework is able to explain many experimentally observed phenomena, such as training on random labels with zero error , the information bottleneck and the phase transition out of it , shattered gradients , and many more. Deep neural networks have been used in image recognition tasks with great success. The first of its kind, AlexNet BID8, led to many other neural architectures have been proposed to achieve start-of-the-art in image processing at the time. Some of the notable architectures include, VGG BID12, Inception BID14 and Residual networks (ResNet) BID3.Understanding the inner workings of deep neural networks remains a difficult task. It has been discovered that the training process ceases when it goes through an information bottleneck until learning rate is decreased to a suitable amount then the network under goes a phase transition. Deep networks appear to be able to regularize themselves and able to train on randomly labeled data BID18 with zero training error. The gradients in deep neural networks behaves as white noise over the layers BID1. And many other unexplained phenomena. A recent work BID0 showed that the ensemble behavior and binomial path lengths BID15 of ResNets can be explained by just a Taylor series expansion to first order in the decoupling limit. They found that the series approximation generates a symmetry breaking layer that reduces the redundancy of weights, leading to a better generalization. Because the ResNet does not contain such symmetry breaking layers in the architecture. They suggest that ResNets are able to break the symmetry by the communication between the layers. Another recent work also employed the Taylor expansion to investigate ResNets BID6.In statistical terms, a quantum theory describes errors from the mean of random variables. We wish to study how error propagate through each layer in the network, layer by layer. In the limit of a continuous sample space, the quantum theory becomes a quantum field theory. The effects of sampling error and labelling error can then be investigated. It is well known in physics that a scalar field can drive a phase transition. Using a scalar field theory we show that a phase transition must exist towards the end of training based on empirical . It is also responsible for the remarkable performance of deep networks compared to other classical models. In Appendix D, We explain that quantum field theory is likely one of the simplest model that can describe a deep network layer by layer in the decoupling limit. Much of the literature on neural network design focuses on different neural architecture that breaks symmetry explicitly, rather than spontaneously. For instance, non-linear layers explicitly breaks the symmetry of affine transformations. There is little discussion on spontaneous symmetry breaking. In neural networks, the Goldstone theorem in field theory states that for every continuous symmetry that is spontaneously broken, there exists a weight with zero Hessian eigenvalue at the loss minimum. No such weights would appear if the symmetries are explicitly broken. It turns out that many seemingly different experimental can be explained by the presence of these zero eigenvalue weights. In this work, we exploit the layer decoupling limit applicable to ResNets to approximate the loss functions with a power series in symmetry invariant quantities and illustrate that spontaneous symmetry breaking of affine symmetries is the sufficient and necessary condition for a deep network to attain its unprecedented power. The organization of this paper is as follows. The on deep neural networks and field theory is given in Section 2. Section 3 shows that remnant symmetries can exist in a neural network and that the weights can be approximated by a scalar field. Experimental that confirm our theory is given in Section 4. We summarize more evidence from other experiments in Appendix A. A review of field theory is given in Appendix B. An explicit example of spontaneous symmetry breaking is shown in Appendix C. In this section we introduce our frame work using a field theory based on Lagrangian mechanics. A deep neural network consists of layers of neurons. Suppose that the first layer of neurons with weight matrix W 1 and bias b 1 takes input x 1 and outputs y 1 DISPLAYFORM0 where x = (x, 1) and W 1 = (W 1, b 1), where W 1 and b 1 are real valued. Now suppose that R 1 denotes a nonlinear operator corresponding to a sigmoid or ReLU layer located after the weight layer, so that DISPLAYFORM1 For a neural network with T repeating units, the output for layer t is DISPLAYFORM2 We now show the necessary and sufficient conditions of preserving symmetry. We explicitly include symmetry transformations in Equation and investigate the effects caused by a symmetry transformation of the input in subsequent layers. Suppose Q t ∈ G is a transformation matrix in some Lie group G for all t. Note that the Q t are not parameters to be estimated. We write y t = y t (Q t), where the dependence on Q t is obtained from a transformation on the input, x t (Q t) = Q t x t, and the weights, DISPLAYFORM0 If G is a symmetry group, then y t is covariant with x t, such that y t (Q t) = Q t y t. This requires two conditions to be satisfied. First, DISPLAYFORM1 t, where Q −1 t Q t = I and the existence of the inverse is trivial because G is a group and Q t ∈ G. The second is the commutativity between R t and Q t, such that R t Q t = Q t R t. For example, if g t ∈ Aff(D), the group of affine transformations, R t may not commute with g t. However, commutativity is satisfied when the transformation corresponds to the 2D rotation of feature maps. Including transformation matrices, the output at layer t is DISPLAYFORM2 Statistical learning requires the loss function to be minimized. It can be written in the form of a mutual information, training error, or the Kullback-Leibler divergence. In this section we approximate the loss function in the continuum limit of samples and layers. Then we define the loss functional to transition into Lagrangian mechanics and field theory. Let z i = (X i, Y i) ∈ X be the i-th input sample in data set X, (X i, Y i) are the features and the desired outputs, respectively, and i ∈ {1, . . ., N}. The loss function is DISPLAYFORM0 where W = (W 1, . . ., W T), and Q = (Q 1, . . ., Q T), Q t ∈ G where G is a Lie group, and T is the depth of the network. Taking the continuum limit, DISPLAYFORM1 where p(X, Y) is the joint distribution of X and Y. Using the first fundamental theorem of calculus and taking the continuous layers (t) limit, we write DISPLAYFORM2 where L x (t = 0) is the value of the loss before training. We let L x,t = dL x /dt be the loss rate per layer. The loss rate L x,t is bounded from below. Therefore DISPLAYFORM3 Minimizing the loss rate guarantees the minimization of the total loss. We require L x,t to be invariant under symmetry transformations. That is, if DISPLAYFORM4 However if Q 1 (t) and Q 2 (t) do not belong in the same symmetry group, the above equality does not necessarily hold. Now we define the loss functional for a deep neural network DISPLAYFORM5 Having defined the loss functional, we can transition into Lagrangian dynamics to give a description of the feature map flow at each layer. Let the minimizer of the loss rate be DISPLAYFORM0 From now on, we combine z = (X, Y) as Y only appears in W * in this formalism, each Y determines a trajactory for the representation flow determined by Lagrangian mechanics. Now we define, for each i-th element of W(t), and a non-linear operator R(t) acting on W(t) such that the loss minimum is centered at the origin, DISPLAYFORM1 We now define the Lagrangian density, DISPLAYFORM2 and L = T − V, where T is the kinetic energy and V is the potential energy. We define the potential energy to be DISPLAYFORM3 The probability density p(z) and the loss rate L x,t are invariant under symmetry transformations. Therefore V is an invariant as well. DISPLAYFORM4 We now set up the conditions to obtain a series expansion of V around the minimum w i (Q t) = 0. First, since V is an invariant. Each term in the series expansion must be an invariant such that DISPLAYFORM5, the orthogonal group and that w T (Q t) = w T Q T and w(Q t) = Q t w. So w i w i is an invariant. Then f = w i w i is invariant for all Q t where the Einstein summation convention was used DISPLAYFORM6 Now we perform a Taylor series expansion about the minimum w i = 0 of the potential, DISPLAYFORM7 where H i j = ∂ w i ∂ w j V is the Hessian matrix, and similarly for Λ ij mn. The overall constant C can be ignored without loss of generality. Because V is an even function in w i around the minimum, we must have DISPLAYFORM8 The O(D) symmetry enforces that all weight Hessian eigenvalues to be H i i = m 2 /2 for some constant m 2. This can be seen in the O case, with constants a, b, a = b, Q ∈ O such that w 1 (Q) = w 2 and w 2 (Q) = w 1, DISPLAYFORM9, so the O symmetry implies a = b. This can be generalized to the O(D) case. For the quartic term, the requirement that V be even around the minimum gives DISPLAYFORM10 ii ii = λ/4 for some constant λ and zero for any other elements, the potential is DISPLAYFORM0 where the numerical factors were added for convention. The power series is a good approximation in the decoupling limit which may be applicable for Residual Networks. 1 For the kinetic term T, we expand in power series of the derivatives, DISPLAYFORM1 where the coefficient for (∂ t w) 2 is fixed by the Hamiltonian kinetic energy 1 2 (∂ t w) 2. Higher order terms in (∂ t w) 2 are negligible in the decoupling limit. If the model is robust, then higher order terms in (∂ z w) 2 can be neglected as well. 2 The Lagrangian density is DISPLAYFORM2 where we have set w 2 = w i w i and absorbed c into z without loss of generality. This is precisely the Lagrangian for a scalar field in field theory. Standard for a scalar field theory can be found in Appendix B. To account for the effect of the learning rate, we employ from thermal field theory BID7 and we identify the temperature with the learning rate η. So that now DISPLAYFORM3 Spontaneous symmetry breaking describes a phase transition of a deep neural network. Consider the following scalar field potential invariant under O(D) transformations, DISPLAYFORM0 where m 2 (η) = −µ 2 + 1 4 λη 2, µ 2 > 0 and learning rate η. There exists a value of η = η c such that m 2 = 0. In the first phase, η > η c, the loss minimum is at w * 0i = 0, where DISPLAYFORM1 When the learning rate η drops sufficiently low, the symmetry is spontaneously broken and the phase transition begins. The loss minimum bifurcates at η = η c into DISPLAYFORM2 This occurs when the Hessian eigenvalue becomes negative, m 2 (η) < 0, when η < η c.This phenomenon has profound implications. It is responsible for phase transition in neural networks and generates long range correlation between representations and the desired output. Details from field theory can be found in Appendix C. FIG0 depicts the shape of the loss rate during spontaneous symmetry breaking with a single weight w, and the orthogonal group O(D) is reduced to a reflection symmetry O = {1, −1} such that w(Q) = ±w. At η > η c, the loss rate has a loss minima at point A. When the learning rate decreases, such that η < η c, the critical point at A becomes unstable and new minima with equal loss rate are generated. The weight must go through B to get to the new minimum C. If the learning rate is too small, the weight will be stuck near A. This explains why a cyclical learning rate can outperform a monotonic decreasing learning rate BID13.Because the loss rate is invariant still to the sponteneously broken symmetry, any new minima generated from spontaneous symmetry breaking must have the same loss rate. If there is a unbroken continuous symmetry remaining, there would be a connected loss rate surface corresponding to the new minima generated by the unbroken symmetry. Spontaneous symmetry breaking splits the weights into two sets, w → (π, σ). The direction along this degenerate minima in weight space corresponds to π. And the direction in weight space orthogonal to π is σ. This has been shown experimentally by BID2 in FIG0. We show the case for the breaking of O to O in FIG2.2 The kinetic term T is not invariant under transformation Q(t). To obtain invariance ∂tw i is to be replaced by the covariant derivative Dtw i so that (Dtw i) 2 is invariant under Q(t) BID10. The covariant derivative is DISPLAYFORM3 with B(z, t, Qt) = Q(t)B(z, t)Q(t) −1. The new fields B introduced for invariance is not responsible for spontaneous symmetry breaking, the focus of this paper. So we will not consider them further.3 Formally, the ∂zw term should be part of the potential V, as T contains only ∂tw terms. However we adhere to the field theory literature and put the ∂zw term in T with a minus sign. In this section we show that spontaneous symmetry breaking occurs in neural networks. First, we show that learning by deep neural networks can be considered solely as breaking the symmetries in the weights. Then we show that some non-linear layers can preserve symmetries across the nonlinear layers. Then we show that weight pairs in adjacent layers, but not within the same layer, is approximately an invariant under the remnant symmetry leftover by the non-linearities. We assume that the weights are scalar fields invariant under the affine Aff(D) group for some D and find that experimental show that deep neural networks undergo spontaneous symmetry breaking. Theorem 1: Deep feedforward networks learn by breaking symmetries Proof: Let A i be an operator representing any sequence of layers, and let a network formed by applying A i repeatedly such that DISPLAYFORM0. Then x out = Lx in for some L ∈ Aff(D) and x out can be computed by a single affine transformation L. When A i contains a nonlinearity for some i, this symmetry is explicitly broken by the nonlinearity and the layers learn a more generalized representation of the input. Now we show that ReLU preserves some continuous symmetries. Theorem 2: ReLU reduces the symmetry of an Aff(D) invariant to some subgroup Aff(D), where D < D. Proof: Suppose R denotes the ReLU operator with output y t and Q t ∈ Aff(D) acts on the input x t, where R(x) = max(0, x). Let x T x be an invariant under Aff(D) and let DISPLAYFORM1 Note that γ i can be transformed into a negative value as it has passed the ReLU already. Corollary If there exists a group G that commutes with a nonlinear operator R, such that QR = RQ, for all Q ∈ G, then R preserves the symmetry G.Definition: Remnant Symmetry If Q t ∈ G commutes with a non-linear operator R t for all Q t, then G is a remnant symmetry at layer t. For the loss function L i (X i, Y i, W, Q) to be invariant, we need the predicted output y T to be covariant with x i. Similarly for an invariant loss rate L x,t we require y t to be covariant with x t. The following theorem shows that a pair of weights in adjacent layers can be considered an invariant for power series expansion. Theorem 3: Neural network weights in adjacent layers form an approximate invariant Suppose a neural network consists of affine layers followed by a continuous non-linearity, R t, and that the weights at layer t, W t (Q t) = Q t W t Q −1 t, and that Q t ∈ H is a remnant symmetry such that Q t R t = R t Q t. Then w t w t−1 can be considered as an invariant for the loss rate. Proof: Consider x(Q t) = Q t x t, then DISPLAYFORM2 where in the last line Q t R t = R t Q t was used, so y t (Q t) = Q t y t is covariant with x t. Now, x t = R t−1 W t−1 x t−1, so that DISPLAYFORM3 The pair (R t W t)(R t−1 W t−1) can be considered an invariant under the ramnant symmetry at layer t. Let w t = R t W t − R t W * t. Therefore w t w t−1 is an invariant. In the continuous layer limit, w t w t−1 tends to w(t)T w(t) such that w(t) is the first layer and w(t)T corresponds to the one after. Therefore w(t) can be considered as D scalar fields under the remnant symmetry. The remnant symmetry is not exact in general. For sigmoid functions it is only an approximation. The crucial feature for the remnant symmetry is that it is continuous so that strong correlation between inputs and outputs can be generated from spontaneous symmetry breaking. In the following we will only consider exact remnant symmetries. We will state the Goldstone Theorem from field theory without proof. Theorem (Goldstone) For every spontaneously broken continuous symmetry, there exist a weight π with zero eigenvalue in the Hessian m 2 π = 0. In any case, we will adhere to the case where the remnant symmetry is an orthogonal group O(D). Note that W is a D × D matrix and D < D. We choose a subset Γ ∈ R D of W such that Γ T Γ is invariant under Aff(D). Now that we have an invariant, we can write down the Lagrangian for a deep feedforward network for the weights responsible for spontaneous symmetry breaking. Now we can use standard field theory and apply it to deep neural networks. A review for field theory is given in Appendix B. The formalism for spontaneous symmetry breaking is given in Appendix C. In this section we assume that the non-linear operator is a piecewise linear function such as ReLU and set R = I to be the identity and restrict our attention to the symmetry preserving part of R (see theorem 2). Our discussion also applies to other piecewise-linear activation functions. According to the Goldstone theorem, spontaneous symmetry breaking splits the set of weight deviations γ into two sets (σ, π) with different behaviors. Weights π with zero eigenvalues and a spectrum dominated by small frequencies k in its correlation function. 4 The other weights σ, have Hessian eigenvalues µ 2 as the weights before the symmetry is broken. In Appendix C, a standard calculation in field theory shows that the correlation functions of the weights have the form Spontaneous symmetry breaking and the information bottleneck The neural network undergoes a phase transition out of the information bottleneck via spontaneous symmetry breaking described in Section 2.5. Before the phase transition, the weights γ have positive Hessian eigenvalues m 2. After the phase transition, weights π with zero Hessian eigenvalues are generated by spontaneous symmetry breaking. The correlation function for the π weights is concentrated around small values of |k|, see Equation FORMULA36, with ω 0 = |k| for any t. This corresponds to a highly correlated representations across the sample (input) space and layers. Because the loss is minimized, the feature maps across the network is highly correlated with the desired output. And a large correlation across the sample space means that the representations are independent of the input. This is shown in FIG2 of BID11. After phase transition, I(Y ; T) 1 bit for all layers T, and I(X; T) is small even for representations in early layers. DISPLAYFORM0 Gradient variance explosion It has been shown that the variance in weight gradients in the same layer grow by an order of magnitude during the end of training BID11. We also connect this to spontaneous symmetry breaking. As two sets of weights, (σ, π) are generated with different distributions. Considering them as the same object would in a larger variance. We find that neural networks are resilient to overfitting. Recall that the fluctuation in the weights can arise from sampling noise. Then (∂ z w i) 2 can be a measure of model robustness. A small value denotes the weights' resistance to sampling noise. If the network were to overfit, the weights would be very sensitive to sampling error. After spontaneous symmetry breaking, weights at the loss minimum with zero eigenvalues obey the Klein-Gordon equation with m DISPLAYFORM0 The singularity in the correlation function suggests |k| 2 0. The zero eigenvalue weights provide robustness to the model. BID18 referred to this phenomenon as implicit regularization. In this work we solved one of the most puzzling mysteries of deep learning by showing that deep neural networks undergo spontaneous symmetry breaking. This is a first attempt to describe a neural network with a scalar quantum field theory. We have shed light on many unexplained phenomenon observed in experiments, summarized in Appendix A.One may wonder why our theoretical model works so well explaining the experimental with just two parameters. It is due to the decoupling limit such that a power series in the loss function is a good approximation to the network. In our case, the two expansion coefficients are the lowest number of possible parameters that is able to describe the phase transition observed near the end of training, where the performance of the deep network improves drastically. It is no coincidence that our model can explain the empirical observations after the phase transition. In fact, our model can describe, at least qualitatively, the behaviors of phase transition in networks that the decoupling limit may not apply to. This suggests that the interactions with nearby layers are responsible for the phase transition. In this section we summarize other experimental findings that can be explained by the proposed field theory and the perspective of symmetry breaking. Here Q ∈ G acts on the the input and hidden variables x, h, as Qx, Qh.• The shape of the loss function after spontaneous symmetry breaking has the same shape observed by BID2 towards the end of training, see FIG0.• The training error typically drops drastically when learning rate is decreased. This occurs when the learning rate drops below η c, forcing a phase transition so that new minima develop. See FIG0 • A cyclical learning rate BID13 helps to get to the new minimum faster, see Section 2.5.• Stochasticity in gradient descent juggles the loss function such that the weights are no longer at the local maximum of FIG0. A gradient descent step is taken to further take the weights towards the local minimum. Stochasticity helps the network to generalize better.• When the learning rate is too small to move away from A in FIG0. PReLU's BID4 could move the weight away from A through the training of the non-linearity. This corresponds to breaking the symmetry explicitly in Theorem 1.• Results from are due to spontaneous symmetry breaking, see Section 4.• Deep neural networks can train on random labels with low training loss as feature maps are highly correlated with their respective desired output. BID18 observed that a deep neural network can achieve zero training error on random labels. This shows that small Hessian eigenvalues is not the only condition that determines robustness.• Identity mapping outperforms other skip connections BID5 ) is a of the residual unit's output being small. Then the residual units can be decoupled leading to a small λ and so it is easier for spontaneous symmetry breaking to occur, from m 2 = −µ 2 + 1 4 λη 2.• Skip connection across residual units breaks additional symmetry. Suppose now an identity skip connection connects x 1 and the output of F 2. Now perform a symmetry transformation on x 1 and x 2, Q 1 and Q 2 ∈ G, respectively. Then the output after two residual untis is Qx 3 = Q 1 x 1 + Q 2 x 2 + Q 2 F 2. Neither Q = Q 1 nor Q = Q 2 can satisfy the covariance under G. This is observed by BID9.• The shattered gradient problem BID1. It is observed that the gradient in deep (non-residual) networks is very close to white noise. This is reflected in the exponential in Equation FORMULA56. This effect on ResNet is reduced because of the decoupling limit λ → 0. This leads to the weight eigenvalues m 2 being larger in non-residual networks owing to m 2 = −µ 2 + 1 4 λη 2. And so a higher oscillation frequency in the correlation function.• In recurrent neural networks, multiplicative gating BID16 combines the input x and the hidden state h by an element-wise product. Their method outperforms the method with an addition x+h because the multiplication gating breaks the covariance of the output. A transformation Qx * Qh = Q(x * h), whereas for addition the output remains covariant Qx + Qh = Q(x + h). In this section we state the relevant in field theory without proof. We use Lagrangian mechanics for fields w(x, t). Equations of motion for fields are the solution to the Euler-Lagrange equation, which is a from the principle of least action. The action, S, is DISPLAYFORM0 where L is the Lagrangian. Define the Lagrangian density DISPLAYFORM1 The action in term of the Lagrangian density is DISPLAYFORM2 The Lagrangian can be written as a kinetic term T, and a potential term V (loss function), DISPLAYFORM3 For a real scalar field w(x, t), DISPLAYFORM4 where we have set the constant c 2 = 1 without loss of generality. The potential for a scalar field that allows spontaneous symmetry breaking has the form DISPLAYFORM5 In the decoupling limit, λ → 0, the equation of motion for w is the Klein-Gordon Equation DISPLAYFORM6 In the limit of m 2 → 0, the Klein-Gordon Equation reduces to the wave equation with solution w(z, t) = e i(ωt−k·z), DISPLAYFORM7 One can treat w as a random variable such that the probability distribution (a functional) of the scalar field w(z, t) is p[w] = exp(−S[w])/Z, where Z is some normalizing factor. The distribution peaks at the solution of the Klein-Gordon equation since it minimizes the action S. Now we can define the correlation function between w(z 1, t 1) and w(z 2, t 2), DISPLAYFORM8 where Dw denotes the integral over all paths from (z 1, t 1) to (z 2, t 2). In the decoupling limit λ → 0, it can be shown that DISPLAYFORM9 where Stokes theorem was used and the term on the boundary of (sample) space is set to zero. The above integral in the exponent is quadratic in w and the integral over Dw can be done in a similar manner to Gaussian integrals. The correlation function of the fields across two points in space and time is w(z 1, t 1)w(z 2, t 2) = G(z 1, t 1, z 2, t 2), where G(z 1, t 1, z 2, t 2) is the Green's function to the Klein-Gordon equation, satisfying DISPLAYFORM10 The Fourier transformation of the correlation function is DISPLAYFORM11 An inverse transform over ω gives DISPLAYFORM12 with ω DISPLAYFORM13 In this section we show that weights π with small, near zero, eigenvalues m 2 π = 1 4 λη 2 are generated by spontaneous symmetry breaking. Note that we can write the Lagrangian in Equation as L = T − V. Consider weights γ that transforms under O(D), from Equation DISPLAYFORM0 When m 2 = −µ 2 + 1 4 λη 2 < 0, it can be shown that in this case the loss minimum is no longer at γ i = 0, but it has a degenerate minima on the surface such that i (γ i) 2 = v, where v = −m 2 /λ. Now we pick a point on this loss minima and expand around it. Write γ i = (π k, v + σ), where k ∈ {1, . . ., D − 1}. Intuitively, the π k fields are in the subspace of degenerate minima and σ is the field orthogonal to π. Then it can be shown that the Lagrangian can be written as DISPLAYFORM1 where, in the weak coupling limit λ → 0, DISPLAYFORM2 V π = O(λ), DISPLAYFORM3 the fields π and σ decouple from each other and can be treated separately. The σ fields satisfy the Klein-Gordon Equation (− m 2)σ = 0, with = ∂ 2 t − ∂ 2 z. The π fields satisfy the waveequation, π = 0. The correlation functions of the weights across sample space and layers, P σ = σ(z, t)σ(z, t) and P π = π(z, t)π(z, t) are the Green's functions of the respective equations of motion. Fourier transforming the correlation functions give P σ,π (t, k) = i 2ω 0 exp − iω 0 t,where ω 0 = |k| 2 + |m 2 σ,π |, and m Even though we formulated our field theory based on the decoupling limit of ResNets, the of infinite correlation is very general and can be applied even if the decoupling limit is not valid. It is a direct of spontaneous symmetry breaking. We state the Goldstone Theorem without proof. Theorem (Goldstone): For every continuous symmetry that is spontaneously broken, a weight π with zero Hessian eigenvalue is generated at zero temperature (learning rate η). In brief, the formalism for spontaneous symmetry breaking is mostly done in quantum field theory. In terms of statistics, quantum mechanics is the study of errors. We also believe that it is a good approximation to deep neural networks in the presence of the non-linear operators. The non-linear operators quantizes the input. Let R denotes the opertor corresponding to a sigmoid, say, then the output is R(W) {0, +1} for the most part. And the negative end of ReLU is zero. Let us take a step back and go through the logical steps to understand that a scalar quantum field theory is perhaps one of the simplest model one can consider to describe a neural network layer by layer, in the decoupling limit. We wish to formulate a dynamical model to describe the weights layer by layer,

Represent the short summary for finding the corresponding scientific article (long)

Closed form results for deep learning in the layer decoupling limit applicable to Residual Networks

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.