Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM's final prediction. Using the phrase-level labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done. In comparison with simpler linear models, techniques from deep learning have achieved impressive accuracy by effectively learning non-linear interactions between features. However, due to our inability to describe the learned interactions, this improvement in accuracy has come at the cost of state of the art predictive algorithms being commonly regarded as black-boxes. In the domain of natural language processing (NLP), Long Short Term Memory networks (LSTMs) BID2 have become a basic building block, yielding excellent performance across a wide variety of tasks BID10 BID7, while remaining largely inscrutable. In this work, we introduce contextual decomposition (CD), a novel interpretation method for explaining individual predictions made by an LSTM without any modifications to the underlying model. CD extracts information about not only which words contributed to a LSTM's prediction, but also how they were combined in order to yield the final prediction. By mathematically decomposing the LSTM's output, we are able to disambiguate the contributions made at each step by different parts of the sentence. To validate the CD interpretations extracted from an LSTM, we evaluate on the problem of sentiment analysis. In particular, we demonstrate that CD is capable of identifying words and phrases of differing sentiment within a given review. CD is also used to successfully extract positive and negative negations from an LSTM, something that has not previously been done. As a consequence of this analysis, we also show that prior interpretation methods produce scores which have document-level information built into them in complex, unspecified ways. For instance, prior work often identifies strongly negative phrases contained within positive reviews as neutral, or even positive. The most relevant prior work on interpreting LSTMs has focused on approaches for computing word-level importance scores, with evaluation protocols varying greatly. BID8 introduced a decomposition of the LSTM's output embedding into a sum over word coefficients, and demonstrated that those coefficients are meaningful by using them to distill LSTMs into rules-based classifiers. BID5 took a more black box approach, called Leave One Out, by observing the change in log probability ing from replacing a given word vector with a zero vector, and relied solely on anecdotal evaluation. presents a general gradient-based technique, called Integrated Gradients, which was validated both theoretically and with empirical anecdotes. In contrast to our proposed method, this line of work has been limited to word-based importance scores, ignoring the interactions between variables which make LSTMs so accurate. Another line of work BID3 BID14 has focused on analysing the movement of raw gate activations over a sequence. BID3 was able to identify some co-ordinates of the cell state that correspond to semantically meaningful attributes, such as whether the text is in quotes. However, most of the cell co-ordinates were uninterpretable, and it is not clear how these co-ordinates combine to contribute to the actual prediction. Decomposition-based approaches to interpretation have also been applied to convolutional neural networks (CNNs) BID0 BID12. However, they have been limited to producing pixel-level importance scores, ignoring interactions between pixels, which are clearly quite important. Our approach is similar to these in that it computes an exact decomposition, but we leverage the unique gating structure of LSTMs in order to extract interactions. Attention based models BID1 offer another means of providing some interpretability. Such models have been successfully applied to many problems, yielding improved performance BID11 ) (. In contrast to other word importance scores, attention is limited in that it only provides an indirect indicator of importance, with no directionality, i.e. what class the word is important for. Although attention weights are often cited anecdotally, they have not been evaluated, empirically or otherwise, as an interpretation technique. As with other prior work, attention is also incapable of describing interactions between words. Given an arbitrary phrase contained within an input, we present a novel decomposition of the output of an LSTM into a sum of two contributions: those ing solely from the given phrase, and those involving other factors. The key insight behind this decomposition is that the gating dynamics unique to LSTMs are a vehicle for modeling interactions between variables. Over the past few years, LSTMs have become a core component of neural NLP systems. Given a sequence of word embeddings x 1, ..., x T ∈ R d1, a cell and state vector c t, h t ∈ R d2 are computed for each element by iteratively applying the below equations, with initialization h 0 = c 0 = 0. DISPLAYFORM0 DISPLAYFORM1 and denotes element-wise multiplication. o t, f t and i t are often referred to as output, forget and input gates, respectively, due to the fact that their values are bounded between 0 and 1, and that they are used in element-wise multiplication. After processing the full sequence, the final state h T is treated as a vector of learned features, and used as input to a multinomial logistic regression, often called SoftMax, to return a probability distribution p over C classes, with DISPLAYFORM2 3.2 CONTEXTUAL DECOMPOSITION OF LSTMWe now introduce contextual decomposition, our proposed method for interpreting LSTMs. Given an arbitrary phrase x q, ..., x r, where 1 ≤ q ≤ r ≤ T, we now decompose each output and cell state c t, h t in Equations 5 and 6 into a sum of two contributions. DISPLAYFORM3 The decomposition is constructed so that β t corresponds to contributions made solely by the given phrase to h t, and that γ t corresponds to contributions involving, at least in part, elements outside of the phrase. β DISPLAYFORM4 Here W β T provides a quantitative score for the phrase's contribution to the LSTM's prediction. As this score corresponds to the input to a logistic regression, it may be interpreted in the same way as a standard logistic regression coefficient. In the cell update Equation 5, neuron values in each of i t and g t are independently determined by both the contribution at that step, x t, as well as prior context provided by h t−1 = β t−1 + γ t−1 . Thus, in computing the element-wise product i t g t, often referred to as gating, contributions made by x t to i t interact with contributions made by h t to g t, and vice versa. We leverage this simple insight to construct our decomposition. First, assume that we have a way of linearizing the gates and updates in Equations 2, 3, 4 so that we can write each of them as a linear sum of contributions from each of their inputs. DISPLAYFORM0 When we use this linearization in the cell update Equation 5, the products between gates become products over linear sums of contributions from different factors. Upon expanding these products, the ing cross-terms yield a natural interpretation as being interactions between variables. In particular, cross-terms can be assigned as to whether they ed solely from the phrase, e.g. DISPLAYFORM1, from some interaction between the phrase and other factors, e.g. DISPLAYFORM2 Mirroring the recurrent nature of LSTMs, the above insights allow us to recursively compute our decomposition, with the initializations DISPLAYFORM3 We derive below the update equations for the case where q ≤ t ≤ r, so that the current time step is contained within the phrase. The other case is similar, and the general recursion formula is provided in Appendix 6.2.For clarity, we decompose the two products in the cell update Equation 5 separately. As discussed above, we simply linearize the gates involved, expand the ing product of sums, and group the cross-terms according to whether or not their contributions derive solely from the specified phrase, or otherwise. Terms are determined to derive solely from the specified phrase if they involve products from some combination of β t−1, β c t−1, x t and b i or b g (but not both). When t is not within the phrase, products involving x t are treated as not deriving from the phrase. DISPLAYFORM4 Having decomposed the two components of the cell update equation, we can attain our decomposition of c t by summing the two contributions. DISPLAYFORM5 Once we have computed the decomposition of c t, it is relatively simple to compute the ing transformation of h t by linearizing the tanh function in 6. Note that we could similarly decompose the output gate as we treated the forget gate above, but we empirically found this to not produce improved . DISPLAYFORM6 We now describe the linearizing functions L σ, L tanh used in the above decomposition. Formally, for arbitrary {y 1, ..., y N} ∈ R, where N ≤ 4, the problem is how to write DISPLAYFORM0 In the cases where there is a natural ordering to {y i}, prior work BID8 has used a telescoping sum consisting of differences of partial sums as a linearization technique, which we show below. DISPLAYFORM1 However, in our setting {y i} contains terms such as β t−1, γ t−1 and x t, which have no clear ordering. Thus, there is no natural way to order the sum in Equation 26. Instead, we compute an average over all orderings. Letting π 1,..., π M N denote the set of all permutations of 1,..., N, our score is given below. Note that when π i (j) = j, the corresponding term is equal to equation 26. DISPLAYFORM2 L σ can be analogously derived. When one of the terms in the decomposition is a bias, we saw improvements when restricting to permutations where the bias is the first term. As N only ranges between 2 and 4, this linearization generally takes very simple forms. For instance, when N = 2, the contribution assigned to y 1 is DISPLAYFORM3 This linearization was presented in a scalar context where y i ∈ R, but trivially generalizes to the vector setting y i ∈ R d2. It can also be viewed as an approximation to Shapely values, as discussed in BID6 and BID12. We now describe our empirical validation of CD on the task of sentiment analysis. First, we verify that, on the standard problem of word-level importance scores, CD compares favorably to prior work. Then we examine the behavior of CD for word and phrase level importance in situations involving compositionality, showing that CD is able to capture the composition of phrases of differing sentiment. Finally, we show that CD is capable of extracting instances of positive and negative negation. Code for computing CD scores is available online 1. We first describe the process for fitting models which are used to produce interpretations. As the primary intent of this paper is not predictive accuracy, we used standard best practices without much tuning. We implemented all models in Torch using default hyperparameters for weight initializations. All models were optimized using Adam BID4 with the default learning rate of 0.001 using early stopping on the validation set. For the linear model, we used a bag of vectors model, where we sum pre-trained Glove vectors BID9 and add an additional linear layer from the word embedding dimension, 300, to the number of classes, 2. We fine tuned both the word vectors and linear parameters. We will use the two data sets described below to validate our new CD method. We trained an LSTM model on the binary version of the Stanford Sentiment Treebank (SST) BID13 ), a standard NLP benchmark which consists of movie reviews ranging from 2 to 52 words long. In addition to review-level labels, it also provides labels for each phrase in the binarized constituency parse tree. Following the hyperparameter choices in , the word and hidden representations of our LSTM were set to 300 and 168, and word vectors were initialized to pretrained Glove vectors BID9. Our LSTM attains 87.2% accuracy, and we also train a logistic regression model with bag of words features, which attains 83.2% accuracy. Originally introduced in , the Yelp review polarity dataset was obtained from the Yelp Dataset Challenge and has train and test sets of sizes 560,000 and 38,000. The task is binary prediction for whether the review is positive (four or five stars) or negative (one or two stars). The reviews are relatively long, with an average length of 160.1 words. Following the guidelines from , we implement an LSTM model which attains 4.6% error, and an ngram logistic regression model, which attains 5.7% error. For computational reasons, we report interpretation on a random subset of sentences of length at most 40 words. When computing integrated gradient scores, we found that numerical issues produced unusable outputs for roughly 6% of the samples. These reviews are excluded. We compare the interpretations produced by CD against four state of the art baselines: cell decomposition BID8, integrated gradients , leave one out BID5, and gradient times input. We refer the reader to Section 2 for descriptions of these algorithms. For our gradient baseline, we compute the gradient of the output probability with respect to the word embeddings, and report the dot product between the word vector and its gradient. For integrated gradients, producing reasonable values required extended experimentation and communication with the creators regarding the choice of baselines and scaling issues. We ultimately used sequences of periods for our baselines, and rescaled the scores for each review by the standard deviation of the scores for that review, a trick not previously mentioned in the literature. To obtain phrase scores for word-based baselines integrated gradients, cell decomposition, and gradients, we sum the scores of the words contained within the phrase. Before examining the novel, phrase-level dynamics of CD, we first verify that it compares favorably to prior work for the standard use case of producing unigram coefficients. When sufficiently accurate in terms of prediction, logistic regression coefficients are generally treated as a gold standard for interpretability. In particular, when applied to sentiment analysis the ordering of words given by their coefficient value provides a qualitatively sensible measure of importance. Thus, when determining the validity of coefficients extracted from an LSTM, we should expect there to be a meaningful relationship between the CD scores and logistic regression coefficients. In order to evaluate the word-level coefficients extracted by the CD method, we construct scatter plots with each point consisting of a single word in the validation set. The two values plotted correspond to the coefficient from logistic regression and importance score extracted from the LSTM. For a quantitative measure of accuracy, we use pearson correlation coefficient. We report quantitative and qualitative in Appendix 6.1.3. For SST, CD and integrated gradients, with correlations of 0.76 and 0.72, respectively, are substantially better than other methods, with correlations of at most 0.51. On Yelp, the gap is not as big, but CD is still very competitive, having correlation 0.52 with other methods ranging from 0.34 to 0.56. Having verified reasonably strong in this base case, we now proceed to show the benefits of CD. We now show that, for phrases of at most five words, existing methods are unable to recognize subphrases with differing sentiments. For example, consider the phrase "used to be my favorite", which is of negative sentiment. The word "favorite", however, is strongly positive, having a logistic regression coefficient in the 93rd percentile. Nonetheless, existing methods consistently rank "favorite" as being highly negative or neutral. In contrast, as shown in TAB1, CD is able to identify "my favorite" as being strongly positive, and "used to be" as strongly negative. A similar dynamic also occurs with the phrase "not worth the time". The main justification for using LSTMs over simpler models is precisely that they are able to capture these kinds of interactions. Thus, it is important that an interpretation algorithm is able to properly uncover how the interactions are being handled. Using the above as a motivating example, we now show that a similar trend holds throughout the Yelp polarity dataset. In particular, we conduct a search for situations similar to the above, where a strongly positive/negative phrase contains a strongly dissenting subphrase. Phrases are scored using the logistic regression with n-gram features described in Section 4.1, and included if their absolute score is over 1.5. We then examine the distribution of scores for the dissenting subphrases, which are analogous to "favorite".For an effective interpretation algorithm, the distribution of scores for positive and negative dissenting subphrases should be significantly separate, with positive subphrases having positive scores, and vice versa. However, as can be seen in Appendix 6.1.1, for prior methods these two distributions are nearly identical. The CD distributions, on the other hand, are significantly separate, indicating that what we observed anecdotally above holds in a more general setting. We now show that prior methods struggle to identify cases where a sizable portion of a review (between one and two thirds) has polarity different from the LSTM's prediction. For instance, consider the review in Table 2, where the first phrase is clearly positive, but the second phrase causes the review to ultimately be negative. CD is the only method able to accurately capture this dynamic. By leveraging the phrase-level labels provided in SST, we can show that this pattern holds in the general case. In particular, we conduct a search for reviews similar to the above example. The search criteria are whether a review contains a phrase labeled by SST to be of opposing sentiment to the review-level SST label, and is between one and two thirds the length of the review. In Appendix 6.1.2, we show the distribution of the ing positive and negative phrases for different attribution methods. A successful interpretation method would have a sizable gap between these two distributions, with positive phrases having mostly positive scores, and negative phrases mostly negative. However, prior methods struggle to satisfy these criteria. 87% of all positive phrases are labelled as negative by integrated gradients, and cell decompositions BID8 even have the distributions flipped, with negative phrases yielding more positive scores than the positive phrases. CD, on the other hand, provides a very clear difference in distributions. To quantify this separation between positive and negative distributions, we examine a two-sample KolmogorovSmirnov one-sided test statistic, a common test for the difference of distributions with values ranging from 0 to 1. CD produces a score of 0.74, indicating a strong difference between positive and negative distributions, with other methods achieving scores of 0 (cell decomposition), 0.33 (integrated gradients), 0.58 (leave one out) and 0.61 (gradient), indicating weaker distributional differences. Given that gradient and leave one out were the weakest performers in unigram scores, this provides strong evidence for the superiority of CD. It's easy to love Robin Tunney -she's pretty and she can actbut it gets harder and harder to understand her choices. Leave one out BID5 It's easy to love Robin Tunney -she's pretty and she can actbut it gets harder and harder to understand her choices. Cell decomposition BID8 It's easy to love Robin Tunney -she's pretty and she can actbut it gets harder and harder to understand her choices. Integrated gradients It's easy to love Robin Tunney -she's pretty and she can actbut it gets harder and harder to understand her choices. It's easy to love Robin Tunney -she's pretty and she can actbut it gets harder and harder to understand her choices. Legend Very Negative Negative Neutral Positive Very Positive Table 2: Heat maps for portion of review from SST with different attribution techniques. Only CD captures that the first phrase is positive. In order to understand an LSTM's prediction mechanism, it is important to understand not just the contribution of a phrase, but how that contribution is computed. For phrases involving negation, we now demonstrate that we can use CD to empirically show that our LSTM learns a negation mechanism. Using the phrase labels in SST, we search over the training set for instances of negation. In particular, we search for phrases of length less than ten with the first child containing a negation phrase (such as "not" or "lacks", full list provided in Appendix 6.3) in the first two words, and the second child having positive or negative sentiment. Due to noise in the labels, we also included phrases where the entire phrase was non-neutral, and the second child contained a non-neutral phrase. We identify both positive negation, such as "isn't a bad film", and negative negation, such as "isn't very interesting", where the direction is given by the SST-provided label of the phrase. For a given negation phrase, we extract a negation interaction by computing the CD score of the entire phrase and subtracting the CD scores of the phrase being negated and the negation term itself. The ing score can be interpreted as an n-gram feature. Note that, of the methods we compare against, only leave one out is capable of producing such interaction scores. For reference, we also provide the distribution of all interactions for phrases of length less than 5.We present the distribution of extracted scores in FIG1. For CD, we can see that there is a clear distinction between positive and negative negations, and that the negation interactions are centered on the outer edges of the distribution of interactions. Leave one out is able to capture some of the interactions, but has a noticeable overlap between positive and negative negations around zero, indicating a high rate of false negatives. Another benefit of using CDs for interpretation is that, in addition to providing importance scores, it also provides dense embeddings for arbitrary phrases and interactions, in the form of β T discussed in Section 3.2. We anecdotally show that similarity in this embedding space corresponds to semantic similarity in the context of sentiment analysis. In particular, for all words and binary interactions, we compute the average embedding β T produced by CD across the training and validation sets. In Table 3, we show the nearest neighbours using a Table 3: Nearest neighbours for selected unigrams and interactions using CD embeddings cosine similarity metric. The are qualitatively sensible for three different kinds of interactions: positive negation, negative negation and modification, as well as positive and negative words. Note that we for positive and negative words, we chose the positive/negative parts of the negations, in order to emphasize that CD can disentangle this composition. In this paper, we have proposed contextual decomposition (CD), an algorithm for interpreting individual predictions made by LSTMs without modifying the underlying model. In both NLP and general applications of LSTMs, CD produces importance scores for words (single variables in general), phrases (several variables together) and word interactions (variable interactions). Using two sentiment analysis datasets for empirical validation, we first show that for information also produced by prior methods, such as word-level scores, our method compares favorably. More importantly, we then show that CD is capable of identifying phrases of varying sentiment, and extracting meaningful word (or variable) interactions. This movement beyond word-level importance is critical for understanding a model as complex and highly non-linear as LSTMs. 6 APPENDIX Figure 4: Logistic regression coefficients versus coefficients extracted from an LSTM on SST. We include a least squares regression line. Stronger linear relationships in the plots correspond to better interpretation techniques. To search for negations, we used the following list of negation words: not, n't, lacks, nobody, nor, nothing, neither, never, none, nowhere, remotely

We introduce contextual decompositions, an interpretation algorithm for LSTMs capable of extracting word, phrase and interaction-level importance score


Question: Represent the scientific article to find a short TLDR

Question: Most deep learning-based models for speech enhancement have mainly focused on estimating the magnitude of spectrogram while reusing the phase from noisy speech for reconstruction. This is due to the difficulty of estimating the phase of clean speech. To improve speech enhancement performance, we tackle the phase estimation problem in three ways. First, we propose Deep Complex U-Net, an advanced U-Net structured model incorporating well-defined complex-valued building blocks to deal with complex-valued spectrograms. Second, we propose a polar coordinate-wise complex-valued masking method to reflect the distribution of complex ideal ratio masks. Third, we define a novel loss function, weighted source-to-distortion ratio (wSDR) loss, which is designed to directly correlate with a quantitative evaluation measure. Our model was evaluated on a mixture of the Voice Bank corpus and DEMAND database, which has been widely used by many deep learning models for speech enhancement. Ablation experiments were conducted on the mixed dataset showing that all three proposed approaches are empirically valid. Experimental show that the proposed method achieves state-of-the-art performance in all metrics, outperforming previous approaches by a large margin. Speech enhancement is one of the most important and challenging tasks in speech applications where the goal is to separate clean speech from noise when noisy speech is given as an input. As a fundamental component for speech-related systems, the applications of speech enhancement vary from speech recognition front-end modules to hearing aid systems for the hearing-impaired BID36 BID32.Due to recent advances in deep learning, the speech enhancement task has been able to reach high levels in performance through significant improvements. When using audio signals with deep learning models, it has been a common practice to transform a time-domain waveform to a time-frequency (TF) representation (i.e. spectrograms) via short-time-Fourier-transform (STFT). Spectrograms are represented as complex matrices, which are normally decomposed into magnitude and phase components to be used in real-valued networks. In tasks involving audio signal reconstruction, such as speech enhancement, it is ideal to perform correct estimation of both components. Unfortunately, complex-valued phase has been often neglected due to the difficulty of its estimation. This has led to the situation where most approaches focus only on the estimation of a magnitude spectrogram while reusing noisy phase information BID9 BID39 BID7 BID15 BID26. However, reusing phase from noisy speech has clear limitations, particularly under extremely noisy conditions, in other words, when signal-to-noise ratio (SNR) is low. This can be easily verified by simply using the magnitude spectrogram of clean speech with the phase spectrogram of noisy speech to reconstruct clean speech, as illustrated in Fig A popular approach to speech enhancement is to optimize a mask which produces a spectrogram of clean speech when applied to noisy input audio. One of the first mask-based attempts to perform the task by incorporating phase information was the proposal of the phase-sensitive mask (PSM). Since the performance of PSM was limited because of reusing noisy phase, later studies proposed using complex-valued ratio mask (cRM) to directly optimize on complex values BID37 BID3. We found this direction promising for phase estimation because it has been shown that a complex ideal ratio mask (cIRM) is guaranteed to give the best oracle performance out of other ideal masks such as ideal binary masks, ideal ratio masks, or PSMs. Moreover, this approach jointly estimates magnitude and phase, removing the need of separate models. To estimate a complex-valued mask, a natural desire would be to use an architecture which can handle complex-domain operations. Recent work gives a solution to this by providing deep learning building blocks adapted to complex arithmetic BID28.In this paper, we build upon previous studies to design a new complex-valued masking framework, based on a proposed variant of U-Net BID19, named Deep Complex U-Net (DCUnet). In our proposed framework, DCUnet is trained to estimate a complex ratio mask represented in polar coordinates with prior knowledge observable from ideal complex-valued masks. With the complex-valued estimation of clean speech, we can use inverse short-time-Fourier-transform (ISTFT) to convert a spectrogram into a time-domain waveform. Taking this as an advantage, we introduce a novel loss function which directly optimizes source-to-distortion ratio (SDR) BID31, a quantitative evaluation measure widely used in many source separation tasks. Our contributions can be summarized as follows:1. We propose a new neural architecture, Deep Complex U-Net, which combines the advantages of both deep complex networks and U-Net, yielding state-of-the-art performance.2. While pointing out limitations of current masking strategies, we design a new complexvalued masking method based on polar coordinates.3. We propose a new loss function weighted-SDR loss, which directly optimizes a well known quantitative evaluation measure. Phase estimation for audio signal reconstruction has been a recent major interest within the audio source separation community because of its importance and difficulty. While iterative methods such as the Griffin-Lim algorithm and its variants BID8 BID17 aimed to address this problem, neural network-based approaches are recently attracting attention as noniterative alternatives. One major approach is to use an end-to-end model that takes audio as raw waveform inputs without using any explicit time-frequency (TF) representation computed via STFT BID16 BID18 BID23 BID5. Since raw waveforms inherently contain phase information, it is expected to achieve phase estimation naturally. Another method is to estimate magnitude and phase using two separate neural network modules which serially estimate magnitude and phase BID0 BID25. In this framework, the phase estimation module uses noisy phase with predicted magnitude to estimate phase of clean speech. There is also a recent study which proposed to use additional layers with trainable discrete values for phase estimation.A more straightforward method would be to jointly estimate magnitude and phase by using a continuous complex-valued ratio mask (cRM). Previous studies tried this joint estimation approach bounding the range of the cRM BID37 BID3. Despite the advantages of the cRM approach, previously proposed methods had limitations with regard to the loss function and the range of the mask which we will be returning with more details in Section 3 along with our proposed methods to alleviate these issues. As a natural extension to the works above, some studies have also undergone to examine whether complex-valued networks are useful when dealing with intrinsically complex-valued data. In the series of two works, complex-valued networks were shown to help singing voice separation performance with both fully connected neural networks and recurrent neural networks BID12 b). However, the approaches were limited as it ended up only switching the real-valued network into a complex-valued counterpart and leaving the other deep learning building blocks such as weight initialization and normalization technique in a realvalued manner. Also, the works do not show whether the phase was actually well estimated either quantitatively or qualitatively, only ending up showing that there was a performance gain. In this section we will provide details on our approach, starting with our proposed model Deep Complex U-Net, followed by the masking framework based on the model. Finally, we will introduce a new loss function to optimize our model, which takes a critical role for proper phase estimation. Before getting into details, here are some notations used throughout the paper. The input mixture signal x(n) = y(n) + z(n) ∈ R is assumed to be a linear sum of the clean speech signal y(n) ∈ R and noise z(n) ∈ R, where estimated speech is denoted asŷ(n) ∈ R. Each of the corresponding time-frequency (t, f) representations computed by STFT is denoted as DISPLAYFORM0 The ground truth mask cIRM is denoted as M t,f ∈ C and the estimated cRM is denoted asM t,f ∈ C, where The U-Net structure is a well known architecture composed as a convolutional autoencoder with skip-connections, originally proposed for medical imaging in computer vision community BID19. Furthermore, the use of real-valued U-Net has been shown to be also effective in many recent audio source separation tasks such as music source separation BID10 BID23 BID26, and speech enhancement BID16. Deep Complex U-Net (DCUnet) is an extended U-Net, refined specifically to explicitly handle complex domain operations. In this section, we will describe how U-Net is modified using the complex building blocks originally proposed by BID28. DISPLAYFORM1 Complex-valued Building Blocks. Given a complex-valued convolutional filter W = A + iB with real-valued matrices A and B, the complex convolution operation on complex vector h = x + iy with W is done by W * h = (A * x − B * y) + i(B * x + A * y). In practice, complex convolutions can be implemented as two different real-valued convolution operations with shared real-valued convolution filters. Details are illustrated in Appendix A. Activation functions like ReLU were also adapted to the complex domain. In previous work, CReLU, an activation function which applies ReLU on both real and imaginary values, was shown to produce the best out of many suggestions. Details on batch normalization and weight initialization for complex networks can be found in BID28.Modifying U-Net. The proposed Deep Complex U-Net is a refined U-Net architecture applied in STFT-domain. Modifications done to the original U-Net are as follows. Convolutional layers of UNet are all replaced to complex convolutional layers, initialized to meet the Glorot's criteria BID6. Here, the convolution kernels are set to be independent to each other by initializing the weight tensors as unitary matrices for better generalization and fast learning BID2. Complex batch normalization is implemented on every convolutional layer except the last layer of the network. In the encoding stage, max pooling operations are replaced with strided complex convolutional layers to prevent spatial information loss. In the decoding stage, strided complex deconvolutional operations are used to restore the size of input. For the activation function, we modified the previously suggested CReLU into leaky CReLU, where we simply replace ReLU into leaky ReLU BID14, making training more stable. Note that all experiments performed in Section 4 are done with these modifications. As our proposed model can handle complex values, we aim to estimate cRM for speech enhancement. Although it is possible to directly estimate the spectrogram of a clean source signal, it has been shown that better performance can be achieved by applying a weighting mask to the mixture spectrogram BID33. One thing to note is that real-valued ratio masks (RM) only change the scale of the magnitude without changing phase, ing in irreducible errors as illustrated in Appendix D. On the other hand, cRM also perform a rotation on the polar coordinates, allowing to correct phase errors. In other words, the estimated speech spectrogramŶ t,f is computed by multiplying the estimated maskM t,f on the input spectrogram X t,f as follows:Published as a conference paper at ICLR 2019 DISPLAYFORM0 In this state, the real and imaginary values of the estimated cRM is unbounded. Although estimating an unbounded mask makes the problem well-posed (see Appendix D for more information), we can imagine the difficulty of optimizing from an infinite search space compared to a bounded one. Therefore, a few techniques have been tried to bound the range of cRM. For example, Williamson et al. tried to directly optimize a complex mask into a cIRM compressed to a heuristic bound BID37. However, this method was limited since it was only able to succeed in training the model by computing the error between cIRM and the predicted cRM which often leads to a degradation of performance BID33 BID41. More recently, Ephrat et al.proposed a rectangular coordinate-wise cRM made with sigmoid compressions onto each of the real and imaginary parts of the output of the model BID3. After then MSE between clean source Y and estimated sourceŶ was computed in STFT-domain to train the model. However, the proposed masking method has two main problems regarding phase estimation. First, it suffers from the inherent problem of not being able to reflect the distribution of cIRM as shown in FIG2 and Appendix E. Second, this approach in a cRM with a restricted rotation range of 0 • to 90• (only clock-wise), which makes it hard to correct noisy phase. To alleviate these problems, we propose a polar coordinate-wise cRM method that imposes nonlinearity only on the magnitude part. More specifically, we use a hyperbolic tangent non-linearity to bound the range of magnitude part of the cRM be which makes the mask bounded in an unit-circle in complex space. The corresponding phase mask is naturally obtained by dividing the output of the model with the magnitude of it. More formally, let g(·) be our neural network and the output of it be O t,f = g(X t,f). The proposed complex-valued maskM t,f is estimated as follows: DISPLAYFORM1 A summarized illustration of cRM methods is depicted in FIG2. A popular loss function for audio source separation is mean squared error (MSE) between clean source Y and estimated sourceŶ on the STFT-domain. However, it has been reported that optimizing the model with MSE in complex STFT-domain fails in phase estimation due to the randomness in phase structure BID37. As an alternative, it is possible to use a loss function defined in the time-domain instead, as raw waveforms contain inherent phase information. While MSE on waveforms can be an easy solution, we can expect it to be more effective if the loss function is directly correlated with well-known evaluation measures defined in the time-domain. Here, we propose an improved loss function weighted-SDR loss by building upon a previous work which attempts to optimize a standard quality measure, source-to-distortion ratio (SDR) BID30. The original loss function loss V en suggested by Venkataramani et al. is formulated upon the observation from Equation 4, where y is the clean source signal andŷ is the estimated source signal. In practice, the negative reciprocal is optimized as in Equation FORMULA4. DISPLAYFORM0 Although using Equation 5 works as a loss function, there are a few critical flaws in the design. First, the lower bound becomes − y 2, which depends on the value of y causing fluctuation in the loss values when training. Second, when the target y is empty (i.e., y = 0) the loss becomes zero, preventing the model to learn from noisy-only data due to zero gradients. Finally, the loss function is not scale sensitive, meaning that the loss value is the same forŷ and cŷ, where c ∈ R.To resolve these issues, we redesigned the loss function by giving several modifications to Equation 5. First, we made the lower bound of the loss function independent to the source y by restoring back the term y 2 and applying square root as in Equation FORMULA5. This makes the loss function bounded within the range [-1, 1] and also be more phase sensitive, as inverted phase gets penalized as well. DISPLAYFORM1 Expecting to be complementary to source prediction and to propagate errors for noise-only samples, we also added a noise prediction term loss SDR (z,ẑ). To properly balance the contributions of each loss term and solve the scale insensitivity problem, we weighted each term proportional to the energy of each signal. The final form of the suggested weighted-SDR loss is as follows: DISPLAYFORM2 where,ẑ = x −ŷ is estimated noise and α = ||y|| 2 /(||y|| 2 + ||z|| 2) is the energy ratio between clean speech y and noise z. Note that although weighted SDR loss is a time-domain loss function, it can be backpropagated through our framework. Specifically, STFT and ISTFT operations are implemented as 1-D convolution and deconvolution layers consisting of fixed filters initialized with the discrete Fourier transform matrix. The detailed properties of the proposed loss function are in Appendix C. Dataset. For all experiments, we used the same experimental setups as previous works in order to perform direct performance comparison BID16 BID18 BID22 BID5. Noise and clean speech recordings were provided from the Diverse Environments Multichannel Acoustic Noise Database (DEMAND) BID27 and the Voice Bank corpus BID29, respectively, each recorded with sampling rate of 48kHz. Mixed audio inputs used for training were composed by mixing the two datasets with four signalto-noise ratio (SNR) settings (15, 10, 5, and 0 (dB)), using 10 types of noise (2 synthetic + 8 from DEMAND) and 28 speakers from the Voice Bank corpus, creating 40 conditional patterns for each speech sample. The test set inputs were made with four SNR settings different from the training set (17.5, 12.5, 7.5, and 2.5 (dB)), using the remaining 5 noise types from DEMAND and 2 speakers from the Voice Bank corpus. Note that the speaker and noise classes were uniquely selected for the training and test sets. Pre-processing. The original raw waveforms were first downsampled from 48kHz to 16kHz. For the actual model input, complex-valued spectrograms were obtained from the downsampled waveforms via STFT with a 64ms sized Hann window and 16ms hop length. Implementation. All experiments were implemented and fine-tuned with NAVER Smart Machine Learning (NSML) platform BID24 BID11. In this subsection, we compare overall speech enhancement performance of our method with previously proposed algorithms. As a baseline approach, Wiener filtering (Wiener) with a priori noise SNR estimation was used, along with recent deep-learning based models which are briefly described as the following: SEGAN: a time-domain U-Net model optimized with generative adversarial networks. Wavenet: a time-domain non-causal dilated wavenet-based network. MMSE-GAN: a timefrequency masking-based method with modified adversarial training method. Deep Feature Loss: a time-domain dilated convolution network trained with feature loss from a classifier network. BID21 3.23 2.68 2.67 2.22 5.07 SEGAN BID16 3 For comparison, we used the configuration of using a 20-layer Deep Complex U-Net (DCUnet-20) to estimate a tanh bounded cRM, optimized with weighted-SDR loss. As a showcase for the potential of our approach, we also show from a larger DCUnet-20 (Large-DCUnet-20) which has more channels in each layer. Both architectures are specified in detail in Appendix B. Results show that our proposed method outperforms the previous state-of-the-art methods with respect to all metrics by a large margin. Additionally, we can also see that larger models yield better performance. We see the reason to this significant improvement coming from the phase estimation quality of our method, which we plan to investigate in later sections. TAB2 shows the jointly combined on varied masking strategies and loss functions, where three models (DCU-10 (1.4M), DCU-16 (2.3M), and DCU-20 (3.5M)) are investigated to see how architectural differences in the model affect quantitative . In terms of masking strategy, the proposed BDT mask mostly yields better than UBD mask in DCU-10 and DCU-16, implying the importance of limiting optimization space with prior knowledge. However, in the case of DCU-20, UBD mask was able to frequently surpass the performance of BDT mask. Intuitively, this indicates that when the number of parameter gets large enough, the model is able to fit the distribution of data well even when the optimization space is not bounded. In terms of the loss function, almost every shows that optimizing with wSDR loss gives the best . However, we found out that Spc loss often provides better PESQ than wSDR loss for DCU-10 and DCU-16 except DCU-20 case where Spc and wSDR gave similar PESQ . Validation on complex-valued network and mask. In order to show that complex neural networks are effective, we compare evaluation of DCUnet (Cn) and its corresponding real-valued UNet setting with the same parameter size (Rn). For the real-valued network, we tested two settings cRMRn and RMRn to show the effectiveness of phase estimation. The first setting takes a complexvalued spectrogram as an input, estimating a complex ratio mask (cRM) with a tanh bound. The second setting takes a magnitude spectrogram as an input, estimating a magnitude ratio mask (RM) with a sigmoid bound. All models were trained with weighted-SDR loss, where the ground truth phase was given while training RMRn. Additionally, all models were trained on different number of parameters (20-layer (3.5M), 16-layer (2.3M), and 10-layer (1.4M)) to show that the are consistent regardless of model capacity. Detailed network architectures for each model are illustrated in Appendix B. In TAB3, evaluation show that our approach cRMCn makes better than conventional method RMRn for all cases, showing the effectiveness of phase correction. Also, cRMCn gives better than cRMRn, which indicates that using complex-valued networks consistently improve the performance of the network. Note that these are consistent through every evaluation measure and model size. We performed qualitative evaluations by obtaining preference scores between the proposed DCUnet (Large-DCUnet-20) and baseline methods. 15 utterance samples with different noise levels were selected from the test set and used for subjective listening tests. For each noisy sample, all possible six pairs of denoised audio samples from four different algorithms were presented to the participants in a random order, ing in 90 pairwise comparisons to be made by each subject. For each comparison, participants were presented with three audio samples -original noisy speech and two denoised speech samples by two randomly selected algorithms -and instructed to choose either a preferred sample (score 1) or "can't decide" (score 0.5). A total of 30 subjects participated in the listening test, and the are presented in TAB4 and in Table 7.: Scatter plots of estimated cRMs with 9 different mask and loss function configurations for a randomly picked noisy speech signal. Each scatter plot shows the distribution of complex values from an estimated cRM. The leftmost plot is from the cIRM for the given input. We can observe that most real-values are distributed around 0 and 1, while being relatively sparse in between. The configuration that fits the most to this distribution pattern is observed in the red dotted box which is achieved by the combination of our proposed methods (Bounded (tanh) and weighted-SDR). TAB4 shows that DCUnet clearly outperforms the other methods in terms of preference scores in every SNR condition. These differences are statistically significant as confirmed by pairwise one-tailed t-tests. Furthermore, the difference becomes more obvious as the input SNR condition gets worse, which supports our motivation that accurate phase estimation is even more important under harsh noisy conditions. This is further confirmed by in-depth quantitative analysis of the phase distance as described in Section 5 and TAB6. In this section, we aim to provide constructive insights on phase estimation by analyzing how and why our proposed method is effective. We first visualized estimated complex masks with scatter plots in FIG3 for each masking method and loss function configuration from scaling the magnitude of noisy speech and fails to correct the phase of noisy speech with rotations (e.g., (X DISPLAYFORM0 . In order to demonstrate this effect in an alternate perspective, we also plotted estimated waveforms for each loss function in FIG4 . As one can notice from FIG4 To explicitly support these observations, we would need a quantitative measure for phase estimation. Here, we define the phase distance between target spectrogram (A) and estimated spectrogram (B) as the weighted average of angle between corresponding complex TF bins, where each bin is weighted by the magnitude of target speech (A t,f) to emphasize the relative importance of each TF bin. Phase distance is formulated as the following: DISPLAYFORM1 where, ∠(A t,f, B t,f) represents the angle between A t,f and B t,f, having a range of.The phase distance between clean and noisy speech (PhaseDist (C, N) ) and the phase distance between clean and estimated speech (PhaseDist(C, E)) are presented in TAB6. The show that the best phase improvement (Phase Improvement = PhaseDist(C, N) − PhaseDist(C, E)) is obtained with wSDR loss under every SNR condition. Also Spc loss gives the worst , again reinforcing our observation. Analysis between the phase improvement and performance improvement is further discussed in Appendix G. In this paper, we proposed Deep Complex U-Net which combines two models to deal with complexvalued spectrograms for speech enhancement. In doing so, we designed a new complex-valued masking method optimized with a novel loss function, weighted-SDR loss. Through ablation studies, we showed that the proposed approaches are effective for more precise phase estimation, ing in state-of-the-art performance for speech enhancement. Furthermore, we conducted both quantitative and qualitative studies and demonstrated that the proposed method is consistently superior to the previously proposed algorithms. n the near future, we plan to apply our system to various separation tasks such as speaker separation or music source separation. Another important direction is to extend the proposed model to deal with multichannel audio since accurate estimation of phase is even more critical in multichannel environments BID34. Apart from separation, our approach can be generalized to various audio-related tasks such as dereverberation, bandwidth extension or phase estimation networks for text-to-speech systems. Taking advantage of sequence modeling, it may also be interesting to find further extensions with complex-valued LSTMs BID1 BID38. In this section, we address the difference between the real-valued convolution and the complexvalued convolution. Given a complex-valued convolution filter W = A + iB with real-valued matrices A and B, the complex-valued convolution can be interpreted as two different real-valued convolution operations with shared parameters, as illustrated in FIG6 (b). For a fixed number of #Channel product = #Input channel(M) × #Output channel(N), the number of parameters of the complex-valued convolution becomes double of that of a real-valued convolution. Considering this fact, we built the pair of a real-valued network and a complex-valued network with the same number of parameters by reducing #Channel product of complex-valued convolution by half for a fair comparison. The detail of models reflecting this configuration is explained in Appendix B. In this section, we describe three different model architectures (DCUnet-20 (#params: 3.5M), DCUnet-16 (#params: 2.3M), and DCUnet-10 (#params: 1.4M)) each in complex-valued network setting and real-valued network setting in FIG7, 8, 9. Both complex-valued network (C) and realvalued network (R) have the same size of convolution filters with different number of channels to set the parameter equally. The largest model, Large-DCUnet-20, in TAB0 is also described in FIG12. Every convolution operation is followed by batch normalization and an activation function as described in FIG0. For the complex-valued network, the complex-valued version of batch normalization and activation function was used following Deep Complex Networks BID28. Note that in the very last layer of every model the batch normalization and leaky ReLU activation was not used and non-linearity function for mask was applied instead. The real-valued network configuration was not considered in the case of largest model. In this section, we summarize the properties of the proposed weighted-SDR loss. First, we show that the range of weighted-SDR loss is bounded and explain the conditions under which the minimum value is obtained. Next, we explain the gradients in the case of noise-only input. FIG0: Description of encoder and decoder block. F f and F t denote the convolution filter size along the frequency and time axis, respectively. S f and S t denote the stride size of convolution filter along the frequency and time axis, respectively. O C and O R denote the different number of channels in complex-valued network setting and real-valued network setting, respectively. The number of channels of O R is set to be roughly √ 2 times the number of channels of O C so that the number of trainable parameters of real-valued network and complex-valued network becomes approximately the same. Let x denotes noisy speech with T time step, y denotes target source andŷ denotes estimated source. Then, loss wSDR (x, y,ŷ) is defined as follows: DISPLAYFORM0 where, α is the energy ratio between target source and noise, i.e., y 2 /(y 2 + x − y 2).Proposition 1. loss wSDR (x, y,ŷ) is bounded on [-1,1]. Moreover, for fixed y = 0 and x − y = 0, the minimum value -1 can only be attained whenŷ = y, if x = cy for ∀c ∈ R.Proof. Cauchy-Schwarz inequality states that for a ∈ R T and b ∈ R T, − a b ≤ < a, b > ≤ a b. By this inequality, [-1,1] becomes the range of loss wSDR. To attain the minimum value, the equality condition of the Cauchy-Schwarz inequality must be satisfied. This equality condition is equivalent to b = 0 or a = tb, for ∃t ∈ R. Applying the equality condition with the assumption (y = 0, x − y = 0) to Equation 9 leads toŷ = t 1 y and x −ŷ = t 2 (x − y), f or ∃t 1 ∈ R and ∃t 2 ∈ R. By adding these two equations, we can get (1 − t 1)x = (t 1 − t 2)y. By the assumption x = cy, which is generally satisfied for large T, we can conclude t 1 = 1, t 1 = t 2 must be satisfied when the minimum value is attained. The following property of the weighted-SDR loss shows that the network can also learn from noiseonly training data. In experiments, we add small number to denominators of Equation 9. Thus for the case of y = 0, Equation 9 becomes DISPLAYFORM1 Proposition 2. When we parameterizeŷ = g θ (x), the loss wSDR (x, y, g θ (x)) has a non-zero gradient with respect to θ even if the target source y is empty. Proof. We can calculate partial derivatives as follows: DISPLAYFORM2 Thus, the non-zero gradients with respect to θ can be back-propagated. In this section, we illustrate two possible irreducible errors. FIG1 (a) shows the irreducible phase error due to lack of phase estimation. FIG1 (b) shows the irreducible error induced when bounding the range of mask. Not bounding the range of the mask makes the problem well-posed but it may suffer from the wide range of optimization search space because of the lack of prior knowledge on the distribution of cIRM. The scatter plots of cIRM from training set is shown in FIG2. We show four different scatter plots according to their SNR values of mixture (0, 5, 10, and 15 (dB) ). Each scattered point of cIRM, M t,f, is defined as follows: DISPLAYFORM0 The scattered points near origin indicate the TF bins where the value of Y t,f is significantly small compared to X t,f. Therefore, those TF bins can be interpreted as the bins dominated with noise rather than source. On the other hand, the scattered points near indicates the TF bins where the value of Y t,f is almost the same as X t,f. In this case, those TF bins can be interpreted as the bins dominated with source rather than noise. Therefore, as SNR becomes higher, the amount of TF bins dominated with clean source becomes larger compared to the lower SNR cases, and consequently the portion of real part close to 1 becomes larger as in FIG2. In this section, we show a supplementary visualization of phase of estimated speech. Although the raw phase information itself does not show a distinctive pattern, the hidden structure can be revealed with group delay, which is the negative derivative of the phase along frequency axis BID40. With this technique, the phase information can be explicitly shown as in FIG3. FIG3 shows the group delay of clean speech and the corresponding magnitude is shown in FIG3 (a). The two representations shows that the group delay of phase has a similar structure to that of magnitude spectrogram. The estimated phase by our model is shown in in FIG3 (c).While the group delay of noisy speech FIG3 ) does not show a distinctive harmonic pattern, our estimation show the harmonic pattern similar to the group delay of clean speech, as shown in the yellow boxes in FIG3 In this section, to show the limitation of conventional approach (without phase estimation), we emphasize that the phase estimation is important, especially under low SNR condition (harsh condition).We first make an assumption that the estimation of phase information becomes more important when the given mixture has low SNR. Our reasoning behind this assumption is that if the SNR of a given mixture is low, the irreducible phase error is likely to be greater, hence a more room for improvement with phase estimation as illustrated in FIG4. This can also be verified in TAB6 columns PhaseDist(C, N) and Phase Improvement where the values of both columns increase as SNR becomes higher. FIG4: (a) The case where SNR of given mixture is high. In this case the source is likely to be dominant in the mixture. Therefore it is relatively easier to estimate ground truth source with better precision even when the phase is not estimated. (b) The case where SNR of given mixture is low. In this case the source is not dominant in the mixture. Therefore, the irreducible phase error is likely to be higher in low SNR conditions than higher SNR conditions. Under this circumstance, we assume the lack of phase estimation will in a particularly bad system performance. To empirically show the importance of phase estimation, we show correlation between phase improvement and performance difference between the conventional method (without phase estimation) and our proposed method (with phase estimation) in TAB8. The performance difference was calculated by simply subtracting the evaluation of conventional method from the evaluation of our method with phase estimation. For fair comparison, both conventional method (RMRn) and proposed method (cRMCn) were set to have the same number of parameters. Also, both models were trained with weighted-SDR loss. The show that when the SNR is low, both the phase improvement and the performance difference are relatively higher than the from higher SNR conditions. Furthermore, almost all show an incremental increase of phase improvement and performance difference as the SNR decreases, which agrees on our assumption. Therefore we believe that phase estimation is important especially in harsh noisy conditions (low SNR conditions). Table 7: Pairwise preference scores of four models including DCUnet. The scores are obtained by calculating the relative frequency the subjects prefer one method to the other method. Hard/Medium/Easy denote 2.5/7.5/17.5 SNR conditions in dB, respectively. Significance for each statistic is also described (n.s.: not significant, * : p<0.05, * * : p<0.01, * * * : p<0.001).

represent the tldr for finding the corresponding scientific article

This paper proposes a novel complex masking method for speech enhancement along with a loss function for efficient phase estimation.

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent this

Question: All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis. We propose that such a search for order amidst chaos might offer a unifying principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing RL (SMiRL). SMiRL trains an agent with the objective of maximizing the probability of observed states under a model trained on all previously seen states. The ing agents acquire several proactive behaviors to seek and maintain stable states such as balancing and damage avoidance, that are closely tied to the affordances of the environment and its prevailing sources of entropy, such as winds, earthquakes, and other agents. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, and control a humanoid to avoid falls, without any task-specific reward supervision. We further show that SMiRL can be used as an unsupervised pre-training objective that substantially accelerates subsequent reward-driven learning The general struggle for existence of animate beings is not a struggle for raw materials, nor for energy, but a struggle for negative entropy. (Ludwig Boltzmann, 1886) All living organisms carve out environmental niches within which they can maintain relative predictability amidst the ever-increasing entropy around them (Boltzmann, 1886; Schrödinger, 1944; ;). Humans, for example, go to great lengths to shield themselves from surprise -we band together in millions to build cities with homes, supplying water, food, gas, and electricity to control the deterioration of our bodies and living spaces amidst heat and cold, wind and storm. The need to discover and maintain such surprise-free equilibria has driven great resourcefulness and skill in organisms across very diverse natural habitats. Motivated by this, we ask: could the motive of preserving order amidst chaos guide the automatic acquisition of useful behaviors in artificial agents? Our method therefore addresses the unsupervised reinforcement learning problem: how might an agent in an environment acquire complex behaviors and skills with no external supervision? This central problem in artificial intelligence has evoked several candidate solutions, largely focusing on novelty-seeking behaviors (; ; ; ; ;). In simulated worlds, such as video games, novelty-seeking intrinsic motivation can lead to interesting and meaningful behavior. However, we argue that these sterile environments are fundamentally lacking compared to the real world. In the real world, natural forces and other agents offer bountiful novelty. The second law of thermodynamics stipulates ever-increasing entropy, and therefore perpetual novelty, without even requiring any agent intervention. Instead, the challenge in natural environments is homeostasis: discovering behaviors that enable agents to maintain an equilibrium, for example to preserve their bodies, their homes, and avoid predators and hunger. Even novelty seeking behaviors may emerge naturally as a means to maintain homeostasis: an agent that is curious and forages for food in unlikely places might better satisfy its hunger. In natural environments (left), an inactive agent will experience a wide variety of states. By reasoning about future surprise, a SMiRL agent can take actions that temporarily increase surprise but reduce it in the long term. For example, building a house initially in novel states, but once it is built, the house allows the agent to experience a more stable and surprise-free environment. On the right we show an interpretation of the agent interaction loop using SMiRL. When the agent observes a state, it updates it belief p(s) over states. Then, the action policy π(a|s, θ) is conditioned on this belief and maximizes the expected likelihood of the next state under its belief. We formalize allostasis as an objective for reinforcement learning based on surprise minimization (SMiRL). In highly entropic and dynamic environments with undesirable forms of novelty, minimizing surprise (i.e., minimizing novelty) causes agents to naturally seek a stable equilibrium. Natural environments with winds, earthquakes, adversaries, and other disruptions already offer a steady stream of novel stimuli, and an agent that minimizes surprise in these environments will act and explore in order to find the means to maintain a stable equilibrium in the face of these disturbances. SMiRL is simple to describe and implement: it works by maintaining a density p(s) of visited states and training a policy to act such that future states have high likelihood under p(s). This interaction scheme is shown in Figure 1 (right) Across many different environments, with varied disruptive forces, and in agents with diverse embodiments and action spaces, we show that this simple approach induces useful equilibrium-seeking behaviors. We show that SMiRL agents can solve Tetris, avoid fireballs in Doom, and enable a simulated humanoid to balance and locomote, without any explicit task reward. More pragmatically, we show that SMiRL can be used together with a task reward to accelerate standard reinforcement learning in dynamic environments, and can provide a simple mechanism for imitation learning. SMiRL holds promise for a new kind of unsupervised RL method that produces behaviors that are closely tied to the prevailing disruptive forces, adversaries, and other sources of entropy in the environment. Videos of our are available at https://sites.google.com/view/surpriseminimization We propose surprise minimization as a means to operationalize the idea of learning useful behaviors by seeking to preserve order amidst chaos. In complex natural environments with disruptive forces that tend to naturally increase entropy, which we refer to as entropic environments, minimizing surprise over an agent's lifetime requires taking action to reach stable states, and often requires acting continually to maintain homeostasis and avoid surprise. The long term effects of actions on the agent's surprise can be complex and somewhat counterintuitive, especially when we consider that actions not only change the state that the agent is in, but also its beliefs about which states are more likely. The combination of these two processes induce the agent to not only seek states where p(s) is large, but to also visit states so as to alter p(s), in order to receive larger rewards in the future. This "meta" level reasoning can in behaviors where the agent might actually visit new states in order to make them more familiar. An example of this is shown in Figure 1 where in order to avoid the disruptions from the changing weather an agent needs to build a shelter or home to protect itself and decrease its observable surprise. The SMiRL formulation relies on disruptive forces in the environment to avoid collapse to degenerate solutions, such as staying in a single state s 0. Fortunately, natural environments typically offer no shortage of such disruption. To instantiate SMiRL, we design a reinforcement learning agent with a reward proportional to how familiar its current state is based on the history of states it has experienced during its "life," which corresponds to a single episode. Formally, we assume a fully-observed controlled Markov process (CMP), though extensions to partially observed settings can also be developed. We use s t to denote the state at time t, and a t to denote the agent's action, ρ(s 0) to denote the initial state distribution, and T (s t+1 |s t, a t) to denote the transition dynamics. The agent has access to a dataset D t = {s 1, . . ., s t} of all states experienced so far. By fitting a generative model p θt (s) with parameters θ t to this dataset, the agent obtains an estimator that can be used to evaluate the negative surprise reward, given by We denote the fitting process as θ t = U(D t). The goal of a SMiRL agent is to maximize the sum t log p θt (s t+1). Since the agent's actions affect the future D t and thus the future θ t's, the optimal policy does not simply visit states that have a high p θt (s) now, but rather those states that will change p θt (s) such that it provides high likelihood to the states that it sees in the future. We now present a practical reinforcement learning algorithm for surprise minimization. Recall that a critical component of SMiRL is reasoning about the effect of actions on future states that will be added to D, and their effect on future density estimates -e.g., to understand that visiting a state that is currently unfamiliar and staying there will make that state familiar, and therefore lead to higher rewards in the long run. This means that the agent must reason not only about the unknown MDP dynamics, but also the dynamics of the density model p θ (s) trained on D. In our algorithm, we accomplish this via an episodic training procedure, where the agent is trained over many episodes and D is reset at the beginning of each episode to simulate a new lifetime. Through this procedure, SMiRL learns the parameters φ of the agent's policy π φ for a fixed horizon. To learn this the policy must be conditioned on some sufficient statistic of D t, since the reward r t is a function of D t. Having trained parameterized generative models p θt as above on all states seen so far, we condition π on θ t and |D t |. This implies an assumption that θ t and |D t | represent the sufficient statistics necessary to summarize the contents of the dataset for the policy, and contain all information required to reason about how p θ will evolve in the future. Of course, we could also use any other summary statistic, or even read in the entirety of D t using a recurrent model. In the next section, we also describe a modification that allows us to utilize a deep density model without conditioning π on a high-dimensional parameter vector. Algorithm 1 provides the pseudocode. SMiRL can be used with any reinforcement learning algorithm, which we denote RL in the pseudocode. As is standard in reinforcement learning, we alternate between sampling episodes from the policy (lines 6-12) and updating the policy parameters (line 13). The details of the updates are left to the specific RL algorithm, which may be on or off-policy. During each episode, as shown in line 11, D 0 is initialized with the first state and grows as each state visited by the agent is added to the dataset. The parameters θ t of the density model are fit to D t at each timestep to both be passed to the policy and define the reward function. At the end of the episode, D T is discarded and the new D 0 is initialized. While SMiRL may in principle be used with any choice of model class for the generative model p θ (s), this choice must be carefully made in practice. As we show in our experiments, relatively simple distribution classes, such as products of independent marginals, suffice to run SMiRL in simple environments with low-dimensional state spaces. However, it may be desirable in more complex environments to use more sophisticated density estimators, especially when learning directly from high-dimensional observations such as images. In particular, we propose to use variational autoencoders (VAEs) to learn a non-linear compressed state representation and facilitate estimation of p θ (s) for SMiRL. A VAE is trained using the standard loss to reconstruct states s after encoding them into a low-dimensional normal distribution q ω (z|s) through the encoder q with parameters ω. A decoder p ψ (s|z,) with parameters ψ computes s from the encoder output z. During this training process, a KL divergence loss between the prior p(z) and q ω (z|s) is used to keep this distribution near the standard normal distribution. We described a VAE-based approach for estimating the SMiRL surprise reward. In our implementation, the VAE is trained online, with VAE updates interleaved with RL updates. Training a VAE requires more data than the simpler density models that can easily be fit to data from individual episodes. We propose to overcome this by not resetting the VAE parameters between training episodes. Instead, we train the VAE across episodes. Instead of passing all VAE parameters to the SMiRL policy, we track a separate episode-specific distribution p θt (z), distinct from the VAE prior, over the course of each episode. p θt (z) replaces p θt (s) in the SMiRL algorithm and is fit to only that episode's state history. We represent p θt (z) as a vector of independent normal distributions, and fit it to the VAE encoder outputs. This replaces the density estimate in line 10 of Algorithm 1. Specifically, the corresponding update U(D t) is performed as follows: Training the VAE online, over all previously seen data, deviates from the recipe in the previous section, where the density model was only updated within an episode. However, this does provide for a much richer state density model, and the within-episode updates to estimate p θt (z) still provide our method with meaningful surprise-seeking behavior. As we show in our experiments, this can improve the performance of SMiRL in practice. We evaluate SMiRL on a range of environments, from video game domains to simulated robotic control scenarios. These are rich, dynamic environments -the world evolves automatically even without agent intervention due to the presence of disruptive forces and adversaries. Note that SMiRL relies on such disruptions to produce meaningful emergent behavior, since mere inaction would otherwise suffice to achieve homeostasis. However, as we have argued above, such disruptions are also an important property of most real world environments. Current RL benchmarks neglect this, focusing largely on unrealistically sterile environments where the agent alone drives change . Therefore, our choices of environments, discussed below, are not solely motivated by suitability to SMiRL; rather, we aim to evaluate unsupervised RL approaches, ours as well as others, in these more dynamic environments. Tetris. The classic game of Tetris offers a naturally entropic environment -the world evolves according to its own rules and dynamics even in the absence of coordinated behavior of the agent, piling up pieces and filling up the board. It therefore requires active intervention to maintain homeostasis. We consider a 4 × 10 Tetris board with tromino shapes (composed of 3 squares), as shown in Figure 2a. The observation is a binary image of the current board with one pixel per square, as well as an indicator for the type of shape that will appear next. Each action denotes one of the 4 columns in which to drop the shape and one of 4 shape orientations. For evaluation, we measure how many rows the agent clears, as well as how many times the agent dies in the game by allowing the blocks to reach the top of the board, within the max episode length of 100. Since the observation is a binary image, we model p(s) as independent Bernoulli. See Appendix A for details. VizDoom. We consider two VizDoom environments from: TakeCover and DefendTheLine. TakeCover provides a dynamically evolving world, with enemies that appear over time and throw fireballs aimed at the player . The observation space consists of the 4 previous grayscale first-person image observations, and the action space consists of moving left or right. We evaluate the agent based on how many times it is hit by fireballs, which we term the "damage" taken by the agent. Images from the TakeCover environment are shown in Fig 2b and Fig??. In DefendTheLine, additional enemies can move towards the player, and the player can shoot the enemies. The agent starts with limited ammunition. This environment provides a "survival" reward function (r = 1 for each timestep alive) and performance is measured by how long the agent survives in the environment. For both environments, we model p(s) as independent Gaussian over the pixels. See Appendix A for details. miniGrid Is a navigation task where the agent has a partial observation of the environment shown by the lighter gray area around the red agent in Figure 2d. The agent needs to navigate down the hallways to escape the enemy agents (blue) to reach the safe room on the right the enemies can not enter, through a randomly placed door. Simulated Humanoid robots. In the last set of environments, a simulated planar Humanoid robot is placed in situations where it is in danger of falling. The action consists of the PD targets for each of the joints. The state space comprises the rotation of each joint and the linear velocity of each link. We evaluate several versions of this task, which are shown in Figure 2. The Cliff tasks starts the agent at the edge of a cliff, in a random pose and with a forward velocity of 1 m/s. Falling off the cliff leads to highly irregular and unpredictable configurations, so a surprise minimizing agent will want to learn to stay on the cliff. In the Treadmill environment, the robot starts on a platform that is moving at 1 m/s backwards; an agent will be carried backwards unless it learns some locomotion. The Pedestal environment is designed to show that SMiRL can learn a more active balancing policy. In this environment the agent starts out on a thin pedestal and random forces are applied to the robots links and boxes of random size are thrown at the agent. The Walk domain is used to evaluate the use of the SMiRL reward as a form of "stability reward" that assists the agent in learning how to walk while reducing the number of falls. This is done by initializing p(s) from example walking data and adding this to the task reward, as discussed in Section 4.2. The task reward in Walk is r walk = exp((v d * v d) * −1.5), where v d is the difference between the x velocity and the desired velocity of 1 m/s. In these environments, we measure performance as the proportion of episodes with a fall. A state is classified as a fall if either the agent's links, except for the feet, are touching the ground, or if the agent is −5 meters or more below the level of the platform or cliff. Since the state is continuous, we model p(s) as independent Gaussian; see Appendix A for details. Our experiments aim to answer the following questions: Can SMiRL learn meaningful and complex emergent behaviors in the environments described in Section 3? First, we evaluate SMiRL on the Tetris, VizDoom, Cliff, and Treadmill tasks, studying its ability to generate purposeful coordinated behaviors after training using only the surprise minimizing objective, in order to answer question. The SMiRL agent demonstrates meaningful emergent behaviors in each of these domains. In the Tetris environment, the agent is able to learn proactive behaviors to eliminate rows and properly play the game. The agent also learns emergent game playing behaviour in the VizDoom environment, acquiring an effective policy for dodging the fireballs thrown by the enemies. In both of these environments, stochastic and chaotic events force the SMiRL agent to take a coordinated course of action to avoid unusual states, such as full Tetris boards or fireball explosions. In the Cliff environment, the agent learns a policy that greatly reduces the probability of falling off of the cliff by bracing against the ground and stabilize itself at the edge, as shown in Figure 2e. In the Treadmill environment, SMiRL learns a more complex locomotion behavior, jumping forward to increase the time it stays on the treadmill, as shown in Figure 2f. A quantitative measurement of the reduction in falls is shown in Figure 4. We also study question in the TakeCover, Cliff, Treadmill and Pedestal environments, training a VAE model and estimating surprise in the latent space of the VAE. In most of these environments, the representation learned by the VAE leads to faster acquisition of the emergent behaviors in Take Comparison to intrinsic motivation. Figure 3 shows plots of the environment-specific rewards over time on Tetris, TakeCover, and the Humanoid domains Figure 4. In order to compare SMiRL to more standard intrinsic motivation methods, which seek out states that maximize surprise or novelty, we also evaluated ICM and RND (b). We also plot an oracle agent that directly optimizes the task reward. On Tetris, after training for 2000 epochs, SMiRL achieves near perfect play, on par with the oracle reward optimizing agent, with no deaths, as shown in Figure 3 (left, middle). ICM seeks novelty by creating more and more distinct patterns of blocks rather than clearing them, leading to deteriorating game scores over time. On TakeCover, SMiRL effectively learns to dodge fireballs thrown by the adversaries, as shown in 3 (right). Novelty-seeking ICM once again yields deteriorating rewards over time due to the method seeking novel events that correspond to damage. The baseline comparisons for the Cliff and Treadmill environments have a similar outcome. The novelty seeking behaviour of ICM causes it to learn a type of irregular behaviour that causes the agent to jump off the Cliff and roll around on the Treadmill, maximizing the variety (and quantity) of falls Figure 4. SMiRL and curiosity are not mutually exclusive. We show that these intrinsic reward functions can be combined to achieve better on the Treadmill environment Figure 4 (right). The combination of methods leads to increased initial learning speed and producing a walking-type gait on that task. To illustrate SMiRL's desire to explore we evaluate over an environment where the agent needs to produce long term planning behaviour. This environment is shown in Figure 2d, where the agent needs to navigate its way through the hallways, avoiding enemies, to reach a safe room through a randomly placed door. We found that SMiRL is able to solve this task. Results from these examples are shown on the accompanying website. While the central focus of this paper is the emergent behaviors that can be obtained via SMiRL, in this section we study more pragmatic applications. We show that SMiRL can be used for joint training to accelerate reward-driven learning of tasks, and also illustrate how SMiRL can be used to produce a rudimentary form of imitation learning. Imitation. We can easily adapt SMiRL to perform imitation by initializing the buffer D 0 with states from expert demonstrations, or even individual desired outcome states. To study this application of SMiRL, we initialize the buffer D 0 in Tetris with user-specified desired board states. An illustration of the Tetris imitation task is presented in Figure 6, showing imitation of a box pattern (top) and a checkerboard pattern (bottom), with the leftmost frame showing the user-specified example, and the other frames showing actual states reached by the SMiRL agent. While a number of prior works have studied imitation without example actions (; a; ; b; ; Lee et al.), this capability emerges automatically in SMiRL, without any further modification to the algorithm. SMiRL as a stability reward. In this next experiment, we study how SMiRL can accelerate acquisition of reward-driven behavior in environments that present a large number of possible actions leading to diverse but undesirable states. Such settings are common in real life: a car can crash in many different ways, a robot can drop a glass on the ground causing it to break in many ways, etc. While this is of course not the case for all tasks, many real-world tasks do require the agent to stabilize itself in a specific and relatively narrow set of conditions. Incorporating SMiRL into the learning objective in such settings can accelerate learning, and potentially improve safety during training, as the agent automatically learns to avoid anything that is unfamiliar. We study this application of SMiRL in the DefendTheLine task and the Walk task. In both cases, we use SMiRL to augment the task reward, such that the full reward is given by r combined (s) = r task (s) + αr SMiRL (s), where α is chosen to put the two reward terms at a similar magnitude. In the Walk task, illustrated in Figure 2g, p θ (s) is additionally initialized with 8 example walking trajectories (256 timesteps each), similarly to the imitation setting, to study how well SMiRL can incorporate prior knowledge into the stability reward (Reward + SMiRL (ours). We include another version that is not initialized with expert data (Reward + SMiRL (no-expert). We measure the number of falls during training, with and without the SMiRL reward term. The in Figure 5b show that adding the SMiRL reward in significantly fewer falls during training, and less when using expert data while learning to walk well, indicating that SMiRL stabilizes the agent more quickly than the task reward alone. In the DefendTheLine task, shown in Figure 2c, we compare the performance of SMiRL as a joint training objective to the more traditional novelty-driven bonus provided by ICM and RND (b). Novelty-driven bonuses are often used to accelerate learning in domains that present an exploration challenge. However, as shown in the in Figure 5a, the SMiRL reward, even without demonstration data, provides for substantially faster learning on this task than novelty-seeking intrinsic motivation. These suggest that SMiRL can be a viable method for accelerating learning and reducing the amount of unsafe behavior (e.g., falling) in dynamic environments. Prior works have sought to learn intelligent behaviors through reinforcement learning with respect to a provided reward function, such as the score of a video game or a hand-defined cost function . Such rewards are often scarce or difficult to provide in practical real world settings, motivating approaches for reward-free learning such as empowerment or intrinsic motivation (; ;). Intrinsic motivation has typically focused on encouraging novelty-seeking behaviors by maximizing model uncertainty (; ; ;), by maximizing model prediction error or improvement , through state visitation counts , via surprise maximization (; ;), and through other novelty-based reward bonuses (; a;). We do the opposite. Inspired by the free energy principle (;, we instead incentivize an agent to minimize surprise and study the ing behaviors in dynamic, entropy-increasing environments. In such environments, which we believe are more reflective of the real-world, we find that prior noveltyseeking environments perform poorly. Prior works have also studied how competitive self-play and competitive, multi-agent environments can lead to complex behaviors with minimal reward information (; ; ;). Like these works, we also consider how complex behaviors can emerge in resource constrained environments. However, our approach can also be applied in non-competitive environments. We presented an unsupervised reinforcement learning method based on minimization of surprise. We show that surprise minimization can be used to learn a variety of behaviors that maintain "homeostasis," putting the agent into stable and sustainable limit cycles in its environment. Across a range of tasks, these stable limit cycles correspond to useful, semantically meaningful, and complex behaviors: clearing rows in Tetris, avoiding fireballs in VizDoom, and learning to balance and hop forward with a bipedal robot. The key insight utilized by our method is that, in contrast to simple simulated domains, realistic environments exhibit dynamic phenomena that gradually increase entropy over time. An agent that resists this growth in entropy must take active and coordinated actions, thus learning increasingly complex behaviors. This stands in stark contrast to commonly proposed intrinsic exploration methods based on novelty, which instead seek to visit novel states and increase entropy. Besides fully unsupervised reinforcement learning, where we show that our method can give rise to intelligent and complex policies, we also illustrate several more pragmatic applications of our approach. We show that surprise minimization can provide a general-purpose risk aversion reward that, when combined with task rewards, can improve learning in environments where avoiding catastrophic (and surprising) outcomes is desirable. We also show that SMiRL can be adapted to perform a rudimentary form of imitation. Our investigation of surprise minimization suggests a number of directions for future work. The particular behavior of a surprise minimizing agent is strongly influenced by the particular choice of state representation: by including or excluding particular observation modalities, the agent will be more or less surprised. where s is a single state, θ i is the sample mean calculated from D t indicating the proportion of datapoints where location i has been occupied by a block, and s i is a binary variable indicating the presence of a block at location i. If the blocks stack to the top, the game board resets, but the episode continues and the dataset D t continues to accumulate states. SMiRL on VizDoom and Humanoid. In these environments the observations placed in the buffer are downsampled 10 × 13 single-frame observations for VizDoom environments and the full state for the Humanoid environments. We model p(s) as an independent Gaussian distribution for each dimension in the observation. Then, the SMiRL reward can be computed as: where s is a single state, µ i and σ i are calculated as the sample mean and standard deviation from D t and s i is the i th observation feature of s. We emphasize that the RL algorithm in SMiRL is provided with a standard stationary MDP (except in the VAE setting, more on that below), where the state is simply augmented with the parameters of the belief over states θ and the timestep t. We emphasize that this MDP is indeed Markovian, and therefore it is reasonable to expect any convergent RL algorithm to converge to a near-optimal solution. Consider the augmented state transition p(s t+1, θ t+1, t + 1|s t, a t, θ t, t). This transition model does not change over time because the updates to θ are deterministic when given s t and t. The reward function R(s t, θ t, t) is also stationary: it is in fact deterministic given s t and θ t. Because SMiRL uses RL in an MDP, we benefit from the same convergence properties as other RL methods. However, the version of SMiRL that uses a representation learned from a VAE is not Markovian because the VAE parameters are not added to the state, and thus the reward function changes over time.. We find that this does not hurt , and note that many intrinsic reward methods such as ICM and RND also lack stationary reward functions. This process is described in Algorithm 1. We do not use entropic to mean that state transition probabilities change over time. Rather, it means that for any state in the environment, random disruptive perturbations may be applied to the state. In such settings, SMiRL seeks to visit state distributions p(s) that are easy to preserve. VAE on-line training When using a VAE to model the surprise of new states, we evaluate the probability of the latent representations z, as described in Section 2.3. The VAE is trained at the end of each episode on all data seen so far across all episodes. This means that the encoder q ω (z|bs) is changing over the course of the SM iRL algorithm, which could lead to difficulty learning a good policy. In practice, the rich representations learned by the VAE help policy learning overall. Training parameters. For the discrete action environment (Tetris and VizDoom), the RL algorithm used is deep Q-learning with a target Q network. For the Humanoid domains, we use TRPO . For Tetris and the Humanoid domains, the policies are parameterized by fully connected neural networks, while VizDoom uses a convolutional network. The encoders and decoders of the VAEs used for VizDoom and Humanoid experiments are implemented as fully connected networks over the same buffer observations as above. The coefficient for the KL-divergence term in the VAE loss was 0.1 and 1.0 for the VizDoom and Humanoid experiments, respectively.

Represent the short summary for finding the corresponding scientific article (long):

Learning emergent behavior by minimizing Bayesian surprise with RL in natural environments with entropy.

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.