Question: Represent the scientific article to find a short TLDR

Question: Our work addresses two important issues with recurrent neural networks: they are over-parameterized, and the recurrent weight matrix is ill-conditioned. The former increases the sample complexity of learning and the training time. The latter causes the vanishing and exploding gradient problem. We present a flexible recurrent neural network model called Kronecker Recurrent Units (KRU). KRU achieves parameter efficiency in RNNs through a Kronecker factored recurrent matrix. It overcomes the ill-conditioning of the recurrent matrix by enforcing soft unitary constraints on the factors. Thanks to the small dimensionality of the factors, maintaining these constraints is computationally efficient. Our experimental on seven standard data-sets reveal that KRU can reduce the number of parameters by three orders of magnitude in the recurrent weight matrix compared to the existing recurrent models, without trading the statistical performance. These in particular show that while there are advantages in having a high dimensional recurrent space, the capacity of the recurrent part of the model can be dramatically reduced. Deep neural networks have defined the state-of-the-art in a wide range of problems in computer vision, speech analysis, and natural language processing BID28 BID36. However, these models suffer from two key issues. They are over-parametrized; thus it takes a very long time for training and inference. Learning deep models is difficult because of the poor conditioning of the matrices that parameterize the model. These difficulties are especially relevant to recurrent neural networks. Indeed, the number of distinct parameters in RNNs grows as the square of the size of the hidden state conversely to convolutional networks which enjoy weight sharing. Moreover, poor conditioning of the recurrent matrices in the gradients to explode or vanish exponentially fast along the time horizon. This problem prevents RNN from capturing long-term dependencies BID22 BID5.There exists an extensive body of literature addressing over-parametrization in neural networks. BID31 first studied the problem and proposed to remove unimportant weights in neural networks by exploiting the second order information. Several techniques which followed include low-rank decomposition BID13, training a small network on the soft-targets predicted by a big pre-trained network BID2, low bit precision training BID12, hashing BID8, etc. A notable exception is the deep fried convnets BID44 which explicitly parameterizes the fully connected layers in a convnet with a computationally cheap and parameter-efficient structured linear operator, the Fastfood transform BID29. These techniques are primarily aimed at feed-forward fully connected networks and very few studies have focused on the particular case of recurrent networks BID1.The problem of vanishing and exploding gradients has also received significant attention. BID23 proposed an effective gating mechanism in their seminal work on LSTMs. Later, this technique was adopted by other models such as the Gated Recurrent Units (GRU) BID10 and the Highway networks BID39 for recurrent and feed-forward neural networks respectively. Other popular strategies include gradient clipping BID37, and orthogonal initialization of the recurrent weights. More recently BID1 proposed to use a unitary recurrent weight matrix. The use of norm preserving unitary maps prevent the gradients from exploding or vanishing, and thus help to capture long-term dependencies. The ing model called unitary RNN (uRNN) is computationally efficient since it only explores a small subset of general unitary matrices. Unfortunately, since uRNNs can only span a reduced subset of unitary matrices their expressive power is limited BID42. We denote this restricted capacity unitary RNN as RC uRNN. Full capacity unitary RNN (FC uRNN) BID42 proposed to overcome this issue by parameterizing the recurrent matrix with a full dimensional unitary matrix, hence sacrificing computational efficiency. Indeed, FC uRNN requires a computationally expensive projection step which takes O(N 3) time (N being the size of the hidden state) at each step of the stochastic optimization to maintain the unitary constraint on the recurrent matrix. BID35 in their orthogonal RNN (oRNN) avoided the expensive projection step in FC uRNN by parametrizing the orthogonal matrices using Householder reflection vectors, it allows a fine-grained control over the number of parameters by choosing the number of Householder reflection vectors. When the number of Householder reflection vector approaches N this parametrization spans the full reflection set, which is one of the disconnected subset of the full orthogonal set. BID25 also presented a way of parametrizing unitary matrices which allows fine-grained control on the number of parameters. This work called as Efficient Unitary RNN (EURNN), exploits the continuity of unitary set to have a tunable parametrization ranging from a subset to the full unitary set. Although the idea of parametrizing recurrent weight matrices with strict unitary linear operator is appealing, it suffers from several issues: Strict unitary constraints severely restrict the search space of the model, thus making the learning process unstable. Strict unitary constraints make forgetting irrelevant information difficult. While this may not be an issue for problems with non-vanishing long term influence, it causes failure when dealing with real world problems that have vanishing long term influence 4.7. BID20 have previously pointed out that the good performance of strict unitary models on certain synthetic problems is because it exploits the biases in these data-sets which favors a unitary recurrent map and these models may not generalize well to real world data-sets. More recently BID41 have also studied this problem of unitary RNNs and the authors found out that relaxing the strict unitary constraint on the recurrent matrix to a soft unitary constraint improved the convergence speed as well as the generalization performance. Our motivation is to address the problems of existing recurrent networks mentioned above. We present a new model called Kronecker Recurrent Units (KRU). At the heart of KRU is the use of Kronecker factored recurrent matrix which provide an elegant way to adjust the number of parameters to the problem at hand. This factorization allows us to finely modulate the number of parameters required to encode N × N matrices, from O(log(N)) when using factors of size 2 × 2, to O(N 2) parameters when using a single factor of the size of the matrix itself. We tackle the vanishing and exploding gradient problem through a soft unitary constraint BID26 BID20 BID11 BID41. Thanks to the properties of Kronecker matrices BID40, this constraint can be enforced efficiently. Please note that KRU can readily be plugged into vanilla real space RNN, LSTM and other variants in place of standard recurrent matrices. However in case of LSTMs we do not need to explicitly enforce the approximate orthogonality constraints as the gating mechanism is designed to prevent vanishing and exploding gradients. Our experimental on seven standard data-sets reveal that KRU and KRU variants of real space RNN and LSTM can reduce the number of parameters drastically (hence the training and inference time) without trading the statistical performance. Our core contribution in this work is a flexible, parameter efficient and expressive recurrent neural network model which is robust to vanishing and exploding gradient problem. The paper is organized as follows, in section 2 we restate the formalism of RNN and detail the core motivations for KRU. In section 3 we present the Kronecker recurrent units (KRU). We present our experimental findings in section 4 and section 5 concludes our work. DISPLAYFORM0 Hidden and output bias σ, L(ŷ, y) Point-wise non-linear activation function and the loss function 2 RECURRENT NEURAL NETWORK FORMALISM TAB0 summarizes some notations that we use in the paper. We consider the field to be complex rather than real numbers. We will motivate the choice of complex numbers later in this section. Consider a standard recurrent neural network BID14. Given a sequence of T input vectors: x 0, x 1,..., x T −1, at a time step t RNN performs the following: DISPLAYFORM0 DISPLAYFORM1 whereŷ t is the predicted value at time step t. The total number of parameters in a RNN is c(DN + N 2 + N + M + M N), where c is 1 for real and 2 for complex parametrization. As we can see, the number of parameters grows quadratically with the hidden dimension, i.e., O(N 2). We show in the experiments that this quadratic growth is an over parametrization for many real world problems. Moreover, it has a direct impact on the computational efficiency of RNNs because the evaluation of Wh t−1 takes O(N 2) time and it recursively depends on previous hidden states. However, other components Ux t and Vh t can usually be computed efficiently by a single matrix-matrix multiplication for each of the components. That is, we can perform U[x 0, . . ., x T] and V[h 0, . . ., h T −1], this is efficient using modern BLAS libraries. So to summarize, if we can control the number of parameters in the recurrent matrix W, then we can control the computational efficiency. The vanishing and exploding gradient problem refers to the decay or growth of the partial derivative of the loss L with respect to the hidden state h t i.e. ∂L ∂ht as the number of time steps T grows BID1. By the application of the chain rule, the following can be shown BID1: DISPLAYFORM0 From Equation 3, it is clear that if the absolute value of the eigenvalues of W deviates from 1 then ∂L ∂ht may explode or vanish exponentially fast with respect to T − t. So a strategy to prevent vanishing and exploding gradient is to control the spectrum of W. Although BID1 and BID42 use complex valued networks with unitary constraints on the recurrent matrix, the motivations for such models are not clear. We give a simple but compelling reason for complex-valued recurrent networks. The absolute value of the determinant of a unitary matrix is 1. Hence in the real space, the set of all unitary (orthogonal) matrices have a determinant of 1 or −1, i.e., the set of all rotations and reflections respectively. Since the determinant is a continuous function, the unitary set in real space is disconnected. Consequently, with the real-valued networks we cannot span the full unitary set using the standard continuous optimization procedures. On the contrary, the unitary set is connected in the complex space as its determinants are the points on the unit circle and we do not have this issue. As we mentioned in the introduction BID25 uses this continuity of unitary space to have a tunable continuous parametrization ranging from subspace to full unitary space. Any continuous parametrization in real space can only span a subset of the full orthogonal set. For example, the Householder parametrization BID35 suffers from this issue. We consider parameterizing the recurrent matrix W as a Kronecker product of F matrices DISPLAYFORM0 Where each W f ∈ C P f ×Q f and DISPLAYFORM1 To illustrate the Kronecker product of matrices, let us consider the simple case when ∀ f {P f = Q f = 2}. This implies F = log 2 N. And W is recursevly defined as follows: DISPLAYFORM2 DISPLAYFORM3 When ∀ f {p f = q f = 2} the number of parameters is 8 log 2 N and the time complexity of hidden state computation is O(N log 2 N). When ∀ f {p f = q f = N} then F = 1 and we will recover standard complex valued recurrent neural network. We can span every Kronecker representations in between by choosing the number of factors and the size of each factor. In other words, the number of Kronecker factors and the size of each factor give us fine-grained control over the number of parameters and hence over the computational efficiency. This strategy allows us to design models with the appropriate trade-off between computational budget and statistical performance. All the existing models lack this flexibility. The idea of using Kronecker factorization for approximating Fisher matrix in the context of natutal gradient methods have recently recieved much attention. The algorithm was originally presented in BID33 and was later extended to convolutional layers, distributed second order optimization BID3 and for deep reinforcement learning BID43. However Kronecker matrices have not been well explored as learnable parameters except BID45 ) used it's spectral property for fast orthogonal projection and BID46 used it as a layer in convolutional neural networks. Poor conditioning in vanishing or exploding gradients. Unfortunately, the standard solution which consists of optimization on the strict unitary set suffers from the retention of noise over time. Indeed, the small eigenvalues of the recurrent matrix can represent a truly vanishing long-term influence on the particular problem and in that sense, there can be good or bad vanishing gradients. Consequently, enforcing strict unitary constraint (forcing the network to never forget) can be a bad strategy. A simple solution to get the best of both worlds is to enforce unitary constraint approximately by using the following regularization: DISPLAYFORM0 Please note that these constraints are enforced on each factor of the Kronecker factored recurrent matrix. This procedure is computationally very efficient since the size of each factor is typically small. It suffices to do so because if each of the Kronecker factors {W 0, . . ., W F −1} are unitary then the full matrix W is unitary BID40 and if each of the factors are approximately unitary then the full matrix is approximately unitary. We apply soft unitary constraints as a regularizer whose strength is cross-validated on the validation set. This type of regularizer has recently been exploited for real-valued models. BID11 showed that enforcing approximate orthogonality constraint on the weight matrices make the network robust to adversarial samples as well as improve the learning speed. In metric learning BID26 have shown that it better conditions the projection matrix thereby improving the robustness of stochastic gradient over a wide range of step sizes as well asthe generalization performance. BID20 and BID41 have also used this soft unitary contraints on standard RNN after identifying the problems with the strict unitary RNN models. However the computational complexity of naively applying this soft constraint is O(N 3). This is prohibitive for RNNs with large hidden state unless one considers a Kronecker factorization. Existing deep learning libraries such as Theano BID6, Tensorflow BID0 and Pytorch BID38 do not support fast primitives for Kronecker products with arbitrary number of factors. So we wrote custom CUDA kernels for Kronecker forward and backward operations. All our models are implemented in C++. We will release our library to reproduce all the which we report in this paper. We use tanh as activation function for RNN, LSTM and our model KRU-LSTM. Whereas RC uRNN, FC uRNN and KRU uses complex rectified linear units BID1. Copy memory problem BID23 tests the model's ability to recall a sequence after a long time gap. In this problem each sequence is of length T + 20 and each element in the sequence come from 10 classes {0, . . ., 9}. The first 10 elements are sampled uniformly with replacement from {1, . . ., 8}. The next T − 1 elements are filled with 0, the'blank' class followed by 9, the'delimiter' and the remaining 10 elements are'blank' category. The goal of the model is to output a sequence of T + 10 blank categories followed by the 10 element sequence from the beginning of the input sequence. The expected average cross entropy for a memory-less strategy is FORMULA3, we choose the training and test set size to be 100K and 10K respectively. All the models were trained using RMSprop with a learning rate of 1e−3, decay of 0.9 and a batch size of 20. For both the settings T = 1000 and T = 2000, KRU converges to zero average cross entropy faster than FC uRNN. All the other baselines are stuck at the memory-less cross entropy. The are shown in figure 1. For this problem we do not learn the recurrent matrix of KRU, We initialize it by random unitary matrix and just learn the input to hidden, hidden to output matrices and the bias. We found out that this strategy already solves the problem faster than all other methods. Our model in this case is similar to a parametrized echo state networks (ESN). ESNs are known to be able to learn long-term dependencies if they are properly initialized BID24. We argue that this data-set is not an ideal benchmark for evaluating RNNs in capturing long term dependencies. Just a unitary initialization of the recurrent matrix would solve the problem. Following BID1 we describe the adding problem BID23. Each input vector is composed of two sequences of length T. The first sequence is sampled from U. In the second sequence exactly two of the entries is 1, the'marker' and the remaining is 0. The first 1 is located uniformly at random in the first half of the sequence and the other 1 is located again uniformly at random in the other half of the sequence. The network's goal is to predict the sum of the numbers from the first sequence corresponding to the marked locations in the second sequence. We evaluate four settings as in BID1 with T =100, T =200, T =400, and T =750. For all four settings, KRU uses a hidden dimension N of 512 with 2x2 Kronecker factors which corresponds to ≈3K parameters in total. We use a RNN of N = 128 (≈ 17K parameters), LSTM of N = 128 (≈ 67K parameters), RC uRNN of N = 512 (≈ 7K parameters), FC uRNN of N = 128 (≈ 33K parameters). The train and test set sizes are chosen to be 100K and 10K respectively. All the models were trained using RMSprop with a learning rate of 1e−3 and a batch size of 20 or 50 with the best are being reported here. The are presented in figure 2. KRU converges faster than all other baselines even though it has much fewer parameters. This shows the effectiveness of soft unitary constraint which controls the flow of gradients through very long time steps and thus deciding what to forget and remember in an adaptive way. LSTM also converges to the solution and this is achieved through its gating mechanism which controls the flow of the gradients and thus the long term influence. However LSTM has 10 times more parameters than KRU. Both RC uRNN and FC uRNN converges for T = 100 but as we can observe, the learning is not stable. The reason for this is that RC uRNN and FC uRNN retains noise since they are strict unitary models. Please note that we do not evaluate RC uRNN for T = 400 and T = 750 because we found out that the learning is unstable for this model and is often diverging. Results on adding problem for T =100, T =200, T =400 and T =750. KRU consistently outperforms the baselines on all the settings with fewer parameters. As outlined by, we evaluate the Pixel by pixel MNIST task. MNIST digits are shown to the network pixel by pixel and the goal is to predict the class of the digit after seeing all the pixels one by one. We consider two tasks: Pixels are read from left to right from top or bottom and Pixels are randomly permuted before being shown to the network. The sequence length for these tasks is T = 28 × 28 = 784. The size of the MNIST training set is 60K among which we choose 5K as the validation set. The models are trained on the remaining 55K points. The model which gave the best validation accuracy is chosen for test set evaluation. All the models are trained using RMSprop with a learning rate of 1e−3 and a decay of 0.9.The are summarized in FIG3 and table 2. On the unpermuted task LSTM achieve the state of the art performance even though the convergence speed is slow. Recently a low rank plus diagonal gated recurrent unit (LRD GRU) have shown to achieves 94.7 accuracy on permuted MNIST with 41.2K parameters whereas KRU achieves 94.5 with just 12K parameters i.e KRU has 3x parameters less than LRD GRU. Please also note that KRU is a simple model without a gating mechanism. KRU can be straightforwardly plugged into LSTM and GRU to exploit the additional benefits of the gating mechanism which we will show in the next experiments with a KRU-LSTM. We now consider character level language modeling on Penn TreeBank data-set BID32. Penn TreeBank is composed of 5017K characters in the training set, 393K characters in the validation set and 442K characters in the test set. The size of the vocabulary was limited to 10K most frequently occurring words and the rest of the words are replaced by a special <UNK> character BID36. The total number of unique characters in the data-set is 50, including the special <UNK> character. All our models were trained for 50 epochs with a batch size of 50 and using ADAM BID27. We use a learning rate of 1e−3 which was found through cross-validation with default beta parameters BID27 ). If we do not see an improvement in the validation bits per character (BPC) after each epoch then the learning rate is decreased by 0.30. Back-propagation through time (BPTT) is unrolled for 30 time frames on this task. We did two sets of experiments to have fair evaluation with the models whose were available for a particular parameter setting BID35 and also to see how the performance evolves as the number of parameters are increased. We present our in table 3. We observe that the strict orthogonal model, oRNN fails to generalize as well as other models even with a high capacity recurrent matrix. KRU and KRU-LSTM performs very close to RNN and LSTM with fewer parameters in the recurrent matrix. Please recall that the computational bottleneck in RNN is the computation of hidden states 2.1 and thus having fewer parameters in the recurrent matrix can significantly reduce the training and inference time. Recently HyperNetworks BID18 have shown to achieve the state of the art performance of 1.265 and 1.219 BPC on the PTB test set with 4.91 and 14.41 million parameters respectively. This is respectively 13 and 38 times more parameters than the KRU-LSTM model which achieves 1.47 test BPC. Also Recurrent Highway Networks (RHN) BID47 proved to be a promising model for learning very deep recurrent neural networks. Running experiments, and in particular exploring meta-parameters with models of that size, requires unfortunately computational means beyond what was at our disposal for this work. However, there is no reason that the consistent behavior and improvement observed on the other reference baselines would not generalize to that type of large-scale models. BID9 our main objective here is to have a fair evaluation of different recurrent neural networks. We took the baseline RNN and LSTM models of BID9 whose model sizes were chosen to be small enough to avoid overfitting. We choose the model size of KRU and KRU-LSTM in such way that it has fewer parameters compared to the baselines. As we can in the table 4 both our models (KRU and KRU-LSTM) overfit less and generalizes better. We also present the wall-clock running time of different methods in the figure 4. BID9 100 ≈20K 10K 8.82 9.10 5.64 9.03 LSTM BID9 Framewise phoneme classification BID16 is the problem of classifying the phoneme corresponding to a sound frame. We evaluate the models for this task on the real world TIMIT data-set BID15. TIMIT contains a training set of 3696 utterances among which we use 184 as the validation set. The test set is composed of 1344 utterances. We extract 12 Mel-Frequency Cepstrum Coefficients (MFCC) BID34 ) from 26 filter banks and also the log energy per frame. We also concatenate the first derivative, ing in a feature descriptor of dimension 26 per frame. The frame size is chosen to be 10ms and the window size is 25ms. The number of time steps to which back-propagation through time (BPTT) is unrolled corresponds to the length of each sequence. Since each sequence is of different length this implies that for each sample BPTT steps are different. All the models are trained for 20 epochs with a batch size of 1 using ADAM with default beta parameters BID27. The learning rate was cross-validated for each of the models from η ∈ {1e−2, 1e−3, 1e−4} and the best are reported here. The best learning rate for all the models was found out to be 1e−3 for all the models. Again if we do not observe a decrease in the validation error after each epoch, we decrease the learning rate by a factor of γ ∈ {1e−1, 2e−1, 3e−1} which is again cross-validated. Figure 5 summarizes Figure 5: KRU and KRU-LSTM performs better than the baseline models with far less parameters in the recurrent weight matrix on the challenging TIMIT data-set BID15. This significantly bring down the training and inference time of RNNs. Both LSTM and KRU-LSTM converged within 5 epochs whereas RNN and KRU took 20 epochs. A similar was obtained by BID16 using RNN and LSTM with 4 times less parameters respectively than our models. However in their work the LSTM took 20 epochs to converge and the RNN took 70 epochs. We have also experimented with the same model size as that of BID16 and have obtained very similar as in the table but at the expense of longer training times. Here we study the properties of soft unitary constraints on KRU. We use Polyphonic music modeling data-sets BID7: JSB Chorales and Piano-midi, as well as TIMIT data-set for this set of experiments. We varied the amplitude of soft unitary constraints from 1e − 7 to 1e − 1, the higher the amplitude the closer the recurrent matrix will be to the unitary set. All other hyper-parameters, such as the learning rate and the model size are fixed. We present our studies in the figure 6. As we increase the amplitude we can see that the recurrent matrix is getting better conditioned and the spectral norm or the spectral radius is approaching towards 1. As we can see that the validation performance can be improved using this simple soft unitary constraints. For JSB Chorales the best validation performance is achieved at an amplitude of 1e − 2, whereas for Piano-midi it is at 1e − 1.For TIMIT phoneme recognition problem, the best validation error is achieved at 1e − 5 but as we increase the amplitude further, the performance drops. This might be explained by a vanishing long-term influence that has to be forgotten. Our model achieve this by cross-validating the amplitude of soft unitary constraints. These experiments also reveals the problems of strict unitary models such as RC uRNN BID1, FC uRNN BID42, oRNN BID35 and EURNN BID25 ) that they suffer from the retention of noise from a vanishing long term influence and thus fails to generalize. A popular heuristic strategy to avoid exploding gradients in RNNs and thereby making their training robust and stable is gradient clipping. Most of the state of the art RNN models use gradient clipping for training. Please note that we are not using gradient clipping with KRU. Our soft unitary constraints offer a principled alternative to gradient clipping. Moreover BID19 recently showed that gradient descent converges to the global optimizer of linear recurrent neural networks even though the learning problem is non-convex. The necessary condition for the global convergence guarantee requires that the spectral norm of recurrent matrix is bounded by 1. This seminal theoretical also inspires to use regularizers which control the spectral norm of the recurrent matrix, such as the soft unitary constraints. We have presented a new recurrent neural network model based on its core a Kronecker factored recurrent matrix. Our core reason for using a Kronecker factored recurrent matrix stems from it's elegant algebraic and spectral properties. Kronecker matrices are neither low-rank nor block-diagonal but it is multi-scale like the FFT matrix. Kronecker factorization provides a fine control over the model capacity and it's algebraic properties enable us to design fast matrix multiplication algorithms. It's spectral properties allow us to efficiently enforce constraints like positive semi-definitivity, unitarity and stochasticity. As we have shown, we used the spectral properties to efficiently enforce a soft unitary constraint. Experimental show that our approach out-perform classical methods which uses O(N 2) parameters in the recurrent matrix. Maybe as important, these experiments show that both on toy problems (§ 4.1 and 4.2), and on real ones (§ 4.3, 4.4,, and § 4.6), while existing methods require tens of thousands of parameters in the recurrent matrix, competitive or better than state-of-the-art performance can be achieved with far less parameters in the recurrent weight matrix. These surprising provide a new and counter-intuitive perspective on desirable memory-capable architectures: the state should remain of high dimension to allow the use of high-capacity networks to encode the input into the internal state, and to extract the predicted value, but the recurrent dynamic itself can, and should, be implemented with a low-capacity model. From a practical standpoint, the core idea in our method is applicable not only to vanilla recurrent neural networks and LSTMS as we showed, but also to a variety of machine learning models such as feed-forward networks BID46, random projections and boosting weak learners. Our future work encompasses exploring other machine learning models and on dynamically increasing the capacity of the models on the fly during training to have a perfect balance between computational efficiency and sample complexity. Given a sequence of T input vectors: x 0, x 1,..., x T −1, let us consider the operation at the hidden layer t of a recurrent neural network: DISPLAYFORM0 By the chain rule, DISPLAYFORM1 where σ is the non-linear activation function and J k+1 = diag(σ (z k+1)) is the Jacobian matrix of the non-linear activation function. DISPLAYFORM2 From equation 14 it is clear the norm of the gradient is exponentially dependent upon two factors along the time horizon:• The norm of the Jacobian matrix of the non-linear activation function J k+1.• The norm of the hidden to hidden weight matrix W.These two factors are causing the vanishing and exploding gradient problem. Since the gradient of the standard non-linear activation functions such as tanh and ReLU are bounded between, J k+1 does not contribute to the exploding gradient problem but it can still cause vanishing gradient problem. LSTM networks presented an elegant solution to the vanishing and exploding gradients through the introduction of gating mechanism. Apart from the standard hidden state in RNN, LSTM introduced one more state called cell state c t. LSTM has three different gates whose functionality is described as follows: DISPLAYFORM0 Decides what information to keep and erase from the previous cell state. DISPLAYFORM1 Decides what new information should be added to the cell state.• Output gate (W o, U o, b o):Decides which information from the cell state is going to the output. In addition to the gates, LSTM prepares candidates for the information from the input gate that might get added to the cell state through the action of input gate. Let's denote the parameters describing the function that prepares this candidate information as W c, U c, b c.Given a sequence of T input vectors: x 0, x 1,..., x T −1, at a time step t LSTM performs the following: DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 where σ and τ are the point-wise sigmoid and tanh functions. indicates element-wise multiplication. The first three are gating operations and the 4th one prepares the candidate information. The 5th operation updates the cell-state and finally in the 6th operation the output gate decided what to go into the current hidden state. Unitary evolution RNN (uRNN) proposed to solve the vanishing and exploding gradients through a unitary recurrent matrix, which is for the form: DISPLAYFORM0 Where: DISPLAYFORM1 Diagonal matrices whose diagonal entries are of the from D kk = e iθ k, implies each matrix have N parameters, (θ 0, . . ., θ N −1).• F and F −1: Fast Fourier operator and inverse fast Fourier operator respectively.• R 1, R 2: Householder reflections. DISPLAYFORM2 The total number of parameters for this uRNN operator is 7N and the matrix vector can be done N log(N) time. It is parameter efficient and fast but not flexible and suffers from the retention of noise and difficulty in optimization due its unitarity. Orthogonal RNN (oRNN) parametrizes the recurrent matrices using Householder reflections. DISPLAYFORM3 where DISPLAYFORM4 and DISPLAYFORM5 where DISPLAYFORM6 The number of parameters in this parametrization is O(N K). When N = K = 1 and v = 1, it spans the rotation subset and when v = −1, it spans the full reflection subset. Consider a matrix W ∈ C N ×N factorized as a Kronecker product of F matrices W 0,..., W F −1, DISPLAYFORM0 Where each W i ∈ C Pi×Qi respectively and DISPLAYFORM1 DISPLAYFORM2 Proof. DISPLAYFORM3 For simplicity here we use real number notations. Consider a dense matrix X ∈ R M ×K and a Kronecker factored matrix DISPLAYFORM0 The computational complexity first expanding the Kronecker factored matrix and then computing the matrix product is O(M N K). This can be reduced by exploiting the recursive definition of Kronecker matrices. For examples when N = K and ∀ f {P f = Q f = 2}, the matrix product can be computed DISPLAYFORM1 The matrix product in 29 can be recursively defined as DISPLAYFORM2 Please note that the binary operator is not the standard matrix multiplication operator but instead it denotes a strided matrix multiplication. The stride is computed according to the algebra of Kronecker matrices. Let us define Y recursively: DISPLAYFORM3 Combining equation 34 and 32 DISPLAYFORM4 We use the above notation for Y in the algorithm. That is the algorithm illustrated here will cache all the intermediate outputs (Y 0, . . ., Y F −1) instead of just Y F −1. These intermediate outputs are then later to compute the gradients during the back-propagation. This cache will save some computation during the back-propagation. If the model is just being used for inference then the algorithm can the organized in such a way that we do not need to cache the intermediate outputs and thus save memory. Algorithm for computing the product between a dense matrix and a Kronecker factored matrix34 is given below 1. All the matrices are assumed to be stored in row major order. For simplicity the algorithm is illustrated in a serial fashion. Please note the lines 4 to 15 except lines 9-11 can be trivially parallelized as it writes to independent memory locations. The GPU implementation exploits this fact. Algorithm 1 That is, the Kronecker layer is parametrized by a Kronecker factored matrix W = ⊗ F −1 f =0 W f stored as it factors {W 0, . . ., W F −1} and it takes an input X and produces output Y = Y F −1 using the algorithm 1.The following algorithm 2 computes the Gradient of the Kronecker factors: {gW 0, . . ., gW F −1} and the Jacobian of the input matrix gX given the Jacobian of the output matrix: gY = gY F −1.Algorithm 2 Gradient computation in a Kronecker layer. Input: Input matrix X ∈ R M ×K, Kronecker factors {W 0, . . ., W F −1}: W f ∈ R p f ×q f, Size of each Kronecker factors {(P 0, Q 0),..., (P F −1, Q F −1)}: DISPLAYFORM5

Represent the TLDR for finding the corresponding scientific article

Out work presents a Kronecker factorization of recurrent weight matrices for parameter efficient and well conditioned recurrent neural networks.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: This paper studies the undesired phenomena of over-sensitivity of representations learned by deep networks to semantically-irrelevant changes in data. We identify a cause for this shortcoming in the classical Variational Auto-encoder (VAE) objective, the evidence lower bound (ELBO). We show that the ELBO fails to control the behaviour of the encoder out of the support of the empirical data distribution and this behaviour of the VAE can lead to extreme errors in the learned representation. This is a key hurdle in the effective use of representations for data-efficient learning and transfer. To address this problem, we propose to augment the data with specifications that enforce insensitivity of the representation with respect to families of transformations. To incorporate these specifications, we propose a regularization method that is based on a selection mechanism that creates a fictive data point by explicitly perturbing an observed true data point. For certain choices of parameters, our formulation naturally leads to the minimization of the entropy regularized Wasserstein distance between representations. We illustrate our approach on standard datasets and experimentally show that significant improvements in the downstream adversarial accuracy can be achieved by learning robust representations completely in an unsupervised manner, without a reference to a particular downstream task and without a costly supervised adversarial training procedure. Representation learning is a fundamental problem in Machine learning and holds the promise to enable data-efficient learning and transfer to new tasks. Researchers working in domains like Computer Vision and Natural Language Processing have already demonstrated the effectiveness of representations and features computed by deep architectures for the solution of other tasks. A case in point is the example of the FC7 features from the AlexNet image classification architecture that have been used for many other vision problems . The effectiveness of learned representations has given new impetus to research in representation learning, leading to a lot of work being done on the development of techniques for inducing representations from data having desirable properties like disentanglement and compactness (; ; ;). Many popular techniques for generating representation are based on the Variational AutoEncoders (VAE) model . The use of deep networks as universal function approximators has facilitated very rapid advancements which samples generated from these models often being indistinguishable from natural data. While the quality of generated examples can provide significant convincing evidence that a generative model is flexible enough to capture the variability in the data distribution, it is far from a formal guarantee that the representation is fit for other purposes. In fact, if the actual goal is learning good latent representations, evaluating generative models only based on reconstruction fidelity and subjective quality of typical samples is neither sufficient nor entirely necessary, and can be even misleading. In this paper, we uncover the problematic failure mode where representations learned by VAEs exhibit over-sensitivity to semantically-irrelevant changes in data. One example of such problematic behaviour can be seen in Figure 1. We identify a cause for this shortcoming in the classical Vari-ational Auto-encoder (VAE) objective, the evidence lower bound (ELBO), that fails to control the behaviour of the encoder out of the support of the empirical data distribution. We show this behaviour of the VAE can lead to extreme errors in the recovered representation by the encoder and is a key hurdle in the effective use of representations for data-efficient learning and transfer. To address this problem, we propose to augment the data with properties that enforce insensitivity of the representation with respect to families of transformations. To incorporate these specifications, we propose a regularization method that is based on a selection mechanism that creates a fictive data point by explicitly perturbing an observed true data point. For certain choices of parameters, our formulation naturally leads to the minimization of the entropy regularized Wasserstein distance between representations. We illustrate our approach on standard datasets and experimentally show that significant improvements in the downstream adversarial accuracy can be achieved by learning robust representations completely in an unsupervised manner, without a reference to a particular downstream task and without a costly supervised adversarial training procedure. Figure 1: An illustration of the intrinsic fragility of VAE representations. Outputs from a Variational Autoencoder with encoder f and decoder g parametrized by η and θ, respectively, trained on CelebA. Conditioned on the encoder input X a = x a the decoder output X = g(f (x a)) = (g • f)(x a) is shown on the top row. When the original example is perturbed with a carefully selected vector d such that X b = X a + d with d ≤, the output X turns out to be perceptually very different. Such examples suggest that either the representations Z a and Z b are very different (the encoder is not smooth), or the decoder is very sensitive to small changes in the representation (the decoder is not smooth), or both. We identify the source of the problem primarily as the encoder and propose a practical solution. It is clear that if learned representations are overly sensitive to irrelevant changes in the input (for example, small changes in the pixels of an image or video, or inaudible frequencies added to an audio signal), models that rely on these representations are naturally susceptible to make incorrect predictions when inputs are changed. We argue that such specifications about the robustness properties of learned representations can be one of the tractable guiding features in the search for good representations. Based on these observations, we make the following contributions: 1. We introduce a method for learning robust latent representations by explicitly targeting a structured model that admits the original VAE model as a marginal. We also show that in the case the target is chosen a pairwise conditional random field with attractive potentials, this choice leads naturally to the Wasserstein divergence between posterior distributions over the latent space. This insight provides us a flexible class of robustness metrics for controlling representations learned by VAEs. 2. We develop a modification to training algorithms for VAEs to improve robustness of learned representations, using an external selection mechanism for obtaining transformed examples and by enforcing the corresponding representations to be close. As a particular selection mechanism, we adopt attacks in adversarial supervised learning to attacks to the latent representation. Using this novel unsupervised training procedure we learn encoders with adjustable robustness properties and show that these are effective at learning representations that perform well across a variety of downstream tasks. 3. We show that alternative models proposed in the literature, in particular β-VAE model used for explicitly controlling the learned representations, or Wasserstein Generative Adversarial Networks (GANs) can also be interpreted in our framework as variational lower bound maximization. 4. We show empirically using simulation studies on MNIST, color MNIST and CelebA datasets, that models trained using our method learn representations that provide a higher degree of adversarial robustness even without supervised adversarial training. Modern generative models are samplers p(X|θ) for generating realizations from an ideal target distribution π(X), also known as the data distribution. In practice π(X) is unknown in the sense that it is hard to formally specify. Instead, we have a representative data set X, samples that are assumed to be conditionally independently drawn from the data distribution π(X) of interest. We will refer to the empirical distribution asπ(X) =, thereby also learning a generator. The VAE corresponds to the latent variable model p(X|Z, θ)p(Z) with latent variable Z and observation X. The forward model p(X|Z = z, θ) (the decoder) is represented using a neural network g with parameters θ, usually the mean of a Gaussian N (X; g(z; θ), vI x ) where v is a scalar observation noise variance and I x is an identity matrix. The prior is usually a standard Gaussian p(Z = z) = N (z; 0, I z). The exact posterior over latent variables p(Z|X = x, θ) is approximated by a probability model q(Z|X = x, η) with parameters η. A popular choice here is a multivariate Gaussian N (Z; µ(x; η), Σ(x; η)), where the mapping f such that (µ, Σ) = f (x, η) is chosen to be a neural network (with parameters η to be learned from data). We will refer to the pair f, g as an encoder-decoder pair. Under the above assumptions, VAE's are trained by maximizing the following form of the ELBO using stochastic gradient descent (SGD), The gradient of the Kullback-Leibler (KL) divergence term above (see A.1) is available in closed form. An unbiased estimate of the gradient of the first term can be obtained via sampling z from q using the reparametrization trick , aided by automatic differentiation. Under the i.i.d. assumption, where each data point x (n), for n = 1... N is independently drawn from the model an equivalent batch ELBO objective can be defined as where the empirical distribution of observed data is denoted asπ (See E.1 for a derivation). This form makes it more clear that the variational lower bound is only calculating the distance between the encoder and decoder under the support of the empirical distribution. To see how this locality leads to a fragile representation, we construct a VAE with discrete latents and observations. We let X ∈ {1, . . ., N x} and Z ∈ {1, . . ., N z} and define the following system of conditional distributions as the decoder and encoder models as: where ω(u) = cos(2πu). These distributions can be visualized by heatmaps of probability tables where i and j are row and column indicies, respectively Figure 2. This particular von-Mises like parametrization is chosen for avoiding boundary effects due to a finite latent and observable spaces. The prior p(Z) is taken as uniform, and is not shown. Note that this parametrization emulates a high capacity network that can model any functional relationship between latent states and observations, while being qualitatively similar to a standard VAE model with conditionally Gaussian decoder and encoder functions. In reality, the true target density is not available but we would have a representative sample. To simulate this scenario, we sample a'dataset' from a discrete target distribution π(X): this is merely a draw from a multinomial distribution, yielding a multinomial vector s with entries s i that gives the count how many times we observe x = i. The of such an experiment are depicted in Figure 3 (a) (see caption for details). This picture reveals several important properties of a VAE approximation. Figure 3: (a) Result by optimizing the ELBO for a VAE that illustrates the fragility of the encoder. Subfigure with the title'Data' (π(X)) is a random sample from the true target'Target' (π(X)) on the right. The ing encoder q(Z|X) and decoder p(X|Z) are shown as'Q' and'P', respectively. The vertical and horizontal axes correspond to latents Z and observations X respectively. Integrating over the decoder distribution using a uniform prior p(Z) over the latents, we obtain the model The obtained by a smooth encoder. Both the decoder and the representation (encoder) are more smooth while essentially having a similar fitting quality. 1. After training, we observe that when j and j are close, the corresponding conditionals p(X|Z = j) and p(X|Z = j) are close (hence corresponding decoder mean parameters m j and m j are close, hence (see middle panel of Fig.3 (a) with the title P showing the decoder). This smoothness is perhaps surprising at a first sight: in this example, we could arbitrarily permute columns of the decoder and still get the same marginal distribution. Technically speaking, given a uniform prior p(Z), the marginal likelihood p(X|θ) is entirely invariant with respect to permutations of the latent state. In fact if the encoder distribution wouldn't be constrained we could also permute the columns of the encoder to keep the ELBO invariant. In the appendix E.2, we provide an argument why the choice of an unimodal encoder model and optimization of the variational objective leads naturally to smooth decoder functions. 2. The encoders found by the VAE on the other hand are not smooth at all, despite the fact that the model shows a relatively good fit. This behaviour alerts us about judging generative models only by the quality of the samples, by traversing the latent space and generating conditional samples from the decoder. The quality of the decoder seems to be not a proxy for the robustness of the representation. The fragility of representations is inherent from the ELBO objective. For the entire dataset, a batch ELBO that involves the counts s i can be written as The last expression is proportional to the negative KL divergence between two tabular distributions:. As such, whenever s i is zero, the contribution of row i of the encoder distribution vanishes and the corresponding parameters µ i and σ i are not effecting the lower bound. In a sense, the objective does not enforce any structure on the encoder outside of the position of the data points in the training set. This figure shows that the outof-sample behaviour (i.e., for i whereπ(X) = 0) the encoder is entirely initialization dependent, hence no learning takes place. We would also expect that the ing representations would be fragile, in the sense that a small perturbation of an observation can in a large change in the encoder output. In this section, we will adopt a strategy for training the encoder that is guaranteed not to change the original objective of the decoder when maximizing the lower bound while obtaining a smoother representation. The key idea of our approach is that we assume an external selection mechanism that is able to provide new fictive data point x in the vicinity of each observation in our data set x. Here, "in the vicinity" means that we desire that the corresponding latent state of the original datapoint z = f (x; η) and the latent state of the fictitious point z = f (x ; η) should be close to each other in some sense. Assuming the existence of such an external selection mechanism, we first define the following augmented distribution where This is a pairwise conditional Markov random field (CRF) model , where we take c(Z a, Z b) as a pairwise cost function. A natural choice here would be, for example, the Euclidean square distance Z a − Z b 2. Moreover, we choose a nonnegative coupling parameter γ ≥ 0. For any pairwise Q(Z a, Z b) distribution, the ELBO has the following form It may appear that the SE has to maintain a pairwise approximation distribution Q(Z a, Z b). However, this turns out to be not necessary. Given the encoder, the marginals of, so the only remaining terms that depend on the pair distribution are the final two terms in. We note that this two terms are just the objective function of the entropy regularized optimal transport problem . If we view Q(Z a, Z b) as a transport plan, the first term is maximal when the expected cost is minimal while the second term is maximal when the variational distribution is factorized as In retrospection, this link is perhaps not that surprising as the Wasserstein distance, the solution of the optimal transport problem, is itself defined as the solution to a variational problem : Consider a set Γ of joint densities Q(Z a, Z b) with the property that Q has fixed marginals The Wasserstein divergence 1, denoted by WD is defined as the solution of the optimization problem with respect to pairwise distribution Q where c(Z a, Z b) is a function that specifies the'cost' of transferring a unit of probability mass from It is important to note that with our choice of the particular form of the variational distribution Q(Z a, Z b) we can ensure that we are still optimizing a lower bound of the original problem. We can achieve this by simply integrating out the X, effectively ignoring the likelihood term for the fictive observations. Our choice does not modify the original objective of the decoder due to the fact that the marginals are fixed given η. To see this, take the exponent of and integrate over the unobserved X log p(X = x|θ) = log dX p(X = x, X |θ) we name this lower bound B SE as the Smooth Encoder ELBO (SE-ELBO). The gradient of B SE with respect to the decoder parameters θ is identical to the gradient of the original VAE objective B. This is intuitive as x is an artificially generated sample, we should use only terms that depend on x and not on x. Another advantage of this choice is that it is possible to optimize the decoder and encoder concurrently as in the standard VAE. Only an additional term enters for the regularization of the encoder where the marginals obtained via amortized inference q(Z a |x a, η) and q(Z b |x b, η) are forced to be close in a regularized Wasserstein distance sense, with the coupling strength γ. Effectively, we are doing data augmentation for smoothing the representations obtained by the encoder without changing the actual data distribution. In the appendix E.3, we also provide an argument about the smoothness of the corresponding encoder mapping, justifying the name. The ing algorithm is actually a simple modification to the standard VAE and is summarized below: Adversarial attacks are one of the most popular approaches for probing trained models in supervised tasks, where the goal of an adversarial attack is finding small perturbations to an input example that would maximally change the output, e.g., flip a classification decision, change significantly a prediction . The perturbed input is named as an adversarial example and these extra examples are used, along with the original data points, for training adversarially robust models . As extra samples are also included, such a training procedure is referred as data augmentation. However, in unsupervised learning and density 1 We use the term divergence to distinguish the optimal transport cost from the corresponding metric. This distinction is reminiscent to the distinction between Euclidian divergence · 2 and the Euclidian distance · estimation, data augmentation is not a valid approach as the underlying empirical distribution would be altered by the introducing new points. However, as we let the encoder to target a different distribution than the actual decoder, we can actually use the extra, self generated samples to improve desirable properties of a given model. Hence this approach could also be interpreted as a'self-supervised' learning approach where we bias our search for a'good encoder' and the data selection mechanism acts like a critique, carefully providing examples that should lead to similar representations. In this paper we will restrict ourselves to Projected Gradient Descent (PGD) attacks popular in adversarial training as a selection mechanism, where the goal of the attacker is finding a point that would introduce the maximum difference in the Wasserstein distance of the latent representation. In other words, we implement our selection mechanism where the extra data point is found by approximately solving the following constrained optimization problem This attack is assigned a certain iteration budget L for a given radius, that we refer as selection iteration budget and the selection radius, respectively. We note a similar attack mechanism is proposed for generative models as described in , where one of the proposed attacks is directly optimizing against differences in source and target latent representations. Note that our method is not restricted to a particular selection mechanism; indeed two inputs that should give a similar latent representation could be used as candidates. Goal and Protocol In our experiments, we have tested and compared the adversarial accuracy of representations learned using a VAE and our smooth encoder approach. We adopt a two step experimental protocol, where we first train encoder-decoder pairs agnostic to any downstream task. Then we fix the representation, that is we freeze the encoder parameters and only use the mean of the encoder as the representation, then train a simple linear classifier based on the fixed representation using standard techniques. In this supervised stage, no adversarial training technique is employed. Ideally, we hope that such an approach will provide a degree of adversarial robustness, without the need for a costly, task specific adversarial training procedure. To evaluate the robustness of the ing classifier, for each data point in the test set, we search for an adversarial example using an untargeted attack that tries to change the classification decision. The adversarial accuracy is reported in terms of percentage of examples where the attack is not able to find an adversarial example. The VAE and SE decoder and encoder are implemented using standard MLP and ConvNet architectures. The selection procedure for SE training is implemented as a projected gradient descent optimization (a PGD attack) with selection iteration budget of L iterations to maximize the Wasserstein distance between q(Z|X = x) and q(Z|X = x + δ) with respect to the perturbation δ where δ ∞ <. Further details about the experiment can be found in the appendix C.1. We run simulations on ColorMNIST, MNIST and CelebA datasets. The ColorMNIST is constructed from the MNIST dataset by coloring each digit artificially with all of the colors corresponding to the seven of the eight corners of the RGB cube (excluding black). We present the with the strongest attack we have experimented: a PGD attack with 100 iterations and 10 restarts. We observe that for weaker attacks (such as 50 iterations with no restarts), the adversarial accuracy is typically much higher. For the ColorMNIST dataset, the are shown in Figure 4 where we test the adversarial accuracy of representations learned by our method and compare it to a VAE. We observe that the adversarial accuracy of a VAE representation quickly drops towards zero while SE can maintain adversarial accuracy in both tasks. In particular, we observe that for the arguably simpler color classification task, we are able to obtain close to perfect adversarial test accuracy using representations learned by the VAE and SE. However, when the classifiers are attacked using PGD, the adversarial accuracy quickly drops with increasing radius size, while the accuracy degrades more gracefully in the SE case. In Figure 5, we show the robustness behaviour of the method for different architectures. A ConvNet seems to perform relatively better than an MLP but these show that the VAE representation is not robust, irrespective of the architecture. We have also carried out controlled experiments with random selection instead of the more costly untargetted adversarial attacks (See appendix C.1 Figure 7(a) for further ). We observe some limited improvements with SE using random selection in adversarial accuracy compared to VAE but training a SE with adversarial selection seems to be much more effective. We note that the selection iteration budget was lower (L = 20 with no restarts) than the attack iteration budget (100 with 10 restarts) during evaluation. It was not practical to train the encoder with more powerful selection attacks, thus it remains to be seen if the tendency of increased accuracy with increased iteration budgets would continue. We also observe that essentially the same level of adversarial accuracy can be obtained with a small fraction of the available labels (See appendix C.1 Figure 8 for further ). We have also repeated our experiments on the CelebA dataset, a large collection of high resolution face images labeled with 40 attribute labels per example. We have used 17 of the attribute labels as the targets of 17 different downstream classification tasks. The are shown in Table. 2. The clearly illustrate that we can achieve much more robust representations than a VAE. It is also informative to investigate specific adversarial examples to understand the failure modes. In Figure 6 we show two illustrative examples from the CelebA. Here we observe that attacks to the SE representations are much more structured and semantically interpretable. In our exploratory investigations, we qualitatively observe that the reconstructions corresponding to the adversarial In the VAE case, the attacker is able to find a perturbation to map the representation to that of a bearded person. However, the perturbation here does not seem to have a particular visible structure. (b) The SE representation is attacked by a perturbation that can clearly be identified as drawing a beard on the image. In this case, the attack is able to fool the classifier and the generated image from the representation is that of a person with beard and mustache. In the second example (c), the VAE representation seems to be attacked by exploiting the non-smooth nature of the encoder; the attacker is able to identify an adversarial example that maps the latent representation to the one in the vicinity of a clearly different person with the desired features, as can be seen from the corresponding reconstruction. In contrast, in the SE case (d), the attack adds a much more structured perturbation, and in this example it was actually not successful in switching the decision. Additionally, from the reconstruction it is evident that a latent feature is attacked that seems to control the receding hairline. examples are almost always legitimate face images with clearly recognizable features. This also seems to support our earlier observation that VAE decoders are typically smooth while the encoders are inferring non-robust features. Our approach seems to be a step towards obtaining more robust representations. Table 1: Comparison of nominal (Nom) and adversarial (Adv) accuracy (in percentage) on 17 downstream tasks using a VAE and a SE trained with a selection radius of = 0.1. The experiment is carried out using the experimental protocol described in the text (Section 4). The adversarial evaluation on CelebA with Attack radius of 0.1 and attack iteration budget of 100 with 10 restarts. The literature on deep generative models and representation learning is quite extensive and is rapidly expanding. There is a plethora of models, but some approaches have been quite popular in recent years: Generative Adversarial Networks (GANs) and VAEs. While the connection of our approach to VAE's is evident, there is also a connection to GANs. In the appendix, we provide the details where we show that a GAN decoder can be viewed as an instance of a particular smooth encoder. Our method is closely related to the β-VAE , used for controlling representations replaces the original variational objective with another one for explicitly trading the data fidelity with that of prior fidelity. In the appendix, we show that the method can be viewed as an instance of the smooth encoders. Wasserstein distance minimization has been applied in generative models as an alternative objective for fitting the decoder. Following the general framework sketched in, the terms of the variational decomposition of the marginal likelihood can be modified in order to change the observation model or the regulariser. For example, Wasserstein AutoEncoders (WAE), or sliced propose to replace data fidelity and/or the KL terms with a Wasserstein distance. Our approach is different from these approaches as we do not propose to replace the likelihood as a fundamental principle for data fitting. In contrast, the Wasserstein distance formulation naturally emerges from the particular model choice and the corresponding variational approximation. Our approach involves an adversarial selection step. The word'Adversarial' is an overloaded term in generative modelling so it is important to mention differences between our approach. Adversarial Variational Bayes is a well known technique in the literature that aims to combine the empirical success of GANs with the probabilistic formulation of VAEs, where the limiting functional form of the variational distribution can be replaced by blackbox inference . This approach also does not modify the original VAE objective, however, the motivation here is different as the aim is developing a more richer family. In our view, for learning useful representations, when the decoder is unknown, the advantage of having a more powerful approximating family is not clear yet; this can even make the task of learning a good representation harder. Adversarial Autoencoders , Adversarially Learned Inference (ALI) and BiGANs (Bidirectional GANs) are also techniques that combine ideas from GANs and VAEs for learning generative models. The key idea is matching an encoder process q(z|x)p(x) and to the decoder process p(z)p(x|z) using an alternative objective, rather than by minimizing the KL divergence as done by the variational inference (see (??)). In this formulation, p(x) is approximated by the empirical data distribution, and p(z) is the prior model of a VAE. The encoder q(z|x) and decoder p(x|z) are modelled using deep networks. This approach is similar to Wasserstein autoencoders that propose to replace the likelihood principle. The idea of improving VAEs by capturing the correlation structure between data points using MRFs and graphical models has been also been recently proposed under the name Correlated Variational Auto-Encoders (CVAEs). Our approach is similar, however we introduce the correlation structure not between individual data points but only between true data points and artificially selected data points. We believe that correctly selecting such a correlation structure of the individual data points can be quite hard in practice, but if such prior knowledge is available, CVAE can be indeed a much more powerful model than a VAE. We note that a proposal for automatically learning such a correlation structure is also recently proposed by . In this paper, we have introduced a method for improving robustness of latent representations learned by a VAE. It must be stressed that our goal is not building the most powerful adversarially robust supervised classifier, but obtaining a method for learning generic representations that can be used for several tasks; the tasks can be even unknown at the time of learning the representations. While the nominal accuracy of an unsupervised approach is expected to be inferior to a supervised training method that is informed by extra label information, we observe that significant improvements in adversarial robustness can be achieved by our approach that forces smooth representations. The KL divergence between two Gaussian distributions translates to a well known divergence in the parameters (in the general case this is a Bregman divergence) where P a = N (µ a, Σ a) and P b = N (µ b, Σ b) are Gaussian densities with mean µ · and covariance matrix Σ ·, and | · | denotes the determinant for a matrix argument, and Tr denotes the trace. The KL divergence consists of two terms, the first term is the scale invariant divergence between two covariance matrices also known as a Itakuro-Saito divergence and the second term is a Mahalonobis distance between the means. The KL divergence is invariant to the choice of parametrization or the choice of the coordinate system. Consider a set Γ of joint densities Q(Z a, Z b) with the property that Q has fixed marginals Q a (Z a) and The Wasserstein divergence WD is defined as the solution of the optimization problem with respect to pairwise distribution Q where c(z a, z b) is a function that specifies the'cost' of transferring a unit of probability mass from z a to z b. The 2 -Wasserstein distance W 2 2 for two Gaussians has an interesting form. The optimum transport plan, where the minimum of is attained, is given where b. It can be checked that this optimal Guassian density is degenerate in the sense that there exists a linear mapping between z a and z b: where A 1/2 denotes the matrix square root, a symmetric matrix such that (A 1/2) 2 = A for a symmetric positive semidefinite matrix A. The 2 -Wasserstein distance is the value attained by the optimum transport plan Entropy Regularized 2 -Wasserstein is the value attained by the minimizer of the following functional where H is the entropy of the joint distribution Q. Using the form in subject to the semidefinite constraint The entropy of a Gaussian Q(z a, z b) is given by the Schur formula Here, D is the dimension of the vector (z a, z b). The entropy regularized problem has a solution where we need to minimizẽ Taking the derivative and setting to zero we obtain a particular Matrix Ricatti equation that gives us a closed form formula for the specific entropy regularized Wasserstein distance For the case of two univariate Gaussians, i.e., when the joint distribution has the form the solution is given by the solution of the scalar quadratic equation. We take the root that gives a feasible solution as the minimizer. In the scalar case, this is the solution that satisfies where we have defined It can be easily checked that the other root is infeasible. For the scalar ψ case we obtain where D z is the dimension of the latent representation, and µ k and Σ k are the k'th component of the output of a neural network with parameters η. Similarly, x i denotes the i'th component of the observation vector x of size D x. For optimization, we need an unbiased estimate of the gradient of the SE-ELBO with respect to encoder parameters η and decoder parameters θ: Given x, we first select a fictive sample x via a selection mechanism, in this case as an adversarial attack as explained in section 3.1. Sample a latent representation and calculate the associated prediction The terms of the SE-ELBO can be calculated as We always train decoder-encoder pairs with identical architectures using both the standard VAE ELBO and the SE ELBO with a fixed γ. Then, in each case by fixing the encoder (that is essentially using the same representation) and by only using the mean activations of the encoders, we train linear classifiers using standard training for solving several downstream tasks. For both encoder and decoder networks we use a 4 layer multi layer perceptron (MLP) and a convolutional network (ConvNET) architectures with 200 units of ReLU activations at each layer. We carried out experiments with latent space dimensions of 32, 64 and 128, corresponding to an output sizes of an encoder with 64, 128 and 256 units, with two units per dimensions to encode the mean and the log-variance parameters of a fully factorized Gaussian condition distribution. The training is done using the Adam optimizer. Each network (both the encoder and decoder) are randomly initialized and trained for 300K iterations. GANs are presented as neural sampling models of observations x of form x = f (ζ; η) where f is typically a deep neural network with parameters η, and ζ is a realization from some simple distribution p(Z). In the context of GANs, the function f is called a generator. When the dimension of x is bigger than the dimension of ζ, the density p(x) induced by the transformation f is inevitably a degenerate distribution. Since f is continuous, and it is concentrated on a subset of the data space X f ≡ {x : ∃ζ, x = f (ζ; η)}. Our use of letter f and parameters η is deliberate and we will illustrate in the sequel that the generator network of a GANs is actually analogous to a smooth encoder, where the roles of the latent variables and observations are switched, but we will first review GANs. To fit a degenerate distribution to a dataset, the GAN approach adopts a strategy where the generator is co-trained with a second neural network d(x; w) with parameters w with the following objective where D real (x) is the empirical data distribution. This objective is (unfortunately) referred as an adversarial objective in the literature, not to be confused with adversarial attack mechanism in the context of supervised learning. The function d is called a discriminator. After replacing expectations with finite sample averages, this objective enforces that in a dataset that contains both synthetically generated (fake) and real examples, the classification function d should increase the correct classification rate by discriminating fakes from real examples while the generator f should decrease the detection rate of fake examples. When 0 ≤ d(·) ≤ 1, which is the case for a classifier, one can also write the objective as where l(x; w) = log d(x; w). This form also highlights an interesting alternative formulation and an interpretation in terms of optimal transport. In fact, not long after the seminal work of , the mechanism beyond the GAN objective and its direct connection to the theory of optimal transport has been recognized by the seminal paper where the problem is further framed as with the constraint that |l(x; w) − l(x; w)| ≤ c(x,x), i.e. l is a Lipschitz function for some L where c(x,x) ≤ L x −x. Here, D fake (x; θ) is the fitted density ofx = f (ζ; η). This is the dual formulation of the optimal transport problem, that can be understood as an economic transaction between a customer and a shipment company. Here, the difference l(x; w) − l(x; w) can be interpreted as the profit made by the company for the shipment of one unit of mass from x and tox, and the Lipschitz condition ensures that it makes still sense for the customer to make use of the services of the company rather than simply doing the transport of her own. The customer wants to pay less, so she should minimize the profit of the company. This can be achieved by changing the desired delivery distribution D fake by adjusting θ, so that the transfer from the fixed source distribution D real is minimized. Ideally, when D fake = D real, there is nothing to transfer and no cost is incurred. This objective also minimizes the Wasserstein distance between the actual data distribution D real and the fake data distribution D fake as given by the generator. Once the GAN objective can be viewed as minimizing a particular Wasserstein distance, it is rather straightforward to view it as a maximizer of a particular ELBO corresponding to a particular smooth encoder, albeit in one where the positions of the observations and the latents are exchanged and a very large coupling coefficient γ is chosen. Moreover, the variational marginals have specific forms: One marginal Q a (X) is chosen as the empirical data distribution and the other marginal is chosen as having the form of a neural sampler The artificial extended target becomes It can be seen that the ELBO in this case becomes Now, by taking the coupling γ sufficiently large, the coupling term dominates the lower bound and we obtain the Wasserstein minimization objective. The random draws from p(Z) become the selection mechanism. Moreover, the terms that depend on the artificial target p(Z|X, θ) become also irrelevant so in this regime the problem becomes just solving the optimal transport problem between Q a and Q b. A link between entropic GANs and VAEs is also pointed at in the literature, albeit for calculating a likelihood for. However, our motivations as well as the interpretation of the connection is quite different and we view the GAN decoder as an instance of the smooth encoder. Targeting the encoder to an augmented distribution different than the decoder us the freedom to express some extensions of VAE in the same framework. One of such extensions is the β-VAE, quite often used for controlling representations replaces the original variational objective with the following objective The justification in the original paper is obtained from an implicit robustness criteria where D KL (q(Z|X a = x, η)||p(Z)) < and β appears in a Lagrangian formulation. have also provided an alternative justification. In our formulation, β can be simply interpreted as a dispersion term that is related to the number of points selected by the selection mechanism. To see this, suppose the selection mechanism chooses β − 1 points x b,i where i = 1... β − 1 that are identical to the true observation x = x b,i = x i for i = 1... β − 1. Now, instead of integrating out Z 1:β−1, we choose a variational distribution with identical marginals of form The variational lower bound becomes identical to the β-VAE objective as where the last step follows due to the functional form of the variational distribution. E TECHNICAL In section 2.2, we have defined a batch ELBO. To see the connection to VAE ELBO we first define the empirical data distribution π(X) = 1 N N i=1 δ(X − x i). We can now write log p(X = x|θ) ≥ E {log p(X = x|Z, θ)} q(Z|X=x,η) − D KL (q(Z|X = x, η)||p(Z)) ≡ B x (η, θ) E {log p(X = x i |Z, θ)} q(Z|X=xi,η) E {log q(Z|X = x i, η)} q(Z|X=xi,η) E {log p(Z)} q(Z|X=xi,η) = E {log p(X|Z, θ)} q(Z|X,η)π(X) − E {log q(Z|X, η)} q(Z|X,η)π(X) +E {log p(Z)} q(Z|X,η)π(X) −E {log π(X)} q(Z|X,η)π(X) + E {log π(X)} π(X) = −D KL (q(Z|X, η)π(X)||p(X|Z, θ)p(Z)) + const This shows that the ELBO is minimizing the KL distance between one exact and one approximate factorization of the joint distribution p(X, Z) = p(X|Z, θ)p(Z) ≈ q(Z|X, η)π(X). In the context of a VAE, the smoothness of the decoder is implicitly enforced by the highly constrained encoder distribution and the dynamics of an SGD based training. In the sequel, we will illustrate that, if two latent coordinates are sufficiently close, the decoder mean mapping is forced to be bounded. In a standard VAE, the encoder output for each data point is conditionally Gaussian as q(Z|X = x; η) = N (f µ (x; η), f Σ (x; η)). The decoder is chosen as p(X|Z = z; η) = N (g(z; θ), vI). The decoder parameters θ under the ELBO depend only on the data fidelity term x − g(z; θ) 2 /v. For a moment, assume that the encoder is fixed and focus on a single data point x. During training, a set of latent state vectors z i for i = 1... T are sampled from the conditionally Gaussian encoder distribution. When the dimension of the latent space D z is large, these samples z i will be with high probability on the typical set. The typical set of a nondegenerate Gaussian distribution is approximately the surface of a Mahalanobis ball, a compact hyper-ellipsoid M (x) centered at f µ (x; η) with scaling matrix f Σ (x; η) 1/2. If we assume that the training procedure is able to reduce the error in the sense that x − g(z i ; θ) ≤ E for all z i where E is a bound on the error magnitude for z i sampled from the encoder, the decoder is forced to give the same output for each point approximately on M (x). For a point z a drawn from q(Z|X = x; η) we have z a − f µ (x; η) K ≈ D z with high probability where K = f Σ (x; η) −1 and x K ≡ √ x Kx. For a point z b independently drawn from q(Z|X = x; η), by the triangle inequality we have g(z a ; θ) − g(z b ; θ) ≤ 2E where the Mahalanobis distance where λ min is the smallest eigenvalue of the covariance matrix. Hence the distance is also bounded when the variance is not degenerate and minimum distance will be on the order of z a − z b ≈ 2 √ D z λ min so we expect the ratio to be bounded We see that the ELBO objective enforces the decoder to be invariant on the typical set of q(Z|X = x; η), where most of the probability mass is concentrated. Now, for each data point x, the corresponding latent space hyper-ellipsoid M (x) are forced to be large in the sense of having a large determinant by the entropy term of the encoder that promotes large log-determinant. The size of M (x) is also controlled by the prior fidelity term, avoiding blowing up. Hence the union ∪ x∈X M (x), where X is the dataset, will approximately cover the latent space when the encoder has converged and on each hyper-ellipsoid M (x) the decoder will be enforced to be smooth. In this section we show that the smooth encoder training forces a small Lipschitz constant for the encoder mean mapping. To simplify the argument, we will assume that the variance mapping of the encoder would be a constant function that does not vary with x, i.e., f Σ (x; η) = Σ(η). The latter assumption could be removed by considering a metric on the joint space of the means and covariance. Using the adversarial selection mechanism, during training we solve the following problem using PGD: x * = arg max x: x −x p ≤ WD(q(Z|X = x, η), q(Z|X = x, η)) Assuming that PGD finds the global maximum at the boundary of the -ball where x − x * p =, under constant variance assumption for the encoder we can see that the Wasserstein divergence simply becomes the square distance between mean mappings WD(q(Z|X = x, η), q(Z|X = x *, η)) = f µ (x; η) − f µ (x * ; η) 2 2 We know that the SE ELBO objective has to minimize this distance for any coupling term γ so the procedure actually tries to reduce the local Lipschitz constant L(x) around data point x L(x) = f µ (x; η) − f µ (x * ; η) x − x * p ≤ E and promotes smoothness where E is an upper bound on the change in the representation f µ (x; η)− f µ (x * ; η) ≤ E.

We propose a method for computing adversarially robust representations in an entirely unsupervised way.

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data. We adapt a recently proposed continuous constrained optimization formulation to allow for nonlinear relationships between variables using neural networks. This extension allows to model complex interactions while being more global in its search compared to other greedy approaches. In addition to comparing our method to existing continuous optimization methods, we provide missing empirical comparisons to nonlinear greedy search methods. On both synthetic and real-world data sets, this new method outperforms current continuous methods on most tasks while being competitive with existing greedy search methods on important metrics for causal inference. Structure learning and causal inference have many important applications in different areas of science such as genetics, biology and economics. Bayesian networks (BN), which encode conditional independencies using directed acyclic graphs (DAG), are powerful models which are both interpretable and computationally tractable. Causal graphical models (CGM) are BNs which support interventional queries like: What will happen if someone external to the system intervene on variable X? Recent work suggests that causality could partially solve challenges faced by current machine learning systems such as robustness to out-of-distribution samples, adaptability and explainability. However, structure and causal learning are daunting tasks due to both the combinatorial nature of the space of structures and the question of structure identifiability. Nevertheless, these graphical models known qualities and promises of improvement for machine intelligence renders the quest for structure/causal learning appealing. The problem of structure learning can be seen as an inverse problem in which the learner tries to infer the causal structure which has generated the observation. In this work, we propose a novel score-based method for structure learning named GraN-DAG which makes use of a recent reformulation of the original combinatorial problem of finding an optimal DAG into a continuous constrained optimization problem. In the original method named NOTEARS, the directed graph is encoded as a weighted adjacency matrix W which represents coefficients in a linear structural equation model (SEM). To enforce acyclicity, the authors propose a constraint which is both efficiently computable and easily differentiable. Most popular score-based methods for DAG learning usually tackle the combinatorial nature of the problem via greedy search procedures relying on multiple heuristics. Moving toward the continuous paradigm allows one to use gradient-based optimization algorithms instead of handdesigned greedy search algorithms. Our first contribution is to extend the work of to deal with nonlinear relationships between variables using neural networks (NN). GraN-DAG is general enough to deal with a large variety of parametric families of conditional probability distributions. To adapt the acyclicity constraint to our nonlinear model, we use an argument similar to what is used in and apply it first at the level of neural network paths and then at the level of graph paths. Our adapted constraint allows us to exploit the full flexibility of NNs. On both synthetic and real-world tasks, we show GraN-DAG outperforms other approaches which leverage the continuous paradigm, including DAG-GNN, a recent nonlinear extension of independently developed which uses an evidence lower bound as score. Our second contribution is to provide a missing empirical comparison to existing methods that support nonlinear relationships but tackle the optimization problem in its discrete form using greedy search procedures such as CAM. We show that GraN-DAG is competitive on the wide range of tasks we considered. We suppose the natural phenomenon of interest can be described by a random vector X ∈ R d entailed by an underlying CGM (P X, G) where P X is a probability distribution over X and G = (V, E) is a DAG. Each node i ∈ V corresponds to exactly one variable in the system. Let π G i denote the set of parents of node i in G and let X π G i denote the random vector containing the variables corresponding to the parents of i in G. We assume there are no hidden variables. In a CGM, the distribution P X is said to be Markov to G which means we can write the probability density function (pdf) as p(. A CGM can be thought of as a BN in which directed edges are given a causal meaning, allowing it to answer queries regarding interventional distributions. In general, it is impossible to recover G only given samples from P X. It is, however, customary to rely on a set of assumptions to render the structure fully or partially identifiable. Definition 1. Given a set of assumptions A on a CGM M = (P X, G), its graph G is said to be identifiable from P X if there exists no other CGMM = (P X,G) satisfying all assumptions in A such thatG = G andP X = P X. There are many examples of graph identifiability for continuous variables as well as for discrete variables. Those are obtained by assuming that the conditional pdf p i ∀i belongs to a specific parametric family P. For example, if one assumes that where f i is a nonlinear function satisfying some mild regularity conditions, then G is identifiable from P X (see for the complete theorem and its proof). We will make use of this in Section 4. Structure learning is the problem of learning G using a data set of n samples {x,..., x (n) } from P X. Score-based approaches cast this estimation problem as an optimization problem over the space of DAGs, i.e. Ĝ = arg max G∈DAG Score(G). The score is usually the maximum likelihood of your data given a certain model. Most score-based methods embrace the combinatorial nature of the problem via greedy search procedures. We now present the work of which approaches the problem from a continuous optimization perspective. To cast the combinatorial optimization problem into a continuous constrained one, proposes to encode the graph G on d nodes as a weighted adjacency matrix d×d which represents (possibly negative) coefficients in a linear structural equation model (SEM) of the form X i:= u i X + N i ∀i where N i is a noise variable. Let G U be the directed graph associated with the SEM and let A U be the (binary) adjacency matrix associated with G U. One can see that the following equivalence holds: To make sure G U is acyclic, the authors propose the following constraint on U: where e M ∞ k=0 M k k! is the matrix exponential and is the Hadamard product. It can be shown that G U is acyclic iff the constraint is satisfied (see for a proof). The authors propose to use a regularized negative least square score (maximum likelihood for a Gaussian noise model). The ing continuous constrained problem is where X ∈ R n×d is the design matrix containing all n samples. The nature of the problem has been drastically changed: we went from a combinatorial to a continuous problem. The difficulties of combinatorial optimization have been replaced by those of non-convex optimization, since the feasible set is non-convex. Nevertheless, a standard numerical solver for constrained optimization such has an augmented Lagrangian method (AL) can be applied to get an approximate solution. 3 GraN-DAG: Gradient-based neural DAG learning We propose a new nonlinear extension to the framework presented in Section 2.3. For each variable X i, we learn a fully connected neural network with L hidden layers parametrized by } where W (i) is the th weight matrix of the ith NN. Each NN takes as input X −i ∈ R d, i.e. the vector X with the ith component masked to zero, and outputs θ (i) ∈ R m, the m-dimensional parameter vector of the desired distribution family for variable X i. The fully connected NNs have the following form where g is a nonlinearity applied element-wise. Let φ {φ,..., φ (d) } represents all parameters of all d NNs. Without any constraint on its parameter φ (i), neural network i models the conditional pdf p i (x i |x −i ; φ (i) ). Note that the product is not a valid joint pdf since it does not decompose according to a DAG. We now show how one can constrain φ to make sure the product of all conditionals outputted by the NNs is a valid joint pdf. The idea is to define a new weighted adjacency matrix A φ similar to the matrix U encountered in Section 2.3, which can be directly used inside the constraint of Equation 3 to enforce acyclicity. Before defining the weighted adjacency matrix A φ, we need to focus on how one can make some NN outputs unaffected by some inputs. Since we will discuss properties of a single NN, we drop the NN subscript (i) to improve readability. We will use the term neural network path to refer to a computation path in a NN. For example, in a NN with two hidden layers, the sequence of weights (W kh2) is a NN path from input j to output k. We say that a NN path is inactive if at least one weight along the path is zero. We can loosely interpret the path product |W h1j ||W h2h1 ||W kh2 | ≥ 0 as the strength of the NN path, where a path product equal to zero if and only if the path is inactive. Note that if all NN paths from input j to output k are inactive (i.e. the sum of their path products is zero), then output k does not depend on input j anymore since the information in input j will never reach output k. The sum of all path products from every input j to every output k can be easily computed by taking the product of all the weight matrices in absolute value. where |W | is the element-wise absolute value of W. It can be verified that C kj is the sum of all NN path products from input j to output k. To have θ independent of variable X j, it is sufficient to have m k=1 C kj = 0. This is useful since, to render our architecture acyclic, we need to force some neural network inputs to be inactive (this corresponds to removing edges in our graph). We now define a weighted adjacency matrix A φ that can be used in constraint of Equation 3. where C (i) denotes the connectivity matrix of the NN associated with variable X i. As the notation suggests, A φ ∈ R d×d ≥0 depends on all weights of all NNs. Moreover, it can effectively be interpreted as a weighted adjacency matrix similarly to what we presented in Section 2.3, since we have that We note G φ to be the directed graph entailed by parameter φ. We can now write our adapted acyclicity constraint: This guarantees acyclicity. The argument is identical to the linear case, except that now we rely on implication instead of. We propose solving the maximum likelihood optimization problem where π φ i denotes the set of parents of variable i in graph G φ. Note that; θ (i) ) is a valid log-likelihood function when constraint is satisfied. As suggested in, we apply an augmented Lagrangian approach to get an approximate solution to program. Augmented Lagrangian methods consist of optimizing a sequence of subproblems for which the exact solutions are known to converge to a stationary point of the constrained problem under some regularity conditions. We approximately solve each subproblem using RMSprop, a stochastic gradient descent variant popular for NN. We empirically compare GraN-DAG to various baselines (both in the continuous and combinatorial paradigm), namely DAG-GNN, NOTEARS, RESIT We first present a comparison on synthetic data sets. We sampled 10 graphs (e.g. with 50 nodes and an average of 200 edges) and data distributions of the form ) with f i ∼ GP and evaluated different methods using SHD and SID (we report the average and the standard deviation over those data sets). Note that we are in the identifiable case presented in Section 2.2. GraN-DAG, NOTEARS and CAM all make the correct gaussian assumption in their respective models. In Table 1 we report a subset of our . GraN-DAG outperforms other continuous approaches while being competitive with the best performing discrete approach we considered. In addition, we considered a well known real world data set which measures the expression level of different proteins and phospholipids in human cells (the ground truth graph has 11 nodes and 17 edges). We found GraN-DAG to be competitive with other approaches. Our implementation of GraN-DAG can be found here.

Represent the short summary for finding the corresponding scientific article (long)

We are proposing a new score-based approach to structure/causal learning leveraging neural networks and a recent continuous constrained formulation to this problem

Represent the TLDR for finding the corresponding scientific article

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.