Question: represent the scientific article to find a short tldr

Question: We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features. In general, our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data. When there is only one latent variable in the superstructure, our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model. We call our model the latent tree variational autoencoder (LTVAE). Whereas previous deep learning methods for clustering produce only one partition of data, LTVAE produces multiple partitions of data, each being given by one super latent variable. This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways. Clustering is a fundamental task in unsupervised machine learning, and it is central to many datadriven application domains. Cluster analysis partitions all the data into disjoint groups, and one can understand the structure of the data by examining examples in each group. Many clustering methods have been proposed in the literature BID0, such as k-means BID18, Gaussian mixture models BID5 and spectral clustering BID30. Conventional clustering methods are generally applied directly on the original data space. However, it is challenging to perform cluster analysis on high dimensional and unstructured data BID26, such as images. It is not only because the dimensionality is high, but also because the original data space is too complex to interpret, e.g. there are semantic gaps between pixel values and objects in images. Recently, deep learning based clustering methods have been proposed that simultanously learn nonlinear embeddings through deep neural networks and perform cluster analysis on the embedding space. The representation learning process learns effective high-level representations from high dimensional data and helps the cluster analysis. This is typically achieved by unsupervised deep learning methods, such as restricted Boltzmann machine (RBM) BID11, autoencoders (AE) BID28, variational autoencoders (VAE) BID16, etc. Previous deep learning based clustering methods BID33 BID10 BID14 BID34 ) assume one single partition over the data and that all attributes define that partition. In real-world applications, however, the assumptions are usually not true. High-dimensional data are often multifaceted and can be meaningfully partitioned in multiple ways based on subsets of attributes BID4. For example, a student population can be clustered in one way based on course grades and in another way based on extracurricular activities. Movie reviews can be clustered based on both sentiment (positive or negative) and genre (comedy, action, war, etc.). It is challenging to discover the multi-facet structures of data, especially for high-dimensional data. To resolve the above issues, we propose an unsupervised learning method, latent tree variational autoencoder (LTVAE) to learn latent superstructures in variational autoencoders, and simultaneously perform representation learning and structure learning. LTVAE is a generative model, where the data is assumed to be generated from latent features through neural networks, while the latent features themselves are generated from tree-structured Bayesian networks with another level of latent variables as shown in Fig. 1. Each of those latent variables defines a facet of clustering. The proposed method automatically selects subsets of latent features for each facet, and learns the dependency structure among different facets. This is achieved through systematic structure learning. Consequently, LTVAE is able to discover complex structures of data rather than one partition. We also propose efficient learning algorithms for LTVAE with gradient descent and Stepwise EM through message passing. The rest of the paper is organized as follows. The related works are reviewed in Section 2. We introduce the proposed method and learning algorithms in Section 3. In Section 4, we present the empirical . The is given in Section 5. Clustering has been extensively studied in the literature in many aspects BID0. More complex clustering methods related to structure learning using Bayesian nonparametrics have been proposed, like Dirichlet Process, Hierarchical Dirichlet Process (HDP). However, those are with conventional clustering methods that apply on raw data. Recently, deep learning based clustering methods have drawn more and more attention. A simple two-stage approach is to first learn low-dimensional embeddings using unsupervised feature learning methods, and then perform cluster analysis on the embeddings. However, without any supervision, the representation learning do not necessarily reveal the true cluster structure of the data. DEC BID33 ) is a method that simultaneously learns feature representations and cluster assignments through deep autoencoders. It gradually improves the clustering by driving the deep network to learn a better mapping. Improved Deep Embedded Clustering BID10 improves DEC by keeping the decoder network and adding reconstruction loss to the original clustering loss in DEC. Variational deep embedding BID14 ) is a generative method that models the data generative process using a Gaussian mixture model combined with a VAE, and also performs joint learning of representations and clustering. Similarly, GMVAE BID7 performs joint learning of a GMM and a VAE, but instead generates the mixture components through neural networks. Deep clustering network (DCN) BID34 is another one that jointly learns an autoencoder and performs k-means clustering. These joint learning methods consistently achieve better clustering than conventional ones. The method proposed in BID35 uses convolutional neural networks and jointly learns the representations and clustering in a recurrent framework. All these methods assume flat partitions over the data, and do not attempt the structure learning issue. An exception is hierarchical nonparametric variational autoencoders proposed in BID9. It uses nCRP as the prior for VAE to allow infinitely deep and branching tree hierarchy structure and focuses on learning hierarchy of concepts. However, it is still one partition over the data, only that the partitions in upper levels are more general partitions, while those in lower levels more fine-grained. Different from it, our work focuses on multifacets of clustering, for example, the model could make one partition based on identity of subjects, while another partition based on pose. In this section, we present the proposed latent tree variational autoencoder and the learning algorithms for joint representation learning and structure learning for multidimensional clustering. Deep generative models assume that data x is generated from latent continuous variable z through some random process. The process consists of two steps: a value z is generated from some prior distribution p(z); the observation x is generated from the conditional distribution p θ (x|z), which is parameterized through deep neural networks. Thus, it defines the joint distribution between Generation Network DISPLAYFORM0 Figure 2: Inference and gradient through message passing. Solid-arrows denote collecting message, and dashed-arrows denote distributing message.observation x and latent variable z: DISPLAYFORM1 This process is hidden from our view, and we learn this process by maximizing the marginal loglikelihood p(x) over the parameters θ and latent variable z from data. After the learning, the latent variable z can be regarded as the deep representations of x since it captures the most relevant information of x. Thus, the learning process is also called representation learning. In order to learn the latent structure of z, for example multidimensional cluster structure, we in-. This essentially forms a Bayesian network. And if we restrict the network to be treestructured, the z and Y together form a latent tree model BID37 BID21 BID19 BID20 BID38 with z being the observed variables and Y being the latent variables. For multidimensional clustering, each latent variable Y is taken to be a discrete variable, where each discrete state y of Y defines a cluster. Each latent variable Y thus defines a facet partition over the data based on subset of attributes and multiple Y's define multiple facets. Given a value y of Y, z b follows a conditional Gaussian distribution P (z b |y) = N (µ y, Σ y) with mean vector µ y and covariance matrix Σ y. Thus, each z b and its parent constitute a Gaussian mixture model (GMM). Suppose the parent of a node is denoted as π(·), the maginal distribution of z is defined as follows DISPLAYFORM2 which sums over all possible combinations of Y states. As a matter of fact, a GMM is a Gaussian LTM that has only one latent variable connecting to all observed variables. Let the latent structure of Y be S, defining the number of latent variables in Y, the number of discrete states in each variable Y and the connectivity structure among all variables in z and Y. And let the parameters for all conditional probabilities in the latent structure be Θ. Both the latent structure S and the latent parameters Θ are unknown. We aim to jointly learn data representations and the latent structure. The proposed LTVAE model is shown in Fig. 1. The latent structure S are automatically learned from data and will be discussed in a later section. Due to the existence of the generation network, the inference of the model is intratable. Instead, we do amortized variational inference for the latent variable z by introducing an inference network BID16 and define an approximate posterior q φ (z|x). The evidence lower bound (ELBO) L ELBO of the marginal loglikelihood of the data given (S, Θ) is: DISPLAYFORM3 where log y p S (z, y; Θ) is the marginal loglikelihood of the latent variable z under the latent tree model, and H[·] is the entropy. The conditional generative distribution p θ (x|z) could be a Gaussian distribution if the input data is real-valued, or a Bernoulli distribution if binary, parameterized by the generation network. Using Monte Carlo sampling, the ELBO can be asymptotically estimated by DISPLAYFORM4 where DISPLAYFORM5 can be computed analytically if we choose the form of DISPLAYFORM6, where J is the dimensionality of z. Furthermore, the marginal loglikelihood log y p S (z (i), y; Θ) can be computed efficiently through message passing. Message passing is an efficient algorithm for inference in Bayesian networks BID17 BID22. In message passing, we first build a clique tree using the factors in the defined probability density. Because of the tree structure, each z b along with its parent form a clique with the potential ψ(z b, y) being the corresponding conditional distribution. This is illustrated in Fig. 2. With the sampled z (i), we can compute the message ψ (y) by absorbing the evidence from z. During collecting message phase, the message ψ (y) are sent towards the pivot. After receiving all messages, the pivot distributes back messages towards all z. Both the posterior of Y and the marginal loglikelihood of z (i) thus can be computed in the final normalization step. In this section, we propose efficient learning algorithms for LTVAE through gradient descent and stepwise EM with message passing. Given the latent tree model (S, Θ), the parameters of neural networks can be efficiently optimized through stochastic gradient descent (SGD). However, in order to learn the model, it is important to efficiently compute the gradient of the marginal loglikelihood log p S (z; Θ) from the latent tree model, the third term in Eq. 4. Here, we propose an efficient method to compute gradient through message passing. Let z b be the variables that we want to compute gradient with respect to, and let Y b be the parent node. The marginal loglikelihood of full z can be written as DISPLAYFORM0 where f (y b) is the collection of all the rest of the terms not containing z b. The gradient g z b of the marginal loglikelihood log p S (z; Θ) w.r.t z b thus can be computed as DISPLAYFORM1 where p(y b |z) is the posterior probability of y b and can be computed efficiently with message passing as described in the previous section. The detailed derivation is in Appendix E. DISPLAYFORM2 With the efficient computation of the third term in Eq. 4 and its gradient w.r.t z through message passing, the parameters of inference network and generation network can be efficiently optimized through SGD.In order to jointly learn the parameters of the latent tree Θ, we propose Stepwise EM algorithm based on mini-batch of data. Specifically, we maximize the third term in Eq. 4, i.e. the marginal loglikelihood of z under the latent tree. In the Stepwise E-step, we compute the distributions P (y, y |z, θ (t−1) ) and P (y|z, θ (t−1) ) for each latent node Y and its parent Y. In the Stepwise M-step, we estimate the new parameter θ (t). Let s(z, y) be a vector the sufficient statistics for a single data case. Lets = E p S (y|z;Θ) [s(z, y)] be the expected sufficient statistics for the data case, where the expectation is w.r.t the posterior distribution of y with current parameter. And let µ = N i=1s i be the sum of the expected sufficient statistics. The update of the parameter Θ is performed as follows: DISPLAYFORM3 where η is the learning rate and l is the complete data loglikelihood. Each iteration of update of LTVAE thus is composed of one iteration of gradient descent update for the neural network parameters and one iteration of Stepwise EM update for the latent tree model parameters with a mini-batch of data. For the latent structure S, there are four aspects need to determine: the number of latent variables, the cardinalities of latent variables, the connectivities among variables. We aim at finding the model m * that maximizes the BIC score BID25 BID17: DISPLAYFORM0 where θ * is the MLE of the parameters and d(m) is the number of independent parameters. The first term is known as the likelihood term. It favors models that fit data well. The second term is known as the penalty term. It discourages complex models. Hence, the BIC score provides a tradeoff between model fit and model complexity. To this end, we perform systematic searching to find a structure with a high BIC score. We use the hill-climing algorithm to search for m * as in BID21, and define 7 search operators: node introduction (NI) and node deletion (ND) to introduce new latent nodes and delete existing nodes, state introduction (SI) and state deletion (SD) to add a new state and delete a state for existing nodes, node relocation (NR) to change links of existing nodes, pouching (PO) and unpouching (UP) operators to combine nodes into a single node and separate variables from a node.. The structure search operators are shown in FIG1. Each operator produces a set of candidates from existing structure, and the best candidate is picked if it improves the previous one. To reduce the number of possible search candidates, we first perform SI, NI and PO to expand the structure and pick the best model. Then we perform NR to adjust the best model. Finally, we perform UP, ND and SD to simplify the current best structure and pick the best one. Acceleration techniques BID22 are adopted that make the algorithm efficient enough. The structure learning is performed iteratively together with the parameter learning of neural networks. The overall learning algorithm is illustrated in Algorithm 1. Starting from a pretrained model, we iteratively improve the structure and parameters of latent tree model while learning the representations of data through neural network in a greedy manner. Using current structure S t as the initial structure, Published as a conference paper at ICLR 2019 Compute log pS (z; Θ) and ∂ log p S (z) ∂z from Eq. 5 and 7 Compute ELBO from Eq. 4 θ, φ ← Back-propagation and SGD step DISPLAYFORM0 we search for a better model. With new latent tree model, we optimize for a better representation until convergence. We first demonstrate the effectiveness of the proposed method through synthetic data. Assume that the data points have two facets Y 1 and Y 2, where each facet controlls a subset of attributes (e.g. two-dimensional domain) and defines one partition over the data. This four-dimensional domain z = {z 1, z 2, z 3, z 4} is a latent representation which we do not observe. What we observe is x ∈ R 100 that is obtained via the following non-linear transformation: DISPLAYFORM0 where W ∈ R 10×4 and U ∈ R 100×10 are matrices whose entries follow the zero-mean unit-variance i.i.d. Gaussian distribution, σ(·) is a sigmoid function to introduce nonlinearity. The generative model is shown in FIG2. We define two clusters in facet Y 1 and two clusters in facet Y 2, and generate 5,000 samples of x. Under the above generative model, recovering the two facets Y 1 and Y 2 structure and the latent z domain from the observation of x seems very challenging. All previous DNN-based methods (AE+GMM, DEC, DCN, etc.) are only able to discover one-facet of clustering (i.e. one partition over the data), and none of these is applicable to solve such a multidimensional clustering problem. FIG2 shows the of the proposed method. As one can see, the LTVAE successfully discovers the true superstructure of Y 1 and Y 2. The 2-d plot of z 1 and z 2 shows the separable latent space clusters under facet Y 1, and it matches the ground-truth cluster assignments. Additionally, the 2-d plot of z 3 and z 4 shows another separable clusters under facet Y 2, and it also matches the ground-truth cluster assignments well in the other facet. We evaluate the proposed LTVAE model on two image datasets and two other datasets, and compare it against other deep learning based clustering algorithms, including two-stage methods, AE+GMM and VAE+GMM, which first learn AE/VAE BID16 ) models then construct a GMM on top of them, and joint learning methods, DEC BID33 and DCN BID34. The datasets include MNIST, STL-10, Reuters BID33 BID14 and the Heterogeneity Human Activity Recognition (HHAR) dataset. When evaluating the clustering performance, for fair of comparison, we follow previous works BID33 BID34 and use the network structures of d−500−500−2000−10 for the encoder network and 10−2000− 500 − 500 − d for the decoder network for all datasets, where d is the data-space dimension, which varies among datasets. All layers are fully-connected. We follow the pretraining procedure as in BID33. We first perform greedy layer-wise pretraining in denoising autoencoder manner, then stack all layers to form deep autoencoder. The deep autoencoder is further finetuned to minimize the reconstruction loss. The weights of the deep autoencoder are used to intialize the weights of encoder and decoder networks of above methods. After the pretraining, we optimze the objectives of those methods. For DEC and DCN, we use the same hyperparameter settings as the original papers. When initializing the cluster centroids for DEC and DCN, we perform 10 random restarts and pick the with the best objective value for k-means/GMM. For the proposed LTVAE, we use Adam optimzer BID15 with initial learning rate of 0.001 and mini-batch size of 128. For Stepwise EM, we set the learning rate to be 0.01. As in Algorithm 1, we set E = 5, i.e. we update the latent tree model every 5 epochs. When optimizing the candidate models during structure search, we perform 10 random restarts and train with EM for 200 iterations. We first show that, by using the marginal loglikelihood defined by the latent tree model as the prior, LTVAE better fits the data than conventional VAE and importance weighted autoencoders (IWAE) BID3. While alternative quantitative criteria have been proposed BID2 BID13 BID24 for generative models, log-likelihood of held-out test data remains one of the most important measures of a generative model's performance BID16 BID3 BID32 BID9. For comparison, we approximate true loglikelihood L 5000 using importance sampling BID3: DISPLAYFORM0, where z (i) ∼ q φ (z|x). The for all datasets are shown in TAB0. The proposed LTVAE obtains a higher test data loglikelihood and ELBO, implying that it can better model the underlying complex data distribution embedded in the image data. The most important features of the proposed model are that it can perform variable selection for model-based clustering, leading to multiple facets clustering. We use the standard unsupervised evaluation metric and protocols for evaluations and comparisons to other algorithms BID36. For baseline algorithms we set the number of clusters to the number of ground-truth categories. While for LTVAE, it automatically determines the number of facets and latent superstructure through structure learning. We evaluate performance with unsupervised clustering accuracy (ACC): DISPLAYFORM0 where l i is the groundtruth label, c i is the cluster assignment produced by the algorithm, and m ranges over all possible mappings between clusters and labels. TAB1 show the quantitative clustering compared with previous works. With z dimension of small value like 10, LTVAE usually discovers only one facet. It can be seen the, for MNIST dataset LTVAE achieves clustering accuracy of 86.32%, better than the of other methods. This is also the case for STL-10, Reuters and HHAR.More importantly, the proposed LTVAE does not just give one partition over the data. Instead, it explains the data in multi-faceted ways. Unlike previous clustering experiments, for this experiment, we choose the z dimension to be 20. FIG3 shows the two facet clustering for MNIST. It can be seen that facet 1 gives quite clean clustering over the identity of the digits and the ten digits are well separated. On the other hand, facet 2 gives a more grand partition based on the shape and pose. Note how up-right "4" and "9" are similar, and how tilted "4","7" and "9" are similar. The facet meanings are more evident in FIG3. FIG4 shows four facets discovered for the STL-10 The digits generated by the proposed model. Digits in the same row come from the same latent code of the latent tree.dataset. Although it is hard to characterize precisely how the facets differ from each other, there are visible patterns. For example, the cats, monkeys and birds in facet 2 have clearly visible eyes, while this is not always true in facet 1. The deers in facet 2 are all showing their antlers/ears, while this is not true in facet 3. In facet 2 we see frontal views of cars, while in facets 1 and 3 we see side view of cars. In facet 1, each cluster consists of the same types of objects/animal. In facet 3/4, images in the same cluster do not necessarily show the same type of objects/animals. However, they have similar overall feel. Since the structure of the data in latent space is automatically learned through the latent tree, we can sample the data in a more structured way. One way is through ancestral sampling, where we first sample the root of the latent tree and then hierarchically sample the children variables to get z, from which the images can be generated through generation network. The other way is to pick one component from the Gaussian mixture and sample z from that component. This produces samples from a particular cluster. FIG5 shows the samples generated in this way. As it can be seen, digits sampled from each component has clear semantic meaning and belong to the same category. Whereas, the samples generated by VAE does not have such structure. Conditional image generation can also be performed to alter the attributes of the same digit as shown in Appendix B. LTVAE learns the dependencies among latent variables Y. In general, latent variables are often correlated. For example, the social skills and academic skills of a student are generally correlated. Therefore, its better to model this relationship to better fit the data. Experiments show that removing such dependencies in LTVAE models in inferior data loglikelihood. In this paper, for the inference network, we simply use mean-field inference network with same structure as the generative network BID16. However, the limited expressiveness of the mean-field inference network could restrict the learning in the generative network and the quality of the learned model BID31 BID6. Using a faithful inference network structure as in BID31 to incorporate the dependencies among latent variables in the posterior, for example one parameterized with masked autoencoder distribution estimator (MADE) model BID8, could have a significant improvement in learning. We leave it for future investigation. In this paper, we propose an unsupervised learning method, latent tree variational autoencoder (LT-VAE), which simultaneously performs representation learning and multidimensional clustering. Different from previous deep learning based clustering methods, LTVAE learns latent embeddings from data and discovers multi-facet clustering structure based on subsets of latent features rather than one partition over data. Experiments show that the proposed method achieves state-of-the-art clustering performance and reals reasonable multifacet structures of the data. For the MNIST dataset, the conditional probability between identity facet Y 1 (x-axis) and pose facet Y 2 (y-axis) is shown in Fig. 8. It can be seen that a cluster in Y 1 facet could correspond to multiple clusters in Y 2 facet due to the conditional probability, e.g. cluster 0, 4, 5, 11 and 12. However, not all clusters in Y 2 facet are possible for a given cluster in Y 1 facet. Figure 8: Conditional probability of Y 1 and Y 2 for the two facets of MNIST discovered by LTVAE. Here we show more on conditional image generation. Interestingly, with LTVAE, we can change the original images by fixing variables in some facet and sampling in other facets. For example, in MNIST we can fix the variables in identity facet and change the pose of the digit by sampling in the pose facet. FIG6 shows the samples generated in this way. As it can be seen, the pose of the input digits are changed in the samples generated by the proposed method. C COMPUTATIONAL TIMEWe compare the computational time of the proposed LTVAE w/ structure learning and that w/ fixed structure. For LTVAE with fixed structure, we fixed the structure of the latent tree model to be a single Y connecting to all zs, in which each z node consists of single z variable.

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.

represent the tldr for finding the corresponding scientific article

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model.


Question: Represent the article for finding a single-sentence TLDR

Question: Many practical robot locomotion tasks require agents to use control policies that can be parameterized by goals. Popular deep reinforcement learning approaches in this direction involve learning goal-conditioned policies or value functions, or Inverse Dynamics Models (IDMs). IDMs map an agent’s current state and desired goal to the required actions. We show that the key to achieving good performance with IDMs lies in learning the information shared between equivalent experiences, so that they can be generalized to unseen scenarios. We design a training process that guides the learning of latent representations to encode this shared information. Using a limited number of environment interactions, our agent is able to efficiently navigate to arbitrary points in the goal space. We demonstrate the effectiveness of our approach in high-dimensional locomotion environments such as the Mujoco Ant, PyBullet Humanoid, and PyBullet Minitaur. We provide quantitative and qualitative to show that our method clearly outperforms competing baseline approaches. In reinforcement learning (RL), an agent optimizes its behaviour to maximize a specific reward function that encodes tasks such as moving forward or reaching a target. After training, the agent simply executes the learned policy from its initial state until termination. In practical settings in robotics, however, control policies are invoked at the lowest level of a larger system by higher-level components such as perception and planning units. In such systems, agents have to follow a dynamic sequence of intermediate waypoints, instead of following a single policy until the goal is achieved. A typical approach to achieving goal-directed motion using RL involves learning goal-conditioned policies or value functions . The key idea is to learn a function conditioned on a combination of the state and goal by sampling goals during the training process. However, this approach requires a large number of training samples, and does not leverage waypoints provided by efficient planning algorithms. Thus, it is desirable to learn models that can compute actions to transition effectively between waypoints. A popular class of such models is called Inverse Dynamics Model (IDM) . IDMs typically map the current state (or a history of states and actions) and the goal state, to the action. In this paper, we address the need of an efficient control module by learning a generalized IDM that can achieve goal-direction motion by leveraging data collected while training a state-of-the-art RL algorithm. We do not require full information of the goal state, or a history of previous states to learn the IDM. We learn on a reduced goal space, such as 3-D positions to which the agent must learn to navigate. Thus, given just the intermediate 3-D positions, or waypoints, our agent can navigate to the goal, without requiring any additional information about the intermediate states. The basic framework of the IDM is shown in Fig. 1. The unique aspect of our algorithm is that we eliminate the need to randomly sample goals during training. Instead, we exploit the known symmetries/equivalences of the system (as is common in many robotics settings) to guide the collection of generalized experiences during training. We propose a class of algorithms that utilize the property of equivalence between transitions modulo the difference in a fixed set of attributes. In the locomotion setting, the agent's transitions are symmetric under translations and rotations. We capture this symmetry by defining equivalence modulo orientation among experiences. We use this notion of equivalence to guide the training of latent representations shared by these experiences and provide them as input to the IDM to produce the desired actions, as shown in Fig. 4. A common challenge faced by agents trained using RL techniques is lack of generalization capability. The standard way of training produces policies that work very well on the states encountered by the agent during training, but often fail on unseen states. Achieving good performance using IDMs requires both these components: collecting generalized experiences, and learning these latent representations, as we demonstrate in Section 6. Our model exhibits high sample efficiency and superior performance, in comparison to other methods involving sampling goals during training. We demonstrate the effectiveness of our approach in the Mujoco Ant environment in OpenAI Gym , and the Minitaur and Humanoid environments in PyBullet . From a limited number of experiences collected during training under a single reward function of going in one direction, our generalized IDM succeeds at navigating to arbitrary goal positions in the 3-D space. We measure performance by calculating the closest distance to the goal an agent achieves. We perform ablation experiments to show that collecting generalized experience in the form of equivalent input pairs boosts performance over all baselines, these equivalent input pairs can be condensed into a latent representation that encodes relevant information, and learning this latent representation is in fact critical to success of our algorithm. Details of experiments and analysis of can be found in Sections 5 and 6. Several recent works learn policies and value functions that are conditioned over not just the state space, but also the goal space (; ;) and then generalize those functions to unseen goals. Goal-conditioned value functions are also largely used in hierarchical reinforcement learning algorithms , where the higher level module learns over intrinsic goals and the lower level control modules learn subpolicies to reach those goals, or the lower level control modules can efficiently execute goals proposed by the higher-level policy . use trained goalconditioned policies to learn actionable latent representations that extract relevant information from the state, and use these pretrained representations to train the agent to excel at other tasks. Pong* et al. learn goal-conditioned value functions, and use them in a model-based control setting. IDMs are functions that typically map the current state of the agent and the goal state that the agent aims to achieve, to the desired action. They have been used in a wide variety of contexts in existing literature. train IDMs using a history of states and actions and full goal state information for transferring models trained in simulation to real robots. and use IDMs in combination with Forward Dynamics Models (FDMs) to predict actions from compressed representations of high-dimensional inputs like RGB images generated by the FDM. use IDMs to provide a curiosity-based reward signal Figure 2: We report the of our algorithm and baselines on the three locomotion environments shown above: Humanoid, Minitaur, and Ant. The red sphere indicates the goal in each environment. in the general RL framework to encourage exploration; use IDMs to provide supervision for the learning of visual features relevant to the task assigned to the robot. We circumvent the need to learn goal-conditioned policies or value functions by combining IDMs with known symmetric properties of the robot. We train an IDM conditioned on the state space and a reduced goal space, using data collected while training any state-of-the-art RL algorithm. Our data collection is unique in that we exploit equivalences in experiences observed during training and learn a latent representation space shared between such equivalent experiences. Our IDM produces the action given this latent representation as an input, leading to generalization over parts of the state and goal spaces unobserved during training. In the general RL framework , a learning agent interacts with an environment modeled as a Markov Decision Process consisting of: 1) a state space S, 2) an action space A, 3) a probability distribution P: S × S × A →, where P (s |s, a) is the probability of transitioning into state s by taking action a in state s, 4) a reward function R: S × A × S → R that gives the reward for this transition, and 5) a discount factor Γ. The agent learns a policy π θ: S → A, parameterized by θ while trying to maximize the discounted expected return. Goal-conditioned RL optimizes for learning a policy that maximizes the return under a goal-specific reward function R g. On-policy RL algorithms, such as Policy Gradient methods and Trust Region methods use deep neural networks to estimate policy gradients, or maximize a surrogate objective function subject to certain constraints. Off-policy RL algorithms; ) incorporate elements of deep Q-learning into the actor-critic formulation. Hindsight Experience Replay (HER) is a popular technique used in conjunction with an off-policy RL algorithm to learn policies in a sample-efficient way from sparse rewards in goal-based environments. Our method leverages samples collected while training a state-of-the-art RL algorithm to train an IDM that maps the current state and desired goal position to the action required to reach the goal. There are four major components involved in this process: 1) collecting data while training the RL algorithm, 2) learning a basic IDM that maps the current state and the desired goal to the required action, 3) collecting experiences that are equivalent to those observed, and using them to train the IDM, and 4) learning a latent representation that generalizes this model to unseen parts of the state space by utilizing the equivalent experiences collected in step 3. We elaborate on each of these in the following sections. Our goal in this steps is to collect data for our IDM in the process of learning a policy under a single reward function. Recall the motivation for learning IDMs: we want a model that can take in the current state of the agent and the next command, in the form of a location in the space that the agent should travel to. Thus, we collect state, action, and position data from the transitions observed during the training process. We emphasize the difference between the state space S and the goal space O. The state space S is high-dimensional, consisting of information related to joint angles, velocities, torques, etc. The goal space, O, is low-dimensional, consisting, in this case, of the 3-D coordinates of the goal position that the agent is supposed to navigate to. Definition 1 (Experiences). We define experiences as tuples (s, o, o, a), where s is the current state of the agent, o is its current 3-D position, and a is the action that the agent performed to move from the state s and position o to the next position, or intermediate goal, o. We write..,T to denote all the experience tuples collected from a trajectory τ of length T. Given a set of experiences E, we can train the IDM using supervised learning techniques. Definition 2 (Inverse Dynamics Model). We define the Inverse Dynamics Model (IDM) as where s, o, o and a represent the current state, current position, desired goal, and action respectively. The IDM can reproduce seen actions on state and observation tuples that have appeared in the training data. However, it can not generalize good behaviour to states and observations that have not appeared in the initial training (see Fig. 6 for qualitative evidence). Our aim in the next steps is to generalize the observed experiences so that they can be used over previously unseen inputs from the S × O × O space. One issue with the current hypothesis of collecting data for a single reward is: the samples we obtain are highly biased in that they predominantly contain samples for motion in one direction. As a , there are certain parts of the input space that our agent is unlikely to ever visit. So it is unreasonable to expect it to generalize its behaviour in those parts. We can see qualitative evidence in Fig. 6, where the Humanoid can navigate to goals seen during training time (6a) using the basic IDM (corresponding method in the plots is RL), but fails when the goal lies outside the training distribution (6b). In order to mitigate this bias in the state space introduced by single reward function, we collect generalized experience comprising experience tuples that are equivalent modulo orientation (e.m.o.) with respect to actions. We use Θ to represent the orientation space. Definition 3 (Equivalence modulo Orientation). For e 1, e 2 ∈ E, e 1 and e 2 are e.m.o. with respect to A =⇒ A(e 1) = A(e 2). This defines an equivalence mod Θ with respect to The LFM generates the latent representations for two e.m.o. samples. We minimize the distance between them to enforce equivalence, and also fit the IDM to predict the action correctly, given these latent representations. Definition 4 (Generalized Experience). We collect unseen experiences equivalent to the observed experiences by taking a trajectory τ ⊆ E, changing the initial orientation of the agent, leading to a different s 0, and repeating the same set of actions observed in τ. We denote this operation by G, so G(τ) is a new trajectory. The full generalized experience set obtained in this way is written as G = E ∪ G(E). s 0 is an unseen state that has not appeared while training the agent. Despite using generalized experiences during training, the IDM does not always show great improvements in tasks like navigating to the desired goal position in an arbitrary direction, as seen in Table 1. We hypothesize that this is due to the agent failing to recognize e.m.o. experiences, and instead learning actions from irrelevant attributes of the state space. We use the e.m.o. experiences from the generalized experience set G to train a Latent Feature Model ψ that discards irrelevant information from the state, and learns only shared information relevant to the IDM to produce actions. Definition 5 (Latent Feature Model). Our Latent Feature Model (LFM) aims to learn the equivalence between e.m.o. experiences from the generalized experience set G. We define the LFM ψ as where γ is a k-dimensional latent representation of the experience sample. We then modify the IDM φ to produce the action from this latent representation as In order to learn these two models: LFM and IDM, we need our objective function to incorporate the property of equivalence modulo actions in the latent representations, and learn a good mapping from these representations to actions. Since the LFM is used to generate latent representations for two e.m.o. experience samples simultaneously, and then optimize their distance, we use a Siamese framework to model ψ. Our objective L 1 minimizes the distance between the latent representations generated for e.m.o. experience samples. we use a simple regression loss such as the L2 distance to fit the output of the IDM to the action. Here, the input to the IDM is the mean of the latent representations generated for the e.m.o. experiences. We jointly learn these two models by minimizing a weighted loss function: Fig. 4 shows the training procedure we use for our IDM. Each e.m.o. sample pair from equivalent trajectories is passed as input to the LFM, which generates the latent representations for the pair. The mean of these latent representations is then passed as input to the IDM, which predicts the action to be taken. These two models are trained simultaneously, ing in rich latent representations that can achieve goal-directed motion, generalizable to any arbitrary location. Remark 1. It is important to note that at test time, only the current state and goal are passed to the LFM to generate the latent representation, which is used by the IDM to predict the required action. We demonstrate the effectiveness of our overall approach by performing a series of ablation experiments, successively including each major component of our algorithm. In addition to a random baseline, we use Vanilla Goal-Conditioned Policy, and Hindsight Experience Replay as baselines for comparison with our algorithm. We demonstrate superior using our algorithm on three locomotion environments: Mujoco Ant environment in OpenAI Gym , and Humanoid and Minitaur environments in PyBullet . In order to fairly evaluate performance, we train each baseline using the same number of environment interactions as our algorithm. We also maintain uniform seeds, initial states and goals across all methods. The details of the test setting, along with network architectures, learning rates, and other hyperparameters, are discussed in the Appendix. Since our method aims at achieving goal-directed motion, we compare it with other on-policy and off-policy RL algorithms that are trained for this specific purpose. In this baseline experiment, we collect (s, o, o, a) samples by taking random actions at each step. We use these samples to train the IDM. Vanilla Goal-Conditioned Policy (VGCP): The second baseline algorithm we consider is VGCP, which takes as input the state and desired goal, and learns a policy on this input space using any state-of-the-art RL algorithm. The policy is given by π VGCP: S, O → A and is learnt using a stateof-the-art model-free RL technique. We use Proximal Policy Optimization (PPO) ( ) for Ant and Minitaur, and Soft Actor-Critic (SAC) ( ) for Humanoid. We select Soft Actor Critic (SAC) and Deep Deterministic Policy Gradient (DDPG) ) as the off-policy algorithms used in conjunction with HER for our algorithms. We report on HER with both sparse and dense rewards. Sparse rewards indicate whether the target was successfully reached, and dense rewards include this information, in addition to control and contact costs. Throughout the paper, HER-Sp refers to HER with sparse rewards, and HER-De refers to HER with dense rewards. Collecting Experience using standard RL algorithm (RL): We collect samples while training state-of-the-art RL algorithms rewarding them for going to the right (as is common in locomotion environments), and use them as the training data for our IDM. More details are listed in the Appendix. Collecting Generalized Experience (GE): We collect generalized experiences in the following manner: we save the trajectories followed by the agent while it learns a policy for locomotion. For some/all of these trajectories (details in A.1.2), we rotate the initial state of the agent by a random angle, and repeat the actions taken in the original trajectory. The samples collected in this modified trajectory are e.m.o. to those in the original trajectory. All of these samples constitute generalized experiences, which we use to train the IDM. Table 1: Quantitative of our methods and all baselines on the three environments: Humanoid, Minitaur, and Ant respectively. In each environment, LR emerges as clearly the best performer, enabling the agent to navigate closest to the goal. We use the generalized experiences collected in the previous step to extend the learned behaviour of the agent to unseen parts of the state space. We use the dual network architecture shown in Fig. 4 in this experiment. We jointly train the LFM and IDM using a weighted loss function that minimizes the distance between the latent representations generated for e.m.o. experiences, and fits the output of the IDM to the desired actions. Remark 2. We preprocess (s, o, o) to (s, d) where d is the unit vector in the direction o → o, and provide it as input to the models. We also show on a task in which the agent has to navigate through a series of waypoints, to see if our IDM has indeed learnt the right actions. There are three questions we wish to answer through this experiment: 1) How much does the agent deviate from the intended optimal trajectory through the intermediate goals/waypoints? 2) How fast is the agent able to accomplish each goal? 3) What is the agent's style/gait of walking? For this qualitative comparison, we found that neither HER nor VGCP with the current setting was able to generate good trajectories. Thus, we trained VGCP on 20 million samples (our method uses 15 million environment samples). We use this trained policy to generate the VGCP trajectories in Fig. 7, and our LR model (trained using 15 million samples) for the LR trajectories. For all trajectories, the agents are given the same initial orientation and goal position to ensure fair comparison. In this section, we analyze the performances of our algorithm and baselines qualitatively and quantitatively. In particular, we discuss the following: 1) overall performances of all methods, 2) distribution of distances to the goal observed at test time for all methods, 3) comparison between performance of best baseline and LR on the waypoint navigation task, and 4) analysis of speed, trajectory, and walking style observed in LR and the best baseline on Humanoid. The first two points are addressed in the following two subsections. The next two questions are discussed in detail in Fig. 7. We report the closest distance from the target that the agent is able to reach, as the evaluation metric for all our experiments. The mean and standard deviation of closest distance are reported in Table 1. It is clear that for each of the three environments, GE and LR outperform all other baselines. In particular, LR is observed to be the best performing algorithm for each environment. This lends validation to our claim that learning latent representations shared between equivalent experiences indeed boosts performance, instead of treating the equivalent experiences as independent training samples. The in table 1 show that for the Minitaur and Humanoid environments, the performance of VGCP is better than most baselines, and nearly comparable to GE. However, it shows poor performance on the Ant environment. This anomaly arises from the fact that 2 million samples are not enough to train a goal-conditioned policy for the Ant. The Minitaur and Humanoid, on the other hand, are trained for 4 million and 15 million time steps respectively, thus enabling better goal-conditioned policies to be learnt, leading to much superior performance. For each of the environments, we observe that HER-Sp and HER-De both show poor performance, compared to that of GE, LR, and even VGCP. In this section, we analyze the violin plots in Fig. 5. These plots show the distributions of the closest distances from targets observed over 10,000 episodes for each algorithm. For each environment, we see that for RL, and in some cases, VGCP, HER, and GE, there is a small peak away from the mean, which gives a much lower distance than the mean distance. This suggests that there is a significant number of episodes in which the Humanoid reaches very close to the target. We analyze this discrepancy in performance qualitatively in 6, and conclude that the small peak consists of episodes in which the initial state and goal position have been observed during training, and thus, the goal-conditioned policy has already learnt the optimal actions for this scenario. Across all environments, Ant and Minitaur in particular, the peak of the LR distribution is much lower than the other methods. This shows that in most episodes, the agent can successfully navigate to the goal; the slightly higher mean value and variance are due to the small number of episodes in which the agent fails at the beginning of the episode. This is to be expected because the actions taken by the IDM depend on the kind of data collected during the training process. In some cases, the model may learn the wrong actions, leading to the agent dying early in those episodes. Figure 7: Given a series of waypoints, we check the ability of our Humanoid to navigate through them efficiently, and compare it with the best baseline, VGCP, trained on 20 million samples of environment interaction (our algorithm uses 15 million). (a) We plot the trajectory followed by each agent as it navigates through the waypoints. We see that LR can navigate through all goals much more easily than VGCP, without deviating. In addition to this, LR enables the agent to reach each goal much faster than VGCP, which is unable to reach the last goal due to its slow speed, as the episode terminates in a fixed number of timesteps. (b) We plot the orientation of the agent at uniform intervals throughout the episode. We observe that while the agent trained using LR walks forward as expected, the agent trained using VGCP, while managing to move slowly closer to the goal, does so in an arbitrary fashion in that its orientation is seldom in the direction of its motion. We also provide qualitative evidence to prove that the distribution of closest distances in episodes is indeed biased by initial state and target configurations seen in the training data. We fix the initial state of the agent and generate goals in a specific region. We plot the of 10 episodes for each method in Fig. 6 for the Humanoid environment. The initial state and goal configurations we show for, consist of those experienced during training (left), and those not experienced during training (right). We see that for the configurations experienced at training time, most baseline methods, and our methods, GE and LR, are able to reach close to the goal. However, when the configuration lies outside the training distribution, only LR (sometimes GE) can navigate to the goal. We include more in the Appendix. We propose a new algorithm to achieve goal-directed motion for a variety of locomotion agents by learning the inverse dynamics model on shared latent representations for equivalent experiences. To this end, we take three important steps: We utilize the experience collected by our agent while training a standard reinforcement learning algorithm, so that our IDM has "good" samples in which the agent walks reasonably well. We generalize this experience by modifying the initial configuration for each observed trajectory in the collected data, and generate the equivalent trajectories. We learn the important shared information between such symmetric pairs of experience samples through a latent representation that is used as an input to the IDM to produce the action required for the agent to reach the goal. We provide extensive qualitative and quantitative evidence to show that our methods surpass existing methods to achieve generalization over unseen parts of the state space. , and Humanoid and Minitaur environments in PyBullet . In each of these environments, for our methods, the agent is trained to perform well on the 3-D locomotion task in one direction. For the baseline methods, the agent is trained to reach goals generated in different 3-D positions. The reward function for collecting data for RL, GE and LR includes contact and control costs, and rewards for moving forward. The reward function for training VGCP includes contact and control costs, but instead of moving forward, the agent is encouraged to move in the direction of the goal. In the case of HER-Sparse, the agent only receives a reward 0 if it reaches the goal, and -1 otherwise. For HER-Dense, the agent receives a weighted sum of distance to the goal and contact and control costs as its reward. Humanoid Humanoid is a bipedal robot, with a 43-D state space and 17-D action space. The state contains information about the agent's height, orientation (yaw, pitch, roll), 3-D velocity, and the 3-D relative positions of each of its joints (knees, shoulders, etc.). The action consists of the torques at these joints. Minitaur Minitaur is a quadrupedal robot, with a 17-D state space and 8-D action space. The state contains information about motor angles, torques, velocities, and the orientation of the base. The action consists of the torque to be applied at each joint. Ant Ant is a quadrupedal robot, with a 111-D state space and 8-D action space. The state space contains the agent's height, 3-D linear and angular velocity, joint velocities, joint angles, and the external forces at each link. The action consists of the torques at all the 8 joints. Our choice of these locomotion environments is driven by the motivation provided earlier: we want an agent to navigate to a desired goal position in the 3-D space. In view of this, we do not use 2-D locomotion environments like Half-Cheetah, Walker, or Hopper. Also, though Reacher and Pusher are goal-based environments, we do not use them as they are manipulation environments, and we are aiming to achieve navigation by agents trained on 3-D locomotion tasks. The details of number of environment interactions, network architectures for the IDM, LFM, policy, and the state-of-the-art RL algorithm used to train the policy (for baselines), or collect data (for our methods), are given in Table 2. We use the Adam optimizer with a learning rate of 1e-3 and a batch size of 512 for all our methods across all environments. For the Humanoid environment, we observed that the first ∼ 5 million samples were predominantly bad samples that led to the Humanoid falling down very early in each episode. Thus, we froze training after 5 million samples, when we had a policy in which the agent was able to walk a few steps. We used the policy at that point to collect 10 million steps of environment interaction. For the Ant and Minitaur environments, we collected 2 million and 4 million steps of environment interactions respectively, during training. For RL, we used all steps encountered in on-policy data. For GE and LR, we used only the first half of training samples encountered while training the policy. The second half of the samples is generated by applying the generalized transformation G, explained in Section 4.3. We use the first 1 million, 2 million and 5 million samples collected for the Ant, Minitaur and Humanoid (trained) respectively, and generate the rest by applying G. The hyperparameters involved in training the LFM and IDM are: the dimension k of the latent representation space, and the ratio of the latent representation loss (L 1) to the regression loss (L 2), λ. We set λ = 0.25 for all 3 environments. We use k = 10 for the Ant and Minitaur environments, and k = 50 for the Humanoid environment. We discuss the impact of the latent representation dimension k on the performance of the model in the next section. At test time, the agent is rotated by a random angle. The target is set at a distance of ∼ 2 − 5 units from the agent at an angle of [−45 •, 45 •] to the agent's orientation in the case of the Ant and Humanoid environments. For Minitaur, we set the target at a distance of ∼ 1.5 − 2.5 units from the agent at an angle of [−45 •, 45 •] to the agent's orientation. The intermediate goals, or waypoints, can be provided by any planning algorithm, since our approach is agnostic to the actual planning algorithm used. In our work, we use Model Predictive Control, i.e. we replan at each step. Each episode consists of a maximum of 1000 steps for each environment, and the episode terminates when the agent reaches the goal, or falls down/dies. We report the closest distance from the target that the agent is able to reach, for each episode. We select 10 random seeds and test the performance of each method on 1000 episodes for each random seed. In order to ensure that the comparison between all methods is indeed fair, we set the initial configuration of the agent and the target to be the same across all methods at test time. Why does our method perform better than baselines in spite of using inferior samples? It is important to note that since we take only the first half of the environment interactions for GE and LR, we are learning from essentially inferior samples consisting of actions that may not be optimal, as they have been encountered earlier in the training process. In spite of learning from inferior actions, our method outperforms the baselines. This is because our IDM formulation enables the learning of actions that can reach the next goal given the current state, which is the kind of behaviour we require in goal-directed motion. The focus of this model is more on learning actions that caused certain transitions, not on learning the most optimal actions that achieve the transition, although that would be the next best thing to do. In addition to this, our LFM framework enables the agent to achieve their equivalent transitions by producing the same action for all transitions that share a common latent representation. This enables our agent to generalize its motion to a larger part of the goal space in comparison with baselines that struggle to achieve goal-directed motion for goals or states lying outside the training distribution. Why is our method more sample-efficient than the baselines? Our methods (GE and LR) do not utilize the latter half of samples collected during the training process, which would consist of more optimal actions, because that would imply using a higher number of actual environment interactions than the baselines. In spite of this, our method outperforms all baselines for all environments. This is because goal-conditioned policies explore the state and goal spaces to achieve goal-directed motion. While exploration is in fact a desirable component in reinforcement learning, sampling from the goal space during training requires the agent to acquire a sense of direction and the skill of locomotion simultaneously. This enlarged state space, comprising the original state and the goal, leads to an increase in the number of samples required by the agent to learn a good policy. Our latent representations, on the other hand, encode the property of equivalence modulo orientation with respect to actions, described in Section 4.3. This enables our method to generalize representations of some samples of training experience to other parts of the state and goal spaces, which have not been encountered during training. As a , the IDM learns to predict actions for states and goals it has never encountered before, if they have the same latent representation as those seen during training, thus boosting performance while remaining sample-efficient. Why does LR show qualitatively better than VGCP? VGCP is trained as a function of the state and goal space, and produces the action that the agent should take in order to reach the goal. The agent is rewarded for moving towards the goal, keeping control and contact costs minimal. There is no constraint on the type of motion exhibited by the agent, or the time taken by the agent to reach the goal. As a , we see in Fig. 7 that the agent trained using VGCP walks very slowly throughout both episodes, and follows a non-optimal path to navigate through 3 out of the 4 waypoints. Due to its slow speed, it is unable to reach the last waypoint. The LR agent, on the other hand, learns an IDM from samples collected while training on the locomotion task. Since the locomotion reward encourages walking forward while also keeping control and contact costs minimal, the LR agent always walks forward facing each waypoint, and follows a smooth trajectory through all the waypoints, validating the superior performance of our IDM. We see that though the LR and VGCP agents are initialized in the same orientation, the VGCP agent walks with its back facing the waypoints, while the LR agent successfully adopts the optimal path and navigates through all waypoints adopting a "natural" walking style. Why does HER not perform as well as other baselines? If we revisit the motivation behind the HER algorithm, we realize that it shows superior performance using fewer training samples over environments that have low-dimensional state and action spaces, and a sparse reward setting. In our case, we are dealing with high-dimensional locomotion agents that require specific actions to walk, and navigate to the goal. If we use the sparse reward setting, most samples in the replay buffer consist of failures, as the agent first needs to learn to walk, after which it can successfully navigate to the goal. Even with the dense reward setting, there is no significant improvement in performance because of this reason. HER performs very well on low-dimensional sparse-reward environments but it is difficult to extend this behaviour to higher dimensions or dense rewards, especially where complex locomotion skills have to be learned in order to navigate to the goal. How important is the latent representation dimension k in achieving good performance? The dimension of the latent representation k is perhaps the most important hyperparameter that determines the performance of LR. Choosing a bad value of k could in information loss if k dimensions are not sufficient to encode the relevant information for producing the action, or irrelevant information being encoded if k is too high. In our experiments, we use k = 10 for Ant and Minitaur, and k = 50 for Humanoid. We show the impact of k on performance in the Ant environment in Fig. 9. We see that performance improves as we keep reducing the dimension up until k = 10. As we try to reduce further, performance suddenly drops, implying loss of information. Thus, the Ant's 111-D state and 3-D goal are reduced to a 10-D representation that is used by the IDM to produce actions to navigate to the goal.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that the key to achieving good performance with IDMs lies in learning latent representations to encode the information shared between equivalent experiences, so that they can be generalized to unseen scenarios.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we first identify \textit{angle bias}, a simple but remarkable phenomenon that causes the vanishing gradient problem in a multilayer perceptron (MLP) with sigmoid activation functions. We then propose \textit{linearly constrained weights (LCW)} to reduce the angle bias in a neural network, so as to train the network under the constraints that the sum of the elements of each weight vector is zero. A reparameterization technique is presented to efficiently train a model with LCW by embedding the constraints on weight vectors into the structure of the network. Interestingly, batch normalization can be viewed as a mechanism to correct angle bias. Preliminary experiments show that LCW helps train a 100-layered MLP more efficiently than does batch normalization. Neural networks with a single hidden layer have been shown to be universal approximators BID6 BID8. However, an exponential number of neurons may be necessary to approximate complex functions. A solution to this problem is to use more hidden layers. The representation power of a network increases exponentially with the addition of layers BID17 BID2. A major obstacle in training deep nets, that is, neural networks with many hidden layers, is the vanishing gradient problem. Various techniques have been proposed for training deep nets, such as layer-wise pretraining BID5, rectified linear units BID13 BID9, variance-preserving initialization BID3, and normalization layers BID7 BID4.In this paper, we first identify the angle bias that arises in the dot product of a nonzero vector and a random vector. The mean of the dot product depends on the angle between the nonzero vector and the mean vector of the random vector. We show that this simple phenomenon is a key cause of the vanishing gradient in a multilayer perceptron (MLP) with sigmoid activation functions. We then propose the use of so-called linearly constrained weights (LCW) to reduce the angle bias in a neural network. LCW is a weight vector subject to the constraint that the sum of its elements is zero. A reparameterization technique is presented to embed the constraints on weight vectors into the structure of a neural network. This enables us to train a neural network with LCW by using optimization solvers for unconstrained problems, such as stochastic gradient descent. Preliminary experiments show that we can train a 100-layered MLP with sigmoid activation functions by reducing the angle bias in the network. Interestingly, batch normalization BID7 can be viewed as a mechanism to correct angle bias in a neural network, although it was originally developed to overcome another problem, that is, the internal covariate shift problem. Preliminary experiments suggest that LCW helps train deep MLPs more efficiently than does batch normalization. In Section 2, we define angle bias and discuss its relation to the vanishing gradient problem. In Section 3, we propose LCW as an approach to reduce angle bias in a neural network. We also present a reparameterization technique to efficiently train a model with LCW and an initialization method for LCW. In Section 4, we review related work; mainly, we examine existing normalization techniques from the viewpoint of reducing the angle bias. In Section 5, we present empirical that show that it is possible to efficiently train a 100-layered MLP by reducing the angle bias using LCW. Finally, we conclude with a discussion of future works. We introduce angle bias by using the simple example shown in FIG1. FIG1 (a) is a heat map representation of matrix W ∈ R 100×100, each of whose elements is independently drawn from a uniform random distribution in the range (−1, 1). Matrix A ∈ R 100×100 is also generated randomly, and its elements range from 0 to 1, as shown in FIG1 (b). We multiply W and A to obtain the matrix shown in FIG1 (c). Unexpectedly, a horizontal stripe pattern appears in the heat map of W A although both W and A are random matrices. This pattern is attributed to the angle bias that is defined as follows: Definition 1. P γ is an m dimensional probability distribution whose expected value is γ1 m, where γ ∈ R and 1 m is an m dimensional vector whose elements are all one. Proposition 1. Let a be a random vector in R m that follows P γ. Given w ∈ R m such that w > 0, the expected value of w · a is |γ| √ m w cos θ w, where θ w is the angle between w and 1 m. where E(x) denotes the expected value of random variable x. Definition 2. From Proposition 1, the expected value of w · a depends on θ w as long as γ = 0. The distribution of w · a is then biased depending on θ w; this is called angle bias. In FIG1, if we denote the i-th row vector of W and the j-th column vector of A by w i and a j, respectively, a j follows P γ with γ = 0.5. The i-th row of W A is biased according to the angle between w i and 1 m, because the (i, j)-th element of W A is the dot product of w i and a j. Note that if the random matrix A has both positive and negative elements, W A also shows a stripe pattern as long as each column vector of A follows P γ with γ = 0.We can generalize Proposition 1 for any m dimensional distributionP, instead of P γ, as follows: DISPLAYFORM0 Letâ be a random vector that follows an m dimensional probability distributionP whose expected value isμ ∈ R m. Given w ∈ R m such that w > 0, it follows that DISPLAYFORM1 whereθ w is the angle between w andμ. Proof. The proof is the same as that of Proposition 1.Proposition 2 states that the distribution of w ·â is biased according toθ w unless μ = 0. and BID21. Repeating the operations through multiple layers, the variance of θ l i and a l i will shrink to small values. We illustrate the effect of angle bias in an MLP by using the CIFAR-10 dataset BID10 ) that includes a set of 32 × 32 color (RGB) images. Each sample in CIFAR-10 is considered an input vector with 32 × 32 × 3 = 3072 real values, in which each variable is scaled into the range [−1, 1]. We consider an MLP with sigmoid activation functions that has 10 hidden layers with m = 128 neurons in each layer. The weights of the MLP are initialized according to BID3. We randomly took 100 samples from the dataset and input them into the MLP. FIG2 shows the activation pattern in layers BID19 BID21 5, 7, and 9 on the selected samples. Please note that the activation in Layer 1 corresponds to a 1 i in Equation, that is, Layer 1 is the layer after the input layer. We see stripe patterns in the layers other than Layers 1 in FIG2 that are caused by angle bias. In Layer 9, the activation value of each neuron is almost constant regardless of the input. In contrast, no stripe pattern appears in Layer 1, because each element of the input vector is scaled into the range [−1, 1] and its mean value is near zero; this corresponds to the case in which μ ≈ 0 in Proposition 2., for each sample 1. FIG6 shows boxplot summaries of θ l i on the first ten neurons in layers 1, 3, 5, 7, and 9, in which the 1%, 25%, 50%, 75%, and 99% quantiles are displayed as whiskers or boxes. We see the mean of θ l i are biased according to the neurons in the layers other than Layer 1. We also see that the variance of θ l i shrink through layers. Next, we consider an MLP with ReLU activation functions that has 50 hidden layers with m = 128 neurons in each layer. The weights are initialized according to BID3. Figure 4 shows the activation pattern in layers BID19 10, 20, 30, and 40 on the randomly selected samples. We see stripe patterns in the layers other than Layer 1 that are caused by the angle bias. Figure 5 shows boxplot summaries of θ l i on the first ten neurons in layers 1, 10, 20, 30, and 40. We see that the mean of θ l i are biased according the neurons in the layers other than Layer 1. We also see that the variance of θ l i shrink through layers, but the shrinking rate is much moderate compared to that in FIG6. This is because ReLU projects a preactivation vector into the unbounded region [0, +∞) m and the activation vector is less likely to concentrate on a specific region. Under the effect of angle bias, the activation of neurons in deeper layers are almost constant regardless of the input in an MLP with sigmoid activation functions, as shown in FIG2. It indicates that ∇ a 0 L = 0, where L is a loss function that is defined based on the output of the MLP and ∇ a 0 L means the gradient with respect to the input vector a 0. From Equation, we have DISPLAYFORM0 DISPLAYFORM1 Assuming that w Equation FORMULA4, with l = 1, indicating that the gradients of weights in the first layer are vanished. From Equation, with l = 1, FORMULA3 and FORMULA4, with l = 2, under the assumption that w DISPLAYFORM2 DISPLAYFORM3 If we use rectified linear activation instead of sigmoid activation, the gradients of weights are less likely to vanish, because ∇ a 0 L will seldom be exactly zero. However, the rate of each neuron being active 2 is biased, because the distribution of preactivation z l i is biased. If a neuron is always active, it behaves as an identity mapping. If a neuron is always inactive, it is worthless because its output is always zero. Such a phenomenon is observed in deep layers in Figure 4. As discussed in BID1, the efficiency of the network decreases in this case. In this sense, angle bias may reduce the efficiency of a network with rectified linear activation. There are two approaches to reduce angle bias in a neural network. The first one is to somehow make the expected value of the activation of each neuron near zero, because angle bias does not occur if μ = 0 from Proposition 2. The second one is to somehow regularize the angle between w. In this section, we propose a method to reduce angle bias in a neural network by using the latter approach. We introduce W LC as follows: DISPLAYFORM0 The following holds for w ∈ W LC:Proposition 3. Let a be an m dimensional random variable that follows P γ. Given w ∈ W LC such that w > 0, the expected value of w · a is zero..., m) will likely be more similar to each other. The activation vector in layer l, each of whose elements is given by Equation, is then expected to follow P γ. Therefore, if the input vector a 0 follows P γ, we can inductively reduce the angle bias in each layer of an MLP by using weight vectors that are included in W LC. We call weight vector w DISPLAYFORM1 We built an MLP with sigmoid activation functions of the same size as that used in Section 2.2.1, but whose weight vectors are replaced with LCWs. We applied the minibatch-based initialization described in Section 3.3. FIG8 shows the activation pattern in layers 1, 3, 5, 7, and 9 of the MLP with LCW on the randomly selected samples that are used in FIG2. When compared with FIG2, we see no stripe pattern in FIG8. The neurons in Layer 9 respond differently to each input sample; this means that a change in the input leads to a different output. Therefore, the network output changes if we adjust the weight vectors in Layer 1, that is, the gradients of weights in Layer 1 do not vanish in FIG8. FIG9 shows boxplot summaries of θ l i on the first ten neurons in layers 1, 3, 5, 7, and 9 of the MLP with LCW. We see that the angle distributes around 90• on each neuron in each layer. This indicates that the angle bias is resolved in the calculation of z l i by using LCW. FIG10 shows the activation pattern in layers of the MLP with LCW after 10 epochs training. A slight stripe pattern is visible in FIG10, but neurons in each layer react differently to each input. FIG11 shows boxplot summaries of θ l i of the MLP after 10 epochs training. We see that the mean of θ l i is slightly biased according to the neurons. However, the variance of θ l i do not shrink even in deeper layers. We built an MLP with ReLU activation functions of the same size as that used in Section 2.2.2, whose weight vectors are replaced with LCWs. We applied the minibatch-based initialization described in Section 3.3. FIG1 shows the activation pattern in layers BID19 10, 20, 30, and 40 of the MLP with LCW. When compared with Figure 4, we see no stripe pattern in FIG1. FIG1 shows boxplot summaries of θ l i on the first ten neurons in layers BID19 10, 20, 30, and 40 of the MLP with LCW. We can observe that the angle bias is resolved by using LCW in the MLP with ReLU activation functions. A straightforward way to train a neural network with LCW is to solve a constrained optimization problem, in which a loss function is minimized under the condition that each weight vector is included in W LC. Although several methods are available to solve such constrained problems, for example, the gradient projection method BID11, it might be less efficient to solve a constrained optimization problem than to solve an unconstrained one. We propose a reparameterization technique that enables us to train a neural network with LCW by using a solver for unconstrained optimization. We can embed the constraints on the weight vectors into the structure of the neural network by reparameterization. DISPLAYFORM0 where I m−1 is the identity matrix of order (m − 1) × (m − 1). In the experiments in Section 5, we used B m in Equation FORMULA9. We also tried an orthonormal basis of W LC as B m; however, there was little difference in accuracy. It is worth noting that the proposed reparameterization can be implemented easily and efficiently by using modern frameworks for deep learning using GPUs. By introducing LCW, we can reduce the angle bias in z l i in Equation, which mainly affects the expected value of z l i. It is also important to regularize the variance of z l i, especially when the sigmoid activation is used, because the output of the activation will likely saturate when the variance of z l i is too large. We apply an initialization method by which the variance of z l i is regularized based on a minibatch of samples. This type of initialization has also been used in previous studies BID12 and BID16. We conducted preliminary experiments using the CIFAR-10 dataset, the CIFAR-100 dataset BID10, and the SVHN dataset BID14. These experiments are aimed not at achieving state-of-the-art but at investigating whether we can train a deep model by reducing the angle bias and empirically evaluating the performance of LCW in comparison to that of BN and WN.Network structure We used MLPs with the cross-entropy loss function. Each network has 32 × 32 × 3 = 3072 input neurons and 10 output neurons, and it is followed by a softmax layer. We refer to an MLP that has L hidden layers and M neurons in each hidden layer as MLP(L, M). Either a sigmoid activation function or a rectified linear activation function was used. MLP LCW denotes an MLP in which each weight vector is replaced by LCW. MLP BN denotes an MLP in which the preactivation of each neuron is normalized by BN. MLP WN denotes an MLP whose weight vectors are reparametrized by WN.Initialization Plain MLP and MLP BN were initialized using the method proposed in BID3. MLP LCW was initialized using the minibatch-based method described in Section 3.3 with σ z = 0.5. MLP WN was initialized according to BID16.Optimization MLPs were trained using a stochastic gradient descent with a minibatch size of 128 for 100 epochs. The learning rate starts from 0.1 and is multiplied by 0.95 after every two epochs. The experiments were performed on a system running Ubuntu 16.04 LTS with NVIDIA R Tesla R K80 GPUs. We implemented LCW using PyTorch version 0.1.12. We implemented BN using the torch.nn.BatchNorm1d module in PyTorch. We implemented WN by ourselves using PyTorch 3. We first consider MLPs with sigmoid activation functions. FIG1 shows the convergence and computation time for training MLPs with CIFAR-10 dataset. FIG1 (a) shows that the training accuracy of the plain MLP is 10% throughout the training, because the MLP output is insensible to the input because of the angle bias, as mentioned in Section 2.2 BID22. By contrast, MLP LCW or MLP BN is successfully trained, as shown in FIG1 (a), indicating that the angle bias is a crucial obstacle to training deep MLPs with sigmoid activation functions. MLP LCW achieves a higher rate of increase in the training accuracy compared to MLP BN in FIG1 (a), (d), and (g). As described in Section 4, WN itself cannot reduce the angle bias, but the bias is reduced immediately after the initialization of WN. From FIG1 (a) and (d), we see that deep MLPs with WN are not trainable. These suggest that starting with weight vectors that do not incur angle bias is not sufficient to train deep nets. It is important to incorporate a mechanism that reduces the angle bias during training, such as LCW or BN.The computational overhead of training of MLP LCW is approximately 55% compared to plain MLP, as shown in FIG1 (b); this is much lower than that of MLP BN. The overhead of MLP WN is large compared to that of MLP BN, although it contradicts the claim of BID16. We think this is due to the implementation of these methods. The BN module we used in the experiments consists of a specialized function developed by GPU vendors, whereas the WN module was developed by ourselves. In this sense, the overhead of LCW may be improved by a more sophisticated implementation. In terms of the test accuracy, MLP LCW has peaks around 20 epochs, as shown in FIG1 (c), (f), and (i). We have no clear explanation for this finding, and further studies are needed to investigate the generalizability of neural networks. Experimental with SVHN and CIFAR-100 datasets are reported in Section B in the appendix. We have experimented with MLPs with rectified linear activation functions. In our experiments, we observed that the plain MLP with 20 layers and 256 neurons per layer was successfully trained. However, the training of MLP LCW of the same size did not proceed at all, regardless of the dataset used in our experiment; in fact, the output values of the network exploded after a few minibatch updates. We have investigated the weight gradients of the plain MLP and MLP LCW. FIG1 shows boxplot summaries of the weight gradients in each layer of both models, in which the gradients are evaluated by using a minibatch of CIFAR-10 immediately after the initialization. By comparing FIG1 (a) and FIG1 (b), we find an exponential increase in the distributions of the weight gradients of MLP LCW in contrast to the plain MLP. Because the learning rate was the same for every layer in our experiments, this exponential increase of the gradients might hamper the learning of MLP LCW. The gradients in a rectifier network are sums of path-weights over active paths BID1. The exponential increase of the gradients therefore implies an exponential increase of active paths. As discussed in Section 2.3, we can prevent neurons from being always inactive by reducing the angle bias, which we think caused the exponential increase in active paths. We need further studies to make MLP LCW with rectified linear activation functions trainable. Possible directions are to apply layer-wise learning rates or to somehow regularize the distribution of the weight gradients in each layer of MLP LCW, which we leave as future work. In this paper, we have first identified the angle bias that arises in the dot product of a nonzero vector and a random vector. The mean of the dot product depends on the angle between the nonzero vector and the mean vector of the random vector. In a neural network, the preactivation value of a neuron is biased depending on the angle between the weight vector of the neuron and the mean of the activation vector in the previous layer. We have shown that such biases cause a vanishing gradient in a neural network with sigmoid activation functions. To overcome this problem, we have proposed linearly constrained weights to reduce the angle bias in a neural network; these can be learned efficiently by the reparameterization technique. Preliminary experiments suggest that reducing the angle bias is essential to train deep MLPs with sigmoid activation functions.

We identify angle bias that causes the vanishing gradient problem in deep nets and propose an efficient method to reduce the bias.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.